Evaluation Study Of The Career Guidance Project (Consultancy)

Introduction   This document describes the Terms of Reference (ToR) for the evaluation study of the Career Guidance project implemented in Marjeyoun area.   The evaluation’s purpose is to evaluate the progress made towards achieving the project’s set objectives, and provide specific recommendations for future interventions.    The evaluation process will adopt a participatory approach and include the Project’s partners as well as members from the local community. 1. Evaluation Summary   Programme/Project Career Guidance Project Phase N/A Evaluation Type End of Project Evaluation Evaluation Purpose To evaluate the progress made towards achieving the Project’s set objectives, and provide specific recommendations for future interventions. Primary Methodologies Mixed methodology: Qualitative (Focus Group Discussions, Key Informant Interviews) and Quantitative (Surveys) Evaluation Start and End Dates   Anticipated Evaluation Report Release Date     2. Description of Project Being Evaluated   Marjeyoun is remote area away from the capital thus having a limited access to proper career guidance opportunities. Students of grades 7th, 8th, and 9th lack career steering knowledge and options by experts and trained school staff who should be provided with tools and skills to be able to assist their students with needed career guidance   This project falls under the focus of the Education project of Marjeyoun ADP, and aims to identify gaps in knowledge and practice in terms of career guidance within the groups of schools targeted as well as within the groups of students and their parents to be followed up by setting individual plans for each school as well as awareness sessions for parents during this first year of implementation.   Major Sector Related Problems and Opportunities   Problem/Opportunity Cause 1 Problem: students in Marjeyoun area schools grow up not aware of the academic and career choices ahead of them. Career guidance programs are lacking in most schools in Lebanon whereby the student finishes school or even drops out without knowing of other alternatives or of the choices he/she could make at certain steps in their life. 2 Opportunity: the ADP has established a great relation with the partner schools in the past years The ADP team was able to build strong relationships with school management and transmit to them the working approach of WV via the constant and long term work it had with them since 2004   Description of project goal:  Children at Marjeyoun ADP schools are protected and participate.   Through this goal, the ADP aims at creating an environment in the area that motivates children to attend school; where school is guiding, safe and fun. The transformation of the school from being merely a learning place to being a context where children receive guidance for their future stages is the utmost aim of this project. In order to achieve this aim, work will be done across several segments that affect the education of the children and these are: the children themselves, the teachers and school staff/management and the parents.   Description of project outcome: 7th, 8th and 9th grade school students in Marjeyoun ADP are able to make educational and career choices based on career guidance program   Career and academic guidance is a major need for the schools of the area where most children spend their school days lost in which direction to set their ambitions and some of them even not having any ambitions because of them not knowing of the interesting windows of opportunity available to them. This outcome will tackle such issues.   Description of the project outputs: Output 1: Staff and directors of Marjeyoun schools are provided with tools and skills to be able to assist their students with needed career guidance   This output tackles the sensitization of school directors in Marjeyoun area to the importance and need of career guidance for their students in 7th, 8th & 9th grades and helping these directors in setting annual plans to tackle this issue in their schools. School teachers also will benefit from this output whereby their capacities will be built on the topic of guidance to be able to sustain such program in their schools after WV’s intervention ends.   Output 2: 7th, 8th & 9th grade school students are exposed to career steering knowledge and options by experts and trained school staff This output  is comprised of the actual delivery of guidance tools and information to students in 7th, 8th & 9th grades in schools of Marjeyoun. Through this output, the students will be provided with choices and with awareness on career making choices parameters.   2. Evaluation Target Audiences   The following parties are involved and have a major interest in the evaluation process:   Community members: Community members including children, teachers and parents will take part in the evaluation process, to ensure the latter is being conducted in a partially participatory manner. A fully participatory approach would entail the participation of community members in all the actual implementation of evaluation process. Results will be disseminated to all beneficiaries who took part in this evaluation, in order to be transparent and accountable towards the community that World Vision Lebanon (WVL) serves.   WVL: The data collected from the evaluation will inform WVL on the successes and challenges faced throughout the project implementation. It will also highlight the levels of impact or differences that the project has made in the lives of targeted beneficiaries and will reveal the relevance to the communities’ needs. Documented information will further contribute to generating lessons learned and recommendations that should guide future career guidance interventions implemented by WVL,   Career guidance project Partners:  Partners of the career guidance project play a key role in the evaluation process as their input is the main source of information. Their participation will be insured throughout the process in analysis of information, validation of findings, and setting recommendations.The final results and recommendations will be shared with them. Please see Annex 1 for the list of career guidance project partners.   MEER Education Hub Lead: The results of the evaluation will inform the Education Hub Lead about the degree to which the objectives of the project have been met and will help develop evidences in this specific Educational area that will be used in other similar projects, whether in Lebanon (Beirut area) or any other potential MEER office.   3. Evaluation Type   The current evaluation is an end of project evaluation of the Career Guidance project. With the project reaching its final phase, an end of project evaluation is a necessary step in order to assess whether the project has met its objectives. The evaluation will assess the outcome and outputs of the project and will also direct future similar projects conducted by WVL in other areas, or in relatable contexts.   4. Evaluation Purpose and Objectives   The purpose of Career guidance project evaluation is to assess the effectiveness of the project and to inform its future replication.  In addition to evaluate the progress made towards achieving the project’s set objectives in terms of outcomes and outputs. In order to measure the progress made, contribution lower level outputs has to be assessed in order to find out if these outputs led to the achievement of the project outcome. Moreover, the status of the outcome level indicators upon the completion of our intervention has to be measured in order to assess the contribution made towards achieving the overall Marjeyoun logframe goal.     The objectives of the current evaluation are as follows: Assess the contribution made by lower level outputs to achieve the project outcome. In trying to do so, the evaluator has to assess whether the proposed outputs led to certain unexpected outcomes Assess whether planned activities were carried out according to plan Assess the quality of the activities carried out as part of the project.   Evaluation achievement matrix: Career Guidance Project Goal indicator Indicator 1 % of 7th, 8th and 9th grade students who perceive that they are confident that they can make decisions that affect their future Outcome  indicator Indicator 1 % of 7th, 8th & 9th grade school students who report being able to make educational and career choices based on career guidance program Output 1 indicators Indicator 1 # of trainings and awareness sessions held for school directors and staff Indicator 2 # of school directors and staff who attend all the trainings and awareness sessions Indicator 3 # of schools that set annual career guidance programs Output 2 indicators Indicator 1 # of awareness and orientation sessions conducted for 7th, 8th & 9th grade school students Indicator 2 # of students who attend these awareness and orientation sessions   5. Evaluation Methodology The methodology section will describe how the evaluation will take place and will be divided into qualitative and quantitative sub-sections. Qualitative data collection: The survey is based on the questionnaires used during the baseline assessment. The results of these surveys will be compared with the baseline results to evaluate the progress made towards program objectives. Before staring the actual data collection, the survey will be pilot tested in the targeted areas; afterwards, the tool will be revised and finalized for large scale use.  The quantitative data will be collected by data collectors and the quality of the data collected will be monitored by the project evaluator. The data will be entered to Microsoft excel. The quality of data entry will be monitored as well in order to ensure clean data that that excludes errors in the records. Descriptive statistics (means, frequencies, etc.) will be reported through Microsoft Excel. The results of the evaluation will be shared with MR ADP staff and the community stakeholders and their input will inform the finalization of the evaluation report, mainly the analysis of results, statement of conclusion and recommendations.    Survey with students of 8th, 9th and 10th grades This survey will be conducted with 209 students of 8th, 9th, and 10th grades in MR ADP schools. The sample size was calculated using a sample size calculator5 with 5% margin of error and 95% confidence level; based on the total number of students (grade 8 to grade 10) in MR ADP schools which is 457 students according to the students list given by each school.  After that a cluster sampling approach (where schools are the clusters) was used to decide on the number of students to survey in each school. The same method was used during baseline; however the sample size is not the same because during baseline the total number of students was considered from grade 7 to grade 9. However, at evaluation, grade 7 students turned out not to be targeted by the intervention, thus they were removed from the sampling unit without compromising the comparability with baseline results. This survey will assess student’s perception of being confident about making decisions that affect their future and their ability to make educational and career choices based on career guidance projects. So it will cover both the goal and the outcome indicators.   Qualitative Data collection:   Focus Group Discussions (FGDs) A total of 16 FGDs (1 with girls and 1 with boys) will be conducted with students of 8th, 9th, and 10th grades; two FGDs (1 with girls and 1 with boys) in each of the 8 schools that participated in the career guidance project. These FGDs will be conducted for the purpose of validating the quantitative data findings with a focus on the underlying reasons for those findings. During FGDs questions about the quality of career guidance sessions and materials will be asked. The number of the FGDs may vary depending on data saturation.   Key Informant Interviews (KIIs)  KIIs will be conducted with eight school directors where the career guidance project was implemented, Balamand University who was responsible about the implementation of the project in all the eight schools and all other related parties who took part of this project.   During the FGDs and the KIIs a note taker will be present to document the conversations; in addition a recorder will be used to register the whole discussion after obtaining the participants consent. After each conversation, a transcription of the whole discussion will follow. For the FGDs and the KIIs, thematic analysis of the data will be done in order to discern patterns and come up with conclusions that would reflect the views and the perceptions of the participants. The information collected during the FGDs and KIIs will be compared with the quantitative data for a holistic understanding of the evaluation results.   Documents Review: In order to evaluate the output indicators, all project documents including monthly reports activity reports, materials distributed, attendance sheets, and any other means of verification available at ADP level and at the disposal of Balamand University should be reviewed, and analysed. This will allow critical analysis of the quality of materials and information provided to beneficiaries.   6. Limitations   Although it is expected that the stakeholders will be very collaborative throughout the evaluation exercise, we can not totally exclude the possibility that the beneficiaries would refuse to answer the survey questionnaires or attend the FGD. Furthermore, the data collection process will be altered in case of any deterioration in the political / security situation in Lebanon in general and/or in Marjeyoun area in particular.     7. Authority and Responsibility   7.1 Team Members and Roles   Team member and name Role/expertise Stage of assessment Specific duties                             External Evaluator Evaluation leader Throughout -        Set a detailed timeline for the evaluation study -        Develop tools to collect primary qualitative and quantitative data including the preparation of interview questions -        Set the interview schedule in collaboration with Marjeyoun Area Manager and Education Coordinator. -        Test the developed tools and incorporate changes prior to the actual data collection -        Conduct KIIs with the help of Education coordinator -        Conduct Document review of project documents -        Summarize findings of these discussions and compare them to the document review findings -        Data analysis, ensuring community participation in the data analysis and in the validation of findings, report writing, incorporation of feedback and finalization of report -        Dissemination of findings DME Manager Technical Support Throughout -        Ensure that the evaluation process is aligned to LEAP standards -        Support in developing an appropriate methodology for the evaluation -        Provide feedback on developed tools and on drafted documents Technical Lead Technical support Throughout -        Ensure proper and reliable data collection -        Ensure timely completion of process -        Ensure full cooperation of Project staff and key community stakeholders -        Assume full ownership of the process -        Manage all related financial aspects of the assessment -        Dissemination of findings Area Manager and Education  Coordinator Coordination on the field level Data Collection, Stake holders analysis &  Validation of finding -        Coordinate needed contact with the community. -        Contribute to data collection (including focus group facilitation, note taking and interviews) -        Contribute to data analysis -        Dissemination of findings     7.2 Partners Involved in the Evaluation     Partner Role Community partners- See Annex 1 Providing input for data collection, validating findings, and providing recommendations WVL Marjeyoun staff Facilitating data collection, mobilizing the community, participating in data analysis WVL DME team Providing DME technical assistance and feedback to the process and lead the completion of the evaluation report MEER Education Hub Lead Providing guidelines and feedback when needed           7.3 Timeframe   The following timeframe is tentative one; it is estimated as if the evaluation is being done internally. Since this will not be the case, the external evaluator will be defining his/her timeframe.   Baseline study phases Timeframe Identification of target population and sampling   Team recruitment   Team training   Development of data collection tools (most are already available, used previously in the baseline evaluation)   Data collection   Data analysis   Report writing   Disseminating report to community partners and stakeholders         8. Team Advisors   An external evaluator will be recruited to lead the evaluation process, the Technical Lead, the DME manager and the PDQA Manager will provide guidance and assistance.   9. Logistics   The Project staff, under the supervision of the Area manager, will follow up with all logistics pertaining to transportation, accommodation, invitation, facilitation, venue, etc.   10. Products     An evaluation report will be produced which summarizes the results of the data collection with projects’ partners and provides relevant recommendations.     11. Documents   The documents to be reviewed in the research phase are the following: LEAP templates and guidelines WVL National Strategy WVL- Marjeyoun Logframe WVL – Career Guidance baseline report and ITT WVL- Career Guidance previous Annual and Semi-annual reports