"area Development Programmes End Of Cycle Evaluation - World Vision Lebanon"

  i.              Introduction   World Vision Lebanon (WVL) operates in different areas in Lebanon (South, Akkar, Bekaa and Beirut) through Area Development Programmes (ADPs). These programmes follow a cycle of assessment, design, implementation, and evaluation. During FY 16, WV Lebanon is engaging in multiple evaluation processes whereby six ADPs are at an evaluation point. This document is a reference for the end-of-cycle evaluation which will cover the first phase of implementation for Bint Jbail, Akkar and Dreib El Awssat ADPs and second phase for West & Central Bekaa. It aims to identify through a qualitative process the progress towards impact and inform the programme on how to proceed with the next steps of re-design & implementation.   1.    Evaluation Summary Programmes/Project: West & Central Bekaa, Bint Jbail, Akkar/El Sahel and Dreib El Awssat   Area Development Programmes Programmes Phase: Second cycle : Central Bekaa ADP (FY10 - FY15) West Bekaa ADP (FY10 - FY15) First cycle: Bint Jbail  ADP (FY12 - FY16) Akkar/El Sahel ADP (FY10 - FY15) Dreib El Awssat ADP (FY12-FY15) Evaluation Type: End-of-Cycle Programme summative evaluation Evaluation Purpose:   This evaluation investigates the entire impact, the chain of efficiency, consistency, effectiveness and impact towards the programme’s goal and Child Well Being Aspirations (CWBA) in general It will determine whether the outcomes and outputs of the programmes have led to sustainable outcomes and goals It will provide the ADP staff and communities with recommendations for consideration for the second and third implementation phase Primary Methodologies Focus Group Discussions (FGD) Key Informant Interviews (KII) Document Review Evaluation Start and end dates: February – March 2016 Anticipated Evaluation Report release date: April 2016     Description of Programmes Being Evaluated   Central Bekaa ADP Operational period for programme FY10 – FY15 Programme target population 27,000 persons 5.152 Registered Children (RCs) Programme/Projects Central Bekaa Area Development Programme (#02340) §  Child Sponsorship Management Project (175310) §  Health Project (191146) §  Education Project (193086). Direct Beneficiaries Schools (trainings for teachers on learning difficulties, activities with RCs/students) Youth groups (scouts, church groups) Emergency Reduction and Disaster Management (ERDM) Committee consisting of head of municipalities (one committee across all Bekaa ADPs) Disaster Risk Reduction (DRR) committee consisting of youth members (one committee across all Bekaa ADPs) 2 Mother Action Groups (MAGs) Children Council (one council across all Bekaa ADPs) Citizen Voice Action (CVA) and Child Protection Advocacy (CPA) committee (one committee across all Bekaa ADPs) Parents Committee Theatrical committee (one committee across all Bekaa ADPs) Partners Youth Information Center (Kfarzabad Municipality) 7 Primary health care centers 6 Schools (Haratch, Secondary Evangelical school Anjar, MSA Ablah, MSA Ferzol, Sainte Anne school – Riak, Sainte Antoine school - Riak) under the education project 7 Municipalities Programme location The ADP spans an area of 103km2 in the Casa of Zahle. It comprises and targets 10 villages: Ferzol, Kfarzabad, Terbol, Ayn Kfarzabad, Qosaya, Ablah, Niha, Riak and Sariin and Anjar     West Bekaa ADP Operational period for programme FY10 – FY15 Programme target population 56,400 persons 2,701 Registered Children (RCs) Programme/Projects West Bekaa Area Development Programme (#02335) §  Child Sponsorship Management Project (175242) §  Health Project (191145) §  Education Project (193085) Direct Beneficiaries Schools (trainings for teachers on learning difficulties, activities with RCs/students) Youth groups (scouts, church groups) Emergency Reduction and Disaster Management (ERDM) Committee consisting of head of municipalities (one committee across all Bekaa ADPs) Disaster Risk Reduction (DRR) committee consisting of youth members (one committee across all Bekaa ADPs) 2 Mother Action Groups (MAGs) Children Council (one council across all Bekaa ADPs) Citizen Voice Action (CVA) and Child Protection Advocacy (CPA) committee (one committee across all Bekaa ADPs) Parents Committee Theatrical committee (one committee across all Bekaa ADPs) Partners Youth Information Center  (Mansoura Municipality) 3 Primary Health Care Centers 6 schools (Al makassed, Al Ahleya, Elementary Public school, Sainte Famille Saghbin, Evangelical school – Kab Elias, Apotre - Kab Elias) under the education project 5 municipalities Programme location The ADP spans an area of 121.72Km2 situated in the west part of the Bekaa governorate, named the West Bekaa area. Villages where the ADP operates: Kab Elias – Ammic - Aanna –Mansoura - Kherbet Kanafar – Ayn zebde - Saghbin – Tel dnoub – Aitanit -Taalabeya     Bent Jbail ADP Operational period for programme FY12 - FY16 Programme target population Total Population 25,400 (7,579 Children) 2,400 Registered Children (RCs) Programme/Projects Bent Jbail Area Development Programme (# 04005) §  Sponsorship Management Project (L194053) §  Cared for and protected Project (L197433) §  Health Project (L197381) Direct Beneficiaries Children of the 9 villages including the RCs Students including RCs Children council members Youth Groups (Church groups, scouts) School Teachers Parents/ caregivers 4 Committees (Health Committee, Cared for committee, Community-based Disaster Preparedness committee and Sponsorship committee) Partners 2 Community Based organizations (Awfa and Charity of Tebnin) 9 Faith Based Organizations 2 NGOs (Lebanese Red Cross and Alpha) 11 Primary Health Care centers (PHC) 2 Governmental Hospitals 14 schools (5 private schools and 9 public schools) 1 Centre de Lecture et d'Animation Culturel (CLAC) Center 9 Municipalities Programme location South Lebanon, Nabatiyeh Governate, Bent Jbail district, 9 villages (Tebnin, Safad el Batikh, Yater, Harees, Beit Life, Ain Ebel, Rmeich, Debel, Qawzah)     Akkar/ Al Sahel ADP: Operational period for programme FY10 – FY15 Programme target population About 35,000 residents Programme/Projects Akkar/El Sahel Area Programme (#L188840) §  Akkar ADP - Education Project (L194092) §  Akkar ADP - Sponsorship Management Project (L188840) §  Akkar ADP - Health Project (L11111) Direct beneficiaries §  Children §  13 Parents councils (1 council for each school) §  Mothers Partners §  6 Municipalities: Tal Hayat, Ta El Bireh, Massoudiyi, Tal Meeyan, Qlayaat and Hissa §  13 Schools Programme location North Lebanon – Akkar District – El Sahel Area Akkar district is bounded by Syria from the North, Tripoli from the South, Bekaa District from the East & the Mediterranean Sea from west.   Dreib El Awsat ADP: Operational period for programme FY12 – FY15 Programme target population About 700 RCs Programme/Projects Dreib Al Awsat Area Development Programme (#4035) §  Dreib El Awsat ADP - Youth Economic Opportunities and Child Protection (194378) Direct beneficiaries Registered and non-registered children Students in schools Parents Youth Volunteers Children council Parents’ council Registered children council Child Protection committee in Mejdel Partners 3 Municipalities: Kweshra, Sfayneh, Ein El Zeit, Priests and Cheikhs 4 schools: Ein Zeit Public school, Lycee Rabih Al Atfal-Ein Zeit’s management, Kweshra Public School, Rihaniyeh Public School Programme location Lebanon, North Lebanon, Akkar District, Dreib Al Awsat     3.  Evaluation Target Audiences This mega-evaluation is done to inform the ADPs on a macro level about programme outcomes as well as to keep the Support Offices (SO) informed about the programmes’ effectiveness. The evaluation is also a tool for the community local government (LG), non-governmental organizations (NGO) and community groups to measure the impact of WVL programmes in the 5 ADPs (West & Central Bekaa, Bint Jbail, Akkar and Dreib El Awssat). The mega-evaluation exercise is intended to include and influence the following groups:   Community members: including children, teachers, volunteers and parents will take part of the evaluation process, to ensure the latter is being conducted in a partially participatory manner. WVL National Office: data collected from the evaluation will inform WVL on the successes and challenges faced throughout the programme implementation. It will also highlight the levels of impact and sustainability and further contribute to generating lessons learned and recommendations that should guide future interventions implemented by WVL with a special focus on Sponsorship Operations. ADP Partners: Partners of the programme will be the main participants in the evaluation as their input is the main source of information. Their participation will be ensured throughout the process in data collection and validation as well as dissemination of findings. Support Offices (US, Malaysia, Canada, and Australia): The results of the evaluation will present qualitative information pertaining to the programme’s impact on children’s wellbeing. It offers the opportunity for WVL to be accountable towards the main donors as the process will be documented in details and the information is generated with the participation of the community. Regional Office: This mega evaluation will have a special focus on Sponsorship operation models in WV Lebanon and will ultimately inform possible ways to improve sponsorship operations and will contribute to enriching the learning within the Regional Office (RO).       4.  Evaluation Type The evaluation is an end of programmes’ evaluation that will be conducted to determine through a qualitative process what has been the programme impact and what should change in order to have a meaningful and sustainable impact in the communities in West & Central Bekaa, Bint Jbail, Akkar and Dreib El Awssat ADPs. The evaluation is scheduled to be carried out between February and March 2016.     5.  Evaluation Purpose and Objectives The purpose of the programme mega-evaluation is to determine effectiveness of programme intervention and use of resources to identify needs of the children, their families and communities. The end of cycle programme evaluation will cover the last phase of implementation in each of the 5 ADPs. The five standard dimensions of this evaluation include: relevance, effectiveness, impact, sustainability and efficiency.  The effectiveness questions will primarily focus on the most recent programme cycle since the latest programme design significantly shifted focus and approaches from the previous programme cycle.      The key questions in each of the evaluation dimensions are the following:   Key Objectives Key questions to be asked RELEVANCE   i.    To describe how the project has been consistent or inconsistent with WV’s integrated focus for programming and WVL’s national strategy  ii.    To identify if the programme was properly designed to meet targeted needs through evaluating and assessing programme theory, logic, conceptual components and assumptions To assess the relevance of the tools developed throughout the project to the area context and to the needs of the communities To assess the degree of involvement of girls, women, boys and men in project components and their participation in planning, implementation, monitoring and evaluation (with a focus on gender equity)     i.   Was the programme designed in a way to meet the needs of the community, and specifically of adolescents and youth?    ii.   Was the sponsorship programme meeting the needs of the RCs and their families?   iii.   Are sponsorship requirements accepted by the community?   iv.   Is the project consistent with WV’s integrated focus for programming and WVL’s national strategy?    v.   Were the established partnerships appropriate for the scope of work of the project?   vi.   Were girls, women, boys and men equally engaged and participating in the project planning, implementation, monitoring and evaluation?   EFFECTIVENESS To assess the progress made towards achieving the project’s goal and objectives based on the logframe, design and monitoring data     i.    How well were each of the projects’ objectives achieved?    ii.    What were the programme’s strengths in terms of the method used and the design?   iii.    How properly were the projects executed?   iv.    Was the frequency and duration of the trainings provided to direct beneficiaries/committees/ MAGs enough to reach objectives and/or to induce change?    v.    Were youth groups well equipped to be able to transmit acquired information and train others?   vi.    Did the partners including schools and local community groups benefit from WVL’s support during this project? To what extent was their capacity improved?  vii.    How did implementation variations from project design affect results? What were the programme’s weaknesses in terms of the method/s used and the design?   ix.    What were the challenges in terms of meeting sponsorship requirements?    x.    What are the reasons behind the low participation rate of RCs in ADP activities?   xi.    What alternative methods would have led to a better achievement of programme’s objectives? Why and how?  xii.    How well did the programme adapt to unexpected problems? How flexible was the programme’s design? How were environmental and financial risks managed? IMPACT To assess the potential impact of the programme on the targeted communities     i.   To what extent the community benefited from implemented activities?    ii.   To what extent did partners schools benefit from the activities?   iii.   What was the unexpected effect of the programme activities?   iv.   How did the refugees’ crisis in Bekaa area affect the development programme? SUSTAINABILITY   i.    To assess the partnering approach and the participation of various partners and stakeholders in design, planning, implementation, monitoring and evaluation, and their capacity and willingness to sustain the programme’s achievements  ii.    To assess local ownership of the project by partners, and to identify key factors to ensure replicability of the project components To extract and analyze the lessons learnt and to provide specific, practical and actionable recommendations for expansion and further development of the programme     i.   What are the individual, organizational, and community level capacities?    ii.   To what extent are the local groups/committees and schools able and willing to sustain the efforts after the programme ends?   iii.   To what extent do local partners/committees members own the project?   iv.   How involved were partners in design, planning, implementation, monitoring and evaluation of the programme?    v.   Was WVL’s partnering approach successful in implementation and building towards sustainability?   vi.   What was the added value of these partnerships?  vii.   What are the essential lessons learnt and recommendations for the further development of WVL area programming?   EFFICIENCY To investigate whether the resources (financial, human, and materials) have been used efficiently and effectively for the well-being of the target community.     i.   Have resources (financial, human, and materials) been used efficiently for the well-being of the target community? With a special focus on Sponsorship processes. a)     Why Sponsorship like this b)     What would be the best way to continue with Sponsorship c)     Cost benefit analysis of Sponsorship    ii.   Were most of the outputs achieved to an acceptable standard?   6.  Evaluation Methodology The evaluation design will follow qualitative methods; primary data will be collected through Focus group discussions and Key Informant Interviews.   Secondary data will be collected through review of documents and secondary data sources such as:  Programme and project design documents (which include the logframe, M&E Plan, etc.); Baseline Reports Semi/annual programme management reports.    These methods will be used in accordance with the specific evaluation objectives, according to the key questions to be asked and depending on the target group. For each indicator the appropriate method will be applied. The target population will be: children, parents, teachers, health workers, local NGOs, local government authorities on education and health.   7.  Authorities and Responsibility Team Members and Roles The evaluation exercise will be led by an external consultant who will function as the Evaluation Team Leader. The evaluation team will be composed of a mixture of representatives from relevant National Office departments and ADPs: DME Manager/Officer, Technical Specialists, Sponsorship staff, and ADPs staffs of West and Central Bekaa, Bent Jbail, El Sahel ADP and Dreib Al Awsat.   Note:  Some other National Office representatives, the Operations Director, other members from PDQA and Resource Acquisition and Management (RAM) departments will participate during the data analysis phase. The Area Managers will be consulted throughout all phases of the evaluation process in order to provide the evaluation team with the knowledge of programme interventions. The Evaluation Team Leader will have overall responsibility for all stages of evaluation implementation with ongoing consultation with the Area Managers and DME staff.   All of the tasks in the evaluation process (interviews, focus groups, data processing, writing of evaluation report, etc.) will be coordinated and overseen by the Evaluation Team Leader in consultation with the Area Managers in accordance with the Evaluation Plan and Design.   Evaluation Phase Role Primary task Planning Area Manager Coordinate collection of information for terms of reference. Advise the Evaluation Team Leader on evaluation priorities from a programme perspective. Introduce Evaluation Team Leader to evaluation partners. DME Officer Advice and assist Area Manager in preparing the evaluation ToR. Develop Evaluation Design document Evaluation Team Leader Propose a methodology Provide input to the evaluation design document Develop the evaluation tools based on the evaluation design document DME Manager Review and approve TOR and evaluation design document EQRG Provide feedback on the TOR, evaluation design methodology and tools Data Collection and Analysis Evaluation Team Leader Lead evaluation process, conduct focus groups discussions and interviews and review of secondary data, conduct data entry, cleaning and analysis. ADP staff Logistical arrangements (organize FGD, interview, etc.) Provide input on what the findings mean to them. Help developing practical recommendations.  DME team Give LEAP guidance to all teams in regard to evaluation requirements (also member of Evaluation Team) Provide input to the analysis plan WV Evaluation team members Assist with the interpretation of data Area Manager Provide feedback on analysis findings and recommendations to be made Community members and partners group Assist in providing recommendations during briefing, starting of evaluation and in the analysis of the data. Reporting and Follow-Up Evaluation Team Leader Draft the report and circulate it for feedback Incorporate feedback into the report Conduct a community validation workshop Conduct a stakeholder interpretation workshop Finalize the report based on the feedback received from both workshops and submit a summary fact sheet WV Evaluation team members Provide critical feedback on draft ADP staff Provide feedback on the understanding of the report EQRG Provide feedback on report Support Office Provide feedback on report Evaluation partners Provide feedback on how well their objectives have been met   8.  Logistics ADP staff will participate in an orientation to the TOR and the evaluation design as planned by the Evaluation Team Leader. ADP staff will select FGD and KII targets and support with the coordination of these. Transport and accommodation will be organized by the ADP but covered by the evaluation budget. Any logistics that relate to the data collection, data processing, or facilitation will be organized/coordinated by ADP staff in consultation with the NO DME staff, and the Evaluation team Logistics related to the community validation workshop and stakeholder interpretation workshop will be organized by the WV Evaluation Team Preliminary analysis will be done by the consultant in coordination with the evaluation team in ADP following data collection (debriefing at ADP level) A Final Evaluation report will be submitted by Evaluation Team Leader for WV evaluation team, ADP, NO, and SO feedback with an estimated approval by SO.     9. Products   As the final product of the evaluation process, the Evaluation Report will be written by the Evaluation Team Leader with the final approval of PST representative of SOs. There will be one Mega evaluation report including separate analysis (and separate written section) for each of the ADPs to better respond to specific Support Office reporting needs.   The expected deliverables throughout the process of evaluation include: Evaluation Methodology Qualitative tools developed Thematic in-depth analysis of qualitative data Evaluation Report (based on LEAP Evaluation Report Template) including a reflection on the evaluation process in itself Fact sheet summary (in English and Arabic) Logbook to document evaluation process   The consultant is also expected to present and co-facilitate a community validation workshop to validate the findings with relevant beneficiaries and participants, as well as a stakeholder interpretation workshop which will join together several local and national WV staff to present the main findings, reflect on the lessons learned and recommendations and develop key action points for future programming. These two processes will feed into the final evaluation report.     10. Fees The consultations fees will include transportation, logistics and accommodation fees identified by the consultant, as well as recruitment of note takers, translation fees, analysis and report writing.   11.  Documents Major documents that need to be reviewed during the research phase of the mega-evaluation will be: FY09 - FY13 Programme Designs FY09 - FY10 -  FY11 – FY12 – FY13 – FY14 – FY15 semi and annual reports     Qualification of Consultant The consultant should have the following competencies and experience: At least 5 years of progressive proven experience in similar studies and in using both qualitative research methodologies and thematic data analysis; Extensive professional experience in the design and implementation of outcome and impact evaluations; Good analytical and critical thinking; Proven knowledge and experience in applying participatory research methods and tools; Advanced degree in relevant field (e.g. Public Health, Epidemiology or any related technical field); Good understanding of the Humanitarian work especially the development field; Familiarity with civil society and Non-Governmental Organization (NGO) engagement; Ability to work on tight schedules with minimal supervision; Good English and Arabic speaking and writing skills.     Duration of Consultancy WVL has set aside 2 months (Beginning of February until end of March, 2016) for this assignment. Bids should include a detailed proposed work-schedule with specific tasks and should also incorporate a budget that outlines all relevant costs that will be associated with this evaluation. Candidates are expected to state how much time they will need to start the assignment and how much time they need to conduct every task. Selected candidates are also expected to abide by the deadlines and the conditions for deadlines specified within the contract/agreement.