Consultancy Services For Program Evaluation

                                                                                Republic of Lebanon Ministry of Social Affairs The World Bank Group National Volunteering Service Program (NVSP)                                                            Terms of Reference for Consultancy Services for                                                                                Program Evaluation   PURPOSE This Terms of Reference is for an individual or a consortium (firm) contractor to develop and implement all aspects of a program evaluation for the National Volunteer Service Program (NVSP) in Lebanon, to be conducted from August to October 2016.     BACKGROUND NVSP is implemented by the Ministry of Social Affairs (MoSA) of Lebanon with a US$2 million grant from the multi-donor State and Peace Building Fund (SPF) at the World Bank. The NVSP aims to expand youth volunteerism (particularly in communities other than the communities from which the volunteers come) and to improve youth employability (through enhanced soft skills) for increasing civic engagement and improving social cohesion in the medium term. The program targets youth aged between 15 and 24 years from the different communities and regions of Lebanon. Implementing Agencies (IAs) are NGOs, schools and universities, which are responsible for identifying and recruiting youth to implement volunteering projects in Lebanon. NVSP was approved by the World Bank in 2011, but only became effective in 2013 following implementation readiness delays. NVSP is due to close in December 31, 2016. To achieve these objectives, NVSP has the following three components: Component 1: Institutional building and capacity development for MoSA officials and participating NGOs, schools and universities. This component aims to improve the capacity of the project’s main stakeholders for identifying, managing, and monitoring volunteering projects through training, communications, and outreach activities. In particular, this component finances: (i) the implementation of a national communication campaign, which seeks to raise awareness of and encourage relevant stakeholders to get involved in NVSP; (ii) the development of an online portal, which enables IAs to post volunteering opportunities and youth to register for them; and (iii) the design of a soft-skill training curriculum that is provided to participating youth by IAs following a Training of Trainers (ToT). Component 2: Small Grants Program (SGP), Weekend Volunteering Programs (WVP) and Youth Summer Camps to increase volunteering opportunities for youth. The SGP is implemented through two Call for Proposals (CfPs): The first CfP targeted eligible NGOs and financed 22 volunteering projects that were implemented in the summer of 2015; The second CfP-- launched in May 2016-- will target eligible schools and universities and will finance 15 volunteering projects to be implemented in the summer of 2016. Selected IAs received a small grant ($30,000) from the NVSP for supporting the projects and training volunteers on soft-skills. In order to be eligible, at least 30% of the volunteers recruited by the IA must come from a community other than the one where the proposed project will be implemented. On average, the youth participating in the SGP benefited from 20h of employability/soft skills training and 80h of volunteering. In addition to the SGP, the NVSP supported two additional types of youth volunteering activities through the Weekend Activities and the Summer Camps Programs. Component 3: Monitoring and Evaluation (M&E) of the program as well as of volunteer civic engagement. 3.      PURPOSE AND SCOPE OF THE PROGRAM EVALUATION The purpose of this program evaluation is to assess the relevance, efficiency, and effectiveness of the NVSP processes and procedures for achieving the program goals and overall objectives. The program evaluation will focus on all the activities carried out as part of the program’s three components. However, the evaluation will exclude the second call for proposals in component 2 from its scope and analysis. Hence, the evaluation will address the following: Detailed description of how the NVSP in its three different components (excluding the second CfP) is operating in reality, based on direct observations in the field and in comparison with program design and operational guidelines described in key program documents --i.e. NVSP results chain, SGP operational manual, etc.; Evaluation of the institutional and organizational structure of the NVSP program; Measurements of key attributes of processes and implementation using different technical methods, such as quality, efficiency, relevance, sustainability, effectiveness, etc.[1]; Identification of key program strengths and weaknesses, as well as risks, challenges, and opportunities in the light of its prospective continuation/scaling up; Recommendations on how to improve the design and/or operational processes of the program to increase its efficiency, effectiveness, and sustainability. The main audience of the program evaluation (i.e. the key stakeholders that the evaluation intends to inform) are the NVSP program manager and team, the Lebanese Ministry of Social Affairs (MoSA), the World Bank team, and the State and Peace-Building Trust Fund (SPF) donors. Further stakeholders may include the IAs (NGOs, schools and universities, volunteers), the private sector, local municipalities, prospective donors interested in volunteerism, and other interested Government agencies (i.e. the Lebanese Ministry of Education, and the Lebanese Ministry of Finance). The findings of the program evaluation of the NVSP may be used to: (i) improve NVSP implementation in case of project scale-up; (ii) inform the design and implementation of future volunteering activities in Lebanon; (iii) shape the policy dialogue and debate in Lebanon across the various stakeholders on how to promote volunteerism and social cohesion; and (iv) provide information to the Government of Lebanon, donors, and the public on implementation for reporting and record keeping purposes. 4.      EVALUATION QUESTIONS Questions for investigation at the overall program design and coordination mechanisms level What is the theory of change of the program? Is the program design aligned with the program goals? What is the organizational structure of the NVSP? Has it been effective in supporting NVSP program implementation? Where there any significant changes made to the program (e.g. organizational structure, program goals, division of tasks among stakeholders, activities, timeline) during implementation? If so, what were the reasons? How was the performance of involved stakeholders in overseeing and implementing the NVSP project (namely the NVSP team, World Bank team members, and MoSA staff)? What are the coordination mechanisms at the administrative level between the key stakeholders (MoSA team, NVSP, WBG)? Are they effective? What are the coordination mechanisms at the service delivery level between the key stakeholders (i.e. NVSP and IAs)? Are they effective? Questions for investigation at the program component level Component one: Institution building and capacity development Planned activities: Develop online portal for publicizing volunteering opportunities and for registration of youth interested in volunteering; Implement a national communication campaign to raise awareness of and encourage relevant stakeholders to get involved in NVSP; Train IA and MoSA staff in volunteering management; Design a soft-skill training curriculum that is provided to participating youth by IAs following a Training of Trainers (ToT). Questions for investigation: A.1 Was the design of each activities well planned and sufficient to achieve the intended goals? A.2 Who were the intended and target beneficiaries of the different capacity building and institutional development activities and how were they targeted? Were all intended beneficiaries reached? A.3 Which of the planned capacity building and institutional development activities were implemented and what was the process for implementation? What were the encountered bottlenecks and how were they resolved? A.4 How was the quality of capacity building and institutional development activities delivered to select IAs and MoSA staff? Were they effective and successful in achieving their intended objectives of improving the capacity of IAs to manage volunteers and to promote social cohesion values? Component two: Small Grants Program Component Planned activities: Launch first CfP for NGOs  to promote youth volunteering projects by financing 22 volunteering projects; Train beneficiary IA trainers in soft skills / social cohesion understanding; Train volunteers from beneficiary NGOs in soft skills / social cohesion understanding. Questions for investigation: B.1 How was the process for applying to the first CfP? Were there any problems encountered at the application and selection levels and what were the underlying reasons? B.2 Was the design of this component sufficient and adequate in achieving the intended goals?  For example, was the design feature with respect to requiring 30 percent of volunteers from eligible NGOs to volunteer in communities other than their own relevant and useful in promoting social cohesion values? Was the “training of trainers (TOT)” design feature useful and relevant in building the institutional capacity of NGOs in carrying out training activities to their own volunteers? B.3 Did the design of the first CfP provide sufficient incentives and the opportunity for all of the Lebanese NGOs and youth from all over the country to participate? B4. How was the quality of submitted proposals? How relevant were they to program goals (improving youth employability and social cohesion values)? Did the application form provide sufficient guidelines to help IAs strengthen the quality of proposed volunteering activities to maximize their efficiency and effectiveness on the ground?   B.4 How was the implementation process for youth volunteering activities and for delivering soft skills training? What were some of the encountered problems (on the technical capacity side, procurement, financial management, etc.)? B.5. Did the carried-out trainings and activities achieve their intended objectives in improving soft skills of volunteers and their understanding of social cohesion? How efficient and effective were they? Component two (continued): Two Youth Volunteering Activities Planned activities: Implement Weekend Activities; Implement Summer Camps Activities. Questions for investigation: C.1. How were Weekend Activities and Summer Camps Activities designed? C.2. Who were the participating IAs in these activities and how were they selected/targeted? C.3 Were the summer camp and weekend volunteering activities carried out? What was their duration and what activities did they cover? C.4. What were some of the encountered problems and bottlenecks at the implementation level? C.5. How was the quality of implemented youth volunteering activities? C.6. How effective and sustainable were they in expanding volunteering opportunities for youth? Component three: Monitoring and evaluation Planned activities: Develop and implement M&E plan at the program level and of specific activities (i.e. SPG projects); Design impact evaluation; Conduct baseline survey (start of program); Conduct follow-up survey (end of program). Conduct a mid-term evaluation. Questions for investigation: D.1. Were M&E resources sufficient and M&E activities well planned? D.2. What was the quality and usefulness of the M&E plan? D.3. Were M&E activities accomplished as planned? D.4. What were the resources available and processes in place for monitoring, evaluating, and auditing the program? D.5. Were the monitoring and reporting requirements for IAs correctly followed? D.6. Were the reporting requirements perceived as too cumbersome by IAs? D.7. Were there any encountered problems? And, if so, how were they solved? 5.      EVALUATION APPROACH The program evaluation will utilize a variety of quantitative and qualitative methods as needed, including: analysis of program documents and administrative records, key informant interviews, focus group discussions, and beneficiary satisfaction surveys. Data will be collected from multiple key stakeholders to get a representative picture and allow for triangulation of responses at the analysis stage. The individual evaluator/firm, based on an agreed discussion with the World Bank and NVSP team, will draft the research methodology and sample selection criteria. It is expected, however, that the sample will be representative of the broader population involved in the NVSP implementation and will include a subset of the key stakeholders, namely: NVSP management and team, World Bank Group, Ministry of Social Affairs, selected NGOs, non-selected NGOs, selected beneficiaries/volunteers, procurement and financial management specialists, the private sector, etc. EVALUATION DELIVERABLES AND TIMELINE The duration of this assignment is from August, 2016 to October, 2016. The individual evaluator/firm will be assigned throughout this period 20 days, whereby they will be asked to prepare, carry out, and finalize the requested deliverables, as below: Evaluation design matrix, detailing the proposed research and sampling methodology, along with the quantitative and qualitative methods and tools to be used for data collection and analysis, based on discussions with key NVSP and WB team members; A detailed work plan on the organization and process for carrying the evaluation, including mechanisms for stakeholder participation and communication, based on discussions with the core team; A draft of the data collection instruments (in both Arabic and English) and the sample design/frame; Bi-weekly progress reports from the field with key outcomes and encountered bottlenecks; An initial draft of the program evaluation report based on the detailed parameters provided in the TOR and based on detailed discussions with the core team. The first draft will be delivered to the NVSP and WB team by October 15, 2016; A final and complete program evaluation report, taking into account feedback and requested revisions from the core team. The report will contain an executive summary that includes a brief description of the NVSP, its context and current situation, the purpose of the evaluation, its methodology and its major findings, conclusions and recommendations. The final version will be delivered to the NVSP and WB team by October 30, 2016. The individual evaluator/firm is free to hire at his/her own cost any additional researchers/enumerators that may provide support while carrying out the evaluation and conducting the field work. The individual evaluator/firm will report to Jumana Alaref (Task Team Leader, World Bank), copying Marwa El Kik (NVSP Program Manager), Carlos Asenjo (World Bank M&E Expert) and Marwan Alawieh (NVSP M&E officer). QUALIFICATIONS OF THE INDIVIDUAL EVALUATOR/FIRM This consultancy is open to both interested individuals and firms, with the following qualifications: A minimum of a master’s degree for the principal investigator/evaluator in a relevant field, such as international development, public policy, survey methodology, program evaluation, etc. is required; At least 5 years of experience in designing and conducting evaluations of social and development programs is required; Proven track record of strong research and analytical skills is required; Experience in combining quantitative and qualitative methods is required ; Knowledge of the topic of volunteerism and youth employment is an asset; Proficiency in English and Arabic is required; French is an asset.   Interested candidates are asked to send their resumes to: Jumana Alaref (jalaref@worldbank.org); copying Rene Leon Solano (rleonsolano@worldbank.org); Carlos Asenjo (Asenjo.Carlos@gmail.com); Marwa El kik (melkik@nvsp-socialaffairs.com); and Marwan Alawiye (malawieh@nvsp-socialaffairs.com).       [1] The evaluator will be free to choose indicators and calculation methods for measuring the performance of processes and implemented along the five mentioned technical methods.