RDPP External Evaluation

Terms of Reference

Final External Evaluation

Background

With the support of the European Regional Development and Protection Programme for Lebanon, Jordan and Iraq (RDPP II), Oxfam and partners Legal Agenda (LA) and Lebanese Observatory for Workers and Employees’ Rights (LOWER) are implementing a project titled “Promotion of decent work and sustainable business development in Bekaa, Lebanon.” In addition, the project worked very closely with My Work, My Rights! Network, an informal group of local Civil Society Organisations (CSOs) that work on advancing decent working conditions in Lebanon.

The project aimed to contribute to a decent and economically sustainable business environment for displacement-affected populations in the Bekaa by strengthening the linkage between the local civil society, the business development service (BDS) providers, and the Micro, Small, and Medium Enterprises (MSMEs) in Bekaa. To achieve this ultimate objective, the project worked towards improving the organizational, technical, and evidence-based advocacy capacities of the My Work My Rights! Network of CSOs to mainstream and collectively monitor decent work conditions; developing the capacities and tools of business development services in Bekaa on decent work with the support of the network; and providing MSMEs in the Bekaa, specifically in sectors where Syrian refugees can work, with technical support through business development services and financial support to strengthen their business and improve their working conditions.

The project intervention logic includes three expected outcomes:

Outcome 1: My Work, My Rights! Network and members are empowered to influence practices related to decent work at the local, national and international level.

Result 1: My Work, My Rights! Network’s governance structure, strategy, action plan and communication plan are developed

Result 2: My Work, My Rights! Network’s tools with integrated decent work agenda

Result 3: Network’s capacity on influencing tactics and policy development is developed

Outcome 2: Targeted private business development organizations and professionals with strong decent work capacities and services are convened and capacitated on the topic of decent work in the Bekaa.

Result 1: Business development organizations and professionals’ capacity is strengthened

Outcome 3: Targeted MSMEs in the Bekaa have improved working conditions and accessible decent job opportunities.

Result 1: MSMEs technical capacity is developed with continuous coaching and follow-up support

Result 2: MSMEs are financially supported to grow

The project period was marked not just by continual multifaceted crises but also by the outstanding events like COVID-19, lockdowns, the revolution, the Beirut Blast, the political deadlock, and the economic collapse. The combined impact of the events has contributed to significant changes in the context, the national priorities, and the operational environment that should be considered against the evaluation criteria.

Below are objectives and scope of the evaluation, evaluation questions, and possible methodologies, timeline, selection of the final evaluation consultant, deliverables, and indicative planning.

Objective

The purpose of the external evaluation is to:

1) Assess the outcomes and initial impact achieved through the implementation of the project activities and verify to what extent the project contributed to a decent, economically sustainable business environment for displacement-affected populations in the Bekaa, Lebanon. It should provide an objective analysis of the project’s performance in terms of progress and process, relevance, sustainability, and the extent to which the project is contributing to enabling the desired impact. The evaluation should focus on the Theory of Change and determine whether the underlying assumptions behind it were relevant despite the changing context. It should present results, conclusions and document key lessons learnt from project implementation. It should also provide clear recommendations that can guide Oxfam for the implementation of future development projects that combine MSME support, advocacy and working with networks.

Noting the project has underwent a mid-term review over July and August 2021 that highlighted some findings and recommendations, some of which have been put into effect.

In particular, the external evaluation is intended to:

1. Assess the project relevance examining whether the funding and support is consistent with the emerging needs, priorities, and possibilities (locally, regionally, and nationally where applicable).
2. Evaluate the adequacy and effectiveness of the project implementation, delivery of project outputs and outcomes and partnerships. Assess the extent to which a human-rights based approach is employed.
3. Assess the synergy and coherence of the project with Oxfam’s strategy, other livelihoods projects at Oxfam, humanitarian-development nexus, and synergy with RDPP’s strategy, etc.
4. Assess results achieved in comparison with the performance indicators outlined in the project Monitoring, Evaluation and Learning framework, including a re-examination of the validity of the project design and relevance of project’s theory of change.
5. Assess the efficiency of the project and identify significant key internal and external factors that have facilitated or impeded the delivery of outcomes. This includes exploring delays in project implementation, their causes and how they have been managed to draw lessons from the delays and provide suggestions for improved implementation.
6. Develop recommendations for future projects that include working with networks, advocacy and MSME support.
7. Evaluate to what extent the project is deepening localization of aid including the partnerships as per Oxfam partnership principles, women’s economic empowerment frameworks, and local humanitarian leadership.

2) Develop a solid descriptive and diagnostic analysis and overview of the MSMEs that were supported in this project. The analysis should include but not be limited to the following: improvements MSMEs had, achievements made, challenges faced, growth, sustainability, job creation/retention, decent work, progress in business plans, grant disbursement, risks, profit and other business ratios. This analytic report will rely on different data sources that Oxfam has collected through enumerators, BDS coaches and staff which will be provided to the evaluators. The various data sources are listed below:

• MSME Profiling data (comprehensive data on MSMEs collected at the beginning of the project and used to determine which enterprises will be selected to receive support based on several criteria)
• Endline survey with 36 MSMEs that have only received technical support (includes data on job creation/retention, decent work conditions, feedback on BDS support)
• Baseline and endline surveys filled with enterprise owners of the 31 MSMEs that have been technically and financially supported (includes data on job creation/retention, decent work conditions, feedback on BDS support)
• Baseline and endline surveys with a representative sample of employees who work at the MSMEs that received technical and financial support (includes data on decent work conditions)
• Monitoring data acquired through monthly monitoring visits to the MSMEs receiving technical and financial support (includes data on progress made in business plans, grant disbursement, internal and external challenges)
• BDS reports (includes data on achievements, topics of technical support, progress made in business plans, profit and other business ratios, challenges, and mitigation measures)
• 31 MSMEs’ Business Plans and Decent Work Action Plans

Scope

The external evaluation will cover the period from March 2020 till September 2022 and build off the midterm review. The evaluation will create an accurate and comprehensive picture of the project implementation, generating findings on evaluation criteria and documenting good practices and lessons learned. The evaluation will integrate gender equality and non-discrimination as a cross-cutting concern throughout its methodology and all deliverables including the final report.

Audience

The audience and users of this external evaluation include the Oxfam project team, management, partners, My Work My Rights! Network members and donor.

Methodology

The evaluation must use quantitative and qualitative data, drawing upon both primary and secondary data collection techniques.  The evaluation will be linked, but not limited, to the project’s Logical Framework.

The following is a guide to the proposed evaluation questions , which must adhere to OECD/DAC criteria for external evaluation (questions below aim to specify direction and should not limit the evaluation. Evaluation methodology may inform more and/or different questions).

Relevance: Does intervention objectives and design respond to target group, national, and partner/institution needs, policies, and priorities, and continue to do so in light of the changes in context?

• Did the project planning and implementation take the local context into account? Was the project based upon an adequate needs assessment? Does it show an understanding of and support for the livelihoods and capacities of the targeted population?
• Was the structure of the network a good fit to work with each other, considering its objectives?
• To what extent are MSMEs in need of the provided technical support? And financial support? Was the project designed in a way that meets their changing needs?
• To what extent are MSMEs able to create/maintain job opportunities and sustain their businesses within current context?

Effectiveness: To what extent have the expected outcomes and objectives of the project been achieved thus far? What limiting factors affected the realization of these outcomes? What was Oxfam’s contribution and added value?

• Were the capacities of the network members to advocate strengthened? If not, what were the constraints that limited the effectiveness of the capacity building? Were network members able to coordinate and jointly and collectively engage in lobbying and advocacy? Were the tools developed by the network effective at advocating for decent work and improving work conditions?
• Were BDS organizations able to roll out the learned concepts on decent work within their work with the MSMEs? If not, what limiting factors affected the above?
• Were MSMEs able to grow and sustain their businesses with the financial support received?
• Were supported MSMEs able to create new decent jobs? Were MSMEs able to apply the learned concepts on decent work and make improvements in their working conditions within their enterprise?
• Which of the project pathways have been most effective and least effective in creating the desired change?

Efficiency: Has the project been implemented efficiently, cost-effectively, and been able to adapt to any changing conditions?

• Were activities cost-efficient? Did the actual results (outputs and outcomes) justify the costs incurred?
• What coordination mechanisms worked and what didn’t? Are the partnership and management arrangements functional and conducive to the project objectives being achieved within the project timeframe?
• How has the relationship been between the network members? The network and Oxfam?
• How efficient was the coordination and overall relationship between MSMEs and Oxfam? Between MSMEs and BDS organization? Between MSMEs and the network?
• How efficient was the technical BDS support that was provided to the MSMEs? If not, what was missing?

Impact: What are the long-term effects, both positive and negative, that can be reasonably attributed either partly or entirely to the intervention whether directly or not, or intentionally or not? Has the project been able to contribute to a decent and economically sustainable business environment for displacement-affected populations in the Bekaa, Lebanon?

• Did the intervention of the network with the MSMEs contribute to improvement in decent working conditions? Have decent work action plans been put into use?
• Have the advocacy initiatives and policy lobbying by the network reached key policy makers in Lebanon? How engaged are policy makers in supporting decent work?
• Has the technical and financial support contributed to the ability of MSMEs in Bekaa to economically grow or sustain their operations?
• To what degree did beneficiaries' access to income and decent work conditions changed due to different approaches? Which approach had the most positive impact, why? Any differences between refugees and the host community?

Sustainability: To what extent are the benefits of the project likely to continue once donor funding has ceased?

• Is the network able to sustain itself and continue with its mission of decent work promotion and advocacy after the project ends?
• To what extent has revisiting the internal bylaws and policies, ways of working, memberships, roles and responsibilities, fundraising and sustainability plans contributed to the establishment of a common and unified vision, mission and strategic objective which network members are committed to?
• Are supported MSMEs technically and financially capable of sustaining and growing their businesses?
• How likely is it that MSMEs remain committed to the improvements that they have made in terms of decent work? Will MSMEs continue to prioritize decent work and improve these conditions in light of the ongoing crises? If yes, what factors contributed to that? If no, why not?

Other crosscutting questions:

• To what extent was the design and implementation of the intervention gender-intentional? To what extent does MSME uptake of decent work measures create more gender equitable and/or inclusive working conditions?  What is the impact for refugees?
• What and how were the contextual factors (COVID19, economic, social, financial, political crisis) affecting strategies, timely implementation and results?
• How are the programme MEAL systems enabling adaptive decision making within the programme?
• Are the partnerships in line with Oxfam’s principles? Does the project design and implementation support Oxfam’s local humanitarian leadership (LHL) agenda? Did it recognize and complement existing local capacities of partners? Did it promote the role and work of partners? Do partners Share the same vision and values? Is there Complementarity of purpose and value-added? Is there space for autonomy and independence? Is there transparency and mutual accountability? Are roles and responsibilities clear?
• Was Oxfam and partners making room and engaging in individual or collective reflection exercises and committed to learning? Have any of the learnings been operationalized? What were the gaps and barriers in the learning aspect?

The evaluation is suggested to include the following components:

1. A thorough secondary review of existing data and reports, including baseline and endline data, monitoring data, BDS reports, workshops, produced advocacy documents and actions, project documentation, quarterly information notes (QINs), partner reports and similar interventions etc.
2. Group or individual interviews with Oxfam staff (3-4 individuals)
3. Group or individual interviews with partners’ staff (3-4 individuals)
4. Group or individual interviews with external stakeholders (other NGOs, local authorities, 2-3 individuals)
5. Group or individual interviews with Network members (up to 15 organizations)
6. Group or individual interviews with BDS organizations and individuals (10 – 21 individuals)
7. Group or individual interviews with MSMEs (up to 55 individuals)
8. Group interviews with employees and workers with the MSMEs supported (up to 80 individuals)
9. Validation workshop with Oxfam and partner staff in Beirut.

Depending on the health and security developments with COVID-19 and roadblocks in Lebanon, meetings, interviews, workshops, and briefings may be moved to phone-based, or online via video-conferencing technology such as Zoom, Microsoft Teams, and Skype.

Timeline

The evaluation is expected to require no more than forty (40) working days, including design, data collection, drafting and finalizing between August and October 2022.

The following are the suggested stages of the evaluation:

• Stage 1: Program documentation review and data collection strategy with detailed timeframe, finalized data collection tools, list of stakeholders that will be involved in data collection in the form of an inception report (2 weeks).
• Stage 2: Interviews and data collection (2 weeks)
• Stage 3: Data analysis, validation, drafting and submission of the evaluation report in addition to data analysis and drafting of the MSMEs Data Overview (2 weeks).
• Stage 4: Validation workshop and finalizing the evaluation report and the MSMEs data report (2 weeks).

The consultant will be expected to provide a proposed timeline and linked budget for the evaluation.

Deliverables

The consultant will deliver the following:

1. A proposed methodology for the evaluation, including budget and timeline.
2. Present the findings of the programme documentation review and data collection strategy with detailed timeframe for the evaluation, finalized data collection tools, list of stakeholders that will be involved in data collection in the form of inception report.
3. The draft evaluation report, with up to 3 rounds of comments/reviews before validation (see Annex 1 for Oxfam’s recommended outline of an evaluation report).
4. The draft MSMEs data report, with up to 2 rounds of comments/reviews
5. Presentation and facilitating the findings to Project Team, donor, including partner organizations and relevant stakeholder representatives through a validation Workshop. The evaluators are expected to deliver the presentation in PPT.
6. The final evaluation report.
7. The final MSMEs data report.

Profile and Requirements:

Qualifications:

• Demonstrated experience (minimum 5 years) in the evaluation of civil society projects in Lebanon.
• At least a master’s or professional qualification relating to Monitoring, Evaluation and Learning.
• Strong verbal communication skills in Arabic and English, including the ability to interview diverse actors and facilitate workshops.
• Excellent report-writing and presentation skills in English and in Arabic.
• Proven evaluation design, analysis, research and report-writing abilities.
• Knowledge of and experience with working with networks in Lebanon, advocacy and campaigning, and decent work.
• Knowledge of and experience with livelihoods programming and gender mainstreaming.
• Knowledge of Oxfam and its work (Desirable).