Final Evaluation Study - Partners for Justice Project

 1. Context

Search is seeking a consultancy team/firm who can conduct a Final Evaluation to understand the extent to which the project has contributed to reducing Violence Against Women and Girls (VAWG) in Lebanon. This study will also aim to assess the project’s progress towards its specific objectives taking into consideration the midterm evaluation study which was conducted at the end of phase 2 in September 2022.

The Project

Despite the various efforts made in Lebanon to achieve gender equality and end Violence Against Women and Girls (VAWG), the picture is still bleak. In 2021, Lebanon ranked 132nd out of 156 countries in the world on the global gender gap index, and 6th among 19 MENA countries (World Economic Forum (2018), Global Gender Gap Report  Insight Report). Patriarchal social structures and practices limit women and girls’ access to social and economic resources, political and leadership positions, and foster their discrimination in private and public spheres. They also contribute to high rates of VAWG in Lebanese society: one third of women in Lebanon have been victims of intimate partner violence, and one in four women are subject to some form of sexual assault )UN Populations Fund (2002), International Men and Gender Equality Survey, IMAGES, Lebanon Summary).

Partners for Justice is a 51 month project started in September 2019 and due to close in December 2023, with the overall objective of reducing violence against women and girls in Lebanon. The project is being implemented in three phases, that logically followed and built on each other. The first two phases were implemented and finalized in 2020 and 2022 respectively, and were evaluated through a midterm evaluation that assessed the project against OECD criteria, and informed the design and implementation of the third phase, which started in October 2022 and will close in December 2023. This has been supported by the following specific objectives:

• SO.1 Empower men and boys to work with women to become allies and champions of gender equality and prevention of violence against women and girls;
  • ER 1.1: Youth-led organizations are more aware of issues and assumptions related to the social constructions of masculinities and violence against women and girls;
  • ER 1.2: Youth-led organizations show increased capacity to reflect on masculine identities and gender equality in their communities.
• SO.2 Promote a shift in communities’ attitudes in target areas around the permissibility of, and assumptions related to violence against women and girls.
  • ER 2.1: Media stakeholders and youth-led organizations are better able to address negative social dynamics that justify violence against women and girls and/or perpetuate victim blaming; and
  • ER 2.2: Target communities and audience, especially male members, change attitudes towards the permissibility or justification of violence against women and girls.

These phases included the below activities:

Under the first objective:

• Youth-led organizations Mapping (phase 1)
• Capacity building program to youth organizations and their representatives on gender equality and positive masculinity to increase their skills on exploring and analyzing narratives of masculinities in the contextual settings where the youth organizations are working, and contributing to providing men and boys with a safe space to reflect on their own attitudes and behaviors. (phase 1 and 2 )
• Participatory, Youth-led Gender and Masculinity Research (phase 1)
• Common Ground Approach (CGA) Training (phase 2)
• Safe Referral and Identification Training (phase 2)
• Period poverty awareness raising sessions and kits distribution (phase 3)
• Gender Mainstreaming training delivered to the CSOs (phase 3)
• CSO-led stakeholder mapping (phase 3)
• Community Dialogues and Action Plans implemented by the CSOs (phase 3)

Under the second objective:

• Media Actors Selection (phase 1)
• Media for Social Change Working Sessions (phase 1)
• National Awareness and Advocacy Campaign (phase 1)
• National Awareness and Advocacy Campaign Launch Event  (phase 1)
• Sexual and Reproductive Health and Rights (SRHR) Youth-led Initiatives (phase 1)
• Grant scheme implemented by the CSOs (phase 3)
• Cafe Chats for men implemented by the CSOs (phase 3)
• Roundtables and Briefing Papers (phase 3)
• Public Screenings of Films (phase 3)

This Final Evaluation will assess phase 3 of the project by collecting data with the project’s direct and indirect participants, but should also take into consideration phase 1 and  2 for the analysis, to draw trends of changes in a holistic approach, as well as to evaluate the integration of strategic learnings.

Target groups
The first two phases of the project targeted 1) 30 youth representatives selected from 16 CSOs targeting the Lebanese population in five geographic areas: Beirut, Mount Lebanon, Bekaa, South, and North, 2) 10 media professionals who have participated in the media for social change working sessions and helped develop a national awareness media campaign in phase 1 and 3) the community members reached by the small grants and the media campaign including men (male local leaders, religious leaders, policy makers, educators, social workers, male family members), boys, women, and girls from a variety of backgrounds, who were encouraged to challenge their own perceptions on masculinities and VAWG, and were provided with safe spaces to constructively discuss these topics.

The third phase of the project targeted 1) 15 youth representatives from 8 CSOs in three geographic areas and 2) the wider community reached by the dialogue sessions, cafe chats and grant scheme.

This Final Evaluation is expected to collect data with the project’s direct and indirect participants of phase 3.

2. Goal and Objectives of Study

The Final Evaluation Study should enable Search to inform future program design and will assess the overall success and effectiveness of the implemented project. The evaluation will also identify the added value, and/or comparative advantage of the intervention by understanding the degree to which the intervention has made a difference, positively or negatively, beyond the sheer of expected results and planned objectives. The evaluation therefore should include  Outcome Harvesting (OH) activities, which will be integrated into the final report of the Final Evaluation.

In specific terms, the Endline Evaluation has the following objectives:

• Assess the overall impact of the project based on evidence-based outcomes whether  intended (according to the project’s logical framework) or unintended outcomes (positive or negative) and determine Search’s contribution to identified changes;
• Update the Project’s logframe and indicators with End-line values;
• Assess whether the project produced the effects as envisioned in the Theory of Change in addition to assessing the project’s Effectiveness and Impact, Efficiency and Sustainability as per the Organization for Economic Co-operation and Development - Development Assistance Committee (OECD DAC) evaluation criteria; and
• Capture good practices, compile lessons learned (i.e. ending September 2022), and draw specific recommendations to inform future programme design in a similar context.

This study is expected to give specific focus to assessing the progress, effectiveness and relevance of the activities done in phase 3 with the organizations and the community members.

3. Key Questions of the Study

Evaluation Questions

The analysis will be focused on the following evaluation questions, organized by criteria. The study is expected to prioritize the collection of feedback from direct and indirect participants from the respective youth and media, and community members.

• Effectiveness and Impact:  
  • To what extent has the intervention been effective in meeting the essence of the Theory of Change and achieving its intended results, and planned outcomes? What unintended outcomes (positive or negative) emerged from project implementation?
  • What factors (positive and negative) have had the greatest influence on the achievement/progress towards the desired change/objectives? To what extent has the intervention been able to contribute to longer-term effects (impact)?
  • Was the level of partnership(s) with the CSOs appropriate to support the effective achievement of the intervention’s objectives?
• Efficiency and Value for Money:
  • To what extent have resources been allocated and utilized in an efficient manner to achieve value for money?
  • How efficient was the delivery of the intervention in terms of expenditure and implementation of activities?
• Sustainability:
  • To what extent has the project advanced in building mechanisms to sustain the benefits produced? What are efforts required to improve and strengthen sustainability?
  • To what extent did the project stakeholders (youths and youth organizations) show efficiency and ownership towards the project?
• Cross-Cutting Issues:
  • To what extent have gender issues, persons with disabilities, Do No Harm, and conflict sensitivity been taken into consideration in design and implementation of phase 3?
•  Adaptability:
  • How and to what degree did the project adapt in response to reflection and learning outputs, monitoring data, and contextual factors and/or changes?
  • What efforts have the project made to adapt to these changes, if any? and to what extent have these adaptations been effective and adequate?

The evaluation questions and methodology for this assignment may need to be further elaborated by the experts in the inception report, particularly with planning of the use of OH (see Methodology section). The expert(s)/consultant(s) may suggest additional questions. The final version of the evaluation questions will be agreed at the end of the inception phase.

4. Geographic Locations

The Endline Evaluation study will provide quantitative and qualitative information, gender-age and area disaggregated data, and analysis in Lebanon in the following areas: North, South, and Bekaa.

5. Methodology and Data Collection Tools

The Final Evaluation study will utilise a mixed-method approach consisting of both qualitative and quantitative primary and secondary data collections. The study should triangulate data from multiple sources and stakeholders in order to infer reliable findings. In addition to the proposed conventional approach the consultancy team will be taken to carry out the evaluation through using the evaluation questions (section 3) included in the ToR. The endline is expected to integrate and apply ‘Outcome Harvesting’ techniques to identify the most significant results and the project’s contributions to these changes.

In addition to assessing the intended outcomes and objectives of the third phase of the project, the consultancy team will be expected to investigate synergy with the second and the first phases of the project and draw trends of changes in a holistic approach.

Desk review: the consultant(s) will review key project documents including the proposal, the baseline report, the midterm evaluation, MEL framework, Reflection and Learning outputs, log frame, and set meetings with key personnel to better understand the project, its goal, and activities. The consultant is expected to review the above background documentation as part of the desk review phase of the study, in addition to the literature review about the Gender and VAWG context in Lebanon. Also, the consultant will utilise primary available M&E data that was formerly collated by project staff including activity surveys and assessments, baseline data, midterm evaluation, project quarterly reports, etc.

Data collection tools and approach: the consultant will develop quantitative and qualitative data collection tools and collect data through surveys, interviews, focus group discussions (FGDs), and possibly Case Studies using a solid and representative sampling approach. To ensure a better understanding of the key issues that will be addressed in the project, the data will be disaggregated by age, nationality and sex, as well as by geographical location.

Sampling strategy and approach: The consultant shall determine the appropriate sample size and approach in consultation with Search MEAL Manager and referring to the M&E plan.

The  sampling strategy will take into consideration the activities carried out, target population and the project areas (95% Confidence Level and 5% Margin of error). The consultant(s) will meet with the project participants, partners, Search staff and relevant stakeholders.

Outcome harvesting approach: In addition to the proposed conventional approach of the evaluation, the evaluation is expected to integrate and apply, where feasible steps 2-5 of the six-steps of ‘Outcome Harvesting’ (https://www.betterevaluation.org/en/plan/approach/outcome_harvesting ) to answer the questions of what actual achieved results are as opposed to those that are planned, what has changed, when and how,  identifying the most significant results, and assessing project’s contributions to these changes. Step 6 will be limited to and where feasible, the presentation of actionable recommendations in the evaluation report. The level of integration should be defined clearly in the proposed methodological approach of the consultant(s) technical proposal.

Evaluation matrix: The consultant(s) is also required to develop a detailed evaluation matrix with judgment criteria that captures the study’s objectives and evaluation questions of which will guide the evaluation process into reliable judgments and outcomes.

Search’s MEAL Manager will validate and approve the final methodology and tools before data collection begins, as well as provide feedback on the report.

Data collection, analysis, and the results should represent inclusiveness—the methodology should include a wide range of viewpoints, specifically gender and age sensitivity when applicable.

6. Deliverables

Search expects the following deliverables from the external consultant(s) as they correspond to the timeline and budget:

• An inception report detailing the methodology, data collection tools, analytical framework, approach and timeline;
• All original data set should be submitted to Search;
• Outcome Harvesting Workshops;
• A draft endline report in English for review by Search staff and partners;
• A final report in English (40 pages max in length, excluding appendices and the list of outcomes and contribution statement) consistent with Search branding and standards for evaluation. The report will:
  • Uses the Search reporting template unless otherwise agreed in the contract;
  • Provides a clear connection between the conflict or context assessment and the intended results,  articulate the project’s ToC, and include other relevant project specifics;
  • Fully explains the objectives and research questions of the study, limitations, and methods chosen  for analysis;
  • Findings respect Search’s evaluation standards, are structured around the main objectives of the  study, and are presented in relation to the intended target groups.
  • The findings should speak to the link between the project, it’s TOC and its contribution to our strategy. It should  explain adaptations that occurred during the project and their impact on results;
  • Recommendations should have a clear audience and be specific, accessible, and actionable;
  • Indicator table showing all indicators;
  • Appendices should include detailed research instruments, a list of interviewees, terms of references and evaluator(s) brief biography and list of outcomes and contribution statement
• A PowerPoint presentation that will be used by the consultancy team to present the findings to Search and the donor after the final report is approved;
• A one to two hours online presentation of research findings to the Search and donor team; and
• A two-pager summary highlighting key lessons learnt in this project that will inform Search and the Gender and VAWG  programming in the future.

7. Logistical Support

Regardless of the modality chosen, the firm/consultant(s) should be fully responsible for the data collection, including working with data collectors to be recruited and trained by them in case needed. In case of in-person data collection, the firm/consultant(s) will be responsible for organizing their own logistics for data collection (vehicles, fuel, and  drivers). All of this must be budgeted into the financial proposal.

In addition, Search will share the following elements with the external consultant(s): background materials including the project proposal and logframe, etc.

8. Ethical Considerations

The consultancy should take into consideration the rights of participants in line with the ethical codes and guidelines  of research. The participant should be asked to provide their consent and made aware that their identities would  not be revealed. Cultural rights will be respected. The consultant shall remain alert and responsive to any child safeguarding risks, acquire relevant knowledge and skills which will enable promoting strong safeguarding practices, understanding the child safeguarding policy and procedures.

The firm/consultants are required to respect the following ethical principles:

• Comprehensive and systematic inquiry: The consultant should make the most of the existing information and  full range of stakeholders available at the time of the review. Consultant should conduct systematic, data-based inquiries. He or she should communicate his or her methods and approaches accurately and in  sufficient detail to allow others to understand, interpret and critique his or her work. He or she should make clear the limitations of the review and its results;
• Competence: Consultant should possess the abilities and skills and experience appropriate to undertake  the tasks proposed and should practice within the limits of his or her professional training and competence;
• Honesty and integrity: Consultant should be transparent with the contractor/constituent about: any  conflict of interest, any change made in the negotiated project plan, and the reasons why those changes were made, any risk that certain procedures or activities produce misleading review information; and 
• Respect for people: Consultant should respect the security, dignity, and self-worth of the respondents,  program participants. The consultant has the responsibility to be sensitive to and respect differences amongst  participants in culture, religion, gender, disability, age, and ethnicity.

Applicants are kindly required to integrate a detailed multiple scenario strategy in their proposed methodologies and work plans that would ensure efficacy in the delivery of the assignment and allow for flexibility to switch between different modalities (i.e., to virtual data collection and vice-versa) whenever there is a need to do so.

9. Data Quality Assurance and Management

All deliverables will be reviewed and approved by the country office and global Institutional Learning Team prior to acceptance of the final product. The consultant(s) are also required to develop a clear and rigorous quality control plan during the inception phase of which will be implemented and deployed throughout all phases of the evaluation process to ensure reliability and validity of findings.

To ensure effective data management, all the qualitative collected data (through surveys, FGDs, KIIs) will be transcribed with the respect to the data collection ethics. All the quantitative surveys will be cleaned and stored in a secured database. Access to the data will be allowed to the staff members who work on the final evaluation and will be limited to those who have the password.

10. Timeline

The evaluation timeline is expected to be kicked off in October 2023 and concluded by the end of  December 2023, by submitting the final deliverables requested above.

11. Budget

The total budget available for this study is 20,000 USD. A detailed budget should be provided, including daily rates for personnel, and costs related to data collection (per total number of people sampled, sites for collection, etc.), analysis, and production of deliverables. The selection committee will give preference to the applicant(s) with a relatively lower budget and higher quality technical proposal.

12. Requirements of Consultant(s)

Education:

• Lead consultant(s) must have at least a Master’s degree (PhD preferable) in research methods and/or evaluations, development, conflict or peacebuilding, M&E or any related social sciences.

Skills and experience:

The following skills and experience are expected by Search for our evaluator for this project:

• Excellent proficiency in English and Arabic - written (including professional English reporting) and spoken;
• More than five years of experience in programme/project evaluation, including collecting data in interviews, surveys and focus groups and analysis;
• Experience in conflict analysis, working with justice, governmental and gender related topics;
• Strong understanding of concepts related to gender and masculinities;
• Experience in doing evaluations for peacebuilding and democracy, human rights and governance programmes in MENA region;
•  Experience working with international organizations;
•  Experience conducting quantitative surveys and analysis;
• Monitoring and Evaluation methods and data collection skills; and
• Familiarity and experience with contextual challenges in the geographic location(s) where the study will take place.

13. Selection Criteria

Consultant proposals will be selected for:

1. Relevance of proposed methodology to the goal, objectives, and research questions for the project.
2. Quality of proposed methods, conflict sensitivity approaches, and quality control measures.
3. Qualifications of the candidate(s).
4. The proposed budget in relation to the proposed methodology, deliverables and team.
5. Timeline for proposed activities.