Consultancy “Palestinian Livelihood” Project

i. Table of Contents i. Table of Contents. 2 ii. Acknowledgement. 2 iii. Affirmation. 3 iv. Glossary. 3 1. Project and Evaluation Summary. 3 2. Description of project being evaluated. 4 3. Evaluation target audiences. 6 4. Evaluation type. 7 5. Evaluation purposes and objectives. 7 5.1 Evaluation Matrix. 7 5.2 Evaluation achievement matrix. 9 6. Evaluation methodology. 10 6.1 Proposed Methodology. 11 6.2 Data collection and analysis. 12 7. Authority and responsibility. 12 8. Team advisors. 13 9. Logistics. 13 10. Products. 13 10. Budget. 15 10. Documents. 15 11. Qualifications of the consultant. 15 12. Duration of consultancy. 16   ii. Acknowledgement Appreciation is extended to all parties who contributed to the release of this document: ü  Soha Karam – Design, Monitoring and Evaluation Manager ü  Yara Shamlati – Design, Monitoring and Evaluation Officer ü  Rebecca Semaha – Senior Design, Monitoring and Evaluation Coordinator   iii. Affirmation “Except as acknowledged by the references in this paper to other authors and publications, the evaluation TOR described herein consists of our own work, undertaken to describe and advance learning, as part of the requirements of World Vision’s Design, Monitoring and Evaluation Learning System.” Primary quantitative and qualitative data collected throughout the evaluation process remains the property of the communities and families described in this document. Information and data must be used only with their consent.   Zeinab Karout – South Area Design Monitoring and Evaluation Officer - February 2015 iv. Glossary AUB American University of Beirut BCC Burj Chemali Camp BD Business Development BDS Business Development Services BF Business facilitation DME Design, Monitoring and Evaluation FGD Focus Group Discussion FY Fiscal Year HR Human Resources ILO International Labour Organization KII Key Informant Interview MFI Micro Finance Institution NGO Non Governmental Organization PAWL Palestinian Arab Women League PDQA Programme Development and Quality Assurance TOR Terms of Reference UNRWA United Nations Relief and Works Agency for Palestine Refugees WV World Vision WVL World Vision Lebanon 1. Project and Evaluation Summary Project Name Palestinian Livelihood Project Number L196210 Project location Tyre, South Lebanon Bourj El Chemali Camp (BCC) and surrounding gatherings Project Goal Families in Palestinian refugee camps have improved livelihoods Project Outcomes # 1: The targeted Palestinian community has increased access to micro-loans and embedded Business Development Services (BDS) # 2: The targeted Palestinian community have improved business development knowledge and skills; business networks and environment. Project duration October FY12- September FY15 Evaluation type End of project evaluation Evaluation Purpose Assess the effectiveness of the programme that goes in line with the projects’ successes and challenges.   Understand what changes have occurred since baseline and understand World Vision’s contribution to these changes, including any unintended consequences. Primary Methodologies Quantitative and Qualitative methodologies Evaluation duration April FY15- June FY15 Anticipated evaluation final report release date Monday, June 29, 2015 2. Description of project being evaluated The Palestinian livelihood project as part of the refugee programme is mainly implemented in one refugee camp, Burj el Chemali Camp (BCC), in Tyre, South of Lebanon. In addition to other nearby camps El Buss Camp, Jal El Baher gathering, Qasmieh gathering, Ma’ashouk gathering.   The project consists of supporting the Palestinian community in their quest to improve their living conditions. The purpose of the project is to increase and stabilise the income and economic resilience of the targeted Palestinian community and caregivers so they can better provide for their children. Providing for their children would include meeting children’s basic needs such as food, shelter, health and education.   This project follows a Business Facilitation (BF) model which has been contextualized to fit the camps’ situation. A study conducted in 2010 by the American University of Beirut (AUB) and UNRWA revealed unsatisfying socio-economic indicators for Palestinians: the housing conditions are below standards, the camps are overcrowded, school enrolment rates decrease with age, 31% suffer from chronic illnesses (83% of people above 55 reported at least one chronic illness), and 24.5% experienced an acute illness in the last 6 months.   As a result of the overwhelming need of Palestinian refugees World Vision Lebanon (WVL) has been working with this target group since 1997. WVL has previously worked on a series of issues in the camps including health and disability. However in recent times WVL staff working in the camps, Non Governmental Organizations (NGOs) and the target communities themselves began to identify the need for WVL to change its focus and prioritise economic development to assist the target group to become economically viable and independent. In response to this identified need WVL conducted a rapid assessment to identify exactly what they should do to support the target group to increase their income. The rapid assessment included a comprehensive literature review and a series of key informant interviews with WVL staff working in the camps and other NGO’s working in the camps. In summary the literature review highlighted that facilitating access to waged employment can be a somewhat limited income generation strategy for the target group because Palestinian people are legally restricted from working in over 70 professions. Those professions they are legally allowed to gain waged employment in are often poorly paid, seasonal and therefore provide irregular employment opportunities and many can be in unsafe environments[1]. As a result the International Labour Organization (ILO) found that facilitating access to employment did not significantly reduce poverty amongst the target group[2].    Conversely, the rapid assessment identified significant opportunities to facilitate self employment or business development as a strategy to increase income. Palestinian people are able to start businesses both inside and outside the camps. Self employment through businesses has no ‘earning’ limits like the waged employment options and is not as legally restrictive. Some indicators for business development opportunities include staff and partners identifying a large number of products people living within the camp demand, but are not available inside the camp. Additional business and market opportunities were confirmed by other NGO staff working in business development and secondary data from the ILO. Interestingly historical evidence shows that in the past the most successful markets were in fact inside the refugee camps, and people who did not live in the camp came into the camp to buy their goods which again illustrates the potential of a business development strategy. Hence WVL decided it was best to focus on business development.   WVL has experience working in the refugee camps, but no experience working in the sector of economic development or business development in the camps.  For this reason WVL decided to partner with a Microfinance Institution (MFI) in the BCC called Palestinian Arab Women League (PAWL) that works in Palestinian camps and gatherings, and it specializes in Business Development through the provision of micro-loans and Business Development Services (BDS). Throughout this project, money was provided to PAWL to increase their revolving micro-loans and associated BDS. With the additional funds, PAWL provided additional beneficiaries with micro loans and hired an additional BDS specialist so more clients can receive BDS more frequently. PAWL was also supported technically to improve these services which will reflect positively on its beneficiaries’ ability to start or expand a business. World Vision Lebanon (WVL) supported PAWL to monitor the usage of the new information in assessing loan applications and provision of BDS and shared Business Development (BD) tools with PAWL that will further help improve and standardize their service.    WVL worked as well in partnership with the target community to improve the business environment in the camps. It is important to create an environment that is good for business so that it is easy for the target group to start or expand businesses. The project improved the environment by supporting Business people in the target group to come together into business networks. In the networks the target group shared information and knowledge with each other to help each other start and expand their business. The types of information and knowledge shared included the following: information about resources and services available to help start and run a business, their experience in business, and technical knowledge such as book keeping and marketing. This process of networking and sharing also improved individual’s business performance and business confidence through new information, knowledge and skills. The shared linkages, networks, relationships, and information improved the business environment and social capital in target. 3. Evaluation target audiences The following parties are involved and have a major interest in the evaluation process:   Community members: Palestinian community members including business owners, consumers, loan receivers, people who benefitted from the capacity building trainings and workshops provided by business networks and their families will take part in the evaluation process, to ensure the latter is being conducted in a partially participatory manner. A fully participatory approach would entail the participation of community members in all the actual implementation of evaluation process. Results will be disseminated to all beneficiaries who took part in the evaluation, in order to be transparent and accountable towards the community that WVL serves. WVL: The data collected from the evaluation will inform WVL on the successes and challenges faced throughout the project implementation. It will also highlight the levels of impact or differences that the project has made to child well being outcomes and the relevance to the communities’ needs. Documented information will further contribute to generating lessons learned and recommendations that should guide future interventions implemented by WVL, particularly in Palestinian camps setting. WVL Partner (PAWL): PAWL will be the main participant in the evaluation as their input is the main source of qualitative information. The information will be used by PAWL to assess whether it’s already established activities have been increased and its services have been improved due to the financial and technical support provided by WVL. Support Office – World Vision Australia (WVAus): The results of the evaluation will present information pertaining to the project’s impact on children’s wellbeing. It offers the opportunity for WVL to be accountable towards the main donors as the process will be documented in details and information generated with the participation of the community groups and stakeholders. 4. Evaluation type The current evaluation is an end of project evaluation. This evaluation will allow the comparison of the findings with the baseline results in order to highlight effectiveness and changes across time. It would also measure if the initial assumptions held true and if the approach is worth replicating in other Palestinian camps where WVL is operating. The evaluation aims to analyze the project implementation approach (which would include but will not be limited to the analysis of the WV partnership approach with PAWL) and to identify best practices and lessons learned. 5. Evaluation purposes and objectives The purpose of livelihood project evaluation is to (1) assess the project strategy and implementation, (2) assess the effectiveness of the project by measuring the achievements of the programme outcomes and goal based on the current logframe design, monitoring and evaluation data and (3) give recommendations to guide future programming. The objectives of the current evaluation are as follows: Assess the project alignment with overall WVL strategy and operations strategy including the project design; Evaluate and validate project achievement of implementation targets and milestones reached; Examine the effectiveness of the project in terms of its purpose and intended outcomes; Evaluate the relevance and appropriateness of actions and strategy; Ascertain the impact the project had on targeted communities (positive, negative, intended, unintended impact...); Examine the management aspects of the project including staffing, resource allocation and use, policies, coordination arrangements, administration, monitoring, reporting and documentation among others; Investigate the connectedness/sustainability of actions promoted by the project; Suggest strategic and operational recommendations that can be used by WVL in the ongoing and future programmes/projects. 5.1 Evaluation Matrix This evaluation matrix is required to be completed by the consultant as part of developing the evaluation methodology. Evaluation Objectives Issues/Questions to be asked Methodology for data collection Sources To identify if the project was properly designed to meet targeted needs (Relevance) What was the added value of the project to the community? To what extent was the project successful in addressing the needs of the community? How appropriate was the project from the communities’ points of views including partner (PAWL)? How appropriate was the project from WVL staff points of view? How appropriate was the selection of the partner for the project? How did the challenges faced in the management affect the achievement of the project’s objectives?     To assess the progress made towards achieving the project’s goal and objectives based on the log frame, design and monitoring data (Effectiveness) How well were the projects’ objectives achieved? What were the projects’ strengths in terms of the method used and the design? How properly was project executed and how effective was the monitoring system?  How did implementation variations (from project design) affect results? What were the projects’ weaknesses in terms of the method used and the design)? What alternatives methods would have led to a better achievement of project’s objectives? How well did the project adapt to unexpected problems? How flexible was the project design?       To assess the potential Impact of the project on the targeted communities To what extent have beneficiaries benefited from project activities and outputs? To what extent has local partner (PAWL) benefited from the activities and outputs? Have there been any unintended or negative changes that can be attributed to the project? What differences have the project made to people’s lives?     To assess the sustainability of the project, particularly focusing on the partnering approach and the capacity and willingness of partner to sustain the project’s achievements (Sustainability) To what extent do community members and partner own the project? What are the individual, organizational, and community level capacities? To what extent are community members able and willing to sustain the efforts WVL phase out? Are there mechanisms in the community to sustain the project? What was the added value of the partnership established with PAWL?     To investigate whether the resources (financial, human, and materials) have been used efficiently and effectively for the well-being of the target community. (Efficiency ) Was the methodology of implementation adequate under the circumstances? Were most of the outputs achieved to an acceptable standard?     5.2 Evaluation achievement matrix Livelihoods for Palestinian Project Goal indicators Indicator 1 % of beneficiaries increased their business profitability (disaggregated by gender ownership) by 15 percent in 2013 (mid-term) and by 30 percent in 2015 (end of project). Indicator 2 % of caregivers who are able to meet the at least 3 basic needs of their children"-measuring health, education, shelter, clothing Sub goal level Indicator 1 50 % of beneficiaries who express increased confidence by end of project Indicator 2 30 % of beneficiaries who expand their business premises, move location and/or report using improved equipment. Indicator 3 Number of new business start-ups Outcome 1 indicators Indicator 1 PAWL beneficiaries satisfied with services provided by PAWL Indicator 2 % of Palestinian community that have accessed loans from PAWL Indicator 3 % of Palestinian community demanding/applying for services from PAWL Indicator 4 Percentage of loans overdue from the loans received from WVL loan pool Indicator 5 PAWL staff who report their operations and product quality has improved as a result of WVs mentoring Outcome 2 indicators Indicator 1 Increase in the degree of collaboration between individual business owners Indicator 2 Change in the business environment caused by networking/collaboration Indicator 3 70 % of participants report having improved business knowledge, skills and information for business operation for at least 4 business practices Indicator 4 70 % of participants that practice at least 4 improved business practices 6. Evaluation methodology The evaluation methodology is informed by the M&E plan specified during project’s design and it is aligned with WV’s guidelines and standards for ensuring good quality evaluation process. The evaluation methods should be a combination of quantitative and qualitative methods with the relevant stakeholders, partners, beneficiaries, project coordinators, manager, monitoring & evaluation staff... Data collection methods may include but are not limited to secondary data information, general observations, surveys, in depth interviews, Key Informant Interviews (KII) with main stakeholders/partners, and focus group discussions (FGD).   The consultant is expected to develop the appropriate methodology and tools that would best capture the objectives of the evaluation, based on the following assessments: The baseline assessment that was conducted in BCC whereby surveys were administered to business owners within the camp. A total of 99 surveys were filled, randomly selected from the total of 286 businesses identified within the market assessment. Furthermore a FGD was conducted with loan beneficiaries from PAWL, grouping 6 beneficiaries, 2 women and 4 men. This Focus Group Discussions (FGDs) aimed to assess the level of satisfaction from PAWL services, ways to improve those services. It also served to validate some of the findings from the baseline surveys and get more depth about them. The market assessment that was based on different data sources and relied on multiple data collection methodologies, aside from the secondary data analysis. a.     Qualitative assessments consisted of conducting: -       Key informant interviews (KII) with local authorities and active people from the community. A total of 3 KIIs were held with prominent stakeholders from the community -       Focus groups discussions (FGD) with entrepreneurs: a total of 3 mixed FGDs were held (about 20 participants in total). -       FGD with consumers: a total of 3 mixed FGDs were held. The FGD and KII were based on a purposeful and convenience sample whereby the most active and interested entrepreneurs, consumers and organizations were involved in the data collection. The FGD and KII were equally representative of the women and men’s voices. b.     The quantitative assessment included: -       Surveys with small business owners. The survey with entrepreneurs was conducted with 4% of the total sample with 26 men and only 4 women being interviewed. The assessed sample was chosen randomly from the list of businesses in the shop. -       A business mapping was also conducted to identify the various businesses in the area. The consultant would also have to review project budget and financial figures and review project policy documents including Human Resources and others and establish how they were applied. 6.1 Proposed Methodology The evaluation will be following but not limited to the below methodologies: -       Quantitative evaluation targeting business owners in the camps based on the baseline tools that are adapted to capture all the changes in the project -       Qualitative evaluation with loan beneficiaries, consumers, people who benefitted from the capacity building trainings and workshops provided by business networks, their families, PAWL representative and project staffs. 6.2 Data collection and analysis For the validation meetings and qualitative data collection, it will be completed by the consultant as a facilitator, and a note taker will be present during the FGDs and the KIIs. For the FGDs and the KIIs, thematic analysis of the data will be completed in order to discern patterns and come up with conclusions that would reflect the views and the perceptions of the participants. The information collected during the FGD will be used to triangulate the quantitative data for a holistic understanding of the evaluation results. The details of the data collection are shared below in section 7 “Authority and Responsibility”. For the quantitative data collection, it will be completed by selected data collectors hired by the consultant and supervised by him/her. Appropriate statistical tests will be employed pending the type and measurement scale of the outcomes and independent variables (e.g., continuous, nominal, or ordinal). For instance, ANOVA will be used for comparisons between means of scores (probably knowledge and attitudes); chi-square test will be used to determine the significance of differences in percentages;  spearman rank correlation coefficients will be used to examine the relationship between different scores used; non-parametric equivalent tests may be used in case the data is not normally distributed etc. In depth statistical analysis is required; data should be analyzed with statistical significance set at 0.05 and confidence intervals should be calculated and added. 7. Authority and responsibility Role Responsibilities and tasks Consultant Develop and submit the evaluation methodology (sample selection, quantitative & qualitative methodology) Submit evaluation matrix Develop data collection tools based on WVL’s standards and aligned with the tools used during the baseline Pilot test the tools Recruit and train data collectors Recruit and train data entry clerk Supervise and assure quality of quantitative data collection Review and validate qualitative data previously collected Conduct in depth data analysis Provide weekly feedback to WVL team Prepare a power point presentation of the main findings Present findings to WV staff and partners Support WVL team in validating the results with business owners, loans beneficiaries, their families and PAWL Draft the Evaluation report Finalize the Evaluation report based on feedback from WVL team Submit a 2 pager summary fact sheet DME officer Finalize the evaluation TOR Provide feedback on evaluation methodology and matrix Provide feedback on data collection tools Follow up with the consultant on data collection and analysis Provide feedback on data analysis Provide feedback on final evaluation report Economic Development Specialist Technical support in finalizing data collection tools Technical input for the analysis, synthesis of data, and drafting of recommendations Provide feedback on the evaluation report   8. Team advisors The team advisors are point people with defined expertise that will serve the purpose of the evaluation. Their input will contribute to the development of a good quality evaluation through their support and orientation to the evaluation team prior to and throughout the process. Those will comprise WVL PDQA Manager, the concerned WVL Technical Specialists and WVL DME Manager.   9. Logistics The recruited consultant will be in charge of finalizing logistical issues related to the transportation to the camp where data collection will take place. As for the location for completing the different meetings (KIIs, FGDs, validation meetings, etc.), it will be in WVL’s office in Tyre ADP or in PAWL’s office inside the camp. The paragraph below will explain in details how the data collection and validation will take place: Data collection: WVL staff of the Palestinian livelihood project will be supporting with the coordination of the different meetings/surveys with beneficiaries, partner and project’s stakeholders. Validation meetings: The validation meetings will be completed at Tyre ADP office. The final presentation of results will take place in one meeting where WVL and partner’s staff will be invited to participate in the discussion and learn about the evaluation results.   10. Products The end product of the evaluation will be the evaluation report. In the preparation of the final report, the consultant is expected to provide the reader with, as much as possible, accurate sources of information and conclusions. In fact, all findings and statements must be backed up by existing data and/or data based on approved methodology for sampling, data collection, and analysis. Expected deliverables throughout the process of evaluation include: Copy of training package and materials used for the evaluation team; Evaluation Methodology (including Evaluation Matrix) Survey guideline - including questionnaires, qualitative tools, sampling methodology, sample selection criteria, data collection and analysis protocols; Copies of all quantitative data (datasets, output analysis, tabulation of results including descriptive analysis and associations...) and qualitative data (thematic in-depth analysis...) Final Evaluation Report in English (as per WV’s Evaluation Report Guidelines to be provided); and in hard copy and electronic forms;   The Final Evaluation Report Structure will consist of: Cover page Table of contents Acknowledgements Affirmation Glossary Introduction Executive summary Evaluation Background Methodology Limitations Findings Conclusions and Recommendations Lessons Learned from the evaluation process Appendices: -       To include copies of all tools, list of enumerators, survey timeline including all KII and FGD participants and discussion transcripts (as many pages as necessary- please add reference to the annexes in the report, but include them in a zip file as separate documents). -       A 2-page evaluation summary fact sheet: this is to include relevant findings from the evaluation, key points and recommendations. -       An Indicator Summary sheet, giving status of all indicators measured in the surveys. -       A power point presentation in English including methodology, main findings, recommendations, best practices and lessons learnt.   10. Budget Financial bids should be submitted by the Consultant, and specific details will be worked out based on discussions and plans between the Consultant and the South DME Officer. The DME staff could be able to support in recruiting enumerators if needed. The funding available for this end of programme evaluation should include transportation and accommodation fees identified by the consultant.   10. Documents Below is the list of the documents to be reviewed for the purpose of the evaluation: -       Market Assessment -       Baseline evaluation report -       Palestinian livelihood project design document -       Monthly reports -       Semi-Annual and Annual reports   11. Qualifications of the consultant The consultant should have the following competencies and experience: ü  At least 5 years of progressive proven experience in similar studies and in using both qualitative and quantitative research methodologies and data analysis; ü  Previous evaluation experience of economic development (livelihood) related projects (to be provided within the CV); ü  Extensive professional experience in the design and implementation of outcome and impact evaluations; ü  Good analytical and critical thinking; ü  Proven knowledge and experience in applying participatory research methods and tools; ü  Advanced degree in relevant field (e.g. Social Sciences, Public Health, Epidemiology or any related technical field); ü  Good understanding of the Humanitarian work especially development field; ü  Familiarity with civil society and Non Governmental Organization (NGO) engagement; ü  Ability to work on tight schedules with minimal supervision; ü  Good English and Arabic speaking (preferable) and writing skills.   12. Duration of consultancy WVL has set aside 3 months (April 1, 2015 till June 30, 2015) for this assignment (Taking into consideration all public holidays during these months). Bids should include a detailed proposed work-schedule with specific tasks and should also incorporate a budget that outlines all relevant costs that will be associated with this evaluation. Candidates are expected to state how much time they will need to start the assignment and how much time they need to conduct every task. Selected candidates are also expected to abide by the deadlines and the conditions for deadlines specified within the contract/agreement.   TASKS TIMELINE Review of TOR 1st week of April Review baseline indicators and surveys, KIIs & FGDs used during baseline 1st week of April Design of the evaluation methodology 2nd week of April Develop all necessary data collection tools including KIIs, FGDs Surveys... (These tools will be reviewed by the Design, Monitoring and Evaluation Team, as well as Technical Specialist for approval) 2nd- 3rd week of April Use the appropriate method to calculate the sample size and carry out sampling 4th week of April Recruit and train enumerators, supervisors and facilitators and oversee pilot testing 4th week of April Supervise the quantitative and qualitative data collection while insuring the quality of data 1st- 3rd week of May Design the database for data entry 3rd week of May Supervise data entry and quality data management 3rd- 4th week of May Carry out data analysis as well as report writing 1st- 3rd week of June Present main findings and a debriefing of the evaluation to the Design, Monitoring and Evaluation team as well as key stakeholders 3rd week of June Present the draft report in a review meeting and incorporate the feedback and compile a final report 4th week of June   In the preparation of the final report, the consultant is requested to provide the reader with, as much as possible, accurate sources of its information and conclusions. In fact, all findings and statements must be backed by existing data and/or data based on approved methodology for sampling, data collection, and analysis. Interested individuals and consultancy firms should send in their applications no later than 16.00 hours GMT 13th of March, 2015 including a detailed Technical and Financial Bids with the following documents: proposed methodology (including a revised timeline and budget), curriculum vitae and/or resume and the names and addresses (including telephone and e-mail) of two non-related referees (Technical and Financial Bids should be in English and not more than 3 pages). Important: If application is sent by email, the subject line should clearly indicate “Palestinian Livelihood Project evaluation”. If sent by post or delivered by hand, the top right-hand side of the envelope must be clearly marked “Palestinian livelihood project evaluation”. Only short-listed candidates will be contacted. N.B.: Selected consultants will be provided with all additional documents required to carry out the work. [1] Chaaban, J. et. Al (2010) Socio-economic survey of Palestinian refugees in Lebanon. American University of Beirut and UNRWA. [2] ILO