End of Project Evaluation of Recovery of COVID-19 affected communities in western Nepal (Recover) Project

End of Project Evaluation of Recovery of COVID-19 affected communities in western Nepal (Recover) Project

TERMS OF REFERENCE (ToR)

1. BACKGROUND AND CONTEXT

Introduction

Lutheran World Relief (LWR) is a U.S.-based 501(c)3 organization founded in 1945 with a mission to end poverty, injustice, and human suffering. LWR focuses its work on humanitarian assistance and long-term development, laying a foundation for resilience, sustainable adaptation to climate change, and the strengthening of value chains. Lutheran World Relief combined with IMA World Health in 2020 to create Corus International.

Corus International leads an ensemble of social impact organizations working together in the world’s most fragile settings to deliver holistic and lasting solutions needed to overcome the interconnected challenges of poverty, health care access and climate change. Corus combines over 150 years of experience of our non-profit and for- profit subsidiary organizations – LWR, IMA, CGA Technologies, Ground Up Investing, and LWR Farmers Market. Together, the organizations take a systematic approach to grow rural economies, eliminate extreme poverty, ensure access to quality healthcare, and respond to urgent humanitarian needs in fragile settings.

LWR in partnership with local implementing partners, Backward Society Education (BASE), Dalit Feminist Upliftment Organization (DAFUO), and National Environment and Equity Development (NEEDS) Nepal was awarded a $1,000,000 grant from Margaret A. Cargill Philanthropies (MACP). The partnership is implementing Recovery of COVID-19 affected community (RECOVER) in selected districts in the provinces of Lumbini and Sudurpaschchim province. These districts are Bardiya, Kailali and Kanchanpur. The project will end in June 2022.

The consulting services, for which this Terms of Reference (ToR) is provided, is to manage the implementation of the final evaluation for RECOVER. This ToR outlines a brief description of the project, the objectives and scope of work, and required qualifications of the consultant. The contract duration is for two months from contract signing.

About Recovery of COVID-19 affected community (RECOVER)

The COVID-19 pandemic, including the more contagious Delta and Omicron variants, has created a health and an economic crisis in Nepal and around the world. The severity of these crises is especially pronounced in Nepal’s western provinces where migrants who regularly cross the border with India are not only at greater risk of contracting COVID-19 but also at greater risk of losing employment. These are also areas where LWR identified a particularly large gap in health care services. With the generous support of MACP, LWR and its partners are facilitating the recovery of vulnerable communities and build preparedness for future disease outbreaks in 9 municipalities in the three districts through the RECOVER project. To do this, LWR took a two-pronged approach:

Immediate relief and recovery

The outbreak of COVID-19 overwhelmed the country’s health system and vaccine hesitancy threatens to prolong and exacerbate the pandemic. To address the immediate needs of the target communities, RECOVER provided hospitals and health posts with much needed supplies and support such as PPEs, handwashing stations (including training on upkeep), and tools to support testing and contact tracing. LWR also provided health facilities and their staff with trainings on disease prevention, detection, management and treatment, and supported information campaigns to raise awareness about COVID-19, disease prevention and treatment options.

In addition to the health crisis in these communities, the economic crisis in the region resulting from the pandemic threatens the lives and livelihoods of thousands of families through a loss of income, food shortages, and long-term reduction in agricultural productivity. To provide immediate economic relief to these communities, RECOVER provided food and non-food items to vulnerable people and short-term employment or income-generation opportunities.

Long-term preparedness

In addition to the activities supporting the immediate relief and recovery of the target communities in Nepal, RECOVER worked with the communities to strengthen their preparedness for future health and economic disasters. Throughout the life of the project, community awareness campaigns on healthy food consumption utilizing local resources, hygiene best practices, and disease prevention and treatment were conducted. Continuous training to local healthcare personnel and create improved linkages between health facilities and the pharmaceutical supply to ensure longer-term stability of the health system were also provided. To support the long-term preparedness of the target communities for economic disasters, RECOVER implemented activities that could strengthen the capacity of local cooperatives to support local farmers, provided skills training to interested households to support income diversification, and established locally-operated community food pantries that could help meet the food needs of these communities in future disasters.

RECOVER’s two-pronged approach is expected to contribute towards achieving the project’s two outcomes.

These two outcomes are:

1. Strengthened health care system and health recovery – This outcome seeks to enable the local health systems to respond to COVID-19 outbreak and future shocks. It will do so by:

• improving the capacity of local health personnel and systems through training and provision of equipment and supplies that will help in the prevention, detection, management, and treatment of COVID-19,
• implementing awareness campaign that will help educate the general population about ways to minimize the risk of COVID-19 transmission, promote health posts services, and healthy food campaigns,
• strengthening Point of Entries (PoEs) and support in functioning health desks for screening migrant returnees and case investigation and contact tracing, and
• supporting local government in COVID-19 Vaccine rollout

2. Restored livelihood impacted by COVID-19 – This outcome seek to restore livelihood affected by COVID- 19. Affected HHs will be supported with:

• quick impact agriculture interventions like seed inputs, tools and equipment, livestock support, vending carts, small groceries, agrovets, tea, etc delivered through cooperatives and community groups so that affected HH can revive their business or livelihood options,
• early action humanitarian support (especially targeting the poor and the marginalized) like relief package, enrollment in government’s health insurance schemes, and community pantries, and
• support for immediate livelihood restoration and provision of quick impact assistance package for identified migrant returnees.

Project location

RECOVER is being implemented in the following municipalities.

2. EVALUATION PURPOSE AND USE

The overall purpose of the consultancy service is to conduct the evaluation of RECOVER in order to determine what the project has achieved, what worked, what did not, and why. The learnings from the evaluation will also contribute towards improving future humanitarian assistance for health-related disasters.

The specific objectives of this assignment/ consultancy are as follows-

• To collect, clean, and analyze data and information (qualitative and quantitative) of the community and population of RECOVER Project sites based on the evaluation criteria specified in this ToR.
• To collect, clean, and analyze data data and information of local government health offices and officials related to RECOVER Project sites based on the evaluation criteria specified in this ToR.
• To prepare a well analyzed and comprehensive evaluation report of the RECOVER Project that addresses the evaluation criteria specified in this ToR.

The result of evaluation will be used by:

1. Lutheran World Relief
2. Local partners
3. MACP
4. Local Government

A more detailed evaluation use plan will be provided to the Consultant during the Inception Phase.

3. EVALUATION SCOPE

The assessment will be based on OECD-DAC criteria given below.

Relevance or appropriateness of the project: IS THE INTERVENTION DOING THE RIGHT THINGS?

To assess the extent in which the project interventions goals and implementation are aligned with beneficiary and stakeholders needs by investigating if they view the interventions as useful and valuable. Where applicable, the assessment of relevance should be analyzed based on its four elements of analysis:

• relevance to beneficiary and stakeholder needs,
• relevance to context,
• relevance of quality and design, and
• relevance over time.

Coherence: HOW WELL DOES THE INTERVENTION FIT?

To assess the compatibility of the project intervention with other interventions in the target geography, be it interventions delivered by government institutions or other development actors. Coherence also looks at the consistency of the project intervention with relevant international norms and standards especially in the context of COVID-19 response and gender empowerment. It should be analyzed based on two dimensions:

• Internal coherence – how well are the project interventions align with the wider policy frameworks of the local or national institutions responsible for implementing the development interventions in the districts. The assessment should look at how the project interventions harmonized with local development activities or if there are any duplications, how they complement. Internal coherence also provides a lens for assessing inclusion.
• External coherence –how well the interventions connect with interventions implemented by other development actors in the three districts, if there are any.

Effectiveness: IS THE INTERVENTION ACHIEVING ITS OBJECTIVES?

To measure the extent to which the project interventions achieved or likely to achieve its desired outcomes and results. The assessment should provide insights into:

• The extent the project has attained its planned results by comparing its actual achievement with its targets
• An evaluation of the process by which this was done
• Which project components were decisive or important to the process
• Whether there were any unintended effects

Effectiveness is different from impact as it is more concerned with the most closely attributable effects while impact examines higher-level effects and broader changes. Assessment of effectiveness should also look at how well it allowed for participants participation in the implementation of the project interventions and developing their ownership and accountability especially during the process evaluation. It should also look at how the project evolve and adapt and how the design changes, if at all, based on feedback from stakeholders, emerging results, and changes in context, in particular the COVID pandemic.

Efficiency: HOW WELL ARE RESOURCES BEING USED?

To measure the extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way. The assessment should look at whether the project use of resources justifies the results it achieved. The focus of this assessment should be in

• Operational efficiency – How well are resources used during implementation. It reviews if human and financial resources were used as planned or redirected as need changes. It should also look at how well the project managed its risks.
• Timeliness – The assessment should look at whether and to what extent the results were achieved within the intended timeframe. It should review and check if the timeframe and work plans were realistic and appropriate, and if the changes to the timeline are necessary given the many external factors that could affect the implementation of the project.

Impact: WHAT DIFFERENCE DOES THE INTERVENTION MAKE?

What were the higher level changes or effects both positive and negative or intended and unintended on the community, beneficiaries, gender and cooperatives/village groups; What are the community/beneficiaries perspective on the immediate and intermediate effects (quotes can be collected to provide evidence on participant’s perspective) and what were the partner staff perspectives on the effects. To what extent is the project contributing towards achieving transformational change especially within the context of disaster recovery?

Part of impact is the report of the project achievement against outcome indicators. The relevant outcome indicators are provided in the table below.

Sustainability: WILL THE BENEFITS LAST?

Measure the extent to which the project results and impacts or positive outcome at the community level are sustainable in the longer terms (quotes can be collected that can provide evidence on participant’s perspective).

A set of evaluation questions reporting against each criterion per project will be given to the Consultant during Contract signing.

4. METHODOLOGY

The evaluations of the RECOVER projects will rely on data collected as described in this TOR. Each data collection activity will form the basis for measuring outcomes for program beneficiaries which is critical for assessing program impact. The Consultant should work in close coordination with LWR team to carry out the study. The Consultant is expected to collect and analyze primary and secondary data needed by the evaluation.

The evaluation will be using both quantitative and qualitative methods. The quantitative method will be used to measure quantitatively the short-term change within the health posts, community or beneficiaries. As the RECOVER did not collect data during baseline, a retrospective post-then-pre design will be used. Retrospective post-then-pre design is a way to control response shift bias in self-reported changes in knowledge, awareness, skills, confidence, attitudes or behaviors among program participants. The qualitative method will focus on providing insights into the changes that happened, how it happened, and why it happened. The qualitative approach is envisioned to go deep into the discussion rather than go wide and cover more groups.

Quantitative Method

The Consultant is expected to administer a Rapid Beneficiary Study that addresses all the research questions and assess the program against the objectives and indicators as specified in the project’s detailed M&E Plan. For convenience, the relevant outcomes and their indicators are provided in the preceding section. The project aimed to reach 4000 households for its core activities. The same number would be used for the population covered by the study. It is expected that the consultant would suggest an appropriate sample size given the total number of households that RECOVER will reach and the following parameters:

• 2700 number of households reached by the livelihood
• 1800 number of households reached by the health insurance
• 600 number of health posts personnel trained
• Each of the three groups are distinct and the evaluation should be able to generalize results for each group
• Target 50% women and 50% men for each group

The Consultant will be provided a master list of beneficiaries. The master list should be used as the sampling frame from where the respondents will be drawn.

An English version of the survey questionnaire will be developed by LWR. Finalization of the questionnaire will be done by the Consultant in consultation with the project partners. The questionnaire needs to be translated in Nepali before fieldwork. Survey data collection and administration should be done using mobile devices. KoboToolbox is LWR’s preferred platform but the Consultant is free to suggest and use other open-source application.

Qualitative Method

The Consultant is expected to use a number of qualitative methods in the evaluation. The qualitative part of the study is intended to provide better insights into the changes that happened as a result of the project as well as inform project implements, donors, and policymakers about how the project were implemented and maintained. The Consultant is expected to conduct:

Document Review

The document review is intended to give the Consultant a better understanding of the project and the overall context upon which RECOVER is being implemented. The topics document review is expected to feed into the discussion of the results in the following OECD DAC Evaluation criteria discussed in the Evaluation Scope Section.

• Relevance
• Coherence
• Efficiency
• Effectiveness
• Sustainability

The documents that the Consultants need to review are the following:

• Report of rapid need assessment of the health care facilities and impact of COVID-19 on the livelihood of the communities conducted by Lutheran World Relief.
• RECOVER project documentation such as Project Design Workbook, detailed M&E Plan, ATT/ITT and technical proposal/project logframe, quarterly progress reports, etc.
• National, Provincial and Local level policies and strategies on health and livelihood in response to COVID-19 response and recovery focusing to western part of the country.
• Other relevant research, study and reports published in National and International peer reviewed journals on health system strengthening impacted by COVID-19, livelihood recovery, employment of migrant youths returnees impacted by COVID-19, local entrepreneurship etc.
• Standard tools and techniques used by different donors, foundations for conducting FGDs, collecting household level information, socio-economic and demographic information aligned to RECOVER project.

In-depth interviews

The Consultant is expected to conduct FGDs and KIIs as part of the evaluation. Both methods will allow for a richer conversation with program participants about their experience with the program and the benefits they reap from it. Respondents will mainly be direct project beneficiaries so valuable insights and feedback about the components of the project could be collected. A number of local government officials will also be interviewed to get their thoughts on the project and the likelihood of its component being sustained.

Focus Group Discussion (FGD)

FGD should be conducted by trained and experienced facilitators. The evaluation team should invite up to maximum of 10 participants for each FGD. The aim is to have a minimum of 4 participants to conduct the FGD; if fewer than 4 participants show up the method would be changed to a KII instead, following the same discussion guide as the FGD.

Key Informant Interview (KII)

KII should be conducted by the Consultant or a senior facilitator as the respondents of the KII will be senior officials of the government.

An interview guide in English will be provided by LWR. This need to be translated to Nepali before data collection. The Consultant must ensure that responses are solicited from the participants by asking neutral probes and without interjecting biases.

The selection of groups for the FGDs and individual for the KIIs will be more purposive so that important information is uncovered. The sites where these groups and individuals are located will follow the result of the location where the survey will be conducted. This is done to make data collection more efficient in terms of both time and money. The proposed groups and individuals are given below.

If group gathering restrictions are in effect during the scheduled interview, the FGDs could be converted into KIIs upon the approval of LWR. Qualitative data shall be captured using key notes using the template that will be provided.

In addition, a Learning Workshop with RECOVER Partners is schedule to be held on May 16, 2022. The Consultant is expected to attend the workshop and gather information that are relevant to the evaluation during the event.

COVID-19 impact on data collection

Ensuring the safety and well-being of the evaluation teams and the respondents are very important. Consultant could propose alternative measures to mitigate the impact of COVID-19 in the conduct of the study. These measures could include but is not limited to using indirect interview methods such as mobile/internet interviews or meetings.

5. KEY TASK AND DELIVERABLES

The duration of the consultancy is eight weeks. Many of the preparatory tasks will be done by LWR to enable the Consultant to complete the engagement within the period of engagement. To successfully complete the evaluation, the Consultant is expected to follow the tasks provided below.

Description of Tasks

Task 1 – Write Inception Report and Work Plan

The Consultant shall submit an inception report and work plan that includes the following:

• Understanding of the engagement and tasks required
• Substantive input into design of draft data collection instruments, methodology and overall study approach
• Team composition and recruitment plan
• Numbers, dates, duration, and location of training sessions; including composition of training teams, agenda, preparation of logistics, and reporting arrangement to the LWR Evaluation Coordinator
• A list of permits and letters required to conduct fieldwork and when those will be obtained
• Expected duration of survey implementation and work plan for Field Teams including supervision plan
• Expected date and duration of the Data Cleaning activities, including delivery dates of the different datasets.

Deliverable 1: Inception Report with Work plan submitted one week after contract signing

Task 2 – Prepare for field work

Preparation for field work entails a number of subtasks that must be completed in order to ensure the quality of data that will be collected. It includes:

Recruitment and selection of field teams and data processing teams:

Interviewers and facilitators will be responsible for collecting the data in the field. Interviewers that will be selected must have demonstrated experiences in administering surveys or facilitating discussions. Ideally, each field team must be headed by a field supervisor which will oversee the over-all management of field work of the team including backchecking and other data quality processes on field. Data processing team are responsible for ensuring that data collected in the field are free from errors prior to analysis. They conduct the data transformations as required by the Consultant.

Prepare research permits:

In the event that the community or municipality will require permits for field work, it is the responsibility of the Consultants to obtain these permits.

Finalization of data collection instruments:

All data collection instruments must be tested before being used. Testing procedures could be pilot testing or cognitive pretesting. The objectives of the testing are:

• allow the Consultant to familiarize itself with the instruments,
• test the logistics (such as field mobilization and demobilization, potential locations of basecamps) and time necessary for the surveys
• test the draft questionnaires. Testing the draft questionnaires will include identifying any exercises/questions that are not working as currently framed.

Familiarize with the use of KoboCollect or alternative data collection tool:

LWR prefers data collection using KoboCollect. Consultant must familiarize themselves with the use of the software before field work.

Training of Interviewers and Facilitators:

LWR strongly prefers that all field team members be trained in one central location. If Consultant proposes training in multiple locations, the proposal should detail how the service provider will ensure consistency across training locations and common understanding of questionnaires and data collection protocols.

Note that the Consultant is required to begin fieldwork within one week of the end of training. If there is a delay of more than one week, the consultant will be required to do a one-day refresher training to all interviewers at its own expense.

Deliverable 2: A short training report should be submitted by the Consultant a week after the training.

Task 3 – Collect Data

Data collection effort should always follow government advisories in preventing the spread of COVID- 19. In the event that face-to-face data collection cannot be conducted due to prevailing government restrictions, the Consultant should attempt to collect data using indirect method. Indirect method could include interviews over mobile or social messaging apps.

Data Collection (Quantitative)

LWR will provide the Consultant with a list of households to interview and a replacement strategy that should be strictly followed. The households will be the same households that were engaged by the project.

The field teams are expected to adhere to the respondent tracking process:

1. If the respondent is not home during the time of the interview: The interviewer should return at a different time and if the respondent still cannot be reached during this second attempt, the interviewer must make a third attempt at a different time/day to find and interview the respondent.
2. If the respondent moved village but within the same municipality: The interviewer must trace the respondent within the municipality and conduct the interview.
3. If the respondent moved outside the municipality: Replace the respondent with another one according to the replacement strategy provided by LWR.

Ideally, no respondents should be replaced. However, replacements should be addressed on a case-to-case basis. The Consultant should inform LWR about any such instances before reaching a decision.

Data Collection (Qualitative)

Appropriate venues for the conduct of FGDs must be identified and booked before the activity. Invitations for participants to participate must be sent out a couple of days before the discussion. In the event that, less than 4 participants showed up for the FGD, the activity will still continue, with the facilitator adopting KII approaches. For KIIs, appointment must be booked in advance and reconfirmed a couple of days before the interview. Data recording, if allowed, and extensive note-taking should be done for each activity.

Data Quality

The Consultant is responsible for collecting the highest quality data possible and subjecting interviewers, facilitators, and data to great scrutiny to ensure quality. It is recommended Consultant implement three kinds of data quality checks.

• Observation: Supervisors accompany the interviewers while they are doing the interview to observe and support interviewers. With this type of checking, supervisors will find out if the interviewer understands interview questions, effectively conveys them, and can coach the interviewer on how to improve if problems arise. Supervisors must accompany interviewers for at least 10% of interviews. The Consultant is expected to submit a report tracking the observation made.
• Supervisors are required to reinterview the respondents over mobile and ask some of the (sample) questions in the questionnaires to the same respondents. The purpose of this procedure is to verify that interviewer actually met the respondents and conducted the interview. If there are significantly different answers between the two interviews, supervisors should clarify the difference. If the difference is because the interviewer wrongly posed the question or misunderstood the question, supervisors should brief the team to clarify the misunderstanding. If any interviewer is found to be falsifying or manipulating data, he/she must be fired and replaced. Replacement interviewers must re- interview all respondents interviewed by the interviewers who falsified data. The Service Provider must ensure that at least 5% of interviews are audited (ie, households revisited). The Consultant is expected to submit a report Detailing the result of the re-interviews.
• Checking the completed interview: supervisors will check 20% of completed interviews to identify errors made by the interviewers. The checks can be done through the KoboCollect online facility and should be done while the team is on the field interviewing.

Task 4 – Data cleaning, analysis, and draft report

Data processing team is responsible for starting the data cleaning process as soon as they receive data and other supporting documents from the field. This process includes checking the accuracy of data

received, such as respondents’ identification and linkages among questions (meaning the skip patterns and relationships across questions). If data processing staff requires clarification from the field and the team is still in the field, they should clarify discrepancies with the field team.

Data processing team should also quality check the data captured qualitatively. The notes will be compared to the recording done during the activity to ensure that all important points are captured in the notes.

Data tables will be generated for quantitative data. Latent level of analysis will be done for qualitative data where relevant notes are tagged and categorized.

All answers or notes that are in the local language must be translated in English.

The Consultant is expected to first submit a draft report for comments and review of LWR. Clean raw data sets for quantitative data including frequencies and cross-tabulations must be submitted along with the draft in CSV format, and when available, Stata 11 format. FGD and KII notes along with a summary of key qualitative findings must be submitted in English and in Word format.

Deliverable 3: Draft report and clean data sets

Task 5 – Final Report submission

The Consultant must submit the final report addressing all the comments made by LWR. A tracking table of comments made and responses must be included in the Annex.

Deliverable 4: Final Evaluation Report

6. IMPLEMENTATION ARRANGEMENTS

Consultant: Preparation of detailed work plan, translation of the data collection tools, desk review, field visit, survey, FGD, KII and interaction with stakeholders at district and local level.

M&E Officer and Program Manager, LWR: M&E Officer is responsible in coordinating with the consultant in survey process from LWR, connecting consultants with partners (BASE, DAFUO and NEEDS), facilitate scheduling meeting with stakeholders at local level through NGO partners, informing communities about visit and coordinate/facilitate consultants on the survey. Key focal person of the survey/assignment will be Program Manager, who exchanges ideas among Regional M&E Advisor for AME, Country Director and Program Director to the consultant.

Regional Technical Advisor for M&E Asia and the Middle East: Is responsible for the over-all design of the evaluation including the development of the data collection tools. He has over-all responsibility for maintaining the over-all quality of the evaluation study.

The consultant will lead and be responsible for the overall process of the evaluation study starting from the implementation, debriefing and reporting. Based on consultant’s assessment plan/inception report M&E Officer links consultant with the partners at field and partners will provide support with organizing the consultative survey and meetings with beneficiaries and others, FGDs, KII etc. The project coordinators of the respective partners will be the point person for field level coordination and communication.

7. TIME FRAME

The timeframe for this assignment will be eight weeks from the date of signing an agreement. The consultant should share their final workplan as part of the first deliverable. A draft work plan and the deliverable schedule is provided below.

Draft Work Plan

Deliverable Schedule

8. CONSULTANT’S QUALIFICATIONS AND APPLICATION PROCEDURES

This Consultancy is open to individual or firm. The submitted proposals will be assessed with the following 2 major criteria:

• Demonstrated capacity to conduct survey, project evaluation, research/studies/assessments of similar magnitude and development themes (especially in the field of health system recovery, resilient livelihood, COVID-19 response and recovery).
• Cost effectiveness of the proposed budget. 

LWR is looking for consultants with the following qualifications and experience:

• At least five years of professional experience in conducting surveys, project evaluation with solid experience in conducting and managing macro and micro level public health, livelihoods, and/or emergency response survey and evaluation.
• At least a bachelor’s degree in public health, agriculture, rural development, statistics, or other relevant discipline. Master’s level degree or higher is an advantage.
• Demonstrated capacity to analyze qualitative and quantitative data in a systematic way
• Demonstrated ability to present complex information in a concise, clear and accessible way
• Experience in the design, monitoring and evaluation of integrated health and livelihood recovery projects focusing to migrant returnees and youth.
• Experience of working in survey and designing of health and livelihood focused projects during and post COVID-19 pandemic.
• Have excellent writing skills (in English), and have outstanding analysis, and communication skills.

Interested consultants are requested to prepare an Expression of Interest and provide evidence to support claims of knowledge, skills and experience:

• Indicative budget
• Resumes/CVs with references
• A sample similar evaluation report.

Expression of Interest with the full proposal is to be sent to Alisha Neupane, Finance and Admin Officer, LWR at lwrnepalcontact@lwr.org, no later than April 25, 2022. Any decision for disqualification rests solely with LWR.