American Red Cross seeks to appoint a:
Title: Consultant for Health and Resilience in Eurasia Final Evaluation
Type of evaluation: Summative/Final evaluation
Expected evaluation methodologies: Desk review and qualitative methodology incorporating Outcome Harvesting and in-depth interviewing and site visits
Number of evaluators: One consultant with experience in HIV/AIDS, Resilience programming and outcome based evaluation methods; Russian speaker with excellent English writing skills
Expected start/end dates, number of work days: April-June 2017; 45 days; 3-4 country visits Countries included in Regional Initiative: Armenia, Ukraine, Kazakhstan, Belarus, Russia, Kyrgyzstan
Description of project/program to be evaluated
Background and objectives of project/program
The American Red Cross has been supporting work to reduce the impact of HIV in Eurasia since 2003 as well as building resilience and preparedness within the region since 2010. Parallel to the Regional Health Initiative (RHI) and Resilience program transition plans for FY17, the American Red Cross also supports national society strengthening in Eurasia that centers on two pillars, organizational development and capacity building. The RHI and PCA programs have been key in enabling the American Red Cross to address several priority outcomes for the Eurasia region during the FY17 transition process. The RHI program is currently implemented by the Red Cross and Red Crescent societies in Armenia, Belarus, Kazakhstan, Russia and Ukraine. This initiative has been successful in providing care and support to people living with HIV while contributing to changed attitudes towards HIV through educating about risky behaviors and HIV prevention to most vulnerable populations. At present, this initiative is undergoing a transition process to ensure that national societies are prepared and equipped to respond appropriately to the HIV epidemic while delivering comprehensive HIV programs.
Similarly, the work for building resilience and preparedness in Eurasia through the Prepare Central Asia Initiative commenced in 2010. This American Red Cross supported initiative has celebrated some successes in increasing information and forging partnerships for improving community preparedness for fires, earthquakes and other natural hazards, as well as road safety awareness. Currently, the program addresses the vulnerabilities to the impact of earthquakes in highly at-risk Central Asia urban centers in Almaty and Shymkent, Kazakhstan, and Bishkek and Osh in Kyrgyzstan. As with the RHI Eurasia program, the resilience and preparedness program in Eurasia is also working toward a phase-out and handover of activities by FY17. With planned complete phase-out of programmatic support in the health and resilience sectors for national societies in Armenia, Kazakhstan, Kyrgyzstan and Russia by FY17 (and in Belarus and Ukraine by FY18), it is hoped that communities become more knowledgeable and healthy, and have the ability to assess, manage and monitor their risks, and better able to self-organize and network while having the capacity to identify problems, establish priorities and act. Part of this also entails national societies being able to independently deliver improved community programs and services as well as effectively responding in the event of an emergency.
Previous evaluation activities
Throughout the implementation of programming, ARC personnel have been monitoring and reviewing lessons learned. ARC has conducted several formal evaluative exercises for the RHI program.
- Evaluation of the Care and Support Project, for People Living with HIV/AIDS, Implemented by the Irkutsk Branch of the Russian Red Cross, 2008
- Regional HIV/AIDS Initiative for Eastern Europe and Central Asia Project Expansion Review in Russia and Ukraine, 2011
- Strategic Revive of ARC Global HIV Program, 2012
- PCA Technical Internal Review, 2013
- RHI Program Transition Report, 2014
Purpose of evaluation
The main purpose of the evaluation will be to capture any changes in the community that the program was able to achieve through the program and identify any changes that may be sustained by the community structures and partner stakeholders following the project. The evaluation will also provide recommendations for how the program can be integrated into future Red Cross programming from the perspective of the implementing national societies given that the grant for this specific project will not continue beyond June 2017.
Limitations of the Evaluation
The evaluation is limited in time and funding and therefore will be limited in scope. It will not seek to capture a comprehensive assessment of the impact or overall performance of the RHI/PCA program. The evaluation will also take into account that each country in the RHI/PCA program has its own unique set of activities and relationship. The evaluator will not be able to travel to every single country. Therefore, the breadth of the program will be captured through a remote survey for NS staff using Outcome Harvesting to collect data of national society experience with the program. The consultant will then travel to 2-3 countries to substantiate outcome statements and further explore findings from the survey.
Objectives of evaluation
The objectives of the evaluation are to:
- Capture any changes that have occurred in the national society, program partners or client groups as a result of the program (intended, unintended and positive or negative)
- Understand which elements of the program are most likely to be sustainable to inform future ARC program design.
- Identify opportunities and learnings that the national societies that host site visits can leverage for future program implementation following the close of ARC project support and funding.
The evaluation will leverage the Outcome Harvesting methodology to capture both the breadth and depth of learning from the program. Outcome Harvesting evaluates the space between output and impact; the methodology assumes that sustainable ecosystems and human wellbeing depend on human behavior and therefore is mainly interested with outcomes that are behavioral changes. For example, while recognizing that impact is the ultimate goal towards which a development intervention works, Outcome Harvesting is based on a philosophy of focusing on the outcomes eventually leading to development impact – rather than on the impact itself – because the complexity and long-term nature of the development process often makes it extremely difficult to link impacts to a specific intervention.
From a practical standpoint, Outcome Harvesting will be applied first through a remote data collection process where the American Red Cross will invite national society staff to respond to a survey monkey asking them to write “outcome statements” describing changes that have taken place in the national society, program stakeholders and the communities as a result of the program. The consultant will then travel to 2-3 countries to “substantiate” a sample of the outcome statements, verifying their accuracy and further exploring the change statements through in depth interviews with staff, partner agencies and government and clients/community members.
Main audience of evaluation
The main audience for this evaluation is ARC leadership, AMEE regional leadership and MACP. Following the evaluation, leadership will work with the MEL team to develop a management response plan to take forward the recommendations of the evaluation in order to inform programmatic planning.
Evaluation criteria and questions
The main evaluation question is: How has RHI/PCA contributed to improving health services and systems in participating countries and will the initiative be able to continue to have a positive impact following the close of the project in June 2017?
Criteria: Program Design
Main evaluation questions:
- How did the program design affect the success of program implementation?
- Was there a clear theory of change shaping the program? If yes, were program activities aligned to the theory of change? If not, what tools did the project use to guide its program design and what was the quality of these tools?
- Did the project have a sound and systematic way of identify target beneficiaries that was relevant to the program objectives?
- Did the program activities address the expressed priorities of the targeted beneficiaries? Did they address relevant priorities that may not be explicitly expressed by target beneficiaries?
Main evaluation questions:
- What changes has ARC been able to effect through its organizational development initiatives within individual national societies?
- Did RHI/PCA reach the people it intended to?
- Do community members feel there was change in their communities as result of the program? What were the main changes in communities as a result of RHI/PCA? (Including positive, negative, intended and unintended).
Main evaluation questions:
- What aspects of the program will be most likely to be sustainable in the community and why? (applies only to countries that participate in the site visit part of the evaluation)
- Were there any activities that the community feels are most likely to continue beyond the project close in June 2017?
- Are there any structures or relationships (such as community committees or partnerships among stakeholders) that are likely to continue beyond June 2017?
- Why will some components of the program be more sustainable than others?
Main evaluation questions:
- What aspects of the program will be most likely to be sustainable in how the national society implements HIV/AIDS or Earthquake Preparedness programming and why? (applies only to countries that participate in the site visit part of the evaluation)
- Is the work of the RHI/PCA relevant to the long term programming goals of the individual national societies?
- Did the project support the national societies to develop a plan following the close of the project? What opportunities are there for the national society
- To either continue the program approach, continue to work in the same communities or to apply lessons from the program to future projects?
- Which departments in the national societies have the interest and capacity to absorb the lessons?
Main evaluation questions:
- Are there any lessons in how the program was implemented that will be useful to national societies that continue to implement similar programming?
- Are there lessons for future implementation that span across countries and apply to the entire initiative?
- Are there specific lessons for future implementation in those countries that the consultant conducted a site visit?
Scope of work
The evaluation will require a comprehensive review of reports, proposals and documentation of RHI/PCA as background. The evaluation will also require the consultant(s) to travel to 3-4 countries to conduct in-depth interviews in order to substantiate a sample of outcome statements and explore additional evaluation questions on sustainability and lessons learned.
The contractor will be responsible for the following:
- Conducting a literature review.
- Developing an inception report for review by ARC.
- Developing data collection tools for review by ARC.
- Conducting 3-4 overseas trips to conduct site visits to meet with national society staff and volunteers.
- Conducting interviews and data collection processes to respond to the evaluation questions
- Conducting analysis and reporting on data collected through the evaluation.
- Drafting the final report addressing the ToR evaluation questions and revising the final report based on input from the ISD department.
- The Evaluation Manager will provide preliminary Outcome Harvesting data that will be collected internally prior to the contracting of the consultant.
- The Evaluation Manager will work with delegations and National Societies to identify countries for site visits and facilitate logistics of travel.
- The Evaluation Manager will serve as the focal point for the consultation and staff feedback on the draft report in order to support the consultant to finalize the report.
There will be a draft inception report prior to traveling to the field to provide an overview of the format of the report and document the literature review portion of the evaluation including the historical documentation of program achievements.
Logistic and Administrative Support
The consultant will be reimbursed for the cost of flights and accommodation expenses during the overseas trips following travel.
Reporting relationship
The contractor will report to Ranjan Mohnot, Senior Quality and Learning delegate, and to Kristin Helz, MEL Advisor for technical support and oversight.
Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American Evaluation Association’s Guiding Principles for Evaluators. A summary of these guidelines is provided below.
- Informed Consent: All participants are expected to provide informed consent following standard and pre-agreed upon consent protocols.
- Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
- Competence: Evaluators provide competent performance to stakeholders.
- Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
- Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. It is expected that the evaluator will obtain the informed consent of participants to ensure that they can decide in a conscious, deliberate way whether they want to participate.
- Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.
All collected data will be the sole property of the American Red Cross. The contractor may not use the data for their own research purposes, nor license the data to be used by others, without the written consent of the American Red Cross.
Expected activities and Deliverables
Expected activities
- Desk review and discussions with key program staff; documentation of historical achievements of program, 5 days by April 2017
- Develop and submit inception report for approval, 3 days by April 2017
- Develop data collection instruments, 2 days by April 2017
- Field Trips and Data Collection, 15-20 days by May 2017
- Data Analysis and Draft Report, 7 days by June 2017
- Incorporate Feedback into Final Report, 3 days by June 2017
- Total expected work days: 35-40 days
- Inception report by April 2017
- Finalized data collection instruments by April 2017
- Draft report (estimated 30-40 pages plus historical documentation of program as annex and 2-4-page executive summary.), Due by June 10
- Final report, Due by June 20
Required experience should be a single consultant with technical experience in both organizational development and evaluation or a consulting team with experience in evaluation and experience in organizational development:
- Demonstrated experience in leading evaluations of HIV/AIDS programs in the non-profit sector internationally
- Demonstrated experience in qualitative data collection and analysis
- Demonstrated experience in outcome based qualitative methodologies
- Fluency in Russian and excellent English writing skills required
Application materials
The proposal should include the following four items. Please note that any proposal which does not contain all four items will be rejected.
- One-page Summary of experience
- Detailed CVs of all professionals who will work on the evaluation. If there is more than one consultant on the proposed evaluation team, please attach a table describing the level of effort (in number of days) of each team member in each of the evaluation activities.
- Methods – One page summary of proposed methods and approach of evaluation and number of days proposed.
- Professional references: please provide two or three references from your previous clients.
- Writing Sample: Please provide one sample of an evaluation written or co-written by consultant.
- Daily rate: please mention the proposed daily rate for each consultant in USD.
Email applications to Ms. Nicolette Prugsamatz at Nicolette.PRUGSAMATZ@ifrc.org with first name, last name in application materials attached to the email by April 15, 2017. Short-listed candidates will be contacted for an interview.
Selection criteria
- Qualifications section
- Cost
- Experience in evaluation and qualitative methodologies
- Experience in subject matter expertize of organizational development
- Number of Days and Timeline availability
loading...
