Position Title:
Institutional consultancy for external evaluation of ECD&F programme in 20
sites in 10 Districts in Rwanda
Level: High-level
professional institution
Location: Kigali,
Rwanda
Duration: 5 months
Reporting to: Chief
of Social Policy and Research, UNICEF Rwanda
Contents 1
Background and Purpose 2 Object of the Evaluation 3 Objectives / Study Aim 4
Scope, Focus and Evaluation Criteria 5 Methodological Approach & Process,
Expected Output 6 Major Tasks, Deliverables & Timeframe 7 Stakeholder
Participation 8 Existing information sources 9 Ethical
Consideration/confidentiality 10 Evaluation team composition / qualifications
and requirements 11 Supervision 12 Terms and conditions 13 How to apply 14
Appendix 1 – Key Indicators Table 15 Appendix 2 – Study measures used in
ECD&F baseline evaluation
1 Background and
Purpose
Support to Early
Childhood Development and Family services has emerged over the years as a key
priority area for development in Rwanda. This priority is outlined in the Early
Childhood Development (ECD) Policy and its Strategic Plan, and the Economic
Development and Poverty Reduction Strategy (EDPRS II 2013–2018). The EDPRS II
defines ECD as a ‘foundational issue’ for sustainable development, and confirms
the Government’s commitment to keeping ECD a high priority on the national
agenda. To respond to this government priority and translate national policies
into action, UNICEF Rwanda and Imbuto Foundation, under the leadership of the
Ministry of Gender and Family Promotion (MIGEPROF) and in collaboration with
other stakeholders, entered into a partnership to bring together multiple Early
Childhood Development and Family (ECD&F) interventions, with the family at
the heart of the approach. The ECD&F programme targets children younger
than 6 years old in Rwanda, offering a combination of centre-based and
home-based ECD interventions. The programme focuses on responding to community
needs around sustainability and relies on community ownership of the programme.
Caregivers and children alike participate in programming, and efforts to
involve fathers in child rearing are emphasized. While some of the
interventions (e.g., provision of centre-based ECD services) focus on targeted
villages, others, such as child protection, health and nutrition, cover the
whole sector/district or are of national relevance. Social protection
interventions, home-based ECD, and water, sanitation and hygiene (WASH)
services are geared towards the most deprived children and families in targeted
villages as well as neighbouring communities. The first ECD&F centre was
opened in July 2013 in Kayonza District in the Eastern Province of Rwanda, and
the programme’s goal is to build an ECD&F centre in each of Rwanda’s 30
districts. In 2014 UNICEF Rwanda commissioned an ECD&F baseline evaluation
in 10 intervention/treatment and 10 control sites in 10 district, to assess the
current conditions related to ECD in these sites. The baseline was carried out
in partnership with Imbuto Foundation and guided by an ECD&F Baseline
Evaluation Technical Committee. The baseline evaluation’s key objectives
included assessing the health and well-being of young children and families in
the surveyed sites and establishing current conditions related to ECD&F, to
inform and adjust such programmes in the selected areas. This baseline
evaluation provided pre-intervention indicators of child health and development
outcomes to allow for future impact evaluations of programme effectiveness. To
support and evaluate the impact of the ECD&F programme an end-line survey
is planned for the third and fourth quarter of 2016. The aim is to capture the
impact of a wide variety of ECD&F interventions, including ECD centers, ECD
Home Visits and ECD community based interventions. The end-line evaluation will
inform future programming in the area of ECD&F. The findings and
recommendations will be used to enhance the design, planning and implementation
of UNICEF and Imbuto Foundation ECD&F programmes, as well as provide
technical guidance to other stakeholders engaged in ECD&F programming in
Rwanda. In order to conduct an end-line evaluation of the ECD&F programme,
there is a need to hire an experienced institution. The ECD&F programme is
implemented in 13 districts in Rwanda and there are plans to scale it up to all
districts in the country. An end-line evaluation is required to analyse
programme impact, programme successes and failures, and to develop new guidance
to inform and improve UNICEF’s and Imbuto Foundation ECD&F programming.
Given the expected workload of documenting and evaluating the initiative at
national and local level, the volume of technical deliveries, and the required
technical expertise, the tasks are beyond the current capacity of UNICEF
Rwanda’s in-house personnel. 2
2 Object of the
Evaluation
The key objective of
the ECD&F end-line evaluation is to evaluate the impact of the ECD&F
programme which UNICEF and Imbuto Foundation have been implementing since 2013
through various interventions, most prominently center-based, home-based
community centers and home visits. The stakeholders involved in the evaluation
are UNICEF, Imbuto, MIGEPROF, District officials and NISR. The time period evaluated
in the end-line is June 2014 – September 2016, with consideration that certain
components such as home based care and home visits were rolled out in late 2015
The geographical scope of the evaluation is approximately 850 households in 20
sites (10 control and 10 intervention) in 10 districts (Gakenke, Gasabo,
Gicumbi, Ngoma, Nyabihu, Nyamagabe, Nyamasheke, Nyarugenge, Ruhango and
Rwamagana). The ToC / Logic Model is elaborated upon in section 4.
3 Objectives / Study
Aim
In order to inform
the scale up of ECD&F programming, an end-line evaluation capturing the
impact of the ECD&F interventions to date is required to assess successes,
shortcomings and the replicability to nationwide scale-up and its contribution
to evidence-based policy change. The overall objective of this end-line
evaluation is to understand whether the intended objectives of the project have
been achieved, in line with the plan, as compared with the results of the
baseline evaluation of 2014.
The evaluation will
assess the impact that the ECD&F programme on a wide range of ECD
indicators, such as child protection, health, nutrition, development and early
learning. Specifically, the evaluation will determine to what extent the
intervention has been able to meet its objective to enhance Early Childhood
Development in selected sites.
The findings of the
evaluation will be used by Imbuto Foundation and UNICEF, and by other ECD
stakeholders in Rwanda, to develop future plans (including scale-up strategy
and sustainability plan) and interventions and to inform policies and
strategies to improve performance of ECD&F programmes in Rwanda.
4 Scope, Focus and
Evaluation Criteria
The scope of the
evaluation is the 20 projects sites (10 intervention/treatment and 10 control
sites) in 10 districts of Rwanda, which were included in the baseline
evaluation (approximately 850 households).
The end-line survey
should be designed to focus on impacts/final welfare outcomes and to capture
project outputs and outcomes as agreed. It will assess the impact of the
ECD&F programme in the surveyed sites, and identify the impact of different
ECD interventions, such as in the area of health, nutrition, water and
sanitation, early stimulation etc. In addition, the survey should result in an
overview of the socio-economic situation of the households in the surveyed
sites. The end-line survey should capture the project’s short-term immediate
objectives, i.e. to establish as system in which families and children living
in catchment areas of the model ECD centers in 10 districts have access to
quality ECD and Family services, enhancing their early childhood and
development outcomes.
The end-line survey
methodology should be in accordance with the methodology implemented under the
baseline survey of 2014 (quasi-experimental, see methodology chapter below).
The end-line survey institution is expected to follow household questionnaires
appropriate for quantitative impact evaluation, and implement the survey in the
field (referring to the baseline survey questionnaire and tools implemented in
2014, (child development tools are adjusted to narrow age groups)). All survey
instruments should be based on a thorough review of relevant literature and
existing questionnaires, including the baseline survey questionnaire, and the
Rwanda Demographic and Health Survey.
Based on the
ECD&F conceptual model (see figure) which looks at children’s development
holistically, the end-line evaluation should focus on and include a wide array
of beneficiaries and stakeholders. These include children and caregivers, the
ECD site coordinators and center directors, the community, social service
providers and national and local decision makers.
The logic model /
theory of change of the ECD&F programme (which the baseline and end-line
are based upon), highlights that several preconditions must co-exist to ensure
that children get the best start in life and the opportunity to thrive. These
are: (i) effective and responsive care of the young child by the primary
caregiver, family and community; (ii) access to and use of quality ECD and
other basic social services for young children and (iii) a supportive policy
environment in place.
The input of the
ECD&F programme is therefore multi-sectoral, and includes good physical
environment, such as adequate sanitation, clean drinking water, as well as
availability of items to stimulate young child development. In addition, the
input includes caregiver’s direct engagement with the child. Improved access to
and use of quality ECD and other basic social services for young children is
another critical input of the ECD&F programme.
The output (what we
do and who reach) is therefore multi-faceted, and includes improved water and
sanitation provided, improved access to quality early learning, improved care
of young children by primary caregivers, among others, and is focused on
populations in the 10 catchment areas. The short-term results, as already
outlined above, focus on establishing effective early childhood development and
family services in selected catchment areas.
The evaluation will
be guided by OECD/DAC evaluation criteria of relevance, effectiveness,
efficiency, sustainability and the impact. The criteria should be analysed from
the perspective of the following evaluation questions, objectives, indicators:
Programme impact: •
To what extent did the programme contribute to improved early childhood
indicators? The indicators, include, among others (Refer to appendix 1 for a
more comprehensive list of indicators to be included in the end-line
evaluation): o Direct child level indicators such as: anthropometric, health
(for example diarrhoea and ARI) and nutrition (for example, minimum meal
frequency and dietary diversity) indicators, cognitive development, communication
skills and language, motor development – gross and fine motor, social and
emotional development, school readiness etc. o Household level indicators such
as: child discipline practices, water and sanitation, hygiene, support for
early learning, care seeking behaviour for child illnesses, inadequate care of
children and other indicators of caregiving practices. o Access to essential
services such as birth registration, health insurance, communication for
behavioural change messaging, social protection etc. o Knowledge, attitudes and
awareness such as parents/caregiver and community knowledge, beliefs and
expectations of early childhood development programmes; Parent/caregiver
(family) and community awareness oof and engagement in early childhood development;
Parent/caregivers and community awareness of ECD and the credible channels of
information/communication on ECD.
Programme relevance
and effectiveness: • Are the objectives of the ECD&F programme relevant and
realisable? • To what extent has the programme management structure, technical
assistance and staffing requirements contributed to achievement of intended
interventions? • How relevant were specific programme components for improving
ECD/welfare outcomes, as perceived by direct programme beneficiaries? • What
were the barriers and motivators/enablers influencing participation in the ECD
programme? • Have the interventions reached the intended number of
beneficiaries according to the stated objectives and timeframe? • To what
extent and how did the intervention improve service providers’ knowledge,
skills and practices on Early Childhood Development? • To what extent do
mothers and families perceive overall change in the health and development of
their children as a result of the ECD&F programme? • To what extent do
beneficiaries report to have been reached by Early Childhood Development
services? • To what extent has gender played a role in creating an optimal
child development environment at household/family level The limitations of the
survey mostly pertain to time lapsed from implementation of the ECD&F
programme starting in 2014, to the time of end-line evaluation. In some
districts the ECD&F programme has been running since mid-2014, however in
other districts the programme has just recently be launched. This needs to be
taken into serious consideration when designing the end-line survey i.e. the
roll-out of these various ECD interventions has been staged, and a possible
approach is to compare matured districts with longer running ECD&F programmes
with new districts.
5 Methodological
Approach & Process, Expected Output
The evaluation needs
to adhere to the Government of Rwanda’s evaluation standards. Further key
policies and performance standards to be referenced in evaluating the programme
are described in the United Nations Evaluation Group (UNEG) “Standards for
Evaluation in the UN System” and in UNICEF Evaluation “Policies and
Principles”. The basics of human rights-based approach and results-based
approach to programming are described in the UNICEF Programme Policy and
Procedure Manual.
The evaluation
methodology will be guided by the norms and standards of the United Nations
Evaluation Group (UNEG), and the UNEG guidelines on integrating Human Rights
(HR), Gender Equity (GE) in Evaluation. In order to be responsive to HR and GE
aspects, special consideration will be given to gender, sex, distance from
service locations and wealth when stakeholders and beneficiaries’ view are
sought in data collection. In the design phase of the evaluation framework,
careful considerations will be given to such inclusion aspects. In the analysis
phase, appropriate disaggregation will be attempted to shed light on HR and GE
elements. The evaluation institution will work with UNICEF, Imbuto Foundation
and the ECD&F Technical Committee to develop and finalize the design and
will conduct the evaluation under the leadership of the Committee. The
evaluation team will be responsible for designing tools, field visits, data
collection, data analysis and drafting of the report. The evaluation team will
work with the Technical Committee and other stakeholders to coordinate the
work, conduct interviews, conduct the data collection and analysis, and
disseminate the findings of the evaluation.
Type of Study: The
evaluation is expected to be a quasi-experimental mixed-method (quantitative
and qualitative), analysing the impact of the ECD&F programme. The
qualitative component will draw on the understanding and perception of the main
stakeholders involved in the ECD&F programme.
Data Source: Primary
data will be collected from all households (intervention/treatment and control)
included in the baseline survey. Primary data from the baseline evaluation will
also be a key data source.
Suggested
qualitative data analysis The qualitative analysis will focus on obtaining key
information from a wide array of ECD&F stakeholders in the selected sites,
including ECD district focal points, ECD&F center directors, caregivers at
centers, and parents/caregivers of children in the baseline intervention and
control groups. In addition, the qualitative analysis will focus on programme
relevance and effectiveness questions outlined in the above chapter. The
qualitative interviews of caregivers of children will be conducted
simultaneously to quantitative data collection. The qualitative analysis will
follow-up on the qualitative analysis of the ECD&F baseline evaluation, to
further understand key barriers and enablers for ECD&F programming.
Suggested
quantitative data analysis The quantitative analysis will include both
descriptive analysis and advanced multivariate regression analysis. It will
provide information on all pre- and post-intervention indicators for the
intervention/treatment and control groups. The 2014 baseline evaluation
included two distinct age cohorts. At the time of baseline evaluation data
collection in June – October 2014 the two age cohorts were 0-11 months and 24 –
35 months. At the time of end-line data collection which is planned for
September – November 2016 these two cohorts will be between 27 – 38 months (the
younger age cohort) which translates to approximately between 2 – 3 years, and
between 51 – 62 months (the older age cohort) which translates to approximately
between 4 – 5 years in September 2016. The consultancy institution will develop
the evaluation sampling strategy in consultation with the ECD&F Technical
Committee, for the key indicators, to estimate the indicators stated above, as
well as additional indicators (see appendix 1) for all the 20 sites in 10
districts. The ECD&F baseline sampling strategy was based on random
sampling of households within selected sites. The eligibility criteria were
that participants had to be the primary caregiver and legal guardian of a child
between the ages of 0-11 months or 24 – 35 months, and live in the same
household as the index child. The sample sizes were estimated based on similar
trials from other settings, powered to provide differences between control and
intervention groups (but not to provide differences between sites). The sample
size of intervention and control sites is approximately 884 caregiver and child
pairs, however attrition will need to be taken into account for the end-line,
which will have occurred from the time that has passed since baseline
evaluation data collection of June to October 2014.
The end-line survey
will be conducted in the following 20 sites in 10 districts: District/ Sector/
ECD&F intervention sites /Comparison/control sites :
Gakenke/ Minazi
/Munyana Cell Kanka, *Nyabitare, Kivuba /Gakenke Cell Kabuga, Nyarubuye
Gasabo /Gikomero
/Murambi Cell Twina* Gasagara Cell Rugwiza /Gicaca Cell Ntganzwa, Nyagasozi
Gicumbi / Miyove
/Miyove Cell Nyamiyaga, *Murehe /Mubuga Cell Kacyiru /
Ngoma /Zaza /Ruhinga
Cell Nyagahandagazi,* Gasebeya /Nyagatugunda Cell Jyambere, Kizenga, Rebero /
Nyabihu /Bigogwe
/Kijote Cell Bikingi */Rega Cell Kagano, Ngaregare
Nyamagabe /Kibirizi/
Bugarura Cell Muyange,Uwinyana /RuhungaCell /Gakoma,Nyagishubi
Nyamasheke/ Cyato/
Rugali Cell Karambo, *Rwumba, Rubeho /Bisumo Cell Kayo, Mutuntu
Nyarugenge
/Mageragere Nyarurenzi Cell Iterambere* /Kankuba Cell Kankuba, Karukina
Ruhango/ Mbuye/
Nyakarekare Cell Jali, *Nyakarekare /Kizibere Cell Ruhuha, Bereshi, Kizibere
Rwamagana
/Munyiginga/ Cyimbazi Cell Ntunga* Cyarukamba Cell Kagarama/ Cyarukamba Cell
Ndago Nyarubuye Cell Kiyovu, Kabeza
- Indicates
the umudugudu where the ECD&F centre is located.
The overall
methodology and sampling strategy needs to take into account how to
differentiate between the impact of the various ECD interventions being rolled
out, such as nutrition programmes, early learning programmes, child protection
advocacy programmes etc. In addition, not all ECD interventions have been
implemented in all intervention sites – which may provide an opportunity to gauge
the impact of different ECD interventions. The random sampling of control /
intervention groups will alone not address how to differentiate between the
different interventions, as it will only provide information on whether the
holistic ECD&F package (water, sanitation, nutrition, stimulation etc.) has
an effect or not. The consultancy institution selected for this evaluation will
explore optimal methods to differentiate between the impact of different
ECD&F interventions which have been rolled out between June 2014 –
September 2016.
The evaluation
process and methodology will include three phases:
Phase 1. Inception:
• Evaluation Plan development – draft work schedule to be submitted to the
ECD&F Evaluation Technical Committee for review and approval.
• In-depth desk
review of available information on Early Childhood Development, including
ECD&F baseline technical report and the baseline database, ECD data from
DHS, ECD policy and strategy, ECD&F programme funding proposal and any
other relevant information – to guide the development of the research
questions. Review the current status of children and women in Rwanda with focus
on ECD components and an integrated picture of ECD particularly in the selected
sites.
• Preliminary
discussions with UNICEF Rwanda, Imbuto Foundation and the ECD&F evaluation
technical committee, to facilitate a common in-depth understanding of the
conceptual framework, refining the evaluation questions and adjusting data
collection methods, tools and sources. The consultancy team will have the
opportunity to discuss and propose amendments to the methodology, as long as
the purpose of the study is maintained and expected deliverables are produced
at the required level of quality. Any proposed changes will be discussed at
planning meetings with the Technical Committee at the beginning of the process.
• Drafting of
Inception report (deliverable 1), including the details of the methodology to
be used, an Evaluation Matrix for each finally agreed evaluation question and a
detailed analysis plan, to be presented to and approved by the members of
ECD&F Evaluation Technical Committee. The proposed methodology needs to be
sufficient to capture all the indicators agreed for this evaluation purpose.
Phase 2. Data
collection: • Develop data gathering tools (building on the tools developed for
the ECD&F baseline evaluation – see appendix 2). The ECD&F Baseline
Assessment Tool comprised several internationally recognized measures covering
household characteristics, child development, nutrition, health, caregiving
practices, and services access. The battery of instruments was based on several
internationally recognized tools, such as the ECD module of the Multiple
Indicator Cluster Survey, indicators from Demographic and Health Surveys, and
the widely used Ages and Stages Questionnaire (ASQ). Measures were selected
that evaluated key areas of the ECD&F intervention and are intended for use
as part of a longitudinal evaluation to assess the impact of the ECD&F
program.
• The development of
data gathering tools, interview and observation guides using a consultative
participatory process, will include the following steps: o Facilitate a meeting
to share the tools for comments with the Technical Committee overseeing the
consultancy. o Pre‐test
the tools and facilitate a meeting with the Technical Committee, to validate
the pre‐
testing data obtained and fine tune the tools based on the experiences from the
team of interviewers/enumerators. o Train research assistants /enumerators for
data collection in all the selected sites to ensure that standardised
methodology and application of the tools are fully understood. o Conduct
collaborative planning for field work with the Technical Committee, including
development of agenda and activities.
• Work with team of
interviewers/observers to collect data using quantitative, qualitative
observations and other appropriate methods (measurable indicators) from
children, parents, caregivers, teachers, opinion leaders and primary health
care providers at community level and from key informants in the selected
sites: o In alignment with the agreed methodology, the consultant firm will
collect quantitative end-line data at intervention/treatment and control sites,
including anthropometric measurement for all households and children included
in the ECD&F baseline evaluation (both intervention and control group). o
The consultant firm will conduct focus group discussions and in-depth interview
with key ECD&F stakeholders, including ECD&F center directors, caregivers
at centers, parents/caregivers of children in the baseline intervention and
control groups
• Convene a briefing
meeting with the supervising task force on the on‐going survey for corrective
measures to be undertaken if need be. • Ensure the quality of information collected
from fields, cross check with the validity and reliability of information
collected and verify • The consultant firm will submit regular progress report
and field reports throughout data collection period (deliverable 2)
Phase 3. Analysis
and reporting phase:
• Process data (data
entry, cleaning, and analysis) using the SPSS/STATA/ EpiInfo or any acceptable
statistical data analysis package for data gathering and analysis.
• Following the data
collection and analysis phase, the evaluation team will make a presentation of
the preliminary key findings (First and preliminary findings report –
deliverable 3) to the ECD&F Evaluation Technical Committee.
• A detailed draft
technical report will be shared with key partners for review, and once all
feedback of ECD&F Evaluation Technical Committee members have been
addressed, the consultant firm will share a final report (deliverable 4) for
review and validation by the ECD&F Evaluation Technical Committee.
• A concise
(approximately 40 pages) summary report (deliverable 5) of the key findings of
the validated technical report drafted and shared with the ECD&F Evaluation
Technical Committee
• All outputs from
the Evaluation are subjection to approval by the ECD&F Evaluation Technical
Committee.
• An academic article
(deliverable 6) which will be drafted by consultant firm and members of the
ECD&F Evaluation Technical Committee is the end product. This academic
article, as well as any publication emanating from this evaluation are subject
to prior approval of the Technical Committee 6 Major Tasks, Deliverables &
Timeframe
Tasks/ Expected
Deliverables &Timeframe
Desk
review of available information on ECD in Rwanda – in particular the ECD&F
baseline evaluation technical report and ECD&F baseline database -Feedback
meeting on findings from desk review -Week 1
Design
of the sampling strategy, data collection and relative tools and preparation of
inception report.
Application
for all relevant research permits for Rwanda -Inception report (deliverable 1)
including work plan, methodological approach, instruments to be used, interview
and field visit protocols, annotated outline of final report, to be presented
and approved by the Technical Committee. -Week 2, 3 & 4 (1st payment, 30%)
Data
collection, cleaning and analysing quantitative data -Quantitative data
analysis progress report and end of field data collection report (deliverable
2) -Week 5,6, 7, 8,9, 10
Focus
group discussion and Key informant interviews
Progress
report -Week 5,6, 7, 8,9, 10
Analysis
of findings and draft report preparation -Preliminary findings report and
presentation of key preliminary findings to the ECD&F Evaluation Technical
Committee (deliverable 3) -Week 11, 12, 13, 14, 15 (2rd payment, 30%)
Validation
of technical report / Incorporate comments from ECD&F Evaluation Technical
Committee and finalize report -Draft technical report shared with ECD&F for
review and validation (deliverable 4). Meeting report confirming the validation
of findings, including how the feedback/recommendations from the committee have
been thoroughly addressed. -Week 16
Draft
a concise, user-friendly for non-technical audience, summary report of key
finding from technical report -Summary report (deliverable 5) -Week 17
article to peer-reviewed publication -Academic article (deliverable 6) -Week 18
(3rd payment, 40%)
The final report
should be in line with the UNICEF evaluation standard and very focused on
practical and implementable recommendations. Specifically, the report should
include at least the following sections: executive summary, description of the
evaluation methodology (as per agreed inception report), assessment of the
methodology (including limitations), findings, analysis, conclusions, lessons
learned and recommendations for improvement. The Annexes to the report should
contain: the TOR, the approved data collection instruments, and any other
relevant information.
The final report
should be in line with agreed Government and UNICEF evaluation standards and
very focused on practical and implementable recommendations. The report
template should include, but not limited to: • Title page and opening pages •
Executive summary • Programme description • Role of UNICEF, Imbuto Foundation and
other stakeholders in programme implementation • Purpose of Evaluation •
Objectives • Evaluation design • Methodology, including sampling strategy and
methodological limitations, and evaluation criteria • Stakeholder participation
• Ethical issues • Major findings • Analysis of results • Key Constraints •
General Conclusions • Recommendations • Lessons learned • Annexes TOR, tools of
data collection used
The report should be
provided in both hard copy and electronic version in English. Complete data
sets (tools, database, filled out questionnaires, records of interviews and
focus group discussions etc.) should also be provided to UNICEF at the end of
the evaluation, and will remain the property of UNICEF. Any use of datasets and
publications emanating from the endline evaluation is subject to prior approval
of UNICEF Rwanda.
Potential use of the
evaluation findings: This study will serve (1) to inform programming and policy
makers on the impact of ECD&F programming in the selected sites (2) to
enrich key stakeholders (MIGEPROF, Imbuto Foundation, UNICEF Rwanda and I/NGOs,
all ECD Ifs (Implementing Partners)) awareness of areas in which ECD&F
programming can be strengthened to enhance child development, and (3) to inform
external stakeholders, including key donors of the programme, on the impact of
ECD&F programming on child development.
Dissemination of
Results: Findings of the evaluation will be summarized and discussed with the
ECD&F Evaluation Technical Committee Members. Findings will also be made
available to relevant stakeholders, in consultation with the Technical
Committee.
7 Stakeholder
Participation
Key stakeholders
include the members of the ECD&F Evaluation Technical Committee, which will
function as a Reference Group for the evaluation and assume the following
responsibilities for the evaluation:
• Plan and design
the evaluation through consultation with the main parties involved and final
approval of evaluation’s terms of reference • Provide technical inputs to the
design of the evaluation • Provide guidelines to evaluators and monitor the
evaluation implementation • Review the evaluators’ inception report (including
proposals for desk review of documents, evaluation instruments, country visits,
annotated outline of the report) • Review preliminary findings for validation
of facts and analyses, and help generate recommendations • Approve the
preliminary report • Review and approve the final report, verify the
evaluators’ findings and propose management response • Ensure that the
evaluation findings are used for future planning and ECD&F programmatic
interventions as well as advocacy purposes.
The variety of
stakeholders in the Technical Committee will ensure that different opinions are
represented and objectivity is achieved.
UNICEF Rwanda will
be responsible for selection of the institution to conduct the evaluation. The
evaluation will be managed by UNICEF. The management of the evaluation will
involve drafting the terms of reference, initiating evaluation selection
process, liaison between the evaluation team and other members of the Technical
Committee, as well as quality assurance of the report.
The consultancy
institution will be responsible for organizing field visits, logistical support
related to data collection, and organizing meeting with different stakeholders,
including convening meetings of the ECD&F Evaluation Technical Committee.
UNICEF Rwanda will be responsible for providing relevant information at country
level, providing access to relevant reports/statistics, and providing inputs
for methodology and for data analysis.
8 Existing
information sources The following information sources are available: •
ECD&F baseline evaluation technical report
- ECD&F
baseline evaluation summary report • ECD&F baseline evaluation full
database • ECD&F baseline evaluation tools (including questionnaire) •
DHS ECD module analysis (will be available in March 2016) • ECD&F
programming funding proposals / logframe
9 Ethical
Consideration/confidentiality Adequate measures should be taken to ensure that
the process responds to quality and ethical requirements as per UNICEF
Evaluation Standards. As per United Nations Evaluation Group (UNEG) Standard
and Norms, and in particular adhering to Rwandan standards and norms, the
consultants should be sensitive to beliefs, manners and customs and act with
integrity and honesty in relationships with all stakeholders. Furthermore,
consultants should protect the anonymity and confidentiality of individual
information. Consultants should respect the confidentiality of the information
which is being handled during the assignment. Consultants are allowed to use
documents and information provided only for the tasks related to the terms of
reference of this evaluation. Data will be stored in a secure location, kept
confidential with access restricted to principal investigators. The study data
will be used only for the purpose of this study.
10 Evaluation team
composition / qualifications and requirements
The selected
evaluation institution will be responsible for the creation of an evaluation
team. The minimum request is that the team consists of at least two experts
(one expert in quantitative research and impact evaluation, and a further
expert team members for qualitative research). The team composition should
include national (Rwandan) experts. The exact division of work will be decided
by the institution, but in general, the team leader will be responsible for
discussions, negotiations, final decisions, shape of the evaluation, while
further team members will be tasked with more technical issues (revision of
technical reports, in-depth interviews with service providers, decision makers,
parents, revision of existing research reports etc.). The team will preferably
include the following profiles: Psychologist/Sociologist; Nutritionist/Health
professional; Statistician ‐Data
entry and analysis staff; Data collection Assistants The qualifications and
skill areas required include:
Technical expert
& team leader: • Extensive quantitative research and impact evaluation
expertise and experience, including expertise in data collection and analysis;
demonstrated skills in similar evaluations; demonstrated technical report
writing skills • Demonstrated experience and expertise in designing and
implementing multi‐sectoral
initiatives in partnership with a wide range of stakeholders including
government and communities • Minimum five years of relevant work experience of
which two at national and international levels in field programmes relevant to
ECD, Education and/or Child Survival and Development and/or Child Protection. •
Understanding of the integrated and cross‐sectoral
nature of ECD. • Advanced university degree in one or more of the disciplines
relevant to the following areas: Evaluation expertise, economics and social
sciences. Understanding of early Childhood Development, Child Survival and
Development, Education, Social Sciences, Public Health and/or Child Protection
• Familiarity with technical aspects related to Early Childhood Programming •
Knowledgeable on institutional issues related to the provision of global public
goods; Experience working with/in the UN or other international development
organizations in the social sector or in national level development assistance
and partnership support to government programmes and priorities is an asset. •
Fluency in English a must and knowledge of French and Kinyarwanda an advantage.
Qualitative research
expert: • Extensive qualitative evaluation expertise and experience, including
data collection skills; demonstrated skills in similar evaluations • Knowledge
of technical aspects of similar programmes • Knowledge of the areas of
intervention
All members of the
team: • Language proficiency: excellent writing skills in English • Advanced
university degree in related field or social science • Work experience in
different countries globally: at least 8 years of field experience for team
leader and research expert; at least 3 years of field experience for all other
team members. Experience in working with UN agencies (desired) • Experience in
evaluations/research: knowledgeable on UN evaluation policy, recommended by
UNICEF regional or global evaluation advisors or other senior managers, skilled
in performing structured interviews and facilitating focus group discussions •
Analytical skills: Demonstrated analytical skills related to the use of
quantitative and qualitative data for decision-making • Process management
skills: Demonstrated skills and experience in conducting and presenting
evaluations • Good communication and advocacy skills: Ability to communicate
with various stakeholders, and to express ideas and concepts concisely and
clearly in written and oral form
Evaluators should be
sensitive to beliefs and act with integrity and respect to all stakeholders.
Evaluators should protect the anonymity and confidentiality of individual
interviewees.
Evaluation and
selection criteria of the consultancy institution: A two stage procedure shall
be utilized in evaluating proposals, with evaluation of the technical proposal
being completed prior to any financial proposal being compared. A 70/30
assessment model for the technical and financial proposal respectively will be
adapted. Cumulative weighted average methodology will then apply in determining
the best value for money proposal.
Applications shall
therefore contain the following required documentation: a. Technical Proposal:
Consultant institution should prepare a proposal on the basis of the tasks and
deliverables (as per the ToR). The proposal should include approach and
methodology with detailed breakdown of inception phase, proposed scope and data
collection methodology and approach that will be used by the consultant. The
proposal shall also include a brief explanation of the data analysis and report
writing and possible dissemination plan. Draft work plan and timeline for the
evaluation should be included. The Technical Proposal shall also include
updated CVs and copies of 2 reports of previous evaluations (ideally
health-related) conducted by the consultants.
b. Financial
Proposal: Expected financial offer with cost breakdown of consultancy fee and
daily subsistence allowance (DSA) during the field work in Rwanda. The
financial proposal shall be submitted in a separate file, clearly named
financial proposal. No financial information should be contained in the
technical proposal as this will lead to proposal cancellation. Financial
Proposals should be filled as per table below:
Deliverable /Number
of person days /Delivery date /Costs -Inception report (Deliverable 1)
-End of data collection report (Deliverable 2)
-Preliminary findings report (Deliverable 3)
-Final report and summary report (Deliverable 4 and 5 )
-Operational Costs ( a detailed addendum budget required)
Total
11 Supervision The
evaluation will be supervised by UNICEF Social Policy and Research Section and
UNICEF’s ECD&F specialist, jointly with Imbuto Foundation (Both ECD and
Research Officers). The ECD&F Technical Committee will provide technical
inputs to the design of the evaluation, provide guidance to the evaluators, and
monitor the evaluation implementation process.
12 Terms and
conditions
Procedures and
logistics
Evaluators are
expected to use their own hired vehicles, equipment, including computers.
Consultant institution is required to ensure translation to Kinyarwanda during
field trips as well as translation of documents from Kinyarwanda to English if
required. UNICEF will be under no operational obligation to pay for operational
costs related to this consultancy, all costs required to operationalise this
consultancy shall be borne by the hired institutional firm and should be
included into the proposed financial proposal.
Terms of payment
The payment will be
in three (3) instalments as follows:
• 30% of the total
payment upon completion of the desk review, submission of inception report with
work plan and methodology, theory of change and research instruments and
protocols.
• 30% of the total
payment upon completion quantitative and qualitative data collection and
analysis, including field visits and submission of the draft final report and
PPT of the evaluation;
• The remaining 40%
will be paid upon completion of all deliverables, as per the above schedule
(validated final report of the evaluation; a set of Power Point slides (25-30
slides) with key salient features of the evaluation; a summary report and
publication in the form of academic articles for submission to peer-reviewed
journals
All the deliverables
need to meet UNICEF requirement and quality standards. Payment will only made
for work satisfactorily completed and accepted by UNICEF. UNICEF reserves the
right to withhold all or a portion of payment if performance is unsatisfactory,
if work/outputs is incomplete, not delivered or for failure to meet deadlines.
All materials
developed by the firm will remain the copyright of UNICEF, who will be free to
adapt and modify the materials for future use.
13 Appendix 1 – Key
Indicators Table
The baseline and
end-line include additional indicators to evaluate, in addition to the ones
listed below: Indicator Household Characteristics Mean household size (persons)
Mean number of children under 18 in household Mean number of children under 5
in household Percentage with a second caregiver for the child living in the
household Percentage of children living with both biological parents Percentage
of children living in households with their biological mother Percentage of
children living in households with their biological father Percentage of
children living in a households in Ubudehe category 1 or 2 Percentage of
households in Ubudehe 1 or 2 receiving Social Protection assistance Percentage
of children registered with civil authorities
Household Assets
Percentage of children living in households with electricity Percentage of
children living in households with radio Percentage of children living in
households with mobile phone Percentage of children living in households with
bike Percentage of children living in households with bed net Reporting Primary
Caregiver Characteristics Percentage of primary caregivers who are the child’s
biological mother Mean age of primary caregiver Percentage of primary
caregivers who are currently married or living with a partner Percentage of
primary caregivers who completed primary school Percentage of primary
caregivers who completed secondary school Percentage of primary caregivers who
are illiterate
Decision Making in
the Household Percentage of households in which both mothers and fathers decide
together if a child attends ECD services Percentage of households in which both
mothers and fathers decide together the action when a child is sick Percentage
of households in which both mothers and fathers decide together what a child
eats Percentage of households in which the mother decides if a child attends
ECD services Percentage of households in which the mother decides the action
when a child is sick Percentage of households in which the mother decides what
a child eats Percentage of households in which the father decides if a child
attends ECD services Percentage of households in which the father decides the
action when a child is sick Percentage of households in which the father
decides what a child eats
Early Childhood
Development Child Development Percent of children below cut-off for
age-specific ASQ:3 in one or more domain of development Percent of children
above cut-off for age-specific ASQ:SE social-emotional development Caregiving
Practices Percentage of children experiencing any inadequate care in the past
week Percentage of children exposed to any violent discipline Percentage of
caregivers who believe physical punishment is necessary to raise a child well
Percentage of caregivers who argue during a typical week about relationship in
child’s presence Percentage of caregivers who say cruel things to each other
during a typical week in child’s presence Percentage of caregivers who have had
serious conflict in the home during the past 6 months Percentage of interviewed
caregivers who have gotten into serious conflict with other household mothers
during the past 6 months
Support for Learning
Father cares for child daily Primary caregiver engages in three activities to
promote learning or school readiness in the past week Availability of
children’s books in the household Availability of playthings in the household
Child has regular playmates his or her age Access to ECD and Community Services
Percentage of children who attend ECD program Percentage of households with
children age 4 to 6 who attend nursery or pre-primary school Mean age of child
attending nursery or pre-primary school Percentage of caregivers who have heard
of the 1,000 days campaign Percentage of caregivers who have heard of parenting
evening talks program Percentage of caregivers who have heard of cooking
demonstrations by CHWs
Water, Sanitation,
and Hygiene Percentage of households with an improved water source Percentage
of households with an unimproved water source that treat water appropriately
before drinking Percentage of households that store water in closed containers
Percentage of households with an improved sanitation facility Percentage of
households with a place for hand washing Percentage of households with a place
for hand washing with water and soap/cleansing agent available Percentage of
households with safe disposal of 0-11 month old children’s faeces
Health Maternal
Health Percentage of primary caregivers with poor mental health Percentage of
primary caregivers who ever consume alcohol Percentage of mothers who are HIV+
Percentage of mothers who knew HIV+ status during pregnancy Percentage of
mother who received prevention of mother to child transmission of HIV services
Percentage of mothers who experienced complications during delivery (including
needing a caesarean section)
Child Health
Percentage of children who had diarrhea in the past two weeks Percentage of
caregivers who sought any treatment for diarrhea Percentage of children who
received oral rehydration salts (ORS) Percentage of children who received
recommended homemade fluid (RHF) Percentage of children who received either ORS
or RHF Percentage of children who slept under mosquito net last night
Percentage of children who slept under mosquito net last night among households
that have nets Percentage of children who had fever in the past two weeks
Percentage of caregivers who sought any treatment for fever Percentage of
children who were dewormed in the past six months Percentage of children who
were visited by CHW in the past month Percentage of children who were growth
monitored in the past month
Health Insurance
Coverage Percentage of households with any member insured Percentage of insured
households with index child insured Percentage of insured households covered by
mutuelle Percentage of insured households with all members insured
Nutrition Percentage
of children who are severely stunted Percentage of children who are stunted
Percentage of children who are severely underweight Percentage of children who
are underweight Percentage of children who are severely wasted Percentage of
children who are wasted Percentage of children with global acute malnutrition
Percentage of children who are receiving the minimum acceptable diet Percentage
of households who did not have enough food or money to buy food at least 1 day
during the past week
14 How to apply:
Qualified institutions
are requested to submit a full proposal, consisting of two separate parts
(technical and financial), which can be downloaded from our website to rwasupply@unicef.org.
The deadline of
submission is Tuesday 07 June 2016 at 5PM Kigali time