- Utility: serve practical information needs of intended users
- Feasibility: be realistic, prudent, diplomatic and frugal.
- Propriety: conducted legally, ethically, and with regard to those involved in and affected by the evaluation.
- Accuracy: reveal and convey technically accurate information.
Capacity and resources for M & E
- Technical capacity for M & E
- M & E unit
- Strategically located individuals who are motivated, committed, competent and interested in M & E
2. Scale of program funding and resources for M & E
- 10% of resources devoted to M & E
- Costs related to data collection systems and information dissemination
- Costs may be higher during the first year
- Other resources include indicator guides, manuals, communication tools
Key steps in developing and implementing an M & E plan
Stakeholder consultation and participation
- Define process for stakeholder involvement
- Translate problem statement, program goals and objectives into M & E framework.
- Establish scope of the M & E plan
- Develop M & E framework
- Determine elements to be monitored and evaluated
- Define indicator and identify data sources
- Determine M & E methods for data and information collection
- Develop data collection plan
- Determine M & E responsibilities
- Set targets
- Define reporting system, utilization and dissemination of results
- Plan for mid-course adjustments
Stakeholder consultation
- Advocate for the need for M & E
- Understand program goals and objectives
- Identify user needs and perspectives
- Learn about existing data collection system and their quality
- Understand indicators that are being collected and used (or not used)
- Determine capacity for collecting and using data
Stakeholder participation
- Developing M & E framework
- Selecting indicators
- Setting target
- Reviewing results
- Require building consensus and commitment and maintaining effective relationships with intended users
Translating program goals and objectives-M & E frameworks
- Translate problem statement, program goals and objectives into framework that can be objectively measured.
- What do we want to know at the end of the program?
- What do we expect to change by the end of the program?
- M & E framework outlines the scope of the M & E plan
- Links program activities to expected outputs, outcomes, and population level impacts
- Informs types of information to be collected/used by different levels of the health system.
- Identifies what needs to be measured to guide indicator selection
Elements to be monitored and evaluated
- Monitoring:
- Resources (inputs)
- Quality of service
- Service statistics
- Client/patient outcomes (behaviour change/morbidity)
- Impact evaluation
- Attributing the change in outcomes to the program
Developing M & E framework
- Review program document with stated goals and objectives
- Ensure that factors that may influence program implementation and success are identified
- Achieve consensus among stakeholders
Defining indicators and identifying data sources
Indicators must be:
- Smart
- Linked to M & E framework
- Useful for program decision-making
- Consistent with international standards and other reporting requirements (as appropriate)
- Realistic to collect (feasible).
Data sources:
- Existing vs. New
Determine M & E methodology
- Assess information systems capabilities to address strategic information needs
- Determine methods by which data will be collected, analyzed and reported e.g. RHIS, surveys, sentinel surveillance system, project information systems/records, new data collection
- Determine whether any special studies will be conducted and what design will be used e.g. qualitative, quantitative or combination of both.
Assigning M & E responsibilities:
Implementing the M & E Plan
Monitoring data
- Who will collect data?
- Who will analyze data?
- Who will report data?
Special studies
- Who will oversee data collection?
- Who will conduct analysis and reporting?
- Is anyone else planning a similar evaluation?
Roles of the M & E unit
- Consensus building among all stakeholders
- Coordination
- Between various program components
- Between other stakeholders (avoid duplication of data collection)
- Data manipulation
- Reporting
- Information dissemination and review
Data collection plan/matrix
- Indicators
- Who is responsible
- Timing
- Data quality notes
Class activity: Data collection plan
Develop a Data Collection Matrix for your Program, Consider the Following Issue:
- Who will be responsible for data collection and its supervision?
- Who will be responsible for ensuring data quality at each stage?
- How will data quality be checked at every stage?
- How often will the data be collected, compiled, sent and analyzed?
- What tools/forms will be derived from each data source?
- What resources (staff, office, supplies, computers, and transportation) will be needed at each stage?
- Who will analyze the data? How often will analysis occur?
- How often will the results be complied into reports?
- To whom and how often will the results be disseminated?
Setting targets
- Focus on what the program should achieve
- Orient stakeholders to the task to be accomplished
- Motivate individuals
- Monitor whether anticipated progress is being made
Factors to consider when setting targets
- What can realistically be achieved given the resources and the program context?
- Baseline levels
- Past trends
- Needs and gaps in services
- Capacity and logistics
Approaches to setting target
- Establish final target then plan progress in between
- Establish annual/intermediate targets.
- Assess progress in attaining targets and readjust, if necessary.
Useful information for setting target
- Past trends
- Expert opinions
- Research findings
- What has accomplished elsewhere
- Client expectations
Developing a data dissemination and utilization plan
- Define users of M & E findings
- Define feedback mechanism to meet user needs
- Strategically timed user meetings/workshops
- Annual report and review meeting
- Database to manage data and facilitate access and use (e.g. PIMS, DSS)
Planning for mid-course adjustment
- Program changes can affect the M & E plan performance monitoring and impact evaluation
- Internal M & E capacity facilitates adjustments
- Flexibility and regular review of program results necessary
How well is the M & E plan working?
- Are M & E activities progressing as planned?
- Are the evaluation questions being answered sufficiently?
- Have other evaluation questions been raised and should they be incorporated into the plan?
- Are there any methodological or evaluation design issue that need to be addressed?
- Are there any outside factors (political, environment) that are affecting the plan?
- Are appropriate staff and funding still available to implement the M & E plan?
- Are M & E finding being disseminate and used by stakeholders for decision making and program improvement?
Remember that an M & E plan is a living document and needs to be adjusted when a program is modified.
