Comments on the AMREF M&E Plan for Katine.

12 May, 2008

Comments on the

KCPP Monitoring and Evaluation Plan Version 3. 29/11/2007

By Rick Davies, Tuesday, 25 December 2007 Version 2

1. It is good to see that the project has already developed a detailed M&E plan, and that has undergone a number of revisions.  My comments that follow below focus on areas where I think there are gaps, potential problems and potential opportunities.

2. As with the earlier Community Needs Assessment document, it is important to identify who is the author of this document. In London it has been suggested that this M&E plan was produced for AMREF by Accenture, but in the absence of author details we don’t know if this was the case or not.

3. An additional section on who participated in the development of this M&E plan would be very useful. There is a section on livelihoods (Objective 4) but it is not clear to what extent, and how, Farm Africa may have contributed to this section. Although Barclay’s plans for work on microfinance are still being developed ideally their plans for M&E of their interventions would be integrated into this plan. Even if the M&E of microfinance was discussed in a section of its own, it would still be useful to highlight how and where there would need to be some joint M&E activities with AMREF staff. Such as with the implementation of baseline surveys and mid-term reviews.

4. It is possible that this document is under-representing the participatory dimension of AMREF’s work in Katine. There is no reference to how Katine people have been involved in the design of this M&E plan. For example in the selection of appropriate indicators. Or, how they will be involved in monitoring and evaluation processes, other than survey respondents. It seems highly likely, and desirable, that AMREF will make extensive use of meetings with different stakeholders groups on a periodic and ad hoc basis, throughout the project period[1]. The plans for the use of these meetings should be described in this M&E plan.

5. The section on the Community Needs Assessment (p.3) does refer to “group discussions with a wide range of community residents in public meetings to ensure fair representations of concerns”. However, the one available report on the Community Needs Assessment provides no information on the consultation process, on the priorities that were expressed, and how priorities varied across stakeholder groups.

6. A brief mention has been made on page 5 of the M&E plan on the intention to use focus group discussions and community meetings during the mid-term and end-of-project reviews. Planning and documentation of these reviews will need to provide much more information than was provided in the summary report on the Community Needs Assessment.

7. Annex A lists the details of a very large number of indicators, on which data will be collected at various intervals. Approximately 136 in all. Of these, 41 will be collected quarterly, another 8 bi-annually, another 43 annually and another 32 at the beginning and end of the project.  My main concern here is how realistic is it for AMREF staff to aim to collect data on this many indicators?  Collecting all this data will take up valuable staff time, and analysing it will take up more time still, and so will communicating the results of this analysis.

8. There are three possible ways of rationalising this long list. The first is to ask how well the size of each set of indicators relates to the scale of AMREF’s planned investment in the objective concerned. Will AMREF be investing most effort (e.g. # of indicators) in monitoring its biggest investments, and the least in its smallest investments? The table below shows what appears to be an over-investment in monitoring the health objective, and under investment in monitoring the other objectives, especially the water and sanitation objective. Is there a good reason for this? Or, could the number of health indicators be reduced?

Objectives

% and # of indicators

Budget allocation

1. Improved community health

41% (62 )

26% (£299,000)

2. Improved access to quality education

23% (35)

26% (£295,000)

3. Improved access to safe water, sanitation and hygiene

10% (16)

18% (£205,000)

4. Improved income generating and food security opportunities

21% (32)

23% (£270,000)

5. Communities empowered to engage in local governance

5% (7)

7% (£76,000)

Total

100% (152)

100% (£1,145,000)

9. The second possibility is that the list of indicators proposed under each sub-objective could be presented to different stakeholder groups in the Katine community in the form of a menu. Those groups could then be asked to identify those changes (represented by the different listed indicators) they thought were most important to achieve. The percentages of groups choosing each indicator could then be used as an “importance” weighting for each indicator. These weightings would help in two ways. Firstly, to identify which indicators could be abandoned, because they were a low priority to most stakeholders. Secondly, to identify those indicator where analysis of progress needed to be done with most care and attention, because they were of widespread interest.  Thirdly, to guide what analysis needed to be communicated to what groups, according to their previously expressed priorities.

10. The third possibility is to look at the coverage of the indicators that have been listed in for each objective. Some focus on activities project staff will carry out, some relate to activities that involve both project staff and Katine peoples, and some focus on changes in Katine people’s lives. This corresponds approximately to a focus on Activities, Output and Purpose (outcome) levels in a Logical Framework. Ideally, with each project objective, AMREF would want to monitor a sequence of indicators that tell us about a sequence of changes, of how activities lead to outputs which then lead to outcomes (Purpose level changes). At present the mix of indicators under some of the objectives is unbalanced. Under the education objective there do not seem to be any education outcome measures like enrolment rates, attendance rates, drop out rates or graduation rates. Yet surely these are of concern to both the Katine community and AMREF. On the other hand, the health objective seems to have many outcome indicators, perhaps many more than the project could hope to significantly influence during its lifespan.

11. Annex B and C are referred to but not included. B refers to the data collection formats to be used for the Rapid Assessments, scheduled to be carried out in December 2007, and some of which have been completed. Annex C presumably refers to the survey instrument to be used for the Baseline Household Survey, scheduled for December 2007-January 2008. Copies of these instruments need to be provided not only to myself, as the external evaluator, but also to Farm Africa and Barclays, since they will need to use the data from these surveys. Ideally copies of these instruments would have been circulated to Farm Africa and Barclays before these surveys began.

12. On page 5 there is a reference to the household survey which will measure “change at the household level to see if targeted community members have benefited as a result of project activities” it is very important that the household survey also includes comparable households who have not participated in different project activities. This should not be difficult to do.

13. The M&E plan refers to an intended sample of 95 households for the household survey. The rationale for this sample size needs a clear explanation. Will the sample be intended to represent the 25,000 people of Katine sub-county, or only of Katine village itself? If the former, it looks like as very small sample[2]. If the latter, then why focus on Katine village alone. The sample size should be big enough to allow simple statistic tests of the significance of differences found between at least two categories of groups of people over two periods of time (participants and non-participants, before and after an intervention). For example, by using Chi Square test with four cells

14. It will be very useful if a modest plan could be drawn up for how the baseline survey results will be analysed. This will help inform decisions about appropriate sample sizes, and what indicators are most central to the baseline survey. This could include

  • Frequency distributions: for which indicators, displayed in what form?
  • Cross tabulations: what indicators against what indicators? There are at least 32 indicators that data will be collected on, which means more than 1000 possible cross tabulations of that data. In reality the most interesting cross-tabulations will be those that compare participation in specific project activities with changes in specific aspects of people’s lives. Differences in outcomes across different locations and groups will be especially important to analyse.

15. The M&E plan provides details on data sources for each of the indicators, but no information on who will be responsible for collecting and analysing what data. That would be a useful addition to the next version of this M&E plan. Also useful would information on what products of the M&E process will be made available to who and when. This could be presented in a Gantt chart form

16. A summary of the main recommendations made above:

  • Consider reducing the number of indicators that will be used, and ensure that in each area of AMREF’s work the selected indicators cover a sequence of expected events (i.e. from activities>outputs>outcome>impact)
  • Explain in detail the nature and rationale of the sampling strategy that will be used. Include a reference to the type of analyses that will be done with the survey results (because this affects the sample size required)
  • Provide more information on who has and who will participate in the monitoring and evaluation processes (both partners such as Farm Africa and Barclays, and various groups within the Katine community). And in what way they will participate.
  • Make this M&E Plan, of the next revised version, publicly available via the AMREF and Guardian websites

[1] In the September 25th 2007 Conceptual Framework paper reference is made to plans to work with farmers groups, VHTs, PTA’s and other community committees

[2] Even the plans for the Rapid Assessment (village component) refer to visits to 66 villages. If this is the total number of villages in Katine sub-country (or less than the total) a household survey sample of 95 households will cover less than two households per village.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: