24 June, 2009
Madeleine Bunting’s first of five articles on the components of the Katine project is very timely, and the intention of the series is spot on. They relate to the forthcoming review of the progress of the Katine project via a process known as a Mid-Term Review (MTR). Her hope is that her pieces “… will provide a useful rough draft with a few pointers for the professionals [i.e. the MTR team] who will follow, which is why I’ve listed my questions – please add any that you have which you would like the mid-term review and our independent evaluators to consider”
The MTR process
The MTR team will begin their work from next Monday 29th June, and are expected to produce a report by late July. This review is probably the most important review process during the whole of the life of the Katine project, more important even than the end-of-project evaluation. This is because the results could influence decisions taken over the next year, about (a) what the project should try to do in the remaining time left and (b) what should happen after the project officially ends in late 2010.
Part of the planning process for a mid-term review is the development of Terms of Reference (ToR) which will guide the work of the MTR team. They normally spell out the purpose of the review, the scope of activities to be reviewed and nature of the final products expected. Along with other information on the background of the project, and expectations about how the review will be undertaken. Normally ToRs are subject of negotiations between the stakeholders involved, including the donors (e.g. Guardian and Barclays), the implementing agency (e.g. AMREF) and local partners (e.g. government bodies and community groups in Katine). This process is already underway and will continue up to early next week when the MTR team visits Kampala and Soroti. As with the first two visits to the Katine project by myself (the external evaluator), the ToRs will be made public via the Guardian website. What is different this time is the opening up of the ToRs consultation process via the Guardian website, and Madeleine Bunting’s articles this week in particular.
The points raised by Madeleine’s posting on the health component
At the beginning of my work on the Katine project I proposed that we should use seven criteria for evaluating the project. Five of these are OECD Development Assistance Committee (DAC) criteria: relevance, effectiveness, efficiency, impact and sustainability. Two others I suggested to be included are: equity (fairness of process and outcome) and transparency.
One of the challenges for the MTR team is which of these criteria to apply to which project component, because applying all would mean a much more time consuming MTR process, which may not be affordable (in the wide sense of the word).
Re the distribution of anti-malaria bednets Madeleine Bunting asked “Has there been any coordination to avoid overlaps between AMREF and other donors on this issue?“, that is the distribution of bednets by different agencies to the same community. This question is concerned with efficiency. We could also ask about equity, who is actually using these within the receiving households…children or adults, mothers or fathers. The intention was, I think, that children should be using them, because they have less resistance to malaria. We could also ask about sustainability: Where will households get replacement nets in the future, after the project ends? My feeling is that the equity and sustainability questions are more important here than the question of efficiency.
A number of Madeleine’s questions relate to the criteria of relevance. Are the existing project interventions the most appropriate means of addressing the pressing health problems? Would some form of ambulance service help ensure women with birth complications were able to get to a doctor in time for a caesarian birth? Is “empowering” villagers in Katine with health information enough, when government services are so inadequate in the delivery of drugs and medical staff? Would some form of community health insurance be a useful means of topping up drug supplies or health centre staff pay.
Relevant – compared to what?
In order to answer these questions the MTR team will need to try to understand the project design – what were the objectives and what was the plan for achieving them. It could be unfair to assess a project in relation to an objective it never prioritised in the first place. Part of this process involves a reading the initial project documents and any official revisions to the project design thereafter.
The Katine project did develop a “Conceptual Framework” at the beginning of the project in September 2007, which spelled out what some people call a “Results Chain, showing how AMREF activities would contribute to the achievement of improvements in health, education, water, sanitation, health, livelihoods and governance.
In the health component the expected outcome was “Increased community awareness of, access to and utilisation of health services in community and health facilities” This presumably covers both the services provided by the Village Health Teams, and the two tiers of government health centres with Katine (HC2 and HC4). This is quite ambitious given that there are only two AMREF staff working on the health component over a three year period. Therefore some prioritisation of the kinds of people who should be using health services more than before, and the kinds of health services they should ideally be using more than before, might be expected. Hopefully these prioritisations would be aligned of those with local government planning bodies and the Katine community (i.e. they would be seen as relevant). These issues are aspects of the health component that the MTR team could be looking at.