Reviewing Katine: What’s happening with governance?

In her recent posting on this topic, Madeleine Bunting said “I’ve listened to Joshua Kyallo, Amref Uganda‘s director, explain how villagers can be empowered to demand better services from the government at district level. But there are plenty of questions in my mind as to how effective this will be in improving the operation of state services in Katine….The district budgets for health and education, for roads and water are desperately inadequate. It is not just the lack of demand for services that causes the state to be so ineffectual at village level here. I find the “rights-based” approach, based on developing in villagers a sense of entitlement to basic health and education, hard to understand”

Before asking whether the rights based approach is affective we need to ask if AMREF is in fact pursuing a rights based approach? As of August 2008 I could find no evidence of this on the ground, though the Country Director did affirm that AMREF supported a rights based approach. There are other interpretations of what empowerment is all about. The simplest and easiest to realise, is individual empowerment through the provision of practically useful information, for example, how to reduce the incidence of diarrhoea by maintaining clean water sources. This sort of empowerment was being addressed by the AMREF project in 2008. But it does not address wider issues such as the willingness and capacity of government to provide basic health services. Perhaps the project strategy has turned more in this direction since August 2008. The Mid-Term Review needs to look at this.

In the same article Madeleine Bunting also noted  “Several of the Amref staff spoke of how they had struggled with huge expectations of the project from Katine villagers. Is that the Guardian’s fault, I asked, with its headlines promising “transformation”? Perhaps partly, they agreed…I wondered how actively Amref has managed expectations and how widely it had communicated with villagers across this very scattered sub-county about what the project was going to do and what it was not going to do”

Expectations are usually about objectives and how they should be reached.  If they are diverse this suggests that communication and negotiation about project objectives may not have been as effective as they should have been. Madeleine Bunting’s article raises two possible causes: (a) insufficient communication with local communities by AMREF, and (b) the influence of the Guardian’s frequent visits to Katine communities. It could of course be both.

Another possibility is lack of clarity within AMREF itself, about what the project was trying to achieve. This was a concern I expressed in the first paragraph of my first visit report in January 2008. “The final objectives of the project may need clarification and agreement, by AMREF, its donors and local stakeholders. This agreement should be evident in a smaller set of indicators that show changes in people’s lives, reflect the impact of all five project components, and which can be easily be monitored by community groups.” At that stage the monitoring and evaluation framework had 35 indicators about expected changes in the lives of individuals and households and 60 indicators about the expected changes in the functioning of community groups and organisations. These are large numbers by the standards of most development projects. Later in 2008 the project staff in Katine made some efforts to prioritise these and focus on some key expected outcomes. One of the questions for the MTR should be looking at this year is the clarity of objectives within each of the components – within AMREF in the first instance, then amongst the wider group of stakeholders.

Reviewing Katine: What’s happening in livelihoods?

In her article on this topic, Madeleine Bunting commented “The livelihoods component of the Katine project has caused ongoing concern. Many times we have reported, and observers have commented, that not enough of the budget has been devoted to improving livelihoods. The vexed question over whether we should be giving what the villagers have repeatedly said they wanted – cattle – has repeatedly been raised.

Some of this disquiet seems to have been taken on by Amref and Farm-Africa because some interesting shifts in policy seem to have taken place. There is more emphasis on giving inputs – this is described as “hardware” in development lingo – such as seeds, tools, wheelbarrows and watering cans. The balance between hardware and “software”, or training, has been reversed in the livelihoods component so that more is being spent on the former.”

Later in the article Madeline rightly asks, in relation to the upcoming Mid-Term Review, “Can we have some explanation of why the approach on livelihoods shifted?” The reasons matter.  A subsequent comment from Farm Africa, who provide technical support to the Livelihoods teams, provided some clarification. They reported that “The software to hardware shift was in response to political pressure rather than internal reflection and learning. Politicians, especially at the sub-county level, were comparing KCPP with Government programmes that are hardware heavy, for instance the Northern Uganda Social Action Fund (NUSAF). The livelihoods component in particular was also being compared to the other components; Water, that sinks and rehabilitates boreholes; Health and Education, that build clinics and classrooms respectively, and being told to put more into hardware.”

While AMREF has obliged by providing more “hardware” such as seeds and tools, Farm Africa “are more convinced than ever that the approach of giving more seeds than trainings is not prudent. A significant number of the seedlings in the nursery had failed, mainly due to poor management, perhaps as a result of a lack of follow up training. This follow up training for the seedlings was not feasible due to the amount of the budget that was diverted to hardware. The backbone of our approach has always been to increase production by introducing new technologies and techniques rather than handing out seeds.”

This development is worrying for a number of reasons. Many development  NGOs, including probably Farm Africa and AMREF, would argue that advocacy is an important part of their work, and that they have competence in this area.  If so, why has Farm Africa caved in on an issue it believes in? Is it because they were unable to provide solid evidence from their projects elsewhere that training does make a difference? Or, was it simply the case that local authorities were impervious to the evidence that was presented, because they were trying to meet their constituents’ needs, regardless of their wisdom? Both prospects should cause some re-thinking. The same questions also apply to AMREF, who have been party to this change in direction. In the last (“Conclusions”) page of their 2007-2017 corporate strategy it is stated that “As we gather knowledge and evidence in our programme work and research, we will develop advocacy initiatives to influence policy makers to promote identified best practices.”

The issue of actual evidence is important. It is not self-evident that training will provide more sustainable development than material aid. Part of the “theory-of-change” behind the provision of training is the assumption that information about good agricultural practices (for example) will be passed on from one farmer to the next. Examples have been identified where this has happened. But comments by others (“Dr Jazz”)  underneath Madeleine Bunting’s article also highlight the fact that in some cases neighbours not only do not cooperate this way, but they actively sabotage each others efforts. Another commentator (“Ugandalife”) noted that that in their experience ” information is not generally shared easily. Information is considered an asset and therefore worth money. We have encountered this several times and it is hard to change this attitude.”

The idea that good practices will be imitated and reproduced by others is widespread amongst development projects, in just about all sectors e.g health, education, water and sanitation, livelihoods, etc. But just as common is the widespread failure by development agencies to invest any time and effort into systematically monitoring when and where (i.e. under what conditions) adoption by others actually takes place. This is worrying, because it suggests that many development agencies are isolated from important important strands of thinking that they could learn from. For example, the considerable body of literature that now exists on the “diffusion of innovation” Ironically, much of the early research in this field was done in relation to the adoption of agricultural research findings.

One of the implications of the concerns outlined above are that the MTR team should pay attention to: (a) where assumptions are being made about good practices being adopted by others, (b) what efforts are being put into monitoring how, when and where this is happening.

….

A second set of questions was asked at the end of Madeleine Bunting’s article: “Has some thought been given as to how to mitigate the tension over the fact that only a few people are benefiting from the free seeds and tools?
How significant are those tensions – are some people benefiting much more from this project than others? Could the project end up causing more disagreement and community fragmentation at a local level?”

Good question, worth trying to answer, under the ambit of equity concerns. If there are tensions there are two possible solutions, but only one of these has been discussed much so far. That is the try to extend coverage to all households. That is an expensive task and apparently beyond the current budget of the project. The other is targeting of households most in need. There has been little explicit discussion of this option, as far as I can see.

Farm Africa’s response to the Madeleine Bunting’s second set of questions was that “As far as we know, there is no significant tension within the community as a result of the intervention. One of the reasons why there is no tension, is because the beneficiaries we are working with were not hand picked by the organisations, but rather selected in a participatory and open process involving different stakeholders through an agreed criteria.” The MTR team needs to find out more about this process, including the agreed criteria. And the results of the selection process. For example, it would be interesting to know what proportion of beneficiaries are from illiterate families (about 16% in the population at large) and from families with high dependency ratios (few able-bodied workers and/or many dependents). Or from the 15% of families that reported only eating one meal a day, in the January 2008 baseline survey.

Opening up the Mid-Term Review process

Madeleine Bunting’s first of five articles on the components of the Katine project is very timely, and the intention of the series is spot on. They relate to the forthcoming review of the progress of the Katine project via a process known as a Mid-Term Review (MTR). Her hope is that her pieces “… will provide a useful rough draft with a few pointers for the professionals [i.e. the MTR team] who will follow, which is why I’ve listed my questions – please add any that you have which you would like the mid-term review and our independent evaluators to consider”

The MTR process

The MTR team will begin their work from next Monday 29th June, and are expected to produce a report by late July. This review is probably the most important review process during the whole of the life of the Katine project, more important even than the end-of-project evaluation. This is because the results could influence decisions taken over the next year, about (a) what the project should try to do in the remaining time left and (b) what should happen after the project officially ends in late 2010.

Part of the planning process for a mid-term review is the development of Terms of Reference (ToR) which will guide the work of the MTR team. They normally spell out the purpose of the review, the scope of activities to be reviewed and nature of the final products expected. Along with other information on the background of the project, and expectations about how the review will be undertaken. Normally ToRs are subject of negotiations between the stakeholders involved, including the donors (e.g. Guardian and Barclays), the implementing agency (e.g. AMREF) and local partners (e.g. government bodies and community groups in Katine). This process is already underway and will continue up to early next week when the MTR team visits Kampala and Soroti. As with the first two visits to the Katine project by myself (the external evaluator), the ToRs will be made public via the Guardian website. What is different this time is the opening up of the ToRs consultation process via the Guardian website, and Madeleine Bunting’s articles this week in particular.

The points raised by Madeleine’s posting on the health component

At the beginning of my work on the Katine project I proposed that we should use seven criteria for evaluating the project. Five of these are OECD Development Assistance Committee (DAC) criteria: relevance, effectiveness, efficiency, impact and sustainability. Two others I suggested to be included are: equity (fairness of process and outcome) and transparency.

One of the challenges for the MTR team is which of these criteria to apply to which project component, because applying all would mean a much more time consuming MTR process, which may not be affordable (in the wide sense of the word).

Re the distribution of anti-malaria bednets Madeleine Bunting asked “Has there been any coordination to avoid overlaps between AMREF and other donors on this issue?“, that is the distribution of bednets by different agencies to the same community. This question is concerned with efficiency. We could also ask about equity, who is actually using these within the receiving households…children or adults, mothers or fathers. The intention was, I think, that children should be using them, because they have less resistance to malaria. We could also ask about sustainability: Where will households get replacement nets in the future, after the project ends? My feeling is that the equity and sustainability questions are more important here than the question of efficiency.

A number of Madeleine’s questions relate to the criteria of relevance. Are the existing project interventions the most appropriate means of addressing the pressing health problems? Would some form of ambulance service help ensure women with birth complications were able to get to a doctor in time for a caesarian birth? Is “empowering” villagers in Katine with health information enough, when government services are so inadequate in the delivery of drugs and medical staff? Would some form of community health insurance be a useful means of topping up drug supplies or health centre staff pay.

Relevant – compared to what?

In order to answer these questions the MTR team will need to try to understand the project design – what were the objectives and what was the plan for achieving them. It could be unfair to assess a project in relation to an objective it never prioritised in the first place. Part of this process involves a reading the initial project documents and any official revisions to the project design thereafter.

The Katine project did develop a “Conceptual Framework” at the beginning of the project in September 2007, which spelled out what some people call a “Results Chain, showing how AMREF activities would contribute to the achievement of improvements in health, education, water, sanitation, health, livelihoods and governance.

In the health component the expected outcome was  “Increased community awareness of, access to and utilisation of health services in community and health facilities” This presumably covers both the services provided by the Village Health Teams, and the two tiers of government health centres with Katine (HC2 and HC4). This is quite ambitious given that there are only two AMREF staff working on the health component over a three year period. Therefore some prioritisation of the kinds of people who should be using health services more than before, and the kinds of health services they should ideally be using more than before, might be expected. Hopefully these prioritisations would be aligned of those with local government planning bodies and the Katine community (i.e. they would be seen as relevant). These issues are aspects of the health component that the MTR team could be looking at.