Reviewing Katine: What’s happening with governance?

In her recent posting on this topic, Madeleine Bunting said “I’ve listened to Joshua Kyallo, Amref Uganda’s director, explain how villagers can be empowered to demand better services from the government at district level. But there are plenty of questions in my mind as to how effective this will be in improving the operation of state services in Katine….The district budgets for health and education, for roads and water are desperately inadequate. It is not just the lack of demand for services that causes the state to be so ineffectual at village level here. I find the “rights-based” approach, based on developing in villagers a sense of entitlement to basic health and education, hard to understand“

Before asking whether the rights based approach is affective we need to ask if AMREF is in fact pursuing a rights based approach? As of August 2008 I could find no evidence of this on the ground, though the Country Director did affirm that AMREF supported a rights based approach. There are other interpretations of what empowerment is all about. The simplest and easiest to realise, is individual empowerment through the provision of practically useful information, for example, how to reduce the incidence of diarrhoea by maintaining clean water sources. This sort of empowerment was being addressed by the AMREF project in 2008. But it does not address wider issues such as the willingness and capacity of government to provide basic health services. Perhaps the project strategy has turned more in this direction since August 2008. The Mid-Term Review needs to look at this.

In the same article Madeleine Bunting also noted  “Several of the Amref staff spoke of how they had struggled with huge expectations of the project from Katine villagers. Is that the Guardian’s fault, I asked, with its headlines promising “transformation”? Perhaps partly, they agreed…I wondered how actively Amref has managed expectations and how widely it had communicated with villagers across this very scattered sub-county about what the project was going to do and what it was not going to do”

Expectations are usually about objectives and how they should be reached.  If they are diverse this suggests that communication and negotiation about project objectives may not have been as effective as they should have been. Madeleine Bunting’s article raises two possible causes: (a) insufficient communication with local communities by AMREF, and (b) the influence of the Guardian’s frequent visits to Katine communities. It could of course be both.

Another possibility is lack of clarity within AMREF itself, about what the project was trying to achieve. This was a concern I expressed in the first paragraph of my first visit report in January 2008. “The final objectives of the project may need clarification and agreement, by AMREF, its donors and local stakeholders. This agreement should be evident in a smaller set of indicators that show changes in people’s lives, reflect the impact of all five project components, and which can be easily be monitored by community groups.” At that stage the monitoring and evaluation framework had 35 indicators about expected changes in the lives of individuals and households and 60 indicators about the expected changes in the functioning of community groups and organisations. These are large numbers by the standards of most development projects. Later in 2008 the project staff in Katine made some efforts to prioritise these and focus on some key expected outcomes. One of the questions for the MTR should be looking at this year is the clarity of objectives within each of the components – within AMREF in the first instance, then amongst the wider group of stakeholders.

Reviewing Katine: What’s happening in livelihoods?

In her article on this topic, Madeleine Bunting commented “The livelihoods component of the Katine project has caused ongoing concern. Many times we have reported, and observers have commented, that not enough of the budget has been devoted to improving livelihoods. The vexed question over whether we should be giving what the villagers have repeatedly said they wanted – cattle – has repeatedly been raised.

Some of this disquiet seems to have been taken on by Amref and Farm-Africa because some interesting shifts in policy seem to have taken place. There is more emphasis on giving inputs – this is described as “hardware” in development lingo – such as seeds, tools, wheelbarrows and watering cans. The balance between hardware and “software”, or training, has been reversed in the livelihoods component so that more is being spent on the former.”

Later in the article Madeline rightly asks, in relation to the upcoming Mid-Term Review, “Can we have some explanation of why the approach on livelihoods shifted?” The reasons matter.  A subsequent comment from Farm Africa, who provide technical support to the Livelihoods teams, provided some clarification. They reported that “The software to hardware shift was in response to political pressure rather than internal reflection and learning. Politicians, especially at the sub-county level, were comparing KCPP with Government programmes that are hardware heavy, for instance the Northern Uganda Social Action Fund (NUSAF). The livelihoods component in particular was also being compared to the other components; Water, that sinks and rehabilitates boreholes; Health and Education, that build clinics and classrooms respectively, and being told to put more into hardware.”

While AMREF has obliged by providing more “hardware” such as seeds and tools, Farm Africa “are more convinced than ever that the approach of giving more seeds than trainings is not prudent. A significant number of the seedlings in the nursery had failed, mainly due to poor management, perhaps as a result of a lack of follow up training. This follow up training for the seedlings was not feasible due to the amount of the budget that was diverted to hardware. The backbone of our approach has always been to increase production by introducing new technologies and techniques rather than handing out seeds.”

This development is worrying for a number of reasons. Many development  NGOs, including probably Farm Africa and AMREF, would argue that advocacy is an important part of their work, and that they have competence in this area.  If so, why has Farm Africa caved in on an issue it believes in? Is it because they were unable to provide solid evidence from their projects elsewhere that training does make a difference? Or, was it simply the case that local authorities were impervious to the evidence that was presented, because they were trying to meet their constituents’ needs, regardless of their wisdom? Both prospects should cause some re-thinking. The same questions also apply to AMREF, who have been party to this change in direction. In the last (”Conclusions”) page of their 2007-2017 corporate strategy it is stated that “As we gather knowledge and evidence in our programme work and research, we will develop advocacy initiatives to influence policy makers to promote identified best practices.”

The issue of actual evidence is important. It is not self-evident that training will provide more sustainable development than material aid. Part of the “theory-of-change” behind the provision of training is the assumption that information about good agricultural practices (for example) will be passed on from one farmer to the next. Examples have been identified where this has happened. But comments by others (”Dr Jazz”)  underneath Madeleine Bunting’s article also highlight the fact that in some cases neighbours not only do not cooperate this way, but they actively sabotage each others efforts. Another commentator (”Ugandalife”) noted that that in their experience ” information is not generally shared easily. Information is considered an asset and therefore worth money. We have encountered this several times and it is hard to change this attitude.”

The idea that good practices will be imitated and reproduced by others is widespread amongst development projects, in just about all sectors e.g health, education, water and sanitation, livelihoods, etc. But just as common is the widespread failure by development agencies to invest any time and effort into systematically monitoring when and where (i.e. under what conditions) adoption by others actually takes place. This is worrying, because it suggests that many development agencies are isolated from important important strands of thinking that they could learn from. For example, the considerable body of literature that now exists on the “diffusion of innovation” Ironically, much of the early research in this field was done in relation to the adoption of agricultural research findings.

One of the implications of the concerns outlined above are that the MTR team should pay attention to: (a) where assumptions are being made about good practices being adopted by others, (b) what efforts are being put into monitoring how, when and where this is happening.

….

A second set of questions was asked at the end of Madeleine Bunting’s article: “Has some thought been given as to how to mitigate the tension over the fact that only a few people are benefiting from the free seeds and tools?
How significant are those tensions – are some people benefiting much more from this project than others? Could the project end up causing more disagreement and community fragmentation at a local level?“

Good question, worth trying to answer, under the ambit of equity concerns. If there are tensions there are two possible solutions, but only one of these has been discussed much so far. That is the try to extend coverage to all households. That is an expensive task and apparently beyond the current budget of the project. The other is targeting of households most in need. There has been little explicit discussion of this option, as far as I can see.

Farm Africa’s response to the Madeleine Bunting’s second set of questions was that “As far as we know, there is no significant tension within the community as a result of the intervention. One of the reasons why there is no tension, is because the beneficiaries we are working with were not hand picked by the organisations, but rather selected in a participatory and open process involving different stakeholders through an agreed criteria.” The MTR team needs to find out more about this process, including the agreed criteria. And the results of the selection process. For example, it would be interesting to know what proportion of beneficiaries are from illiterate families (about 16% in the population at large) and from families with high dependency ratios (few able-bodied workers and/or many dependents). Or from the 15% of families that reported only eating one meal a day, in the January 2008 baseline survey.

Opening up the Mid-Term Review process

Madeleine Bunting’s first of five articles on the components of the Katine project is very timely, and the intention of the series is spot on. They relate to the forthcoming review of the progress of the Katine project via a process known as a Mid-Term Review (MTR). Her hope is that her pieces “… will provide a useful rough draft with a few pointers for the professionals [i.e. the MTR team] who will follow, which is why I’ve listed my questions – please add any that you have which you would like the mid-term review and our independent evaluators to consider“

The MTR process

The MTR team will begin their work from next Monday 29th June, and are expected to produce a report by late July. This review is probably the most important review process during the whole of the life of the Katine project, more important even than the end-of-project evaluation. This is because the results could influence decisions taken over the next year, about (a) what the project should try to do in the remaining time left and (b) what should happen after the project officially ends in late 2010.

Part of the planning process for a mid-term review is the development of Terms of Reference (ToR) which will guide the work of the MTR team. They normally spell out the purpose of the review, the scope of activities to be reviewed and nature of the final products expected. Along with other information on the background of the project, and expectations about how the review will be undertaken. Normally ToRs are subject of negotiations between the stakeholders involved, including the donors (e.g. Guardian and Barclays), the implementing agency (e.g. AMREF) and local partners (e.g. government bodies and community groups in Katine). This process is already underway and will continue up to early next week when the MTR team visits Kampala and Soroti. As with the first two visits to the Katine project by myself (the external evaluator), the ToRs will be made public via the Guardian website. What is different this time is the opening up of the ToRs consultation process via the Guardian website, and Madeleine Bunting’s articles this week in particular.

The points raised by Madeleine’s posting on the health component

At the beginning of my work on the Katine project I proposed that we should use seven criteria for evaluating the project. Five of these are OECD Development Assistance Committee (DAC) criteria: relevance, effectiveness, efficiency, impact and sustainability. Two others I suggested to be included are: equity (fairness of process and outcome) and transparency.

One of the challenges for the MTR team is which of these criteria to apply to which project component, because applying all would mean a much more time consuming MTR process, which may not be affordable (in the wide sense of the word).

Re the distribution of anti-malaria bednets Madeleine Bunting asked “Has there been any coordination to avoid overlaps between AMREF and other donors on this issue?“, that is the distribution of bednets by different agencies to the same community. This question is concerned with efficiency. We could also ask about equity, who is actually using these within the receiving households…children or adults, mothers or fathers. The intention was, I think, that children should be using them, because they have less resistance to malaria. We could also ask about sustainability: Where will households get replacement nets in the future, after the project ends? My feeling is that the equity and sustainability questions are more important here than the question of efficiency.

A number of Madeleine’s questions relate to the criteria of relevance. Are the existing project interventions the most appropriate means of addressing the pressing health problems? Would some form of ambulance service help ensure women with birth complications were able to get to a doctor in time for a caesarian birth? Is “empowering” villagers in Katine with health information enough, when government services are so inadequate in the delivery of drugs and medical staff? Would some form of community health insurance be a useful means of topping up drug supplies or health centre staff pay.

Relevant – compared to what?

In order to answer these questions the MTR team will need to try to understand the project design – what were the objectives and what was the plan for achieving them. It could be unfair to assess a project in relation to an objective it never prioritised in the first place. Part of this process involves a reading the initial project documents and any official revisions to the project design thereafter.

The Katine project did develop a “Conceptual Framework” at the beginning of the project in September 2007, which spelled out what some people call a “Results Chain, showing how AMREF activities would contribute to the achievement of improvements in health, education, water, sanitation, health, livelihoods and governance.

In the health component the expected outcome was  “Increased community awareness of, access to and utilisation of health services in community and health facilities” This presumably covers both the services provided by the Village Health Teams, and the two tiers of government health centres with Katine (HC2 and HC4). This is quite ambitious given that there are only two AMREF staff working on the health component over a three year period. Therefore some prioritisation of the kinds of people who should be using health services more than before, and the kinds of health services they should ideally be using more than before, might be expected. Hopefully these prioritisations would be aligned of those with local government planning bodies and the Katine community (i.e. they would be seen as relevant). These issues are aspects of the health component that the MTR team could be looking at.

Money ‘wasted’ on water projects in Africa

This is the headline for a story on the Guardian katine blog, where Annie Kelly discusses a report that “criticises donors, governments and NGOs for installing boreholes and wells in rural Africa without ensuring their long-term sustainability”

Four comments have been made so far, including one by AMREF and myself.

AMREF pointed out that ” In 2002, AMREF helped communities in the Kathonzweni area to develop 50 shallow wells…By 2007 — five years later, and several years after the projects end — the communities had maintained the existing wells and built 20 new ones.”

This sounds very good, almost too good.

My comment was that “It would be useful if AMREF could provide (on this Guardian website) some information on (a) how they are monitoring the functioning of the wells they have helped to establish in Katine so far, and (b) the data that has been collected. Hopefully information is available on both the functioning of the wells and the functioning of the management committees responsible for each well.

There are problems with wells in Katine, like just about everywhere else. This is to be expected. Equipment does get damaged and worn. What matters more is how long those problems remain unresolved.

There will be a mid-term review of the Katine project in a few months time, which could look at the functioning of the wells then. But that would be a once-off inquiry and the risk is that (as elsewhere) extra efforts might be made to make sure things are working well at that time. Its the monitoring that matters.

regards, rick davies, external evaluator for Katine”

Are short-term development projects like Katine a good idea?

This was the headline on the Guardian katine website, on Tuesday 24th March 2009.  Eliza Anyangwe asked “Are short-term development projects like Katine a good idea? The Katine project in Uganda is scheduled to last for three years. But with such a short timescale can it deliver lasting change, or will it leave a bitter aftertaste?”

There were 12 comments including mine, which ran as follows:

“I have been working for aid organisations for about 28 years, and doing monitoring and evaluation of aid projects for about 19 years. Throughout this time most people I have spoken to who have been associated with aid projects have expressed the view that 3 years is not long enough to make a substantial and sustainable development impact. Especially if you are trying improve the way in which government services work. Many of the comments above seem to share this view.

In the case of the Katine project, AMREF have expressed the view that ” the Katine project really will contribute to lasting change in one of Ugandas poorest and most vulnerable regions”. The Guardian editor also seems to share this view. Are they right? Lets try to test those views.

I have already proposed to the Guardian and AMREF that at the end of the three year period AMREF, and its local partners, should make a number of testable predictions about what things will look like in Katine, in another three years time. That is six years after the project started and three years after it ended. Then the Guardian, or perhaps some more independent organisation (with no prior involvement in Katine), should fund what is called an “ex-post evaluation” team to come in and see how many of the predictions were successful, and how many have failed. And equally importantly, why some were successful and others not so. Then let those results be shared and discussed in Katine and elsewhere via the Guardian blog.

regards, rick davies, external evaluator for the Katine (KCPP) project

PS: The problem with this proposal is the uncertainty about who will be around, six years after the start of the Katine project. Will any of the Guardian journalists now engaged with the project be around and interested to see what has happened, or will they have moved on? Will any of the key people in AMREF still be around and interested, or will they have moved on? The big question seems to be how do aid (and other) organisations develop and maintain a longer term view on what they are doing at any one time.

Tiriri health centre: Questions still unanswered

Sarah Bosley’s post on the Guardian Katine website on December 2nd has raised some very important questions about why this health service is not delivering the basic services that it should be delivering: providing anti-malarial drugs, drugs for preventing mother-to-child transmission of HIV and other essential services. Instead there are empty shelves in the dispensary, the ambulance is elsewhere and there has not been an assigned doctor present for months. In her view the health centre “was in a worse state than when the Guardian/Amref project began, 12 months earlier.”

She then questions the role of AMREF, and whether there was not something the AMREF staff could have done to address these problems, in addition to their work of installing clean water supplies, supplying mosquito-nets and the training of village health teams and clinic staff.

Visitors to the website have been both sympathetic and critical of these comments. Some emphasised that these problems are not unique to Katine, or Uganda even, and that their systematic causes need to be addressed. Not the provision of stop gap supplies from other local NGOs or AMREF itself. But meanwhile the problem of lack of drugs and a doctor is a very real and pressing one for the people of Katine, regardless of what is happening elsewhere in Africa.

AMREF have responded, and there are now some interim supplies on their way. But these are recognised as stop gap solutions. They have commented:

Given the limited scope of our resources, AMREF must make strategic, sustainable investments in health. Until the national health system in Uganda is robust enough to reach and truly serve rural communities, we must build the system from the bottom up, empowering communities to take care of their own health.

This means focusing on long-term but essential activities such as training communities in disease prevention and early detection, improving health information systems so that a true picture of community health needs is understood by the district health authorities, and for better planning.

The other key aspect of our work is building the health system from the top down. We continue to advocate at the national, district, and local levels for improved drug delivery and improved health care delivery and resources (doctors, nurses, equipment, and so on). Changing government policy is slow, difficult, and necessary work. The alternative is a perpetually weak health care system based on short-term fixes.

To me the debate that Sarah has promoted raises important questions of what should donors, and other stakeholders, expect of AMREFs work in Katine on health issues at the end of three years. What sort of outcomes should be expected? This question is closely tied into a similar question, about what the Katine community should expect from the Tiriri health centre.

Simply listing the inputs that have been provided (e.g. number of people trained, meetings held with government officials etc.) will not be very satisfactory.  AMREF have already been very active, but to outsiders the health centre seems to be going backwards.

What would be much more useful are some realistic performance objectives for the health centre as a whole. These need to be ones that local officials agree with, and which local communities could understand and relate to. They would be largely about the availability and quality of services that should be provided. Once established, there then needs to be some transparent public reporting on how the Tiriri, and other local health centres, are performing on these criteria. If AMREF wants to empower people then providing public information is essential, both about what they can officially expect from their health centres and about what is actually being provided. Ultimately it is the people’s evaluation that matters and which should be heard.

One possible way is to move forward with a local adaptation of the Yellow Star program, already trialled elsewhere in Uganda. Fortunately, one member of the AMREF Katine staff have had some experience with this program. In doing so it would be important to see this not just as an improved health information system, but also as a means of community empowerment, to demand better services.

The evaluator evaluated

Here are…

AMREF REFLECTIONS ON 2^ND EXTERNAL EVALUATION OF KCPP

tabled at the Quarterly Governance meeting, Barclays, 1 Churchill Place, 15^th October 2.30-4.30 pm

along with my responses in red

Strengths

• Mobilised different stakeholders in AMREF to critically reflect on the progress, key priorities over next 12 months – key development questions on what, when and why
• Acknowledged the different pieces of work going on in Katine and the sheer volume of the work
• Participatory and inclusive – got insights from different stakeholders and beneficiary groups
• The process of responding to the report enabled the whole of AMREF to work together and crystallise and document our approach to development with a particular focus on KCPP
• Useful recommendation and learning on how to improve our M&E processes

Areas for improvement

• Very long/heavy report – really difficult to analyse especially when English is not a first language
  • There is a four page summary at the front, and a contents page, and separate annexes
  • The next report will aim to be 20 pages max (excluding Exec Summ and Annexes)
    • But this proposal is subject to discussions re plans for the AMREF Mid Term Revew (MTR)
• Intensive process/time consuming for different stakeholders in KCPP (PIT, country office team, AMREF HQ, AMREF UK, Farm Africa).
  • Following discussions with Guardian in August my aim now is to progressively reduce the frequency of visits and, where possible, to integrate these with AMREF review processes.
• The evaluation visits are too regular to enable staff on ground to learn, take action before participating in another evaluation (So far we have had 2 evaluations in 7 months). Most 3 year projects have 2 – midterm and end of project). KCPP will have 8 over the 3 years (6 + midterm and end of project).  This is not feasible if we want staff to value the evaluation  and deliver on the specific targets in their workplans
  • Following discussions with Guardian in August my aim now is to progressively reduce the frequency of visits and, where possible, to integrate these with AMREF review processes.
• Analysis of KCPP did not fully take into account the wider AMREF context, its policies and procedures)
  • Examples of important missing contextual information would be useful
  • Not sure how this gap could be addressed while also reducing the size of the report
• Analysis of KCPP did not fully take into account the wider development context of Katine, Soroti district and Uganda
  • Examples of important missing contextual information would be useful
  • Not sure how this gap could be addressed while also reducing the size of the report
• The evaluation focussed a lot on the processes. To the staff, who are working under difficult circumstances, highlighting both the interim tangible and intangible outcomes could have been more motivating
  • In the early stages of a project it is the work processes that are most visible and important, and outcomes tend to be less visible. Future visits should focus progressively more on outcomes and impact.
• In some instances the report  did not take into account key sensitivities about staff and impact on the relationships that AMREF has with different stakeholders
  • Details are needed here before I can respond. But on reflection it seemed as though  it was sections of AMREF who were the most sensitive, and sections of government who were quite robust (in wanting their views expressed).

Suggestions for the future

• Make the report shorter and in simple, user-friendly language (maximum 10 pages)
  • 10 pages is too short. 20 pages is more realistic. But final agreement here will depend on the ambit of the next visit, which is under discussion.
• Consider the implications of the evaluation recommendations on the capacity of the PIT and practical realities on the ground – deliver initial targets, address new issues within very tight deadlines
  • Noted: Recommendations should be limited in number
• The report has multiple target audiences.  It would be good to have a summary which is user friendly and targeting our stakeholders in Uganda, especially the district officials (i.e beyond the UK donors, UK public through the website).  Otherwise we stand to be accused of using extractive evaluation processes
  • Noted: There is a need for an Executive Summary that can fulfil this function (in addition to the existing list of Recommendations). But I am not sure if I should produce customised versions for different stakeholders. This might be better done by AMREF.
    
• Place analysis of findings and recommendations within the contextual realities of Katine sub-county, Soroti District and Uganda development context.  Co-evaluation with someone from Africa/Uganda would add a lot of value
  • Need explanation of what “contextual realities” are being referred to, and how they would be covered within a 10 page, or 20 page, report
  • Co-evaluation with “someone from Africa/Uganda” will happen if we integrate my next visit with the AMREF mid-term review
• Extend the time and if possible limit the number of evaluations bearing in mind that we shall also have midterm and end of project
  • Agreed, as noted above
• Next evaluation should focus more on interim outcomes and the foundations we are making for sustainability
  • Sounds appropriate at that point in time in the lifespan of the project. As part of this next review, I would like to see some systematic documentation of what has been done / happened in all the villages of Katine sub-country. A starting point would be a spreadsheet of villages x activities (including non-AMFREF activities).
• Clarify roles and scope of the evaluation – technical and programmatic; operations and management.
  • The Terms of Reference (ToRs) for each visits should be where this is done. Draft ToRs were shared with AMREF, Barclays and the Guardian for comment prior to each of the two visits to Katine so far. The same will be the case with visits in the future.
    

Learning from other projects?

In October I visited and reviewed the progress of UNICEF’s Women and Children’s Health Program in Papua (WCHPP), as part of a wider team including staff from UNICEF, AusAID, GTZ and specialists in maternal and neonatal health. This is the third of a series of such reviews conducted annually since 2006.

How is that relevant to Katine, a very different community in a very different country, assisted by a very different organisation, funded by a very different donor?

Well, it does seem that the two projects do share some common issues, which I will list here in the hope that someone might react to them (including the good staff of AMREF Soroti).

• Both projects involve the assisting agency working closely with government structures. In both project designs there is an important role for community mobilisation and empowerment, especially in relation to health matters. In Papua UNICEF wants to increase public awareness and use of maternal and child health services. It sees increasing demand for services as an essential complement to increasing the supply of such services. I think AMREF has a similar view in Katine. But the progress with this part of the project in Papua has been slow. UNICEF has led, but the government has not been that enthusiastic about this part of the program, compared to others. Questions were raised by the review team as to whether UNICEF should continue working in this area, especially given the fact that their staff were already overburdened with work. It might be better to find or encourage an independent NGO to work on the demand side, especially one that was more independent of the government.

• Both projects are engaged in many activities, so much so that it was not easy at first glance to appreciate the strategy guiding these activities. In the WCHPP there were two overlapping strategies. One could be described as humanitarian assistance. This involved funding support for staff training, repair of health facilities, procurement of supplies, etc. All of this was useful, and appreciated, but it was essentially gap filling, filling in where government funds should have been at work. Government funds were available, but they were being invested elsewhere, in new hospital buildings and in sectors other than health. The other strategy was the introduction of new and improved ways of doing things (computerised health information systems, partnerships between midwives and traditional birth attendants, etc). This is where UNICEF was arguably doing more development work, potentially adding value to the way in which government health systems were already working.
  • With both the humanitarian and development work there were some important limitations, that needed to be addressed. With the humanitarian work, there were no agreed performance targets or milestones that UNICEF would help the government reach, and then phase out its assistance. For example, at what point do you stop supporting the operations of a sub-district health centre, and move your assistance elsewhere? With the development work, the introduction of new or improved methods was not consistently associated with assessment of existing methods, and then a follow up comparison of the value of the new method. Nor was there any systematic packaging and promotion of the results of studies that were carried out. De factor “experiments” were being carried out, but they were not being systematically assessed and results publicised. This is despite the fact that the idea of development projects as “policy experiments” has been around for decades (See Rondinelli, 1993).

• There was a third feature that seemed to be shared by both projects, relating to decentralisation. Both Uganda and Indonesia have decentralised government administration, Indonesia probably the most radically. If ever there was a need for decentralised administration, it would be in Papua, which is culturally and geographically a long way from Jakarta. Yet, UNICEF’s response to providing assistance to its Papua office was remarkably centralised. A KAP (Knowledge Attitude Practice) survey was centrally commissioned for the whole of Indonesia as one entity, and this will be used to measure the impact of health education and awareness raising activities in Papua (and elsewhere), where the issues facing women and children are not carbon copies of those found elsewhere in Indonesia. A M&E framework was introduced, exactly as used by similar UNICEF projects elsewhere in Indonesia. Yet the capacities of local government, and associated health services in Papua bear no comparison with those found in Java. All this raises questions of how much can we expect a highly centralised aid agency to help a highly decentralised government?

• The fourth feature related to the behaviour of UNICEF and GTZ staff as members of the annual review teams I mentioned above. While they have now participated in reviews of each others projects for three years in a  row, it still surprises me how much they are still very wedded to their view that their own approach is the best possible under the circumstances, and how little acknowledgement (and even less follow up action) they have given to potentially positive features of each other’s projects. This experience resonates with what we have seen on the Guardian blog, in the way that AMREF has responded to comments by another NGO that it is possible to build schools significantly cheaper that the way AMREF has done. That is, with a little interest and enthusiasm to learn, and a considerable amount of defensiveness.

In my report on my second visit to Katine in August 2008 I raised a very similar set of issues

• The risks of overly centralised management of the project, within AMREF
• Questions about to what extent AMREF should do community empowerment, and whether it might be better done by another party more independent of government
• The need to set targets for what AMREF was trying to achieve, in consultation with its local partners
• The need to take the idea of generating development models seriously, by building operational research into all new activities that were trying to do things differently

The issues raised here also relate to my argument in a recent posting: How well is the KCPP doing? Compared to what? In that posting I argued that comparisons of AMREF’s performance with other projects could be as important as comparisons of with its own stated intentions and objectives

August 2008 Visit Report by the external evaluator

The August 2008 visit report is accessible here

• Main report
• The annexes

1. Summary of issues and main recommendations

1.      Recommendations about changes almost by definition do not identify the good that is present and which needs to be maintained. With this in mind the following points should be noted:

• The KCPP staff are committed to their work, and appear to work well together as a group. The management style is team oriented, which is essential if there is to be effective integration of the project components.
• There is a breadth of experience and expertise within the team e.g. the Health component Project Officer’s experience with the Yellow Star programme, and the Project Manager’s experience with training in rights- based approaches.
• There are good relationships with the district and sub-county authorities. Although the district authorities have made critical comparisons with NUSAF they are also openly appreciative of the good working relationship that they have with AMREF
• Both the material aid and training provided by the different components are appreciated by the communities concerned within Katine. There have been no substantial criticisms of their quality or relevance.

2.      This section summarises some major recommendations. The main text of the report lists specific recommendations by number, where the issues arise. There are 14 sets of recommendations (28 in all). And a further seven recommendations relating to the work of CARE and UWESO in Annex E. Please read them all. Search for: Recommendations*

3.      There have been significant delays in the project implementation, with the possibility of more than a quarter of the planned activities not being completed by the end of the first year (and possibly a third of the year 1 budget remaining unspent). However what is not necessary is a rush to complete activities by the end of September, simply for the sake of completing as much of the original plan on time as possible. What is important is attention to the cause of these delays.  The results of this examination should be evident in more timely implementation in the October 2008 – March 2009 period. See section 3.

PS1: AMREF have since pointed out that the project workplans outline “a sequence of activities as opposed to specific deadlines” (through the workplans do assign specific activities to specific time periods). In their view delays are to be expected because AMREF is involved in community development, and thus AMREF does not set the pace on its own. However, as noted later in this report, delays have also arisen from inefficiencies of process within AMREF. The question now is how big do the delays need to be before the project donors can legitimately express concern?

PS2: Re reviewing the causes of delays, AMREF’s view is that their existing planning and review processes are adequate

4.      The importance of particular project activities depends on their place within a wider strategic view of how the project will achieve its ends. A project with many activities but little strategy could easily be accused of gap-filling, and little more. This is a serious criticism that can be made of many aid projects, and one that AMREF needs to be able to respond to KCPP. The issue of KCPP’s strategy is discussed in section 8, on the project’s theory-of-change (ToC).  One high level statement of KCPP’s ToC is that the Development of Models + Creation of Demand = Development impact. This report argues (in 3.5 and that the project’s strategy for the encouragement of public demand (e.g. for access to quality health, education and water services) is underdeveloped and needs attention including greater use of communication channels outside of the government administration (e.g. via mass media and elected representatives at different levels).  The approach to the “development of models” also needs to be articulated more explicitly, because the options here are more complex than appear at first glance. In section 8.4 a number of different types of model creation have been identified. What would now be useful is clarification from AMREF on which of these are important in the KCPP. See section 8.

PS: AMREF’s view here is that its corporate (10 year) and Uganda country strategy provide a sufficient strategic guidance to the KCPP

5. Another high level statement of KCPP’s  ToC is that the Community Partnering + Capacity Building + Operations Research and Advocacy = Development Impact. There is ample evidence that Community Partnering and Capacity Building are already taking place. What is not yet present is sufficient investment in the analysis of the efficiency and effectiveness of the work being by the KCPP, which could then be used by staff and other stakeholders to advocate the use of certain development approaches (aka models). Doing so would be consistent with AMREF’s stated support for evidence-based advocacy (versus “assumption based advocacy”).  Progress with operations research and advocacy would be evident in (a) dedicated activities and budget allocations for this work within each of the components’ workplans, as well as for the M&E officer, and (b) documentation of progress made with advocacy efforts. See section 8.

PS: AMREF has commented that “AMREF is in the process of developing an advocacy strategy for the Katine project. The advocacy strategy will provide guidelines on how the project will gather the evidence and utilise it to influence policies and best practices at the local and national levels through the various foras that AMREF participates in such as the health policy action committee.”

6. Two important sources of risk for the project have been identified during this July visit. One is external, and one internal. The external risk is that the KCPP will face significant competition for attention from other development projects, such as NUSAF[1]. NUSAF is more decentralised in its management, and seen to be more cost-effective by Soroti district officials[2].  With the beginning of NUSAF Phase 2 District and sub-county officials may be less open to the adoption of KCPP models, unless the KCPP engages in serious evidence-based advocacy about the value of what it is doing.  There are also other development projects elsewhere in Uganda which will be competing for national and international attention along with the KCPP[3]. AMREF needs to identify how the KCPP approach is different and what difference these differences make (i.e. talking about its theory-of-change in practical terms) (See section 8.3). It needs to be able to show how it is adding value in ways that would not be possible if the same aid funds were to flow through different channels (either government or non-government).

PS: Re comparisons with other projects, AMREF have said “However we would like to desist from direct comparison with NUSAF because no formal comparable review has been done. ” Although AMREF may not wish to have comparisons made, other stakeholders are likely to persist in doing so, throughout the life of the project, and perhaps that reality needs to be acknowledged and responded to. This accommodation may have already begun, in that AMREF have also said that it “acknowledges issues raised by the district and sub-county authority and aims to take their comments on board. As part of the project steering committee agenda, AMREF will aim to do a comparative financial analysis of the project’s interventions vis a vis funds channelled through Katine sub-county and Soroti district.”

7. A significant internal risk to project success is the degree of centralised management within the Uganda country program.  Expenditures over UGX 100,000 (£33.50) .need to be approved by the Kampala office, whereas the Sub-county chief can authorise up to UGX 30 million. Quarterly workplans by all KCPP staff need to be approved by the Kampala office. Both processes take more time than more localised decision making. Strategies for communications and advocacy apparently need to be developed at the country level first. Country office staff represent the KCPP in both the monthly Project Management Committee and the quarterly Project Steering Committee meetings held in Katine and Soroti respectively. All these processes have implications for the efficiency, effectiveness and credibility of AMREF’s work in Katine.  It should be noted that concerns about the degree of centralisation have been made by members of the District Steering Committee, Sub-county Project Management Committee members, KCPP staff and Guardian visitors. There is, in my view, a strong argument in a favour of AMREF HQ undertaking a review of existing practices  Where it is decided that specific processes do need to remain centralised then there should be clear and transparent performance targets for their completion times. See section 3.7.

PS: AMREF’s response is that they use “standard management processes for Katine and all the other 23 projects in Uganda. These provide efficiency through scale as well as accountability that would be expensive and burdensome to replicate (sic) separately for every project. Moreover, these processes have been developed to support implementation across AMREF, based on our 51 years of work in Africa.” They also note that “The PMC and SC were established through participatory discussions with all the stakeholders and are subject to review according to local needs.

8. Within the project there are two areas of strategy which need particular attention. In the health component the investment in upgrading of health centre facilities and training of village health teams has involved a de facto assumption that the necessary drug supplies will be available, especially anti-malarials. Reports from the VHTs and Health Centre staff indicate otherwise, than in fact shortage of anti-malarials is a major problem. This is in the context where the AMREF’s 2007 needs assessment identified malaria as the single main cause of morbidity and mortality. This is clearly one area where a research and advocacy strategy needs to be developed as an integral part of the KCPP approach. See section 3.1.

PS: AMREF has commented that “Drug stock outs are a common problem in Uganda and AMREF has taken this as an advocacy issue.  In another project in Uganda, AMREF is undertaking an operations research into the issue of drug supply chain. The results of this research will be used as evidence for wider advocacy and policy influencing and will benefit KCCP. AMREF also has other malaria specific projects in Uganda that are addressing similar issues.”

9 The governance component also faces some challenges. On the ground in Katine the IEC[4] component has not yet started to address the issues of “the community’s rights to basic services…”, instead pre-existing conventional IEC materials are being re-used.. The piloting of community partnering “from a rights based approach” seems to be on hold. Discussions in Kampala suggested that this office favours a conservative interpretation of how issues of empowerment and rights can be dealt with. In Katine staff seem to have a more open view. AMREF should clarify its position by developing a strategy statement saying how its existing (or revised) governance activities in Katine will help achieve the governance objectives as stated in the project proposal. This strategy statement should be shared and discussed with its donors. See section 3.5.

PS: AMREF have confirmed that “Rights Based Approach (RBA) is prioritised as a core implementation strategy in the AMREF global and country strategic plans. The RBA model is being developed and rolled out across Africa since this is a relatively new area for AMREF.”…”Based on the agreed project design, the community empowerment component is broader than RBA. Progress on governance is based on the capacity of communities to engage with their decision and policy makers. AMREF’s role is to catalyze community action since taking on that role would disempower the community”.

10 Both of the two problems identified above may be reflections of what could be called a strategy gap in the current design of the KCPP. That is the absence of documented thinking about how activities in each component should connect causally to the project objectives of that component. The recent introduction of a new planning format for proposed activities may help fill this gap, if it asks the right questions. If not, they could be a form of micro-management.

11.  The project has its own objectives and indicators, and data on some of these indicators is now being collected, in addition to the work done on the baseline survey. However further steps are needed. The definition of specific levels of achievement on the various project indicators is essential, if the success of the project is to be evaluable. In the longer term, the sustainability of all project aided activities will be dependent of capable monitoring of those activities by the relevant community groups and government structures (e.g. PTAs, SMCs, PDCs, VSLAs, etc). This is more likely to happen if KCPP objectives (especially the associated targets) are closely aligned with those of its local government and community partners. Although some progress is being made with identifying targeted levels of achievement on indicators of shared concern, this work does need to be finalised soon, preferably by the end of this year. See section 4.

PS: AMREF believes the existing comprehensive M & E framework for the KCCP is adequate, but that “it will be updated periodically on the basis based on new project information and in response to community priorities”.

12. Regarding transparency, AMREF has been open in its dealings with local government, though local government would like to see more information on costs of activities. District and sub-county authorities have in turn expressed willingness to share information on their budgets, an offer worth pursuing. Although the importance of transparency seems to be well recognised in the KCPP office more attention does need to be given to improving transparency with the public at large in Katine, both about AMREF activities and the work of its partners. Simple measures could be useful, such as robust public notice boards at schools, health centres, markets and sub-county offices showing meeting minutes, survey results, annotated maps, and AMREF’s own plans. The use of talk-back radio has also been suggested[5]. Internationally, progress with greater transparency via the AMREF websites has been slow, but with a major improvement during my visit, hopefully now to be sustained. See section 7.

13. Transparency will not be painless.  It will almost inevitably bring with it some critical judgements of how well the KCPP is doing, when compared to others. For example, decentralised projects like NUSAF, and activities by smaller NGOs with lower overheads. Ideally AMREF will respond positively to these challenges, and provide in turn a demanding comparator for other development projects. This expectation is in effect the external evaluator’s theory-of-change: about the relationship between transparency, the value of comparisons and increased aid effectiveness.

PS: AMREF has replied that it is “confident that our development model is rigorous enough to enable us to engage in informed development debates with other likeminded organisations.”

Footnotes:

[1] Northern Uganda Social Action Fund. NUSAF Phase 2 is about to start in January 2009

[2] But not without its own problems e.g. corruption. See Sh2.5b NUSAF funds missing New Vision, Thursday, 10th July, 2008 http://www.newvision.co.ug/D/8/12/638475

[3] Including those announced by President Museveni, in his recent visit to Soroti.

[4] Information, Education and Communication

[5] There are three radio stations in the area, and many households have radios.

How well is the KCPP doing? Compared to what?

One conventional approach to the evaluation of aid projects is to identify the project’s objectives, and then see what has been achieved on the ground and how it compares to the stated objectives on paper.

This approach has its own difficulties. For example, the objectives might not be stated very clearly. Or if there are clear objectives, there may not be any useful indicators of progress with these objectives. Or if there are useful indicators, there may not be any targeted level of achievement on those indicators. I have already discussed these types of problems in the KCPP, both directly with the AMREF KCPP staff, and via this blog. (e.g. “Appropriate Goals“)

But this approach may be irrelevant to many people. What may be more important is how the KCPP seems to be progressing, when compared to other development projects that they know of.

Yesterday’s article on the Guardian Katine website (by Richard Kavuma) on President Museveni’s visit to Soroti mentioned a number of different development initiatives that have been promised, and in some cases already happened. Richard’s article mentions Northern Uganda Social Action Fund (NUSAF), the National Agricultural Advisory Services (NAADS), Savings and Credit Cooperative Societies (SACCOs), and now “model farms”

“Presenting highlights of his anti-poverty campaign, Museveni announced a fund of 98 billion Ugandan shillings ($60 million) to support six homesteads in each of Uganda’s 5,000 parishes to set up model farms. The idea is that the remaining villages will learn from the model homesteads how poverty can be overcome and will – hopefully – start their own journeys.” **

The KCPP project is also about developing convincing models of good practice (re health, education, water, sanitation, livelihoods, governance), that government, local communities and individual households will hopefully adopt.

However, the NUSAF (Phase 2), the SACCOs, and the model farms will all be in competition with the work AMREF is doing via the KCPP. In competition for the attention of local elected leaders and administrators, who need to decide how to best use their scare resources (both their own time and the budgets they are responsible for). And in competition for the attention of households as well.

Competition could be useful. But it will depend on who sets the rules, about what kinds of development achievements are really important. That will be an important arena of competition itself. AMREF could well argue that transparency (about how a project is managed) is most important of all, because without open access to information how can people really tell what is going on? Have the billions of shillings really been spent as intended?

AMREF could also argue that sustainability is very important, important enough to sometimes justify its own development costs being higher than others (e.g. for installing a borehole and establishing a working water source committee).

But to do this it will need some hard facts, not just optimistic assumptions. For example about the percentage of KCPP boreholes still functioning after two years, compared to those established by NUSAF.

And it will also need a strategy, a sense of what facts are the most important, amongst the many that could be collected. What are the most important differences between how AMREF is doing things in the KCPP, and how others (government and non-government) are doing them? And what difference is it expected that these differences make to people’s lives?

These questions can be asked at two levels: (a) about AMREF’s work generally, and (b) much more specifically, about individual project activities. Such as providing furniture and books for schools, or training Village Health Teams, or working with farmers groups to improve their incomes. Facts and findings about concrete activities like these are probably going to be of most interest to the people of Soroti district.

If it can successfully compete for people’s attention, then the impact of the KCPP will have a better chance of spreading, beyond the currently limited number of schools with new equipment or tthe few villages with new boreholes. It will also be in a better position to convince its current (and prospective) donors that there is good reason to fund organisations like AMREF, rather than channeling money directly to poor communities, or to local government, or to other NGOs.  It will be able to show how its particular way of working adds value in a way that justifies the cost involved.

—————————

Footnote**: $60 million = $2,000 per model farm. There could be 30 model farms set up in Katine sub-county, which has five parishes.