Coordinating aid to Katine

Richard Kavuma has writen an interesting article about the role of insecticide treated bednets in preventing malaria, especially amongst young children in Katine (See Net gains in preventing malaria). His article notes that while bednets have reduced malaria there are other important prevention measures that are not being addressed, most notably ensuring a continually available stock of anti-malaria drugs in the local clinics.

I was very interested to read that there was another agency that was providing bednets in Katine, and in other areas of Soroti District. In fact “Since August 2007, … PSI Malaria Control, which supports governments with malaria prevention programmes, has given out 89,660 nets to households in the district’s 17 sub-counties – including Katine.” This compares to the 5,478 bednets given out by AMREF in Katine.

Amongst international aid agencies, especially bilateral (government) and multilateral (inter-govermental) there has been a strong movement over the last decade towards greater coordination and harmonisation of their aid efforts. These intentions were documented in the 2005 Parish Declaration, and have been systematically monitored since then.

Richard’s story raises questions in my kind, and perhaps others, about the reasons for AMREF being involved in bednet distribution, when there another agency present in the district doing the same thing, but on a much larger scale, and possibly with more specialist knowledge in this area. Did PSI start their distributions after those made by AMREF? Could AMREF’s resources (funding and staff) now be better directed elsewhere? Or are there good reasons for AMREF to continue providing this additional channel for bednet distribution?

The reference in Richard’s story to families buying ordinary (untreated) bednets raises a related set of issues. What has happened to the private sector suppliers of  bednets to Katine (and Soroti as a whole), since the distributions of free treated bednets by PSI Malaria Control and AMREF?  Have their sales collapsed, or have they expanded? (and does AMREF know what is happening here?)  It seems highly likely that the only way treated bednets will continue to be made available to families in Katine after the current project comes to an end will be through the private sector suppliers, if theyare still in this business. Is AMREF thinking in these terms?

Mobile phones, the internet and innovation in Katine

One of the often claimed advantages of providing aid through NGOs is their ability to be more innovative, especially when compared to government structures. In practice innovations by NGOs are not as common as one might expect, at least in my experience.

One of the more interesting features of the KCPP is the setting up of a “community media centre” which has four computers connected to the internet. This is in a large room in the same building as the AMREF office, which is about a hundred metres from the Katine sub-county local government offices, and just off the main road that runs north-south through the centre of the county. In the last progress report provided by AMREF, earlier this year, it was reported that

“More than 100 community members have received basic training in computer skills so they can write simple stories. There is a committee who oversee the resource centre from the community. 30 people were specifically selected and trained as Trainer of Trainers. About 40 school children also received similar training from both primary and secondary schools. Over 25 community members visit the media centre daily to seek knowledge and learn. Some read and contribute to blogs on the project website. A media centre committee has been formed by the community to be part of its running”

AMREF’s Mid-Term Review subsequently reported in August that:

Five centre users were interviewed together with Joseph Malinga the Community Media Facilitator.  More than 100 people have been trained in basic computer skills.  All five of the users have posted blogs on the Guardian website, they are know as ‘Katine Informer’.  All five use email, four email outside Katine and two of those email outside Uganda. The users considered that the service would be sustainable; they saw the possibility of an internet café provider from Soroti running the centre although they were fearful of the cost as twenty minutes of internet time in Soroti costs the equivalent of a meal.  Two on-line chat sessions have been held, linking school children in Uganda and the UK.  Some equity issues exist; people with poor, or no, English are disadvantaged.

What is the objective being pursued, and how will we know when it is being achieved? One seems to be to provide opportunities for Katine residents (other than AMREF staff) to participate in the discussions taking place on the Guardian website. This has happened, on a modest scale. The other may be to provide internet café facilities which are more accessible, in terms of location and cost. This is happening, though as noted above, there are big questions about sustainability that need to be addressed. In the short term it will be useful if AMREF could provide some detailed statistics on usage of the internet facility since it started, including total number of users (broken down by gender at least) and some form of frequency distribution showing numbers of visitors x number of visits by them. Are a small number of people using the facility frequently and the rest only once or twice, or is use more evenly spread?

Is this all there is? Or could more be done with this facility? AMREF reported on the media centre under the heading of “Empowering communities with information, knowledge and exchange of ideas”. If that is the wider objective, then how could it be achieved?

The Economist of 26^th September has an interesting article on telecoms in emerging markets, that if read could prompt some more imaginative thinking by AMREF about the possibilites. Uganda is mentioned a number of times, as a place where there is rapidly expanding mobile phone coverage, and where there are some interesting innovations in the types of services being provided via mobile phones. Some of these are being supported by Google, the Grameen Foundation and the mobile phone networks themselves. They include provision of agricultural information via text messages, as well as the better known money transfer services.

Despite the poverty of many households in Katine, mobile phones are not a rarity. There are enough around to make it worthwhile for at least one person in Katine’s Tuesday market to earn income by charging people’s mobile phones using a truck battery (See Guardian article on this). On my first visit to Katine I was surprised to find that mobile phone reception in the sub-county is better than in the part of outer Melbourne where my mother lives. Katine’s open and flat landscape helps.

AMREF’s  baseline survey of Katine, carried out in late 2007, did not ask about mobile phone use, nor did the more recent survey contracted by CARE in early 2009. The focus of both of those surveys , like many development project baseline surveys, was on deprivation and constraint, rather than opportunities. Perhaps it is now time to investigate the opportunities people see around them, and work on how access to those opportunities could be improved. Time could be spent finding out more about the access to mobile phones (owned and borrowed) and the purposes they are used for. The new road is also likely to be seen as an opportunity by many people in Katine, and not just a risk (e.g. more vehicle accidents and more HIV cases).

The Economist article highlights a number of possibilities that could be explored in Katine. One is the role of intermediaries. In Bangladesh this first took the form of a woman (a Grameen group member) in each village selling mobile phone use (for incoming and outgoing calls)  to others in the same village on a call by call basis, while buying the phone from Grameen on an instalment basis. Would this be possible in Katine? Is it already happening on an informal basis? More recently Grameen is looking at the possible role of “information intermediaries”, who can help others get access to particular mobile phone based information services. In Katine that role could be expanded to helping others access useful information via the internet, without having to use the Community Media Centre computers themselves (remembering that a high percentage of Katine residents are not literate). These “infomediaries” might the ones would could afford to pay for internet access, and be willing to do so.

More information is needed, about the use of mobile phones and the use of the internet in Katine. With the current users of the internet in the Community Media Centre what services are they using? Email, text chat or skype for communications? Visiting social networking sites or using Google for information searches? For information about Uganda matters or about the rest of the world?  Looking forward it would then be useful to try to idnetify Uganda -specific information is available on the internet, which might be of most interest to Katine residents. Provided by the government and others.

Re the use of mobile phones I suspect there are many innovatory uses already being explored by people in Katine, but not yet widely appreciated by aid agencies and donors. For example, to what extent are government officials already using mobile phones to submit verbal reports, or at least texted data on key indicators? How often are meetings being convened through networks of phone calls? Are any farmers receiving commodity price information through their telephone contacts? Further afield is the whole area of money transfer. Are households receiving money transfers from Kampala via mobile phone, and at what cost?

According to the Economist a recent study found that “adding an extra 10 mobile phones per 100 people in a typical developing country boost growth in GDP by 0.8 percentage points” Would connecting up a local population of mobile phone users with skilled internet users lift this figure further? Or should any investments that can be made focus on exploiting the uses of the most widely used technology, i.e. mobile phones rather than the internet?

---

Reviewing Katine: What’s happening with governance?

In her recent posting on this topic, Madeleine Bunting said “I’ve listened to Joshua Kyallo, Amref Uganda’s director, explain how villagers can be empowered to demand better services from the government at district level. But there are plenty of questions in my mind as to how effective this will be in improving the operation of state services in Katine….The district budgets for health and education, for roads and water are desperately inadequate. It is not just the lack of demand for services that causes the state to be so ineffectual at village level here. I find the “rights-based” approach, based on developing in villagers a sense of entitlement to basic health and education, hard to understand“

Before asking whether the rights based approach is affective we need to ask if AMREF is in fact pursuing a rights based approach? As of August 2008 I could find no evidence of this on the ground, though the Country Director did affirm that AMREF supported a rights based approach. There are other interpretations of what empowerment is all about. The simplest and easiest to realise, is individual empowerment through the provision of practically useful information, for example, how to reduce the incidence of diarrhoea by maintaining clean water sources. This sort of empowerment was being addressed by the AMREF project in 2008. But it does not address wider issues such as the willingness and capacity of government to provide basic health services. Perhaps the project strategy has turned more in this direction since August 2008. The Mid-Term Review needs to look at this.

In the same article Madeleine Bunting also noted  “Several of the Amref staff spoke of how they had struggled with huge expectations of the project from Katine villagers. Is that the Guardian’s fault, I asked, with its headlines promising “transformation”? Perhaps partly, they agreed…I wondered how actively Amref has managed expectations and how widely it had communicated with villagers across this very scattered sub-county about what the project was going to do and what it was not going to do”

Expectations are usually about objectives and how they should be reached.  If they are diverse this suggests that communication and negotiation about project objectives may not have been as effective as they should have been. Madeleine Bunting’s article raises two possible causes: (a) insufficient communication with local communities by AMREF, and (b) the influence of the Guardian’s frequent visits to Katine communities. It could of course be both.

Another possibility is lack of clarity within AMREF itself, about what the project was trying to achieve. This was a concern I expressed in the first paragraph of my first visit report in January 2008. “The final objectives of the project may need clarification and agreement, by AMREF, its donors and local stakeholders. This agreement should be evident in a smaller set of indicators that show changes in people’s lives, reflect the impact of all five project components, and which can be easily be monitored by community groups.” At that stage the monitoring and evaluation framework had 35 indicators about expected changes in the lives of individuals and households and 60 indicators about the expected changes in the functioning of community groups and organisations. These are large numbers by the standards of most development projects. Later in 2008 the project staff in Katine made some efforts to prioritise these and focus on some key expected outcomes. One of the questions for the MTR should be looking at this year is the clarity of objectives within each of the components – within AMREF in the first instance, then amongst the wider group of stakeholders.

Reviewing Katine: What’s happening in livelihoods?

In her article on this topic, Madeleine Bunting commented “The livelihoods component of the Katine project has caused ongoing concern. Many times we have reported, and observers have commented, that not enough of the budget has been devoted to improving livelihoods. The vexed question over whether we should be giving what the villagers have repeatedly said they wanted – cattle – has repeatedly been raised.

Some of this disquiet seems to have been taken on by Amref and Farm-Africa because some interesting shifts in policy seem to have taken place. There is more emphasis on giving inputs – this is described as “hardware” in development lingo – such as seeds, tools, wheelbarrows and watering cans. The balance between hardware and “software”, or training, has been reversed in the livelihoods component so that more is being spent on the former.”

Later in the article Madeline rightly asks, in relation to the upcoming Mid-Term Review, “Can we have some explanation of why the approach on livelihoods shifted?” The reasons matter.  A subsequent comment from Farm Africa, who provide technical support to the Livelihoods teams, provided some clarification. They reported that “The software to hardware shift was in response to political pressure rather than internal reflection and learning. Politicians, especially at the sub-county level, were comparing KCPP with Government programmes that are hardware heavy, for instance the Northern Uganda Social Action Fund (NUSAF). The livelihoods component in particular was also being compared to the other components; Water, that sinks and rehabilitates boreholes; Health and Education, that build clinics and classrooms respectively, and being told to put more into hardware.”

While AMREF has obliged by providing more “hardware” such as seeds and tools, Farm Africa “are more convinced than ever that the approach of giving more seeds than trainings is not prudent. A significant number of the seedlings in the nursery had failed, mainly due to poor management, perhaps as a result of a lack of follow up training. This follow up training for the seedlings was not feasible due to the amount of the budget that was diverted to hardware. The backbone of our approach has always been to increase production by introducing new technologies and techniques rather than handing out seeds.”

This development is worrying for a number of reasons. Many development  NGOs, including probably Farm Africa and AMREF, would argue that advocacy is an important part of their work, and that they have competence in this area.  If so, why has Farm Africa caved in on an issue it believes in? Is it because they were unable to provide solid evidence from their projects elsewhere that training does make a difference? Or, was it simply the case that local authorities were impervious to the evidence that was presented, because they were trying to meet their constituents’ needs, regardless of their wisdom? Both prospects should cause some re-thinking. The same questions also apply to AMREF, who have been party to this change in direction. In the last (“Conclusions”) page of their 2007-2017 corporate strategy it is stated that “As we gather knowledge and evidence in our programme work and research, we will develop advocacy initiatives to influence policy makers to promote identified best practices.”

The issue of actual evidence is important. It is not self-evident that training will provide more sustainable development than material aid. Part of the “theory-of-change” behind the provision of training is the assumption that information about good agricultural practices (for example) will be passed on from one farmer to the next. Examples have been identified where this has happened. But comments by others (“Dr Jazz”)  underneath Madeleine Bunting’s article also highlight the fact that in some cases neighbours not only do not cooperate this way, but they actively sabotage each others efforts. Another commentator (“Ugandalife”) noted that that in their experience ” information is not generally shared easily. Information is considered an asset and therefore worth money. We have encountered this several times and it is hard to change this attitude.”

The idea that good practices will be imitated and reproduced by others is widespread amongst development projects, in just about all sectors e.g health, education, water and sanitation, livelihoods, etc. But just as common is the widespread failure by development agencies to invest any time and effort into systematically monitoring when and where (i.e. under what conditions) adoption by others actually takes place. This is worrying, because it suggests that many development agencies are isolated from important important strands of thinking that they could learn from. For example, the considerable body of literature that now exists on the “diffusion of innovation” Ironically, much of the early research in this field was done in relation to the adoption of agricultural research findings.

One of the implications of the concerns outlined above are that the MTR team should pay attention to: (a) where assumptions are being made about good practices being adopted by others, (b) what efforts are being put into monitoring how, when and where this is happening.

….

A second set of questions was asked at the end of Madeleine Bunting’s article: “Has some thought been given as to how to mitigate the tension over the fact that only a few people are benefiting from the free seeds and tools?
How significant are those tensions – are some people benefiting much more from this project than others? Could the project end up causing more disagreement and community fragmentation at a local level?“

Good question, worth trying to answer, under the ambit of equity concerns. If there are tensions there are two possible solutions, but only one of these has been discussed much so far. That is the try to extend coverage to all households. That is an expensive task and apparently beyond the current budget of the project. The other is targeting of households most in need. There has been little explicit discussion of this option, as far as I can see.

Farm Africa’s response to the Madeleine Bunting’s second set of questions was that “As far as we know, there is no significant tension within the community as a result of the intervention. One of the reasons why there is no tension, is because the beneficiaries we are working with were not hand picked by the organisations, but rather selected in a participatory and open process involving different stakeholders through an agreed criteria.” The MTR team needs to find out more about this process, including the agreed criteria. And the results of the selection process. For example, it would be interesting to know what proportion of beneficiaries are from illiterate families (about 16% in the population at large) and from families with high dependency ratios (few able-bodied workers and/or many dependents). Or from the 15% of families that reported only eating one meal a day, in the January 2008 baseline survey.

Tiriri health centre: Questions still unanswered

Sarah Bosley’s post on the Guardian Katine website on December 2nd has raised some very important questions about why this health service is not delivering the basic services that it should be delivering: providing anti-malarial drugs, drugs for preventing mother-to-child transmission of HIV and other essential services. Instead there are empty shelves in the dispensary, the ambulance is elsewhere and there has not been an assigned doctor present for months. In her view the health centre “was in a worse state than when the Guardian/Amref project began, 12 months earlier.”

She then questions the role of AMREF, and whether there was not something the AMREF staff could have done to address these problems, in addition to their work of installing clean water supplies, supplying mosquito-nets and the training of village health teams and clinic staff.

Visitors to the website have been both sympathetic and critical of these comments. Some emphasised that these problems are not unique to Katine, or Uganda even, and that their systematic causes need to be addressed. Not the provision of stop gap supplies from other local NGOs or AMREF itself. But meanwhile the problem of lack of drugs and a doctor is a very real and pressing one for the people of Katine, regardless of what is happening elsewhere in Africa.

AMREF have responded, and there are now some interim supplies on their way. But these are recognised as stop gap solutions. They have commented:

Given the limited scope of our resources, AMREF must make strategic, sustainable investments in health. Until the national health system in Uganda is robust enough to reach and truly serve rural communities, we must build the system from the bottom up, empowering communities to take care of their own health.

This means focusing on long-term but essential activities such as training communities in disease prevention and early detection, improving health information systems so that a true picture of community health needs is understood by the district health authorities, and for better planning.

The other key aspect of our work is building the health system from the top down. We continue to advocate at the national, district, and local levels for improved drug delivery and improved health care delivery and resources (doctors, nurses, equipment, and so on). Changing government policy is slow, difficult, and necessary work. The alternative is a perpetually weak health care system based on short-term fixes.

To me the debate that Sarah has promoted raises important questions of what should donors, and other stakeholders, expect of AMREFs work in Katine on health issues at the end of three years. What sort of outcomes should be expected? This question is closely tied into a similar question, about what the Katine community should expect from the Tiriri health centre.

Simply listing the inputs that have been provided (e.g. number of people trained, meetings held with government officials etc.) will not be very satisfactory.  AMREF have already been very active, but to outsiders the health centre seems to be going backwards.

What would be much more useful are some realistic performance objectives for the health centre as a whole. These need to be ones that local officials agree with, and which local communities could understand and relate to. They would be largely about the availability and quality of services that should be provided. Once established, there then needs to be some transparent public reporting on how the Tiriri, and other local health centres, are performing on these criteria. If AMREF wants to empower people then providing public information is essential, both about what they can officially expect from their health centres and about what is actually being provided. Ultimately it is the people’s evaluation that matters and which should be heard.

One possible way is to move forward with a local adaptation of the Yellow Star program, already trialled elsewhere in Uganda. Fortunately, one member of the AMREF Katine staff have had some experience with this program. In doing so it would be important to see this not just as an improved health information system, but also as a means of community empowerment, to demand better services.

The evaluator evaluated

Here are…

AMREF REFLECTIONS ON 2^ND EXTERNAL EVALUATION OF KCPP

tabled at the Quarterly Governance meeting, Barclays, 1 Churchill Place, 15^th October 2.30-4.30 pm

along with my responses in red

Strengths

• Mobilised different stakeholders in AMREF to critically reflect on the progress, key priorities over next 12 months – key development questions on what, when and why
• Acknowledged the different pieces of work going on in Katine and the sheer volume of the work
• Participatory and inclusive – got insights from different stakeholders and beneficiary groups
• The process of responding to the report enabled the whole of AMREF to work together and crystallise and document our approach to development with a particular focus on KCPP
• Useful recommendation and learning on how to improve our M&E processes

Areas for improvement

• Very long/heavy report – really difficult to analyse especially when English is not a first language
  • There is a four page summary at the front, and a contents page, and separate annexes
  • The next report will aim to be 20 pages max (excluding Exec Summ and Annexes)
    • But this proposal is subject to discussions re plans for the AMREF Mid Term Revew (MTR)
• Intensive process/time consuming for different stakeholders in KCPP (PIT, country office team, AMREF HQ, AMREF UK, Farm Africa).
  • Following discussions with Guardian in August my aim now is to progressively reduce the frequency of visits and, where possible, to integrate these with AMREF review processes.
• The evaluation visits are too regular to enable staff on ground to learn, take action before participating in another evaluation (So far we have had 2 evaluations in 7 months). Most 3 year projects have 2 – midterm and end of project). KCPP will have 8 over the 3 years (6 + midterm and end of project).  This is not feasible if we want staff to value the evaluation  and deliver on the specific targets in their workplans
  • Following discussions with Guardian in August my aim now is to progressively reduce the frequency of visits and, where possible, to integrate these with AMREF review processes.
• Analysis of KCPP did not fully take into account the wider AMREF context, its policies and procedures)
  • Examples of important missing contextual information would be useful
  • Not sure how this gap could be addressed while also reducing the size of the report
• Analysis of KCPP did not fully take into account the wider development context of Katine, Soroti district and Uganda
  • Examples of important missing contextual information would be useful
  • Not sure how this gap could be addressed while also reducing the size of the report
• The evaluation focussed a lot on the processes. To the staff, who are working under difficult circumstances, highlighting both the interim tangible and intangible outcomes could have been more motivating
  • In the early stages of a project it is the work processes that are most visible and important, and outcomes tend to be less visible. Future visits should focus progressively more on outcomes and impact.
• In some instances the report  did not take into account key sensitivities about staff and impact on the relationships that AMREF has with different stakeholders
  • Details are needed here before I can respond. But on reflection it seemed as though  it was sections of AMREF who were the most sensitive, and sections of government who were quite robust (in wanting their views expressed).

Suggestions for the future

• Make the report shorter and in simple, user-friendly language (maximum 10 pages)
  • 10 pages is too short. 20 pages is more realistic. But final agreement here will depend on the ambit of the next visit, which is under discussion.
• Consider the implications of the evaluation recommendations on the capacity of the PIT and practical realities on the ground – deliver initial targets, address new issues within very tight deadlines
  • Noted: Recommendations should be limited in number
• The report has multiple target audiences.  It would be good to have a summary which is user friendly and targeting our stakeholders in Uganda, especially the district officials (i.e beyond the UK donors, UK public through the website).  Otherwise we stand to be accused of using extractive evaluation processes
  • Noted: There is a need for an Executive Summary that can fulfil this function (in addition to the existing list of Recommendations). But I am not sure if I should produce customised versions for different stakeholders. This might be better done by AMREF.
    
• Place analysis of findings and recommendations within the contextual realities of Katine sub-county, Soroti District and Uganda development context.  Co-evaluation with someone from Africa/Uganda would add a lot of value
  • Need explanation of what “contextual realities” are being referred to, and how they would be covered within a 10 page, or 20 page, report
  • Co-evaluation with “someone from Africa/Uganda” will happen if we integrate my next visit with the AMREF mid-term review
• Extend the time and if possible limit the number of evaluations bearing in mind that we shall also have midterm and end of project
  • Agreed, as noted above
• Next evaluation should focus more on interim outcomes and the foundations we are making for sustainability
  • Sounds appropriate at that point in time in the lifespan of the project. As part of this next review, I would like to see some systematic documentation of what has been done / happened in all the villages of Katine sub-country. A starting point would be a spreadsheet of villages x activities (including non-AMFREF activities).
• Clarify roles and scope of the evaluation – technical and programmatic; operations and management.
  • The Terms of Reference (ToRs) for each visits should be where this is done. Draft ToRs were shared with AMREF, Barclays and the Guardian for comment prior to each of the two visits to Katine so far. The same will be the case with visits in the future.
    

Learning from other projects?

In October I visited and reviewed the progress of UNICEF’s Women and Children’s Health Program in Papua (WCHPP), as part of a wider team including staff from UNICEF, AusAID, GTZ and specialists in maternal and neonatal health. This is the third of a series of such reviews conducted annually since 2006.

How is that relevant to Katine, a very different community in a very different country, assisted by a very different organisation, funded by a very different donor?

Well, it does seem that the two projects do share some common issues, which I will list here in the hope that someone might react to them (including the good staff of AMREF Soroti).

• Both projects involve the assisting agency working closely with government structures. In both project designs there is an important role for community mobilisation and empowerment, especially in relation to health matters. In Papua UNICEF wants to increase public awareness and use of maternal and child health services. It sees increasing demand for services as an essential complement to increasing the supply of such services. I think AMREF has a similar view in Katine. But the progress with this part of the project in Papua has been slow. UNICEF has led, but the government has not been that enthusiastic about this part of the program, compared to others. Questions were raised by the review team as to whether UNICEF should continue working in this area, especially given the fact that their staff were already overburdened with work. It might be better to find or encourage an independent NGO to work on the demand side, especially one that was more independent of the government.

• Both projects are engaged in many activities, so much so that it was not easy at first glance to appreciate the strategy guiding these activities. In the WCHPP there were two overlapping strategies. One could be described as humanitarian assistance. This involved funding support for staff training, repair of health facilities, procurement of supplies, etc. All of this was useful, and appreciated, but it was essentially gap filling, filling in where government funds should have been at work. Government funds were available, but they were being invested elsewhere, in new hospital buildings and in sectors other than health. The other strategy was the introduction of new and improved ways of doing things (computerised health information systems, partnerships between midwives and traditional birth attendants, etc). This is where UNICEF was arguably doing more development work, potentially adding value to the way in which government health systems were already working.
  • With both the humanitarian and development work there were some important limitations, that needed to be addressed. With the humanitarian work, there were no agreed performance targets or milestones that UNICEF would help the government reach, and then phase out its assistance. For example, at what point do you stop supporting the operations of a sub-district health centre, and move your assistance elsewhere? With the development work, the introduction of new or improved methods was not consistently associated with assessment of existing methods, and then a follow up comparison of the value of the new method. Nor was there any systematic packaging and promotion of the results of studies that were carried out. De factor “experiments” were being carried out, but they were not being systematically assessed and results publicised. This is despite the fact that the idea of development projects as “policy experiments” has been around for decades (See Rondinelli, 1993).

• There was a third feature that seemed to be shared by both projects, relating to decentralisation. Both Uganda and Indonesia have decentralised government administration, Indonesia probably the most radically. If ever there was a need for decentralised administration, it would be in Papua, which is culturally and geographically a long way from Jakarta. Yet, UNICEF’s response to providing assistance to its Papua office was remarkably centralised. A KAP (Knowledge Attitude Practice) survey was centrally commissioned for the whole of Indonesia as one entity, and this will be used to measure the impact of health education and awareness raising activities in Papua (and elsewhere), where the issues facing women and children are not carbon copies of those found elsewhere in Indonesia. A M&E framework was introduced, exactly as used by similar UNICEF projects elsewhere in Indonesia. Yet the capacities of local government, and associated health services in Papua bear no comparison with those found in Java. All this raises questions of how much can we expect a highly centralised aid agency to help a highly decentralised government?

• The fourth feature related to the behaviour of UNICEF and GTZ staff as members of the annual review teams I mentioned above. While they have now participated in reviews of each others projects for three years in a  row, it still surprises me how much they are still very wedded to their view that their own approach is the best possible under the circumstances, and how little acknowledgement (and even less follow up action) they have given to potentially positive features of each other’s projects. This experience resonates with what we have seen on the Guardian blog, in the way that AMREF has responded to comments by another NGO that it is possible to build schools significantly cheaper that the way AMREF has done. That is, with a little interest and enthusiasm to learn, and a considerable amount of defensiveness.

In my report on my second visit to Katine in August 2008 I raised a very similar set of issues

• The risks of overly centralised management of the project, within AMREF
• Questions about to what extent AMREF should do community empowerment, and whether it might be better done by another party more independent of government
• The need to set targets for what AMREF was trying to achieve, in consultation with its local partners
• The need to take the idea of generating development models seriously, by building operational research into all new activities that were trying to do things differently

The issues raised here also relate to my argument in a recent posting: How well is the KCPP doing? Compared to what? In that posting I argued that comparisons of AMREF’s performance with other projects could be as important as comparisons of with its own stated intentions and objectives

How well is the KCPP doing? Compared to what?

One conventional approach to the evaluation of aid projects is to identify the project’s objectives, and then see what has been achieved on the ground and how it compares to the stated objectives on paper.

This approach has its own difficulties. For example, the objectives might not be stated very clearly. Or if there are clear objectives, there may not be any useful indicators of progress with these objectives. Or if there are useful indicators, there may not be any targeted level of achievement on those indicators. I have already discussed these types of problems in the KCPP, both directly with the AMREF KCPP staff, and via this blog. (e.g. “Appropriate Goals“)

But this approach may be irrelevant to many people. What may be more important is how the KCPP seems to be progressing, when compared to other development projects that they know of.

Yesterday’s article on the Guardian Katine website (by Richard Kavuma) on President Museveni’s visit to Soroti mentioned a number of different development initiatives that have been promised, and in some cases already happened. Richard’s article mentions Northern Uganda Social Action Fund (NUSAF), the National Agricultural Advisory Services (NAADS), Savings and Credit Cooperative Societies (SACCOs), and now “model farms”

“Presenting highlights of his anti-poverty campaign, Museveni announced a fund of 98 billion Ugandan shillings ($60 million) to support six homesteads in each of Uganda’s 5,000 parishes to set up model farms. The idea is that the remaining villages will learn from the model homesteads how poverty can be overcome and will – hopefully – start their own journeys.” **

The KCPP project is also about developing convincing models of good practice (re health, education, water, sanitation, livelihoods, governance), that government, local communities and individual households will hopefully adopt.

However, the NUSAF (Phase 2), the SACCOs, and the model farms will all be in competition with the work AMREF is doing via the KCPP. In competition for the attention of local elected leaders and administrators, who need to decide how to best use their scare resources (both their own time and the budgets they are responsible for). And in competition for the attention of households as well.

Competition could be useful. But it will depend on who sets the rules, about what kinds of development achievements are really important. That will be an important arena of competition itself. AMREF could well argue that transparency (about how a project is managed) is most important of all, because without open access to information how can people really tell what is going on? Have the billions of shillings really been spent as intended?

AMREF could also argue that sustainability is very important, important enough to sometimes justify its own development costs being higher than others (e.g. for installing a borehole and establishing a working water source committee).

But to do this it will need some hard facts, not just optimistic assumptions. For example about the percentage of KCPP boreholes still functioning after two years, compared to those established by NUSAF.

And it will also need a strategy, a sense of what facts are the most important, amongst the many that could be collected. What are the most important differences between how AMREF is doing things in the KCPP, and how others (government and non-government) are doing them? And what difference is it expected that these differences make to people’s lives?

These questions can be asked at two levels: (a) about AMREF’s work generally, and (b) much more specifically, about individual project activities. Such as providing furniture and books for schools, or training Village Health Teams, or working with farmers groups to improve their incomes. Facts and findings about concrete activities like these are probably going to be of most interest to the people of Soroti district.

If it can successfully compete for people’s attention, then the impact of the KCPP will have a better chance of spreading, beyond the currently limited number of schools with new equipment or tthe few villages with new boreholes. It will also be in a better position to convince its current (and prospective) donors that there is good reason to fund organisations like AMREF, rather than channeling money directly to poor communities, or to local government, or to other NGOs.  It will be able to show how its particular way of working adds value in a way that justifies the cost involved.

—————————

Footnote**: $60 million = $2,000 per model farm. There could be 30 model farms set up in Katine sub-county, which has five parishes.

Distributed accountability

In my recent visit to the Katine project I raised the issue of whether administrative processes in AMREF’s Uganda Country Office might present a significant project risk, increasing the likelihood that the KCPP might not achieve its stated goals by the end of 2010. In particular I was concerned with the efficiency and effectiveness of their procedures for procurement, the approval of quarterly workplans, and some other related management practices. Compared to many other development organisations I have worked with in the past, these procedures seemed to be unusually centralised. My concern here was not with AMREF’s internal management as an end in itself, but with its effects on development activities in Katine: the ability of AMREF field staff in Katine to implement planned activities on schedule, with some efficiency, effectiveness and (cumulatively) with some credibility.

In discussions in Kampala it became clear that many of the requirements for these processes (but not all) were set further up the organisational hierarchy, outside of Uganda. There are some upcoming opportunities for proposing changes (an Auditors review, and a Board meeting), but changes may not happen simply through lobbying done on behalf of one project, or even one country programme. Nevertheless, they would be worth proposing. Of particular concern to me was the Katine Project Manager’s very limited authority to buy goods and services locally (UGX 100,000 maximum), in surprising contrast to the Ugandan government’s emphasis on local procurement (the Katine Sub-country chief can spend up to UGX 30 million).

In parallel to discussions about possible changes in these requirements I also suggested the use of some management process indicators, with some demanding targets. Ideally information about monthly performance on these process indicators would be made known to all AMREF staff, and be included in the subsequent six months progress reports that are sent to the donors and made available to the Project Steering Committee in Soroti. This would provide a more distributed form of accountability, in contrast to the current progress reporting that focuses on the implementation of activities in Katine by AMREF field staff. Of course the country office could fairly say that they are already accountable, as representatives of the whole country programme. But this is not so true for the specific contributions they make to the KCPP and other projects in their country programme, through procurement and workplan approvals. If these supporting activities were given the same amount of monitoring attention as project activities on the ground then improvements in performance might make it easier to justify the share of KCPP money spent on country office salaries, to not only the donors, but more importantly to the stakeholders in Soroti.

Whether there will be any progress either through devolution, and/or more intensive monitoring, remains to be seen. Ideally, when I next return to Uganda it will be the country office staff who first alert me to any internal bottlenecks and delays. I will not have to discover them by working back from implementation delays as seen in Katine.

AMREF makes progress with its Open Information Policy

When I first met with AMREF staff in London in September(?) last year, at the very beginning of the Katine project, I raised the issue of transparency, and whether AMREF would consider developing a Disclosure Policy, which would spell out what documents would automatically be made publicly available, what would remain confidential and where case-by-case approval might be needed.

I was pleased to see some months later, in November 2007, that the AMREF Board had approved an Open information Policy (possibly already under development before my discussion in September). This was followed by an Implementation Plan, which is expected to be completed by the end of 2008. While I was impressed by the range of documents to be made available, I was a bit concerned at how slow this process was (the project runs from Sept 2007 to the end of 2010), and that the disclosure of project documentation was not explicitly mentioned in the Implementation Plan.

However, in the last few days AMREF have taken a major step forward, with the listing of eight project documents, on both its AMREF UK and AMREF HQ websites. These include four documents making up the first six months progress report, and four background project documents (relating to design and baseline data collection).

In my next visit I will be checking to see how many staff, especially in Katine, are aware of the Open information Policy. And their understanding of how it is to be operationalised. Often policies get developed, but then sit on the shelf, unknown to most people, and therefore only partially implemented at best.

It will also be important to identify how many local stakeholders are aware of this policy, especially in the Project Steering Committee at the district level, and the Project Management Committee level, in Katine sub-county. Issues of access to information, regarding project costs, have already been the focus of discussion in meetings held so far.