One conventional approach to the evaluation of aid projects is to identify the project’s objectives, and then see what has been achieved on the ground and how it compares to the stated objectives on paper.

This approach has its own difficulties. For example, the objectives might not be stated very clearly. Or if there are clear objectives, there may not be any useful indicators of progress with these objectives. Or if there are useful indicators, there may not be any targeted level of achievement on those indicators. I have already discussed these types of problems in the KCPP, both directly with the AMREF KCPP staff, and via this blog. (e.g. “Appropriate Goals“)

But this approach may be irrelevant to many people. What may be more important is how the KCPP seems to be progressing, when compared to other development projects that they know of.

Yesterday’s article on the Guardian Katine website (by Richard Kavuma) on President Museveni’s visit to Soroti mentioned a number of different development initiatives that have been promised, and in some cases already happened. Richard’s article mentions Northern Uganda Social Action Fund (NUSAF), the National Agricultural Advisory Services (NAADS), Savings and Credit Cooperative Societies (SACCOs), and now “model farms”

“Presenting highlights of his anti-poverty campaign, Museveni announced a fund of 98 billion Ugandan shillings ($60 million) to support six homesteads in each of Uganda’s 5,000 parishes to set up model farms. The idea is that the remaining villages will learn from the model homesteads how poverty can be overcome and will – hopefully – start their own journeys.” **

The KCPP project is also about developing convincing models of good practice (re health, education, water, sanitation, livelihoods, governance), that government, local communities and individual households will hopefully adopt.

However, the NUSAF (Phase 2), the SACCOs, and the model farms will all be in competition with the work AMREF is doing via the KCPP. In competition for the attention of local elected leaders and administrators, who need to decide how to best use their scare resources (both their own time and the budgets they are responsible for). And in competition for the attention of households as well.

Competition could be useful. But it will depend on who sets the rules, about what kinds of development achievements are really important. That will be an important arena of competition itself. AMREF could well argue that transparency (about how a project is managed) is most important of all, because without open access to information how can people really tell what is going on? Have the billions of shillings really been spent as intended?

AMREF could also argue that sustainability is very important, important enough to sometimes justify its own development costs being higher than others (e.g. for installing a borehole and establishing a working water source committee).

But to do this it will need some hard facts, not just optimistic assumptions. For example about the percentage of KCPP boreholes still functioning after two years, compared to those established by NUSAF.

And it will also need a strategy, a sense of what facts are the most important, amongst the many that could be collected. What are the most important differences between how AMREF is doing things in the KCPP, and how others (government and non-government) are doing them? And what difference is it expected that these differences make to people’s lives?

These questions can be asked at two levels: (a) about AMREF’s work generally, and (b) much more specifically, about individual project activities. Such as providing furniture and books for schools, or training Village Health Teams, or working with farmers groups to improve their incomes. Facts and findings about concrete activities like these are probably going to be of most interest to the people of Soroti district.

If it can successfully compete for people’s attention, then the impact of the KCPP will have a better chance of spreading, beyond the currently limited number of schools with new equipment or tthe few villages with new boreholes. It will also be in a better position to convince its current (and prospective) donors that there is good reason to fund organisations like AMREF, rather than channeling money directly to poor communities, or to local government, or to other NGOs.¬† It will be able to show how its particular way of working adds value in a way that justifies the cost involved.

—————————

Footnote**: $60 million = $2,000 per model farm. There could be 30 model farms set up in Katine sub-county, which has five parishes.

Advertisements

Distributed accountability

14 August, 2008

In my recent visit to the Katine project I raised the issue of whether administrative processes in AMREF’s Uganda Country Office might present a significant project risk, increasing the likelihood that the KCPP might not achieve its stated goals by the end of 2010. In particular I was concerned with the efficiency and effectiveness of their procedures for procurement, the approval of quarterly workplans, and some other related management practices. Compared to many other development organisations I have worked with in the past, these procedures seemed to be unusually centralised. My concern here was not with AMREF’s internal management as an end in itself, but with its effects on development activities in Katine: the ability of AMREF field staff in Katine to implement planned activities on schedule, with some efficiency, effectiveness and (cumulatively) with some credibility.

In discussions in Kampala it became clear that many of the requirements for these processes (but not all) were set further up the organisational hierarchy, outside of Uganda. There are some upcoming opportunities for proposing changes (an Auditors review, and a Board meeting), but changes may not happen simply through lobbying done on behalf of one project, or even one country programme. Nevertheless, they would be worth proposing. Of particular concern to me was the Katine Project Manager’s very limited authority to buy goods and services locally (UGX 100,000 maximum), in surprising contrast to the Ugandan government’s emphasis on local procurement (the Katine Sub-country chief can spend up to UGX 30 million).

In parallel to discussions about possible changes in these requirements I also suggested the use of some management process indicators, with some demanding targets. Ideally information about monthly performance on these process indicators would be made known to all AMREF staff, and be included in the subsequent six months progress reports that are sent to the donors and made available to the Project Steering Committee in Soroti. This would provide a more distributed form of accountability, in contrast to the current progress reporting that focuses on the implementation of activities in Katine by AMREF field staff. Of course the country office could fairly say that they are already accountable, as representatives of the whole country programme. But this is not so true for the specific contributions they make to the KCPP and other projects in their country programme, through procurement and workplan approvals. If these supporting activities were given the same amount of monitoring attention as project activities on the ground then improvements in performance might make it easier to justify the share of KCPP money spent on country office salaries, to not only the donors, but more importantly to the stakeholders in Soroti.

Whether there will be any progress either through devolution, and/or more intensive monitoring, remains to be seen. Ideally, when I next return to Uganda it will be the country office staff who first alert me to any internal bottlenecks and delays. I will not have to discover them by working back from implementation delays as seen in Katine.

When I first met with AMREF staff in London in September(?) last year, at the very beginning of the Katine project, I raised the issue of transparency, and whether AMREF would consider developing a Disclosure Policy, which would spell out what documents would automatically be made publicly available, what would remain confidential and where case-by-case approval might be needed.

I was pleased to see some months later, in November 2007, that the AMREF Board had approved an Open information Policy (possibly already under development before my discussion in September). This was followed by an Implementation Plan, which is expected to be completed by the end of 2008. While I was impressed by the range of documents to be made available, I was a bit concerned at how slow this process was (the project runs from Sept 2007 to the end of 2010), and that the disclosure of project documentation was not explicitly mentioned in the Implementation Plan.

However, in the last few days AMREF have taken a major step forward, with the listing of eight project documents, on both its AMREF UK and AMREF HQ websites. These include four documents making up the first six months progress report, and four background project documents (relating to design and baseline data collection).

In my next visit I will be checking to see how many staff, especially in Katine, are aware of the Open information Policy. And their understanding of how it is to be operationalised. Often policies get developed, but then sit on the shelf, unknown to most people, and therefore only partially implemented at best.

It will also be important to identify how many local stakeholders are aware of this policy, especially in the Project Steering Committee at the district level, and the Project Management Committee level, in Katine sub-county. Issues of access to information, regarding project costs, have already been the focus of discussion in meetings held so far.

Taken as it stands, the recent article on the Guardian Katine website: “Katine community welcomes Amref’s empowerment drive” reinforces the concerns I expressed to AMREF in my January 2008 visit to the KCPP (the Katine project).

That was about the risk of confusing mobilisation with empowerment. These two processes are at the opposite end of a continuum, they are not the same kind of thing. Mobilisation is about pursuasion, getting people to do things you want them to do, or what you think they should be doing (e.g. getting their children immunised). Empowerment is about enabling people to do what they want to do, whatever that might be. Diversity of people’s behavior is often a good indicator of successful empowerment. For example, increased diversity of ways of earning income in a local economy, as an indicator of economic empowerment. Related to this diversity definition is the willingness to express differences of opinion with others. To argue, not just to cooperate. For example, to demand a health service which has essential drugs in stock, or to demand a school where teachers are present in their post.

It could easily be argued that because there can be place in a development project for mobilisation, as well as empowerment, that these two roles should be the responsibility of two different people in an organisation. Not the responsibility of the same person.

Unfortunately in Anne’s article Richard is introduced both as “the Amref officer for empowerment,…” and later quoted as saying “I lead all ‘mobilisation’ processes” If I was a UK reader of Anne’s article I would be wondering if there is risk that one of Richard’s tasks will end up being neglected, and whether that might be the most troublesome one: empowerment