Negotiated agreements on AMREF-assisted developments

On Tuesday 22nd September Joseph Malinga wrote an article titled

Building work begins on Katine produce store: Katine farmers dig the foundations of a produce store that should help to improve livelihoods in the sub-county

I posted a Comment as follows:

Negotiated agreements on AMREF-assisted developments

It is good to see the issue of uncertain and conflicting expectations being dealt with in this article about a specific development activity assisted by AMREF: the building of a grain store for use by farmers groups in Katine. The need for written agreements about expectations has been raised by the farmers and AMREF has responded to (or has anticipated) this need. The only question in my mind, and perhaps others, was whether this agreement should have been negotiated and signed, before the commencement of the work on the store. On the other hand, a positive feature of this agreement is the willingness of government to make its contribution, along with that of the farmers groups (though this has its risks as well). And the agreement does not seem to be one sided in its expectations. Such agreements could go further than specifying the inputs each will provide, and management arrangements once completed. Reference could also be made to the objectives of this investment, to help forestall any future misuse of the store, e.g. the private use of the store by one of the farmers group members, or someone else completely. How ever the agreement does develop, it would be good if AMREF could share this example agreement via the Guardian website.

In my August 2009 comments on the future of the KCPP I had suggested that Associated with this clarification of expectations, agreements need to be developed that will spell out not only what AMREF will provide, but also what communities will provide, AND what the government will provide. Multiple agreements may be needed, perhaps component by component. One generic agreement will probably not work, because responsibilities will become too generalised and fuzzy. My proposal did not go far enough, agreements about individual developments, such as the grain store, are better still. They are smaller in ambit, and more manageable.

“When the community owns the project Giving the community a say in how their schools are built seems to be the most sensible option, says Joseph Malinga”

This is the headline of a recent post on the Guardian Katine website, by Joseph Malinga, a Guardian journalist based in Katine.

I posted the following reply…

Could Joseph Malinga do a  follow-up to this story explaining what the government (local and/or central) is doing to support community initiatives like this one? Joseph’s story describes what the community is contributing and what AMREF is providing but there does not seem to be any matching contribution from the government. The government appointed head teacher was there already. The only government contribution so far seem to be doubt, about whether the quality of the school building will be adequate. Yet government is supposed to be one of the partners in the Katine Partnership Project, and advocacy is reported to be an important part of AMREF’s development strategy. What contributions or commitments have AMREF secured from government, in return for the work AMREF is doing with these two community schools? For example, the provision of more government paid teachers. At present the community is paying the teachers salaries.

This is an issue I have raised in my review of the Mid-Term Review, at http://evaluatingkatine.files.wordpress.com/2009/09/a-process-review-of-the-katine-mtr.pdf

“The Worst Question to Ask About Charity”

In the recent Mid-term Review (MTR) of the KCCP some local officials questioned the amounts being spent by AMREF on management, and the amounts remaining for what they thought was most important, the construction of infrastructure, like clinics and schools in the Katine. They wanted more for infrastructure and less on management.

In her report, the MTR consultant said “ Just how much money is available is  a figure that can be misrepresented but the entire sum when broken into management, transport and staff costs leaves a reasonable – but not excessive – amount for activities and work on the ground.  The figure is at least 70 per cent of the total budget and this is regarded as an acceptable amount in development projects worldwide.”

In the minutes of a subsequent meeting between AMREF and its donors (Guardian and Barclays) it was reported that the Guardian “asked for costs to be streamlined however [AMREF noted that] management support costs need to be factored in since this came up as an important resource to consider in the review. The MTR recommended a 70%/30% split as common with other projects”

I have real disquite about this position for several reasons, which I will explain. The first reason is it is actually not so easy to calculate this percentage in a standardised way that is applicable across all organisations and understood by all donors. The 70/30 split is probably a common view because everyone in the aid agency world has formed the view that this is what is acceptable to everyone. In other words, it is a herd judgement. I doubt that it is a percentage that has been found through any systematic assessment of aid agency costs.

My second reason is that this crude measure of overhead costs is based on a false assumption of how aid agencies work, a view which is captured by the simplistic but appealing notion that what matters is “whether the money gets there”. In practice in most aid programs very little money or goods actually reach the hands of poor households, because that is the way projects are designed. In Katine more than 50% of the activities in the workplan are training activities, directed at different members of the community and local government. The money spent here goes on staff salaries, and allowances for other trainers, which are spent mainly in the towns. Building costs are another important part of the project. Until recently, most of the money spent on infrastructure was spent on contractors hired from Kampala. Only in the livelihoods component is there much in the way of actual transfer of project resources directly to poor households, in the form of seed supplies. But in parallel to these activities is the UWESO-assisted savings and loan project, where instead of giving people things, the aim is to help people save and make best us of the money they already have. Only in humanitarian emergencies is there any deliberate and substantial real transfer of assets to poor households.

My third reason is that the percentage spent on management costs versus delivery of services is an efficiency measure. Notionally, the smaller the proportion spent on management the better. But this ignores completely the issue of  how effective the services are that are delivered on the ground. Low overheads will not redeem poor quality services. High overheads may contribute to better services. What matters here is cost effectiveness – what can be achieved for a given unit of cost. And cost here include not just immediate costs like building materials, but also the management costs, and (proportionally) the costs of the managers of the managers, etc.

In a recent posting by Dan Pallota on his blog “Free the Non-Profits” he quotes some findings from America which may also apply in the UK:

In 2002 the Better Business Bureau Wise Giving Alliance commissioned a study that asked respondents what information they wanted when considering donating to a charity. Seventy-nine percent wanted to know what percentage of their donation went to charitable programs. Remarkably, only 6% wanted to know if the donation would make a difference. How can that be, you ask? Well, the media, the watchdogs, and the sector itself have done an amazing job of training the public to think that the two things are the same, i.e., that if a charity has low overhead, it must be making a difference. Major studies on the relationship between organizational strength and impact find otherwise.

My advice to AMREF, the Guardian and Barclays is to forget about the 70%/30% ratio, however it is constructed. I agree with Dan Pollota when he says the worst question to ask about charity is, “What percentage of my donation goes to the cause?”, also known as the admin:program ratio, the “efficiency” measure, or the overhead ratio. Whatever you call it, it’s hopelessly flawed, widely abused, utterly useless, a pathetic substitute for meaningful information about a nonprofit’s work, inept at exposing fraud, and a danger to human life”

Okay, then how do we best address the concerns expressed by local authorities, during the Mid-Term Review. In my process review of the MTR I made the following suggestion:

The Guardian and Barclays Bank could take a further step, and request that each six monthly narrative report on the KCPP include seperate sections on the activities of the AMREF London and Kampala offices and the costs they have incurred in carrying out these activities. If this step is taken, these narrative reports should then be routinely shared with the Steering Committee and Management Committee, as well as being made publicly available via the Guardian website as at present.

These sections would detail not only the costs incurred by different sections of AMREF (London, Nairobi, Kampala), but also what they were able to do with that money i.e. some description of their effectiveness

There is an important larger lesson here. Aid projects like the KCPP involve long and complex supply chains, bringing funds and technical expertise to communities of concern, from distant locations. In the private sector intense effort is often invested into making every part of supply changes work as quickly and efficiently as possible. But in the world of development aid often the focus is almost wholly on the final link in the chain, the organisations delivering assistance at the grassroots level. Very little attention is given to the more expensive[1] parts of the supply chain lined up behind them. The Guardian needs to turn its journalistic attention towards the issue of supply chain costs in international aid delivery. The diagram below shows just how complex these supply chains can be, even in a modest project like the KCPP

Thick blue lines = financial transfers. Broken blue lines = information transfers (not including most of those between yellow nodes (intermediaries between donors and recipients))

For further reading see  Dan Pollota’s posts on

The Worst Question to Ask About Charity 9:44 AM Tuesday June 16, 2009

“Efficiency” Measures Miss the Point 3:56 PM Monday June 22, 2009

Beware of Highly “Efficient” Charities 10:44 AM Monday June 29, 2009

Efficiency Measures Discriminate Against Lesser-Known Causes 10:40 AM Wednesday July 8, 2009

Efficiency Measures Short-Change Individual Action 2:20 PM Monday July 13, 2009

---

[1] In terms of the costs of staff time and transport costs involved

Money ‘wasted’ on water projects in Africa

This is the headline for a story on the Guardian katine blog, where Annie Kelly discusses a report that “criticises donors, governments and NGOs for installing boreholes and wells in rural Africa without ensuring their long-term sustainability”

Four comments have been made so far, including one by AMREF and myself.

AMREF pointed out that ” In 2002, AMREF helped communities in the Kathonzweni area to develop 50 shallow wells…By 2007 — five years later, and several years after the projects end — the communities had maintained the existing wells and built 20 new ones.”

This sounds very good, almost too good.

My comment was that “It would be useful if AMREF could provide (on this Guardian website) some information on (a) how they are monitoring the functioning of the wells they have helped to establish in Katine so far, and (b) the data that has been collected. Hopefully information is available on both the functioning of the wells and the functioning of the management committees responsible for each well.

There are problems with wells in Katine, like just about everywhere else. This is to be expected. Equipment does get damaged and worn. What matters more is how long those problems remain unresolved.

There will be a mid-term review of the Katine project in a few months time, which could look at the functioning of the wells then. But that would be a once-off inquiry and the risk is that (as elsewhere) extra efforts might be made to make sure things are working well at that time. Its the monitoring that matters.

regards, rick davies, external evaluator for Katine”

Are short-term development projects like Katine a good idea?

This was the headline on the Guardian katine website, on Tuesday 24th March 2009.  Eliza Anyangwe asked “Are short-term development projects like Katine a good idea? The Katine project in Uganda is scheduled to last for three years. But with such a short timescale can it deliver lasting change, or will it leave a bitter aftertaste?”

There were 12 comments including mine, which ran as follows:

“I have been working for aid organisations for about 28 years, and doing monitoring and evaluation of aid projects for about 19 years. Throughout this time most people I have spoken to who have been associated with aid projects have expressed the view that 3 years is not long enough to make a substantial and sustainable development impact. Especially if you are trying improve the way in which government services work. Many of the comments above seem to share this view.

In the case of the Katine project, AMREF have expressed the view that ” the Katine project really will contribute to lasting change in one of Ugandas poorest and most vulnerable regions”. The Guardian editor also seems to share this view. Are they right? Lets try to test those views.

I have already proposed to the Guardian and AMREF that at the end of the three year period AMREF, and its local partners, should make a number of testable predictions about what things will look like in Katine, in another three years time. That is six years after the project started and three years after it ended. Then the Guardian, or perhaps some more independent organisation (with no prior involvement in Katine), should fund what is called an “ex-post evaluation” team to come in and see how many of the predictions were successful, and how many have failed. And equally importantly, why some were successful and others not so. Then let those results be shared and discussed in Katine and elsewhere via the Guardian blog.

regards, rick davies, external evaluator for the Katine (KCPP) project

PS: The problem with this proposal is the uncertainty about who will be around, six years after the start of the Katine project. Will any of the Guardian journalists now engaged with the project be around and interested to see what has happened, or will they have moved on? Will any of the key people in AMREF still be around and interested, or will they have moved on? The big question seems to be how do aid (and other) organisations develop and maintain a longer term view on what they are doing at any one time.

Katine stakeholder meeting and preliminary project steering committee meeting: April 3rd, 2008

This AMREF document is available on the Guardian newspaper Katine website, in an article titled “Stakeholders question timescale of Katine project“

I posted the following comments:

“I was pleased to see the article about the recent stakeholder meeting held in Katine (“Stakeholders question timescale of Katine project”) and to be able to access online a record of the meeting (“Katine stakeholder meeting and Preliminary project steering committee meeting April 3rd 2008″)

I would hope that further meetings like this are also shared on the Katine website. In the process, I hope that some more detail is provided on the participants who are not government representatives. It seemed that in this meeting almost all participants were government officials or representatives. If that was meant to be so, it would be useful to make that clear. Similarly, records of future meetings should be very clear on who they were meant to include.

My main reason for emphasising this point is that one of the evaluation criteria I will be using (as the external monitor/evaluator) is equity. This means fairness of process as well as fairness of result. There is a second dimension here as well, that of transparency. If a record is kept of the participants in such meetings it will be possible for myself, the Guardian journalist (Richard Kavuma) and anyone else, to make follow up contact with the meeting participants later on, both to hear their views of the meeting, and of what has happened thereafter.”

Comment on Amref’s first six months

A response to

Comment on Amref’s first six months, May 7th 2008

I was pleased to see the article about the recent stakeholder meeting held in Katine (“Stakeholders question timescale of Katine project“) and to be able to access online a record of the meeting (“Katine stakeholder meeting and Preliminary project steering committee meeting April 3rd 2008“)
.
I would hope that further meetings like this are also shared on the Katine website. In the process, I hope that some more detail is provided on the participants who are not government representatives. It seemed that in this meeting almost all participants were government officials or representatives. If that was meant to be so, it would be useful to make that clear. Similarly, records of future meetings should be very clear on who they were meant to include.
.
My main reason for emphasising this point is that one of the evaluation criteria I will be using (as the external monitor/evaluator) is equity. This means fairness of process as well as fairness of result. There is a second dimension here as well, that of transparency. If a record is kept of the participants in such meetings it will be possible for myself, the Guardian journalist (Richard Kavuma) and anyone else, to make follow up contact with the meeting participants later on, both to hear their views of the meeting, and of what has happened thereafter.

Assessing the Katine project

(excerpt from Guardian website…) Rick Davies, the independent evaluator for our Katine project, is off to the region next week. But what are his aims during his first trip to Katine and what do you think of his plan?
January 11, 2008 3:54 PM

This weekend I will head off to Uganda for two weeks, to meet the Amref staff working on the Katine project, and to see the sub-county itself – the place and the people. This will be the first of a series of twice-yearly visits that I will be making over the next few years.

As part of the preparation for this visit, I attended a meeting in London on Monday 7 January to go over my Terms of Reference (ToRs) for my visit with staff from Amref and from the Guardian. One of the things we discussed was my request last year that Amref develop a disclosure policy, which will spell what sort of information it will make publicly available, and under what circumstances.

Much to my surprise, that policy has already been developed and approved by its Board in November, but nobody had told me, nor had it’s existence been made public via the Amref website. This does seem to almost defeat the purpose of the policy, which is unfortunate, since the intentions expressed in the policy do seem positive.

I have made a copy available here, for public comment. My questions to you, the reader, are: What do you think of it? How could it be improved? For comparison, here is a similar sort of policy developed by ActionAid.

In the same meeting we also discussed my visit schedule in Uganda. My draft ToRs are available here. As you can probably see, the list of things to do is quite long, probably too long to complete in this visit. So my first meeting with Amref in Uganda will have to focus on prioritising these tasks.

Top of my own to-do list is to meet all the Amref staff in Katine, find out about their various roles, and to talk about their expectations about my role as the external evaluator – what they would and would not like to me doing. I will be bring along all the comments made so far by participants in an online survey of people’s views on this subject, which you can find here.

So far this online survey has focused on a limited number of stakeholders: the staff of Amref, Guardian and Barclays. But I hope to open it up to wider public participation on return from Uganda.

As well as the tasks listed in my Terms of Reference there are many other questions I would like to explore during my visit. Most of these have been prompted by my reading off Amref’s project documents over the last month, and by reading the Guardian Katine blogs. Here are some of them:

People’s participation: What did the community needs assessments find out about the existence of different community views on development needs in Katine? It is highly unlikely that in a population of 25,000 they all had the same set of priorities. People’s views are likely to vary by gender, age, and location, at least. How have these views affected the project design?

And in Amref’s Monitoring and Evaluation Plan for the project, what role will community groups have in the monitoring and evaluation of the project? How often will their views be sought? How will those views then feed into decision-making about how the project develops?

Project strategy: Will the project be aiming to assist the whole population evenly, or will it be targeting some groups more than others? Does Amref have enough staff and financial resources to reach the whole population? Will the various developments in water supply, health and livelihoods be focused at different target groups, or it is essential that a given group of people experience the combined impact of all these developments? How much information is available at this stage about the distribution of the population through the sub-county, and various government services? Could a map of these be made available on the Guardian Katine website, which could be continually updated and unfilled with information, as the project progresses?

Project impact: Where will the impact of the project be most visible in three years time? Will it be in changes in school attendance and completion, changes in people’s health, or changes in their livelihoods? Will the proposed baseline survey enable Amref to track the changes that are taking place, and separate out the effects of Amref’s input, from the effects of other changes taking place in the society and economy? What about unexpected changes that may not have been planned for? How will they be given adequate attention? Is the monitoring and evaluation plan realistic? Is it too ambitious in terms of the information that will be collected?

Sustainability: How will the impact of the assistance provided by Amref be sustained in the future? Will government be better able, or more willing, to take responsibility for delivering good quality health and education services?

Transparency: What mechanisms does Amref have for transparency at the local (Katine) level, as distinct from via its website and that of the Guardian? Which of the various project documents produced so far has Amref made publicly available? What else could be made available right now? What problems, if any, are arising because of this transparency?

If there are other issues you think I should be looking at, please add your comments below (< see this page on the Guardian website)

Why aren’t you African?

(from Guardian website…) Have you asked Katine’s people what they want? And why aren’t you African? Rick Davies, our independent assessor, answers your questions.
October 23, 2007 10:41 AM

The independent assessor for the Katine project, Rick Davies, answers questions from blog readers about his work.

My role in this project is to independently assess the work carried out by Amref in Katine, a process explained in detail elsewhere on the site.

Already, some readers have posted comments and questions on this blog about my work. Rather than posting a long reply myself, I thought it best to write a new entry.

One reader, Go4it2day, said:

Have you asked the people of Katine what they want? And I do not mean consultations between an NGO and some people called to a meeting. Believe it or not market research is available in Africa.

In my January 2008 visit, one of my tasks will be to examine the ways in which Amref staff have worked with people in Katine to identify their needs and priorities. My expectation is that they will use a variety of means, possibly including one to one surveys.

The same reader asked:

Why is the ‘independent auditor’ not African? Is it because his responsibility is to the Guardian and its readers, and not to the people of Katine?

I will not be the only person who will be monitoring and evaluating the performance of Amref’s work. In the first instance, Amref will have its own dedicated staff and procedures for monitoring and evaluating the implementation and outcomes of its work.There may also be a community level committee of local stakeholders, with similar responsibilities. My role will be more ‘meta-monitoring and evaluation’. One of my tasks will be to check on the adequacy of Amref’s systems and procedures for monitoring and evaluating its work, and to provide to them, where wanted, with help to build their monitoring and evaluation capacity.

My work will involve talking to Amref staff, reading their documents, and talking to people in the community. In the longer term, my role should become less and less important, as the Guardian and its readers feel increasingly confident in the abilities of Amref.

Go4it2day also asked:

Why are the KPIs [key performance indicators] of the project so vague? Why not use simple KPIs like average income, child survival rates, deaths per 1,000. Is it because nice and vague KPIs allow a whitewash of the project?

The objectives need to be agreed on first, through community consultations. Then, during the process, verifiable indicators of their achievement need to be identified. Again, this is best done through consultations with the groups/communities involved.

Java1930 commented:

Having read Amref’s structured programme to tackle the issues of health, education, sanitation and safe water, income generation and good governance I am concerned to see there appears to be no specific acknowledgment of the centrality of women to the success or otherwise of the outcomes in each area.

One of the criteria that I will be using to assess the work of Amref will be how they address issues of equity. And one important dimension of equity is gender equity. I will examine not only how women benefit, versus men, but also how they are involved in planning, implementation and review processes.In a later comment, Java1930 said:

Also I urge Mr Davies when he is making his assessments on the project to ensure he is able to speak to women and girls in the right environment – they should have an opportunity to speak in the absence of their men folk whose presence may inhibit them discussing real issues. And when he feeds back he should be careful to ensure individual women are not identified as raising particular issues for fear of retribution – domestic violence is an issue there as it is anywhere else in the world.

In the first instance I will try to identify to what extent Amref staff have been able to do this, and if not, why not. Secondly, I will seek to follow this advice myself, when in Katine.

The same reader commented:

I would also like this project to be very clear and transparent on how much of the money (cash) raised is actually spent on the people in Katine ie how much pence of each pound raised. For the last 25/30 years Uganda’s been awash with government and non-governmental NGOs who receive millions of pounds and yet we see precious little reaching the people in whose name it is all being raised for. Sad to say but charity is a very big lucrative business.

I have suggested to Amref that the prepare a disclosure policy, which would state what types of information will, by default be publicly available, and what will not, but which may be available on request. Disclosure policies are used by major organisations such as the World Bank, and the IMF, but also by progressive NGOs such as ActionAid.Costs can be cut horizontally (eg the amounts spent in UK, spent in Kampala, and spent in Katine) or vertically (the total costs incurred at all levels for delivering a specific outcome on the ground eg building a new school or assisting a specific community).

The latter is more useful, if you want to compare costs against benefits. Analysing costs horizontally can involve mistaken assumptions: that all assistance will be in the form of things or money given directly to people in Katine, and that purchases made in Kampala or London will be of no benefit to them.

In another comment, Go4it2day said:

But this brings up the question in what will happen to Katine in 4 years, 10 years, 20 years after the Guardian campaign has ended. Africa is littered with ‘Aid Experiments’ that have failed. And to successfully develop somewhere takes decades not 3 years.

I will suggest that when the current ‘project’ comes to an end in 2010 that Amref tries to make a number of verifiable predictions about what will happen to the various achievements to date, over the next three years (ie by 2013).

This does not mean we expect Amref staff to be clairvoyant. Just that they try to make a reasonable best guess assessment of what is likely to happen.

I have asked the Guardian to consider committing myself (or another independent evaluator) to a follow-up visit and review of Katine three years after the end of the current project (ie in 2013). As well as looking for unexpected changes, that visit should also try to assess to what extent Amref’s predictions have been correct or not.