23 September, 2009
On Tuesday 22nd September Joseph Malinga wrote an article titled
Building work begins on Katine produce store: Katine farmers dig the foundations of a produce store that should help to improve livelihoods in the sub-county
I posted a Comment as follows:
Negotiated agreements on AMREF-assisted developments
It is good to see the issue of uncertain and conflicting expectations being dealt with in this article about a specific development activity assisted by AMREF: the building of a grain store for use by farmers groups in Katine. The need for written agreements about expectations has been raised by the farmers and AMREF has responded to (or has anticipated) this need. The only question in my mind, and perhaps others, was whether this agreement should have been negotiated and signed, before the commencement of the work on the store. On the other hand, a positive feature of this agreement is the willingness of government to make its contribution, along with that of the farmers groups (though this has its risks as well). And the agreement does not seem to be one sided in its expectations. Such agreements could go further than specifying the inputs each will provide, and management arrangements once completed. Reference could also be made to the objectives of this investment, to help forestall any future misuse of the store, e.g. the private use of the store by one of the farmers group members, or someone else completely. How ever the agreement does develop, it would be good if AMREF could share this example agreement via the Guardian website.
In my August 2009 comments on the future of the KCPP I had suggested that Associated with this clarification of expectations, agreements need to be developed that will spell out not only what AMREF will provide, but also what communities will provide, AND what the government will provide. Multiple agreements may be needed, perhaps component by component. One generic agreement will probably not work, because responsibilities will become too generalised and fuzzy. My proposal did not go far enough, agreements about individual developments, such as the grain store, are better still. They are smaller in ambit, and more manageable.
“When the community owns the project Giving the community a say in how their schools are built seems to be the most sensible option, says Joseph Malinga”
21 September, 2009
This is the headline of a recent post on the Guardian Katine website, by Joseph Malinga, a Guardian journalist based in Katine.
I posted the following reply…
Could Joseph Malinga do a follow-up to this story explaining what the government (local and/or central) is doing to support community initiatives like this one? Joseph’s story describes what the community is contributing and what AMREF is providing but there does not seem to be any matching contribution from the government. The government appointed head teacher was there already. The only government contribution so far seem to be doubt, about whether the quality of the school building will be adequate. Yet government is supposed to be one of the partners in the Katine Partnership Project, and advocacy is reported to be an important part of AMREF’s development strategy. What contributions or commitments have AMREF secured from government, in return for the work AMREF is doing with these two community schools? For example, the provision of more government paid teachers. At present the community is paying the teachers salaries.
This is an issue I have raised in my review of the Mid-Term Review, at https://evaluatingkatine.files.wordpress.com/2009/09/a-process-review-of-the-katine-mtr.pdf
21 September, 2009
In the recent Mid-Term Review (MTR) of the KCCP some local officials questioned the amounts being spent by AMREF on management, and the amounts remaining for what they thought was most important, the construction of infrastructure, like clinics and schools in the Katine. They wanted more for infrastructure and less on management.
In her report, the MTR consultant said “ Just how much money is available is a figure that can be misrepresented but the entire sum when broken into management, transport and staff costs leaves a reasonable – but not excessive – amount for activities and work on the ground. The figure is at least 70 per cent of the total budget and this is regarded as an acceptable amount in development projects worldwide.”
In the minutes of a subsequent meeting between AMREF and its donors (Guardian and Barclays) it was reported that the Guardian “asked for costs to be streamlined however [AMREF noted that] management support costs need to be factored in since this came up as an important resource to consider in the review. The MTR recommended a 70%/30% split as common with other projects”
I have real disquite about this position for several reasons, which I will explain. The first reason is it is actually not so easy to calculate this percentage in a standardised way that is applicable across all organisations and understood by all donors. The 70/30 split is probably a common view because everyone in the aid agency world has formed the view that this is what is acceptable to everyone. In other words, it is a herd judgement. I doubt that it is a percentage that has been found through any systematic assessment of aid agency costs.
My second reason is that this crude measure of overhead costs is based on a false assumption of how aid agencies work, a view which is captured by the simplistic but appealing notion that what matters is “whether the money gets there”. In practice in most aid programs very little money or goods actually reach the hands of poor households, because that is the way projects are designed. In Katine more than 50% of the activities in the workplan are training activities, directed at different members of the community and local government. The money spent here goes on staff salaries, and allowances for other trainers, which are spent mainly in the towns. Building costs are another important part of the project. Until recently, most of the money spent on infrastructure was spent on contractors hired from Kampala. Only in the livelihoods component is there much in the way of actual transfer of project resources directly to poor households, in the form of seed supplies. But in parallel to these activities is the UWESO-assisted savings and loan project, where instead of giving people things, the aim is to help people save and make best us of the money they already have. Only in humanitarian emergencies is there any deliberate and substantial real transfer of assets to poor households. [PS: But a small number of aid agencies have been experimenting with cash transfers to poor hoursholds, in non-emergency contexts]
My third reason is that the percentage spent on management costs versus delivery of services is an efficiency measure. Notionally, the smaller the proportion spent on management the better. But this completely ignores the issue of how effective the services are that are delivered on the ground. Low overheads will not redeem poor quality services. High overheads may contribute to better services. What matters here is cost effectiveness – what can be achieved for a given unit of cost. And cost here include not just immediate costs like building materials, but also the associated management costs, and (proportionally) the costs of the managers of the managers, etc.
In a recent posting by Dan Pallota on his blog “Free the Non-Profits” he quotes some findings from America which may also apply in the UK:
In 2002 the Better Business Bureau Wise Giving Alliance commissioned a study that asked respondents what information they wanted when considering donating to a charity. Seventy-nine percent wanted to know what percentage of their donation went to charitable programs. Remarkably, only 6% wanted to know if the donation would make a difference. How can that be, you ask? Well, the media, the watchdogs, and the sector itself have done an amazing job of training the public to think that the two things are the same, i.e., that if a charity has low overhead, it must be making a difference. Major studies on the relationship between organizational strength and impact find otherwise.
My advice to AMREF, the Guardian and Barclays is to forget about the 70%/30% ratio, however it is constructed. I agree with Dan Pollota when he says the worst question to ask about charity is, “What percentage of my donation goes to the cause?”, also known as the admin:program ratio, the “efficiency” measure, or the overhead ratio. Whatever you call it, it’s hopelessly flawed, widely abused, utterly useless, a pathetic substitute for meaningful information about a nonprofit’s work, inept at exposing fraud, and a danger to human life”
Okay, then how do we best address the concerns expressed by local authorities, during the Mid-Term Review. In my process review of the MTR I made the following suggestion:
The Guardian and Barclays Bank could take a further step, and request that each six monthly narrative progress report on the KCPP include seperate sections on the activities of the AMREF London and Kampala offices and the costs they have incurred in carrying out these activities. If this step is taken, these narrative reports should then be routinely shared with the Steering Committee and Management Committee, as well as being made publicly available via the Guardian website as at present.
These additional sections would detail not only the costs incurred by different sections of AMREF (London, Nairobi, Kampala), but also what they were able to do with that money i.e. some description of their effectiveness.
There is an important larger lesson here. Aid projects like the KCPP involve long and complex supply chains, bringing funds and technical expertise to communities of concern, from distant locations. In the private sector intense effort is often invested into making every part of supply changes work as quickly and efficiently as possible. But in the world of development aid often the focus is almost wholly on the final link in the chain, the organisations delivering assistance at the grassroots level (e.g. the AMREF office in Katine). Very little attention is given to the more expensive parts of the supply chain lined up behind them. The Guardian needs to turn its journalistic attention towards the issue of supply chain costs in international aid delivery. The diagram below shows just how complex these supply chains can be, even in a modest project like the KCPP
Thick blue lines = financial transfers. Broken blue lines = information transfers (not including most of those between yellow nodes (intermediaries between donors and recipients))
For further reading see Dan Pollota’s posts on
The Worst Question to Ask About Charity 9:44 AM Tuesday June 16, 2009
“Efficiency” Measures Miss the Point 3:56 PM Monday June 22, 2009
Beware of Highly “Efficient” Charities 10:44 AM Monday June 29, 2009
Efficiency Measures Discriminate Against Lesser-Known Causes 10:40 AM Wednesday July 8, 2009
Efficiency Measures Short-Change Individual Action 2:20 PM Monday July 13, 2009
 In terms of the costs of staff time and transport costs involved
28 March, 2009
This is the headline for a story on the Guardian katine blog, where Annie Kelly discusses a report that “criticises donors, governments and NGOs for installing boreholes and wells in rural Africa without ensuring their long-term sustainability”
Four comments have been made so far, including one by AMREF and myself.
AMREF pointed out that ” In 2002, AMREF helped communities in the Kathonzweni area to develop 50 shallow wells…By 2007 — five years later, and several years after the projects end — the communities had maintained the existing wells and built 20 new ones.”
This sounds very good, almost too good.
My comment was that “It would be useful if AMREF could provide (on this Guardian website) some information on (a) how they are monitoring the functioning of the wells they have helped to establish in Katine so far, and (b) the data that has been collected. Hopefully information is available on both the functioning of the wells and the functioning of the management committees responsible for each well.
There are problems with wells in Katine, like just about everywhere else. This is to be expected. Equipment does get damaged and worn. What matters more is how long those problems remain unresolved.
There will be a mid-term review of the Katine project in a few months time, which could look at the functioning of the wells then. But that would be a once-off inquiry and the risk is that (as elsewhere) extra efforts might be made to make sure things are working well at that time. Its the monitoring that matters.
regards, rick davies, external evaluator for Katine”
28 March, 2009
This was the headline on the Guardian katine website, on Tuesday 24th March 2009. Eliza Anyangwe asked “Are short-term development projects like Katine a good idea? The Katine project in Uganda is scheduled to last for three years. But with such a short timescale can it deliver lasting change, or will it leave a bitter aftertaste?”
There were 12 comments including mine, which ran as follows:
“I have been working for aid organisations for about 28 years, and doing monitoring and evaluation of aid projects for about 19 years. Throughout this time most people I have spoken to who have been associated with aid projects have expressed the view that 3 years is not long enough to make a substantial and sustainable development impact. Especially if you are trying improve the way in which government services work. Many of the comments above seem to share this view.
In the case of the Katine project, AMREF have expressed the view that ” the Katine project really will contribute to lasting change in one of Ugandas poorest and most vulnerable regions”. The Guardian editor also seems to share this view. Are they right? Lets try to test those views.
I have already proposed to the Guardian and AMREF that at the end of the three year period AMREF, and its local partners, should make a number of testable predictions about what things will look like in Katine, in another three years time. That is six years after the project started and three years after it ended. Then the Guardian, or perhaps some more independent organisation (with no prior involvement in Katine), should fund what is called an “ex-post evaluation” team to come in and see how many of the predictions were successful, and how many have failed. And equally importantly, why some were successful and others not so. Then let those results be shared and discussed in Katine and elsewhere via the Guardian blog.
regards, rick davies, external evaluator for the Katine (KCPP) project
PS: The problem with this proposal is the uncertainty about who will be around, six years after the start of the Katine project. Will any of the Guardian journalists now engaged with the project be around and interested to see what has happened, or will they have moved on? Will any of the key people in AMREF still be around and interested, or will they have moved on? The big question seems to be how do aid (and other) organisations develop and maintain a longer term view on what they are doing at any one time.
1 September, 2008
The August 2008 visit report is accessible here
1. Recommendations about changes almost by definition do not identify the good that is present and which needs to be maintained. With this in mind the following points should be noted:
- The KCPP staff are committed to their work, and appear to work well together as a group. The management style is team oriented, which is essential if there is to be effective integration of the project components.
- There is a breadth of experience and expertise within the team e.g. the Health component Project Officer’s experience with the Yellow Star programme, and the Project Manager’s experience with training in rights- based approaches.
- There are good relationships with the district and sub-county authorities. Although the district authorities have made critical comparisons with NUSAF they are also openly appreciative of the good working relationship that they have with AMREF
- Both the material aid and training provided by the different components are appreciated by the communities concerned within Katine. There have been no substantial criticisms of their quality or relevance.
2. This section summarises some major recommendations. The main text of the report lists specific recommendations by number, where the issues arise. There are 14 sets of recommendations (28 in all). And a further seven recommendations relating to the work of CARE and UWESO in Annex E. Please read them all. Search for: Recommendations*
3. There have been significant delays in the project implementation, with the possibility of more than a quarter of the planned activities not being completed by the end of the first year (and possibly a third of the year 1 budget remaining unspent). However what is not necessary is a rush to complete activities by the end of September, simply for the sake of completing as much of the original plan on time as possible. What is important is attention to the cause of these delays. The results of this examination should be evident in more timely implementation in the October 2008 – March 2009 period. See section 3.
PS1: AMREF have since pointed out that the project workplans outline “a sequence of activities as opposed to specific deadlines” (through the workplans do assign specific activities to specific time periods). In their view delays are to be expected because AMREF is involved in community development, and thus AMREF does not set the pace on its own. However, as noted later in this report, delays have also arisen from inefficiencies of process within AMREF. The question now is how big do the delays need to be before the project donors can legitimately express concern?
PS2: Re reviewing the causes of delays, AMREF’s view is that their existing planning and review processes are adequate
4. The importance of particular project activities depends on their place within a wider strategic view of how the project will achieve its ends. A project with many activities but little strategy could easily be accused of gap-filling, and little more. This is a serious criticism that can be made of many aid projects, and one that AMREF needs to be able to respond to KCPP. The issue of KCPP’s strategy is discussed in section 8, on the project’s theory-of-change (ToC). One high level statement of KCPP’s ToC is that the Development of Models + Creation of Demand = Development impact. This report argues (in 3.5 and 8) that the project’s strategy for the encouragement of public demand (e.g. for access to quality health, education and water services) is underdeveloped and needs attention including greater use of communication channels outside of the government administration (e.g. via mass media and elected representatives at different levels). The approach to the “development of models” also needs to be articulated more explicitly, because the options here are more complex than appear at first glance. In section 8.4 a number of different types of model creation have been identified. What would now be useful is clarification from AMREF on which of these are important in the KCPP. See section 8.
PS: AMREF’s view here is that its corporate (10 year) and Uganda country strategy provide a sufficient strategic guidance to the KCPP
5. Another high level statement of KCPP’s ToC is that the Community Partnering + Capacity Building + Operations Research and Advocacy = Development Impact. There is ample evidence that Community Partnering and Capacity Building are already taking place. What is not yet present is sufficient investment in the analysis of the efficiency and effectiveness of the work being by the KCPP, which could then be used by staff and other stakeholders to advocate the use of certain development approaches (aka models). Doing so would be consistent with AMREF’s stated support for evidence-based advocacy (versus “assumption based advocacy”). Progress with operations research and advocacy would be evident in (a) dedicated activities and budget allocations for this work within each of the components’ workplans, as well as for the M&E officer, and (b) documentation of progress made with advocacy efforts. See section 8.
PS: AMREF has commented that “AMREF is in the process of developing an advocacy strategy for the Katine project. The advocacy strategy will provide guidelines on how the project will gather the evidence and utilise it to influence policies and best practices at the local and national levels through the various foras that AMREF participates in such as the health policy action committee.”
6. Two important sources of risk for the project have been identified during this July visit. One is external, and one internal. The external risk is that the KCPP will face significant competition for attention from other development projects, such as NUSAF. NUSAF is more decentralised in its management, and seen to be more cost-effective by Soroti district officials. With the beginning of NUSAF Phase 2 District and sub-county officials may be less open to the adoption of KCPP models, unless the KCPP engages in serious evidence-based advocacy about the value of what it is doing. There are also other development projects elsewhere in Uganda which will be competing for national and international attention along with the KCPP. AMREF needs to identify how the KCPP approach is different and what difference these differences make (i.e. talking about its theory-of-change in practical terms) (See section 8.3). It needs to be able to show how it is adding value in ways that would not be possible if the same aid funds were to flow through different channels (either government or non-government).
PS: Re comparisons with other projects, AMREF have said “However we would like to desist from direct comparison with NUSAF because no formal comparable review has been done. ” Although AMREF may not wish to have comparisons made, other stakeholders are likely to persist in doing so, throughout the life of the project, and perhaps that reality needs to be acknowledged and responded to. This accommodation may have already begun, in that AMREF have also said that it “acknowledges issues raised by the district and sub-county authority and aims to take their comments on board. As part of the project steering committee agenda, AMREF will aim to do a comparative financial analysis of the project’s interventions vis a vis funds channelled through Katine sub-county and Soroti district.”
7. A significant internal risk to project success is the degree of centralised management within the Uganda country program. Expenditures over UGX 100,000 (£33.50) .need to be approved by the Kampala office, whereas the Sub-county chief can authorise up to UGX 30 million. Quarterly workplans by all KCPP staff need to be approved by the Kampala office. Both processes take more time than more localised decision making. Strategies for communications and advocacy apparently need to be developed at the country level first. Country office staff represent the KCPP in both the monthly Project Management Committee and the quarterly Project Steering Committee meetings held in Katine and Soroti respectively. All these processes have implications for the efficiency, effectiveness and credibility of AMREF’s work in Katine. It should be noted that concerns about the degree of centralisation have been made by members of the District Steering Committee, Sub-county Project Management Committee members, KCPP staff and Guardian visitors. There is, in my view, a strong argument in a favour of AMREF HQ undertaking a review of existing practices Where it is decided that specific processes do need to remain centralised then there should be clear and transparent performance targets for their completion times. See section 3.7.
PS: AMREF’s response is that they use “standard management processes for Katine and all the other 23 projects in Uganda. These provide efficiency through scale as well as accountability that would be expensive and burdensome to replicate (sic) separately for every project. Moreover, these processes have been developed to support implementation across AMREF, based on our 51 years of work in Africa.” They also note that “The PMC and SC were established through participatory discussions with all the stakeholders and are subject to review according to local needs.
8. Within the project there are two areas of strategy which need particular attention. In the health component the investment in upgrading of health centre facilities and training of village health teams has involved a de facto assumption that the necessary drug supplies will be available, especially anti-malarials. Reports from the VHTs and Health Centre staff indicate otherwise, than in fact shortage of anti-malarials is a major problem. This is in the context where the AMREF’s 2007 needs assessment identified malaria as the single main cause of morbidity and mortality. This is clearly one area where a research and advocacy strategy needs to be developed as an integral part of the KCPP approach. See section 3.1.
PS: AMREF has commented that “Drug stock outs are a common problem in Uganda and AMREF has taken this as an advocacy issue. In another project in Uganda, AMREF is undertaking an operations research into the issue of drug supply chain. The results of this research will be used as evidence for wider advocacy and policy influencing and will benefit KCCP. AMREF also has other malaria specific projects in Uganda that are addressing similar issues.”
9 The governance component also faces some challenges. On the ground in Katine the IEC component has not yet started to address the issues of “the community’s rights to basic services…”, instead pre-existing conventional IEC materials are being re-used.. The piloting of community partnering “from a rights based approach” seems to be on hold. Discussions in Kampala suggested that this office favours a conservative interpretation of how issues of empowerment and rights can be dealt with. In Katine staff seem to have a more open view. AMREF should clarify its position by developing a strategy statement saying how its existing (or revised) governance activities in Katine will help achieve the governance objectives as stated in the project proposal. This strategy statement should be shared and discussed with its donors. See section 3.5.
PS: AMREF have confirmed that “Rights Based Approach (RBA) is prioritised as a core implementation strategy in the AMREF global and country strategic plans. The RBA model is being developed and rolled out across Africa since this is a relatively new area for AMREF.”…”Based on the agreed project design, the community empowerment component is broader than RBA. Progress on governance is based on the capacity of communities to engage with their decision and policy makers. AMREF’s role is to catalyze community action since taking on that role would disempower the community”.
10 Both of the two problems identified above may be reflections of what could be called a strategy gap in the current design of the KCPP. That is the absence of documented thinking about how activities in each component should connect causally to the project objectives of that component. The recent introduction of a new planning format for proposed activities may help fill this gap, if it asks the right questions. If not, they could be a form of micro-management.
11. The project has its own objectives and indicators, and data on some of these indicators is now being collected, in addition to the work done on the baseline survey. However further steps are needed. The definition of specific levels of achievement on the various project indicators is essential, if the success of the project is to be evaluable. In the longer term, the sustainability of all project aided activities will be dependent of capable monitoring of those activities by the relevant community groups and government structures (e.g. PTAs, SMCs, PDCs, VSLAs, etc). This is more likely to happen if KCPP objectives (especially the associated targets) are closely aligned with those of its local government and community partners. Although some progress is being made with identifying targeted levels of achievement on indicators of shared concern, this work does need to be finalised soon, preferably by the end of this year. See section 4.
PS: AMREF believes the existing comprehensive M & E framework for the KCCP is adequate, but that “it will be updated periodically on the basis based on new project information and in response to community priorities”.
12. Regarding transparency, AMREF has been open in its dealings with local government, though local government would like to see more information on costs of activities. District and sub-county authorities have in turn expressed willingness to share information on their budgets, an offer worth pursuing. Although the importance of transparency seems to be well recognised in the KCPP office more attention does need to be given to improving transparency with the public at large in Katine, both about AMREF activities and the work of its partners. Simple measures could be useful, such as robust public notice boards at schools, health centres, markets and sub-county offices showing meeting minutes, survey results, annotated maps, and AMREF’s own plans. The use of talk-back radio has also been suggested. Internationally, progress with greater transparency via the AMREF websites has been slow, but with a major improvement during my visit, hopefully now to be sustained. See section 7.
13. Transparency will not be painless. It will almost inevitably bring with it some critical judgements of how well the KCPP is doing, when compared to others. For example, decentralised projects like NUSAF, and activities by smaller NGOs with lower overheads. Ideally AMREF will respond positively to these challenges, and provide in turn a demanding comparator for other development projects. This expectation is in effect the external evaluator’s theory-of-change: about the relationship between transparency, the value of comparisons and increased aid effectiveness.
PS: AMREF has replied that it is “confident that our development model is rigorous enough to enable us to engage in informed development debates with other likeminded organisations.”
 Northern Uganda Social Action Fund. NUSAF Phase 2 is about to start in January 2009
 Including those announced by President Museveni, in his recent visit to Soroti.
 Information, Education and Communication
 There are three radio stations in the area, and many households have radios.
20 May, 2008
I posted the following comments:
“I was pleased to see the article about the recent stakeholder meeting held in Katine (“Stakeholders question timescale of Katine project”) and to be able to access online a record of the meeting (“Katine stakeholder meeting and Preliminary project steering committee meeting April 3rd 2008″)
I would hope that further meetings like this are also shared on the Katine website. In the process, I hope that some more detail is provided on the participants who are not government representatives. It seemed that in this meeting almost all participants were government officials or representatives. If that was meant to be so, it would be useful to make that clear. Similarly, records of future meetings should be very clear on who they were meant to include.
My main reason for emphasising this point is that one of the evaluation criteria I will be using (as the external monitor/evaluator) is equity. This means fairness of process as well as fairness of result. There is a second dimension here as well, that of transparency. If a record is kept of the participants in such meetings it will be possible for myself, the Guardian journalist (Richard Kavuma) and anyone else, to make follow up contact with the meeting participants later on, both to hear their views of the meeting, and of what has happened thereafter.”