In the recent Mid-Term Review (MTR) of the KCCP some local officials questioned the amounts being spent by AMREF on management, and the amounts remaining for what they thought was most important, the construction of infrastructure, like clinics and schools in the Katine. They wanted more for infrastructure and less on management.

In her report, the MTR consultant said “ Just how much money is available is  a figure that can be misrepresented but the entire sum when broken into management, transport and staff costs leaves a reasonable – but not excessive – amount for activities and work on the ground.  The figure is at least 70 per cent of the total budget and this is regarded as an acceptable amount in development projects worldwide.”

In the minutes of a subsequent meeting between AMREF and its donors (Guardian and Barclays) it was reported that the Guardian “asked for costs to be streamlined however [AMREF noted that] management support costs need to be factored in since this came up as an important resource to consider in the review. The MTR recommended a 70%/30% split as common with other projects”

I have real disquite about this position for several reasons, which I will explain. The first reason is it is actually not so easy to calculate this percentage in a standardised way that is applicable across all organisations and understood by all donors. The 70/30 split is probably a common view because everyone in the aid agency world has formed the view that this is what is acceptable to everyone. In other words, it is a herd judgement. I doubt that it is a percentage that has been found through any systematic assessment of aid agency costs.

My second reason is that this crude measure of overhead costs is based on a false assumption of how aid agencies work, a view which is captured by the simplistic but appealing notion that what matters is “whether the money gets there”. In practice in most aid programs very little money or goods actually reach the hands of poor households, because that is the way projects are designed. In Katine more than 50% of the activities in the workplan are training activities, directed at different members of the community and local government. The money spent here goes on staff salaries, and allowances for other trainers, which are spent mainly in the towns. Building costs are another important part of the project. Until recently, most of the money spent on infrastructure was spent on contractors hired from Kampala. Only in the livelihoods component is there much in the way of actual transfer of project resources directly to poor households, in the form of seed supplies. But in parallel to these activities is the UWESO-assisted savings and loan project, where instead of giving people things, the aim is to help people save and make best us of the money they already have. Only in humanitarian emergencies is there any deliberate and substantial real transfer of assets to poor households. [PS: But a small number of aid agencies have been experimenting with cash transfers to poor hoursholds, in non-emergency contexts]

My third reason is that the percentage spent on management costs versus delivery of services is an efficiency measure. Notionally, the smaller the proportion spent on management the better. But this completely ignores  the issue of  how effective the services are that are delivered on the ground. Low overheads will not redeem poor quality services. High overheads may contribute to better services. What matters here is cost effectiveness – what can be achieved for a given unit of cost. And cost here include not just immediate costs like building materials, but also the associated management costs, and (proportionally) the costs of the managers of the managers, etc.

In a recent posting by Dan Pallota on his blog “Free the Non-Profits” he quotes some findings from America which may also apply in the UK:

In 2002 the Better Business Bureau Wise Giving Alliance commissioned a study that asked respondents what information they wanted when considering donating to a charity. Seventy-nine percent wanted to know what percentage of their donation went to charitable programs. Remarkably, only 6% wanted to know if the donation would make a difference. How can that be, you ask? Well, the media, the watchdogs, and the sector itself have done an amazing job of training the public to think that the two things are the same, i.e., that if a charity has low overhead, it must be making a difference. Major studies on the relationship between organizational strength and impact find otherwise.

My advice to AMREF, the Guardian and Barclays is to forget about the 70%/30% ratio, however it is constructed. I agree with Dan Pollota when he says the worst question to ask about charity is, “What percentage of my donation goes to the cause?”, also known as the admin:program ratio, the “efficiency” measure, or the overhead ratio. Whatever you call it, it’s hopelessly flawed, widely abused, utterly useless, a pathetic substitute for meaningful information about a nonprofit’s work, inept at exposing fraud, and a danger to human life

Okay, then how do we best address the concerns expressed by local authorities, during the Mid-Term Review. In my process review of the MTR I made the following suggestion:

The Guardian and Barclays Bank could take a further step, and request that each six monthly narrative progress report on the KCPP include seperate sections on the activities of the AMREF London and Kampala offices and the costs they have incurred in carrying out these activities. If this step is taken, these narrative reports should then be routinely shared with the Steering Committee and Management Committee, as well as being made publicly available via the Guardian website as at present.

These additional sections would detail not only the costs incurred by different sections of AMREF (London, Nairobi, Kampala), but also what they were able to do with that money i.e. some description of their effectiveness.

There is an important larger lesson here. Aid projects like the KCPP involve long and complex supply chains, bringing funds and technical expertise to communities of concern, from distant locations. In the private sector intense effort is often invested into making every part of supply changes work as quickly and efficiently as possible. But in the world of development aid often the focus is almost wholly on the final link in the chain, the organisations delivering assistance at the grassroots level (e.g. the AMREF office in Katine). Very little attention is given to the more expensive[1] parts of the supply chain lined up behind them. The Guardian needs to turn its journalistic attention towards the issue of supply chain costs in international aid delivery. The diagram below shows just how complex these supply chains can be, even in a modest project like the KCPP

supply network2

Thick blue lines = financial transfers. Broken blue lines = information transfers (not including most of those between yellow nodes (intermediaries between donors and recipients))

For further reading see  Dan Pollota’s posts on

The Worst Question to Ask About Charity 9:44 AM Tuesday June 16, 2009

“Efficiency” Measures Miss the Point 3:56 PM Monday June 22, 2009

Beware of Highly “Efficient” Charities 10:44 AM Monday June 29, 2009

Efficiency Measures Discriminate Against Lesser-Known Causes 10:40 AM Wednesday July 8, 2009

Efficiency Measures Short-Change Individual Action 2:20 PM Monday July 13, 2009


[1] In terms of the costs of staff time and transport costs involved


In September 2007 I suggested that AMREF should develop a Disclosure Policy to govern what sort of information it should make publicly available, and under what circumstances. Such policies have been adopted by other international organisations such as the World Bank and Action Aid.

In November 2007 AMREF’s Board approved their Open Information Policy, and it was subsequently made available on the AMREF website. ( NB: It is possible this document was already under discussion before my suggestion was made)

After reading the Open Information Policy (OIP) and associated Implementation Plan I provided feedback on a number of areas:

Positive developments

  • The OIP has been drafted, approved and made public
  • There is an associated implementation plan, with targets and responsible persons
  • There is a proposal in the OIP to annually review compliance/implementation of the policy
  • The OIP will include information on AMREF’s human resource policies
  • The intention to provide information on AMREF’s partners on AMREF’s websites. It is surprising how often they are not listed on aid agencies websites, or if they are, how hard it is to find this information

Areas for further attention

  • Although the OIP refers to public sharing of information about “Feedback: A periodic presentation of our stakeholders’ views of us and our performance through annual reports and reports of external evaluations conducted by donors, in conjunction with AMREF”, this has not yet been included in the Implementation Plan
  • The one conspicuous gap/omission in the Implementation Plan is project documentation. Possibly the most important of these would be project memorandums/agreements with donors, progress reports and evaluation reports. Given the number of these it would make sense for AMREF to do what they do in libraries when re-classifying books: start uploading this info on all new projects started this year and thereafter, and when time permits upload info on old projects (esp. those still in operation).
  • Both the household and community baseline surveys recently carried out in Katine have generated sets of data that would be of interest and value to other parties outside AMREF, both within Uganda and outside. Unfortunately it is common practice for aid organisations to not routinely make this information public. Ideally the Katine survey data should routinely be made public. The only proviso being that in the case of household survey data respondents’ names should be replaced by ID numbers, to preserve anonymity. Or, where the survey is about politically or socially sensitive issues). The village profile data should be especially useful to other NGOs working in the Katine area (8 or more). It would also be useful to external evaluators, both myself and any mid-term review teams, as a continuing record of where AMREF is doing what.
  • Given the volume of information that will be involved it will be essential that the new AMREF website has clear and usable ways that visitors can navigate around the webs site. There are at least three ways, all of which need to be offered: (a)a good key word search facility, such as that provided by Google, focused on AMREF’s own website, (b) a site map showing how the pages of the website are linked to each other, usually shown in a tree diagram form, (c) well structured pages that show what is available on other web pages e.g. by using tabbed menus at the top, etc.

Comments on the

KCPP Monitoring and Evaluation Plan Version 3. 29/11/2007

By Rick Davies, Tuesday, 25 December 2007 Version 2

1. It is good to see that the project has already developed a detailed M&E plan, and that has undergone a number of revisions.  My comments that follow below focus on areas where I think there are gaps, potential problems and potential opportunities.

2. As with the earlier Community Needs Assessment document, it is important to identify who is the author of this document. In London it has been suggested that this M&E plan was produced for AMREF by Accenture, but in the absence of author details we don’t know if this was the case or not.

3. An additional section on who participated in the development of this M&E plan would be very useful. There is a section on livelihoods (Objective 4) but it is not clear to what extent, and how, Farm Africa may have contributed to this section. Although Barclay’s plans for work on microfinance are still being developed ideally their plans for M&E of their interventions would be integrated into this plan. Even if the M&E of microfinance was discussed in a section of its own, it would still be useful to highlight how and where there would need to be some joint M&E activities with AMREF staff. Such as with the implementation of baseline surveys and mid-term reviews.

4. It is possible that this document is under-representing the participatory dimension of AMREF’s work in Katine. There is no reference to how Katine people have been involved in the design of this M&E plan. For example in the selection of appropriate indicators. Or, how they will be involved in monitoring and evaluation processes, other than survey respondents. It seems highly likely, and desirable, that AMREF will make extensive use of meetings with different stakeholders groups on a periodic and ad hoc basis, throughout the project period[1]. The plans for the use of these meetings should be described in this M&E plan.

5. The section on the Community Needs Assessment (p.3) does refer to “group discussions with a wide range of community residents in public meetings to ensure fair representations of concerns”. However, the one available report on the Community Needs Assessment provides no information on the consultation process, on the priorities that were expressed, and how priorities varied across stakeholder groups.

6. A brief mention has been made on page 5 of the M&E plan on the intention to use focus group discussions and community meetings during the mid-term and end-of-project reviews. Planning and documentation of these reviews will need to provide much more information than was provided in the summary report on the Community Needs Assessment.

7. Annex A lists the details of a very large number of indicators, on which data will be collected at various intervals. Approximately 136 in all. Of these, 41 will be collected quarterly, another 8 bi-annually, another 43 annually and another 32 at the beginning and end of the project.  My main concern here is how realistic is it for AMREF staff to aim to collect data on this many indicators?  Collecting all this data will take up valuable staff time, and analysing it will take up more time still, and so will communicating the results of this analysis.

8. There are three possible ways of rationalising this long list. The first is to ask how well the size of each set of indicators relates to the scale of AMREF’s planned investment in the objective concerned. Will AMREF be investing most effort (e.g. # of indicators) in monitoring its biggest investments, and the least in its smallest investments? The table below shows what appears to be an over-investment in monitoring the health objective, and under investment in monitoring the other objectives, especially the water and sanitation objective. Is there a good reason for this? Or, could the number of health indicators be reduced?

Objectives

% and # of indicators

Budget allocation

1. Improved community health

41% (62 )

26% (£299,000)

2. Improved access to quality education

23% (35)

26% (£295,000)

3. Improved access to safe water, sanitation and hygiene

10% (16)

18% (£205,000)

4. Improved income generating and food security opportunities

21% (32)

23% (£270,000)

5. Communities empowered to engage in local governance

5% (7)

7% (£76,000)

Total

100% (152)

100% (£1,145,000)

9. The second possibility is that the list of indicators proposed under each sub-objective could be presented to different stakeholder groups in the Katine community in the form of a menu. Those groups could then be asked to identify those changes (represented by the different listed indicators) they thought were most important to achieve. The percentages of groups choosing each indicator could then be used as an “importance” weighting for each indicator. These weightings would help in two ways. Firstly, to identify which indicators could be abandoned, because they were a low priority to most stakeholders. Secondly, to identify those indicator where analysis of progress needed to be done with most care and attention, because they were of widespread interest.  Thirdly, to guide what analysis needed to be communicated to what groups, according to their previously expressed priorities.

10. The third possibility is to look at the coverage of the indicators that have been listed in for each objective. Some focus on activities project staff will carry out, some relate to activities that involve both project staff and Katine peoples, and some focus on changes in Katine people’s lives. This corresponds approximately to a focus on Activities, Output and Purpose (outcome) levels in a Logical Framework. Ideally, with each project objective, AMREF would want to monitor a sequence of indicators that tell us about a sequence of changes, of how activities lead to outputs which then lead to outcomes (Purpose level changes). At present the mix of indicators under some of the objectives is unbalanced. Under the education objective there do not seem to be any education outcome measures like enrolment rates, attendance rates, drop out rates or graduation rates. Yet surely these are of concern to both the Katine community and AMREF. On the other hand, the health objective seems to have many outcome indicators, perhaps many more than the project could hope to significantly influence during its lifespan.

11. Annex B and C are referred to but not included. B refers to the data collection formats to be used for the Rapid Assessments, scheduled to be carried out in December 2007, and some of which have been completed. Annex C presumably refers to the survey instrument to be used for the Baseline Household Survey, scheduled for December 2007-January 2008. Copies of these instruments need to be provided not only to myself, as the external evaluator, but also to Farm Africa and Barclays, since they will need to use the data from these surveys. Ideally copies of these instruments would have been circulated to Farm Africa and Barclays before these surveys began.

12. On page 5 there is a reference to the household survey which will measure “change at the household level to see if targeted community members have benefited as a result of project activities” it is very important that the household survey also includes comparable households who have not participated in different project activities. This should not be difficult to do.

13. The M&E plan refers to an intended sample of 95 households for the household survey. The rationale for this sample size needs a clear explanation. Will the sample be intended to represent the 25,000 people of Katine sub-county, or only of Katine village itself? If the former, it looks like as very small sample[2]. If the latter, then why focus on Katine village alone. The sample size should be big enough to allow simple statistic tests of the significance of differences found between at least two categories of groups of people over two periods of time (participants and non-participants, before and after an intervention). For example, by using Chi Square test with four cells

14. It will be very useful if a modest plan could be drawn up for how the baseline survey results will be analysed. This will help inform decisions about appropriate sample sizes, and what indicators are most central to the baseline survey. This could include

  • Frequency distributions: for which indicators, displayed in what form?
  • Cross tabulations: what indicators against what indicators? There are at least 32 indicators that data will be collected on, which means more than 1000 possible cross tabulations of that data. In reality the most interesting cross-tabulations will be those that compare participation in specific project activities with changes in specific aspects of people’s lives. Differences in outcomes across different locations and groups will be especially important to analyse.

15. The M&E plan provides details on data sources for each of the indicators, but no information on who will be responsible for collecting and analysing what data. That would be a useful addition to the next version of this M&E plan. Also useful would information on what products of the M&E process will be made available to who and when. This could be presented in a Gantt chart form

16. A summary of the main recommendations made above:

  • Consider reducing the number of indicators that will be used, and ensure that in each area of AMREF’s work the selected indicators cover a sequence of expected events (i.e. from activities>outputs>outcome>impact)
  • Explain in detail the nature and rationale of the sampling strategy that will be used. Include a reference to the type of analyses that will be done with the survey results (because this affects the sample size required)
  • Provide more information on who has and who will participate in the monitoring and evaluation processes (both partners such as Farm Africa and Barclays, and various groups within the Katine community). And in what way they will participate.
  • Make this M&E Plan, of the next revised version, publicly available via the AMREF and Guardian websites

[1] In the September 25th 2007 Conceptual Framework paper reference is made to plans to work with farmers groups, VHTs, PTA’s and other community committees

[2] Even the plans for the Rapid Assessment (village component) refer to visits to 66 villages. If this is the total number of villages in Katine sub-country (or less than the total) a household survey sample of 95 households will cover less than two households per village.

Download
MS Word copy of this Comment
, written by Rick Davies, Wednesday, 02 April 2008

Objectives

1. On page 1 it is noted that “This same survey will be conducted again in 2010 in the same households to assess what improvements have occurred as a result of project interventions.” To come to such conclusions AMREF will need with-without and before-after measures of households’ wellbeing. Unfortunately, the sample does not include any “without” parishes that AMREF will not be working in. However, there will be internal differences in the extent to which parishes, villages and households participate in AMREF assisted activities. Before-after comparisons could be made of the wellbeing of these groups, contrasting those who had higher versus lower levels of participation. The household survey will enable comparisons of parishes and households (those re-surveyed), but only after some years have passed. Data on the participation of villages could be accumulated on a more continuous basis, through day to day contact by AMREF staff. Data on village level outcomes should also be available via the various groups concerned with water, sanitation, health, livelihoods and education. While this data may not be as accurate as household survey data, it will more continuously available and more immediately useful.

Differences between parishes

2. On page 5 the Summary states that “Overall baseline conditions in Katine sub county were uniformly poor with regard in access to safe water, latrine coverage, hygiene practices, prevalence of malaria and diarrhoea in young children, delivery in health facilities, food security, and use of livelihood support services” But this point seems to be contradicted later in the same summary section where differences between parishes are described in Table 2, and elsewhere noted in a number of areas, including access to drinking water, sources of treatment for malaria, and food security. As noted on in table 2 such differences have implications for “priority interventions in each parish[1] If AMREF is to provide effective development aid that is relevant to people’s needs, sensitively to differences will be essential.

National – local differences

3. The statement that “A comparison of Katine indicators with that from the 2006 Uganda Demographic and Health Survey shows that overall baseline conditions for the Katine population are worse than the average for rural areas of the country” This important claim needs to be supported by a table comparing national and local figures. This has been done with water related indicators on page 23 and HIV/AIDS knowledge on page 26.

Gender differences

4. The behaviour of men and women has been differentiated in many areas of the report e.g. school participation, water collection, and diarrhoea prevalence. However, the gender of the respondents to the questions is less clear. Four different questionnaires were used in each respondent household, one addressed to “the household”, one to men, one to women and one to caretakers of children under five years of age. It would be useful to know where (on what questions) men responding on behalf of their “household” differed in their responses from women responding on behalf of their “household”. The answers could have practical implications for how AMREF staff engage with communities on specific issues.

Key program indicators

5. The one page summary of key program indicators is useful. Somewhat surprisingly it does not include any household level education indicators, although education data was collected. Children’s education outcomes are probably the closest thing to an aggregate measure of development impact that will be available to AMREF, since educational achievement is affected by health, water, sanitation, livelihoods and governance, and all within a reasonable span of time. This is in contrast to mortality rates which usually reflect changes over taking place over longer periods of time, and which are harder to document. There are also no household level governance / empowerment indicators in the table although there is at least one in the AMREF M&E framework

6. These 25 indicators are a selection from at least 34 other household level impact indicators in the Katine M&E framework, which relate to household level wellbeing. The baseline survey in turn included many other questions about household knowledge, activities and attitudes, covered by four separate questionnaires. It would be useful to know how these 25 indicators were selected as “key” indicators.

Community engagement with the results

7. The recommendations do not make any reference to what should or could happen next, with the survey results. Given this survey is part of a development project, rather than part of another national statistics gathering operation, some form of participatory process would seem appropriate. This could include:

  • Checking the results against people’s expectations of the results. This could help highlight possible measurement problems, and identify changes that people were not aware of, and is a sure-fire way of engaging people in a discussion of the survey.
  • Prioritisation of the problems evident in the various “key indicators”, as seen by different groups within the community (men/women, young/old, etc)

Securing the data

8. Baseline survey data can easily get lost, as time moves on and priorities and interests change. AMREF needs to make good provision for the storage and backup (in other locations) of both digital and hard copies of the survey results, the survey instruments and any associated documentation, such as that concerning the sampling process and the names of staff involved.

9. Survey data such as this should be seen as a public resource which should be made publicly available. So long as simple steps are taken to secure the anonymity of the respondents. It should be possible to make the survey data files available, at minimal cost, via AMREF’s website. And then publicise their availability via relevant email lists.

For more comments on aid organisations and public goods, such as these, please see my blog posting on “Aid organisations as self-interested businesses?”[2]

[1] Though other factors would also need to be taken into account as well, such as local residents own priorities.

[2] http://mandenews.blogspot.com/2008/03/aid-organisations-as-self-interested.html

Comments on

Community Needs Assessment – Overview

Monday, 19 November 2007, Rick Davies

1. The purpose of these brief comments is to provide constructive feedback to AMREF on the Community Needs Assessment – Overview (CAN) document. These comments have not been shared with anyone but AMREF at this stage.

2. Minor but important point: Over the next few years it is likely that AMREF will produce many documents about Katine, for internal and external use. It would be helpful if all these documents had date and author information on the front cover (in addition to the AMREF label).

3. Does the title of this report mean there is going to be a full report that goes into more detail than this Overview? If so, make this clear on page 3.

4. The methodology section says that interviews with key informants “were complemented by group discussions with a wide range of community residents in public meetings to ensure fair representation of concerns“.  This statement needs to be supported by information in the Appendix on the nature of these meetings. Such as the number, type and location of different meetings e.g. 4 meetings with women’s groups in w,x,y,z locations, 4 meetings with men’s’ groups in …The Appendix already list details of the secondary data sources and the key informants. The Appendix could also note that the names of all key informants and (perhaps the group meeting participants) will be available in the full report of the Community Needs Assessment, if there is to be one.

5. Re Health (page 4), my impression is that malaria and HIV/AIDS are the two main causes of death in Katine. Is this correct? It would be useful if this page could provide a clear list of the main causes of death, in order of incidence, according to whatever information is currently available. And differentiated by gender, at least.

6. It is often the case that different members of a community have a different view on what are the priority needs in a community. The list of “Key Needs” needs to be supported by some comments on where there was agreement and disagreement within the community on these different needs. Which groups expressed what differing views on what?

7. Location is often an important factor contributing to people’s poverty. Schools and health centres may be nearer to some people in Katine than others. A simple map showing the boundaries of Katine, and the locations of schools and health services within Katine would be useful, it could highlight what areas were more and less geographically disadvantaged (in relative terms).

8. As in the health section, the education section discusses problems and possible responses to those problems. It would be useful if there was a clearer sense of the relative importance of the problems, and of the proposed responses. AMREF cannot do everything, and will have to make some choices. As above, it would also be good to see some information here on the opinions of different sections of the community on these problems and on the proposed responses. The same comments also apply to the section on livelihoods. This could be in table form: groups down the left column, problems across the top row / responses across the top row.

9. The section on community empowerment makes an important point about “lack of participation of the vulnerable in setting priorities and deciding on resource allocation“. If so, then this Overview document should itself model good practice in this area.

10. A summary section would be useful, which highlighted different community groups’ views on the relative importance on the larger categories of needs, which are each discussed in detail in this report: health, education, water, livelihood and community empowerment issues. If these are “all equally important” then does this mean that AMREF’s Katine budget will be split equally across these categories of need?

11. The same summary section could also point out where the CAN fits in the planned sequence of project activities.  How will the results of the CAN inform the design of the baseline survey referred to on page 3 (if the survey comes next)? Or, how has the baseline survey results informed the design of the CAN (if the survey came first)?

12. I would like to see this CAN document (or a re-edited version of this document) made publicly available, via the AMREF website and/or my own web page (Monitoring and Evaluating Success in Katine Frequently Asked Questions[1] (FAQs)). It would signal confidence in the approach being taken, and allow the public at large, beyond Katine, to learn more about the project.


[1] http://www.mande.co.uk/uganda/Katine%20M&E%20FAQs.htm