Negotiated agreements on AMREF-assisted developments

On Tuesday 22nd September Joseph Malinga wrote an article titled

Building work begins on Katine produce store: Katine farmers dig the foundations of a produce store that should help to improve livelihoods in the sub-county

I posted a Comment as follows:

Negotiated agreements on AMREF-assisted developments

It is good to see the issue of uncertain and conflicting expectations being dealt with in this article about a specific development activity assisted by AMREF: the building of a grain store for use by farmers groups in Katine. The need for written agreements about expectations has been raised by the farmers and AMREF has responded to (or has anticipated) this need. The only question in my mind, and perhaps others, was whether this agreement should have been negotiated and signed, before the commencement of the work on the store. On the other hand, a positive feature of this agreement is the willingness of government to make its contribution, along with that of the farmers groups (though this has its risks as well). And the agreement does not seem to be one sided in its expectations. Such agreements could go further than specifying the inputs each will provide, and management arrangements once completed. Reference could also be made to the objectives of this investment, to help forestall any future misuse of the store, e.g. the private use of the store by one of the farmers group members, or someone else completely. How ever the agreement does develop, it would be good if AMREF could share this example agreement via the Guardian website.

In my August 2009 comments on the future of the KCPP I had suggested that Associated with this clarification of expectations, agreements need to be developed that will spell out not only what AMREF will provide, but also what communities will provide, AND what the government will provide. Multiple agreements may be needed, perhaps component by component. One generic agreement will probably not work, because responsibilities will become too generalised and fuzzy. My proposal did not go far enough, agreements about individual developments, such as the grain store, are better still. They are smaller in ambit, and more manageable.

“When the community owns the project Giving the community a say in how their schools are built seems to be the most sensible option, says Joseph Malinga”

This is the headline of a recent post on the Guardian Katine website, by Joseph Malinga, a Guardian journalist based in Katine.

I posted the following reply…

Could Joseph Malinga do a  follow-up to this story explaining what the government (local and/or central) is doing to support community initiatives like this one? Joseph’s story describes what the community is contributing and what AMREF is providing but there does not seem to be any matching contribution from the government. The government appointed head teacher was there already. The only government contribution so far seem to be doubt, about whether the quality of the school building will be adequate. Yet government is supposed to be one of the partners in the Katine Partnership Project, and advocacy is reported to be an important part of AMREF’s development strategy. What contributions or commitments have AMREF secured from government, in return for the work AMREF is doing with these two community schools? For example, the provision of more government paid teachers. At present the community is paying the teachers salaries.

This is an issue I have raised in my review of the Mid-Term Review, at http://evaluatingkatine.files.wordpress.com/2009/09/a-process-review-of-the-katine-mtr.pdf

“The Worst Question to Ask About Charity”

In the recent Mid-Term Review (MTR) of the KCCP some local officials questioned the amounts being spent by AMREF on management, and the amounts remaining for what they thought was most important, the construction of infrastructure, like clinics and schools in the Katine. They wanted more for infrastructure and less on management.

In her report, the MTR consultant said “ Just how much money is available is  a figure that can be misrepresented but the entire sum when broken into management, transport and staff costs leaves a reasonable – but not excessive – amount for activities and work on the ground.  The figure is at least 70 per cent of the total budget and this is regarded as an acceptable amount in development projects worldwide.”

In the minutes of a subsequent meeting between AMREF and its donors (Guardian and Barclays) it was reported that the Guardian “asked for costs to be streamlined however [AMREF noted that] management support costs need to be factored in since this came up as an important resource to consider in the review. The MTR recommended a 70%/30% split as common with other projects”

I have real disquite about this position for several reasons, which I will explain. The first reason is it is actually not so easy to calculate this percentage in a standardised way that is applicable across all organisations and understood by all donors. The 70/30 split is probably a common view because everyone in the aid agency world has formed the view that this is what is acceptable to everyone. In other words, it is a herd judgement. I doubt that it is a percentage that has been found through any systematic assessment of aid agency costs.

My second reason is that this crude measure of overhead costs is based on a false assumption of how aid agencies work, a view which is captured by the simplistic but appealing notion that what matters is “whether the money gets there”. In practice in most aid programs very little money or goods actually reach the hands of poor households, because that is the way projects are designed. In Katine more than 50% of the activities in the workplan are training activities, directed at different members of the community and local government. The money spent here goes on staff salaries, and allowances for other trainers, which are spent mainly in the towns. Building costs are another important part of the project. Until recently, most of the money spent on infrastructure was spent on contractors hired from Kampala. Only in the livelihoods component is there much in the way of actual transfer of project resources directly to poor households, in the form of seed supplies. But in parallel to these activities is the UWESO-assisted savings and loan project, where instead of giving people things, the aim is to help people save and make best us of the money they already have. Only in humanitarian emergencies is there any deliberate and substantial real transfer of assets to poor households. [PS: But a small number of aid agencies have been experimenting with cash transfers to poor hoursholds, in non-emergency contexts]

My third reason is that the percentage spent on management costs versus delivery of services is an efficiency measure. Notionally, the smaller the proportion spent on management the better. But this completely ignores  the issue of  how effective the services are that are delivered on the ground. Low overheads will not redeem poor quality services. High overheads may contribute to better services. What matters here is cost effectiveness – what can be achieved for a given unit of cost. And cost here include not just immediate costs like building materials, but also the associated management costs, and (proportionally) the costs of the managers of the managers, etc.

In a recent posting by Dan Pallota on his blog “Free the Non-Profits” he quotes some findings from America which may also apply in the UK:

In 2002 the Better Business Bureau Wise Giving Alliance commissioned a study that asked respondents what information they wanted when considering donating to a charity. Seventy-nine percent wanted to know what percentage of their donation went to charitable programs. Remarkably, only 6% wanted to know if the donation would make a difference. How can that be, you ask? Well, the media, the watchdogs, and the sector itself have done an amazing job of training the public to think that the two things are the same, i.e., that if a charity has low overhead, it must be making a difference. Major studies on the relationship between organizational strength and impact find otherwise.

My advice to AMREF, the Guardian and Barclays is to forget about the 70%/30% ratio, however it is constructed. I agree with Dan Pollota when he says the worst question to ask about charity is, “What percentage of my donation goes to the cause?”, also known as the admin:program ratio, the “efficiency” measure, or the overhead ratio. Whatever you call it, it’s hopelessly flawed, widely abused, utterly useless, a pathetic substitute for meaningful information about a nonprofit’s work, inept at exposing fraud, and a danger to human life”

Okay, then how do we best address the concerns expressed by local authorities, during the Mid-Term Review. In my process review of the MTR I made the following suggestion:

The Guardian and Barclays Bank could take a further step, and request that each six monthly narrative progress report on the KCPP include seperate sections on the activities of the AMREF London and Kampala offices and the costs they have incurred in carrying out these activities. If this step is taken, these narrative reports should then be routinely shared with the Steering Committee and Management Committee, as well as being made publicly available via the Guardian website as at present.

These additional sections would detail not only the costs incurred by different sections of AMREF (London, Nairobi, Kampala), but also what they were able to do with that money i.e. some description of their effectiveness.

There is an important larger lesson here. Aid projects like the KCPP involve long and complex supply chains, bringing funds and technical expertise to communities of concern, from distant locations. In the private sector intense effort is often invested into making every part of supply changes work as quickly and efficiently as possible. But in the world of development aid often the focus is almost wholly on the final link in the chain, the organisations delivering assistance at the grassroots level (e.g. the AMREF office in Katine). Very little attention is given to the more expensive[1] parts of the supply chain lined up behind them. The Guardian needs to turn its journalistic attention towards the issue of supply chain costs in international aid delivery. The diagram below shows just how complex these supply chains can be, even in a modest project like the KCPP

Thick blue lines = financial transfers. Broken blue lines = information transfers (not including most of those between yellow nodes (intermediaries between donors and recipients))

For further reading see  Dan Pollota’s posts on

The Worst Question to Ask About Charity 9:44 AM Tuesday June 16, 2009

“Efficiency” Measures Miss the Point 3:56 PM Monday June 22, 2009

Beware of Highly “Efficient” Charities 10:44 AM Monday June 29, 2009

Efficiency Measures Discriminate Against Lesser-Known Causes 10:40 AM Wednesday July 8, 2009

Efficiency Measures Short-Change Individual Action 2:20 PM Monday July 13, 2009

---

[1] In terms of the costs of staff time and transport costs involved

Money ‘wasted’ on water projects in Africa

This is the headline for a story on the Guardian katine blog, where Annie Kelly discusses a report that “criticises donors, governments and NGOs for installing boreholes and wells in rural Africa without ensuring their long-term sustainability”

Four comments have been made so far, including one by AMREF and myself.

AMREF pointed out that ” In 2002, AMREF helped communities in the Kathonzweni area to develop 50 shallow wells…By 2007 — five years later, and several years after the projects end — the communities had maintained the existing wells and built 20 new ones.”

This sounds very good, almost too good.

My comment was that “It would be useful if AMREF could provide (on this Guardian website) some information on (a) how they are monitoring the functioning of the wells they have helped to establish in Katine so far, and (b) the data that has been collected. Hopefully information is available on both the functioning of the wells and the functioning of the management committees responsible for each well.

There are problems with wells in Katine, like just about everywhere else. This is to be expected. Equipment does get damaged and worn. What matters more is how long those problems remain unresolved.

There will be a mid-term review of the Katine project in a few months time, which could look at the functioning of the wells then. But that would be a once-off inquiry and the risk is that (as elsewhere) extra efforts might be made to make sure things are working well at that time. Its the monitoring that matters.

regards, rick davies, external evaluator for Katine”

Are short-term development projects like Katine a good idea?

This was the headline on the Guardian katine website, on Tuesday 24th March 2009.  Eliza Anyangwe asked “Are short-term development projects like Katine a good idea? The Katine project in Uganda is scheduled to last for three years. But with such a short timescale can it deliver lasting change, or will it leave a bitter aftertaste?”

There were 12 comments including mine, which ran as follows:

“I have been working for aid organisations for about 28 years, and doing monitoring and evaluation of aid projects for about 19 years. Throughout this time most people I have spoken to who have been associated with aid projects have expressed the view that 3 years is not long enough to make a substantial and sustainable development impact. Especially if you are trying improve the way in which government services work. Many of the comments above seem to share this view.

In the case of the Katine project, AMREF have expressed the view that ” the Katine project really will contribute to lasting change in one of Ugandas poorest and most vulnerable regions”. The Guardian editor also seems to share this view. Are they right? Lets try to test those views.

I have already proposed to the Guardian and AMREF that at the end of the three year period AMREF, and its local partners, should make a number of testable predictions about what things will look like in Katine, in another three years time. That is six years after the project started and three years after it ended. Then the Guardian, or perhaps some more independent organisation (with no prior involvement in Katine), should fund what is called an “ex-post evaluation” team to come in and see how many of the predictions were successful, and how many have failed. And equally importantly, why some were successful and others not so. Then let those results be shared and discussed in Katine and elsewhere via the Guardian blog.

regards, rick davies, external evaluator for the Katine (KCPP) project

PS: The problem with this proposal is the uncertainty about who will be around, six years after the start of the Katine project. Will any of the Guardian journalists now engaged with the project be around and interested to see what has happened, or will they have moved on? Will any of the key people in AMREF still be around and interested, or will they have moved on? The big question seems to be how do aid (and other) organisations develop and maintain a longer term view on what they are doing at any one time.

August 2008 Visit Report by the external evaluator

The August 2008 visit report is accessible here

• Main report
• The annexes

1. Summary of issues and main recommendations

1.      Recommendations about changes almost by definition do not identify the good that is present and which needs to be maintained. With this in mind the following points should be noted:

• The KCPP staff are committed to their work, and appear to work well together as a group. The management style is team oriented, which is essential if there is to be effective integration of the project components.
• There is a breadth of experience and expertise within the team e.g. the Health component Project Officer’s experience with the Yellow Star programme, and the Project Manager’s experience with training in rights- based approaches.
• There are good relationships with the district and sub-county authorities. Although the district authorities have made critical comparisons with NUSAF they are also openly appreciative of the good working relationship that they have with AMREF
• Both the material aid and training provided by the different components are appreciated by the communities concerned within Katine. There have been no substantial criticisms of their quality or relevance.

2.      This section summarises some major recommendations. The main text of the report lists specific recommendations by number, where the issues arise. There are 14 sets of recommendations (28 in all). And a further seven recommendations relating to the work of CARE and UWESO in Annex E. Please read them all. Search for: Recommendations*

3.      There have been significant delays in the project implementation, with the possibility of more than a quarter of the planned activities not being completed by the end of the first year (and possibly a third of the year 1 budget remaining unspent). However what is not necessary is a rush to complete activities by the end of September, simply for the sake of completing as much of the original plan on time as possible. What is important is attention to the cause of these delays.  The results of this examination should be evident in more timely implementation in the October 2008 – March 2009 period. See section 3.

PS1: AMREF have since pointed out that the project workplans outline “a sequence of activities as opposed to specific deadlines” (through the workplans do assign specific activities to specific time periods). In their view delays are to be expected because AMREF is involved in community development, and thus AMREF does not set the pace on its own. However, as noted later in this report, delays have also arisen from inefficiencies of process within AMREF. The question now is how big do the delays need to be before the project donors can legitimately express concern?

PS2: Re reviewing the causes of delays, AMREF’s view is that their existing planning and review processes are adequate

4.      The importance of particular project activities depends on their place within a wider strategic view of how the project will achieve its ends. A project with many activities but little strategy could easily be accused of gap-filling, and little more. This is a serious criticism that can be made of many aid projects, and one that AMREF needs to be able to respond to KCPP. The issue of KCPP’s strategy is discussed in section 8, on the project’s theory-of-change (ToC).  One high level statement of KCPP’s ToC is that the Development of Models + Creation of Demand = Development impact. This report argues (in 3.5 and 8) that the project’s strategy for the encouragement of public demand (e.g. for access to quality health, education and water services) is underdeveloped and needs attention including greater use of communication channels outside of the government administration (e.g. via mass media and elected representatives at different levels).  The approach to the “development of models” also needs to be articulated more explicitly, because the options here are more complex than appear at first glance. In section 8.4 a number of different types of model creation have been identified. What would now be useful is clarification from AMREF on which of these are important in the KCPP. See section 8.

PS: AMREF’s view here is that its corporate (10 year) and Uganda country strategy provide a sufficient strategic guidance to the KCPP

5. Another high level statement of KCPP’s  ToC is that the Community Partnering + Capacity Building + Operations Research and Advocacy = Development Impact. There is ample evidence that Community Partnering and Capacity Building are already taking place. What is not yet present is sufficient investment in the analysis of the efficiency and effectiveness of the work being by the KCPP, which could then be used by staff and other stakeholders to advocate the use of certain development approaches (aka models). Doing so would be consistent with AMREF’s stated support for evidence-based advocacy (versus “assumption based advocacy”).  Progress with operations research and advocacy would be evident in (a) dedicated activities and budget allocations for this work within each of the components’ workplans, as well as for the M&E officer, and (b) documentation of progress made with advocacy efforts. See section 8.

PS: AMREF has commented that “AMREF is in the process of developing an advocacy strategy for the Katine project. The advocacy strategy will provide guidelines on how the project will gather the evidence and utilise it to influence policies and best practices at the local and national levels through the various foras that AMREF participates in such as the health policy action committee.”

6. Two important sources of risk for the project have been identified during this July visit. One is external, and one internal. The external risk is that the KCPP will face significant competition for attention from other development projects, such as NUSAF[1]. NUSAF is more decentralised in its management, and seen to be more cost-effective by Soroti district officials[2].  With the beginning of NUSAF Phase 2 District and sub-county officials may be less open to the adoption of KCPP models, unless the KCPP engages in serious evidence-based advocacy about the value of what it is doing.  There are also other development projects elsewhere in Uganda which will be competing for national and international attention along with the KCPP[3]. AMREF needs to identify how the KCPP approach is different and what difference these differences make (i.e. talking about its theory-of-change in practical terms) (See section 8.3). It needs to be able to show how it is adding value in ways that would not be possible if the same aid funds were to flow through different channels (either government or non-government).

PS: Re comparisons with other projects, AMREF have said “However we would like to desist from direct comparison with NUSAF because no formal comparable review has been done. ” Although AMREF may not wish to have comparisons made, other stakeholders are likely to persist in doing so, throughout the life of the project, and perhaps that reality needs to be acknowledged and responded to. This accommodation may have already begun, in that AMREF have also said that it “acknowledges issues raised by the district and sub-county authority and aims to take their comments on board. As part of the project steering committee agenda, AMREF will aim to do a comparative financial analysis of the project’s interventions vis a vis funds channelled through Katine sub-county and Soroti district.”

7. A significant internal risk to project success is the degree of centralised management within the Uganda country program.  Expenditures over UGX 100,000 (£33.50) .need to be approved by the Kampala office, whereas the Sub-county chief can authorise up to UGX 30 million. Quarterly workplans by all KCPP staff need to be approved by the Kampala office. Both processes take more time than more localised decision making. Strategies for communications and advocacy apparently need to be developed at the country level first. Country office staff represent the KCPP in both the monthly Project Management Committee and the quarterly Project Steering Committee meetings held in Katine and Soroti respectively. All these processes have implications for the efficiency, effectiveness and credibility of AMREF’s work in Katine.  It should be noted that concerns about the degree of centralisation have been made by members of the District Steering Committee, Sub-county Project Management Committee members, KCPP staff and Guardian visitors. There is, in my view, a strong argument in a favour of AMREF HQ undertaking a review of existing practices  Where it is decided that specific processes do need to remain centralised then there should be clear and transparent performance targets for their completion times. See section 3.7.

PS: AMREF’s response is that they use “standard management processes for Katine and all the other 23 projects in Uganda. These provide efficiency through scale as well as accountability that would be expensive and burdensome to replicate (sic) separately for every project. Moreover, these processes have been developed to support implementation across AMREF, based on our 51 years of work in Africa.” They also note that “The PMC and SC were established through participatory discussions with all the stakeholders and are subject to review according to local needs.

8. Within the project there are two areas of strategy which need particular attention. In the health component the investment in upgrading of health centre facilities and training of village health teams has involved a de facto assumption that the necessary drug supplies will be available, especially anti-malarials. Reports from the VHTs and Health Centre staff indicate otherwise, than in fact shortage of anti-malarials is a major problem. This is in the context where the AMREF’s 2007 needs assessment identified malaria as the single main cause of morbidity and mortality. This is clearly one area where a research and advocacy strategy needs to be developed as an integral part of the KCPP approach. See section 3.1.

PS: AMREF has commented that “Drug stock outs are a common problem in Uganda and AMREF has taken this as an advocacy issue.  In another project in Uganda, AMREF is undertaking an operations research into the issue of drug supply chain. The results of this research will be used as evidence for wider advocacy and policy influencing and will benefit KCCP. AMREF also has other malaria specific projects in Uganda that are addressing similar issues.”

9 The governance component also faces some challenges. On the ground in Katine the IEC[4] component has not yet started to address the issues of “the community’s rights to basic services…”, instead pre-existing conventional IEC materials are being re-used.. The piloting of community partnering “from a rights based approach” seems to be on hold. Discussions in Kampala suggested that this office favours a conservative interpretation of how issues of empowerment and rights can be dealt with. In Katine staff seem to have a more open view. AMREF should clarify its position by developing a strategy statement saying how its existing (or revised) governance activities in Katine will help achieve the governance objectives as stated in the project proposal. This strategy statement should be shared and discussed with its donors. See section 3.5.

PS: AMREF have confirmed that “Rights Based Approach (RBA) is prioritised as a core implementation strategy in the AMREF global and country strategic plans. The RBA model is being developed and rolled out across Africa since this is a relatively new area for AMREF.”…”Based on the agreed project design, the community empowerment component is broader than RBA. Progress on governance is based on the capacity of communities to engage with their decision and policy makers. AMREF’s role is to catalyze community action since taking on that role would disempower the community”.

10 Both of the two problems identified above may be reflections of what could be called a strategy gap in the current design of the KCPP. That is the absence of documented thinking about how activities in each component should connect causally to the project objectives of that component. The recent introduction of a new planning format for proposed activities may help fill this gap, if it asks the right questions. If not, they could be a form of micro-management.

11.  The project has its own objectives and indicators, and data on some of these indicators is now being collected, in addition to the work done on the baseline survey. However further steps are needed. The definition of specific levels of achievement on the various project indicators is essential, if the success of the project is to be evaluable. In the longer term, the sustainability of all project aided activities will be dependent of capable monitoring of those activities by the relevant community groups and government structures (e.g. PTAs, SMCs, PDCs, VSLAs, etc). This is more likely to happen if KCPP objectives (especially the associated targets) are closely aligned with those of its local government and community partners. Although some progress is being made with identifying targeted levels of achievement on indicators of shared concern, this work does need to be finalised soon, preferably by the end of this year. See section 4.

PS: AMREF believes the existing comprehensive M & E framework for the KCCP is adequate, but that “it will be updated periodically on the basis based on new project information and in response to community priorities”.

12. Regarding transparency, AMREF has been open in its dealings with local government, though local government would like to see more information on costs of activities. District and sub-county authorities have in turn expressed willingness to share information on their budgets, an offer worth pursuing. Although the importance of transparency seems to be well recognised in the KCPP office more attention does need to be given to improving transparency with the public at large in Katine, both about AMREF activities and the work of its partners. Simple measures could be useful, such as robust public notice boards at schools, health centres, markets and sub-county offices showing meeting minutes, survey results, annotated maps, and AMREF’s own plans. The use of talk-back radio has also been suggested[5]. Internationally, progress with greater transparency via the AMREF websites has been slow, but with a major improvement during my visit, hopefully now to be sustained. See section 7.

13. Transparency will not be painless.  It will almost inevitably bring with it some critical judgements of how well the KCPP is doing, when compared to others. For example, decentralised projects like NUSAF, and activities by smaller NGOs with lower overheads. Ideally AMREF will respond positively to these challenges, and provide in turn a demanding comparator for other development projects. This expectation is in effect the external evaluator’s theory-of-change: about the relationship between transparency, the value of comparisons and increased aid effectiveness.

PS: AMREF has replied that it is “confident that our development model is rigorous enough to enable us to engage in informed development debates with other likeminded organisations.”

Footnotes:

[1] Northern Uganda Social Action Fund. NUSAF Phase 2 is about to start in January 2009

[2] But not without its own problems e.g. corruption. See Sh2.5b NUSAF funds missing New Vision, Thursday, 10th July, 2008 http://www.newvision.co.ug/D/8/12/638475

[3] Including those announced by President Museveni, in his recent visit to Soroti.

[4] Information, Education and Communication

[5] There are three radio stations in the area, and many households have radios.

Katine stakeholder meeting and preliminary project steering committee meeting: April 3rd, 2008

This AMREF document is available on the Guardian newspaper Katine website, in an article titled “Stakeholders question timescale of Katine project“

I posted the following comments:

“I was pleased to see the article about the recent stakeholder meeting held in Katine (“Stakeholders question timescale of Katine project”) and to be able to access online a record of the meeting (“Katine stakeholder meeting and Preliminary project steering committee meeting April 3rd 2008″)

I would hope that further meetings like this are also shared on the Katine website. In the process, I hope that some more detail is provided on the participants who are not government representatives. It seemed that in this meeting almost all participants were government officials or representatives. If that was meant to be so, it would be useful to make that clear. Similarly, records of future meetings should be very clear on who they were meant to include.

My main reason for emphasising this point is that one of the evaluation criteria I will be using (as the external monitor/evaluator) is equity. This means fairness of process as well as fairness of result. There is a second dimension here as well, that of transparency. If a record is kept of the participants in such meetings it will be possible for myself, the Guardian journalist (Richard Kavuma) and anyone else, to make follow up contact with the meeting participants later on, both to hear their views of the meeting, and of what has happened thereafter.”

AMREF’s Open Information Policy

In September 2007 I suggested that AMREF should develop a Disclosure Policy to govern what sort of information it should make publicly available, and under what circumstances. Such policies have been adopted by other international organisations such as the World Bank and Action Aid.

In November 2007 AMREF’s Board approved their Open Information Policy, and it was subsequently made available on the AMREF website. ( NB: It is possible this document was already under discussion before my suggestion was made)

After reading the Open Information Policy (OIP) and associated Implementation Plan I provided feedback on a number of areas:

Positive developments

• The OIP has been drafted, approved and made public
• There is an associated implementation plan, with targets and responsible persons
• There is a proposal in the OIP to annually review compliance/implementation of the policy
• The OIP will include information on AMREF’s human resource policies
• The intention to provide information on AMREF’s partners on AMREF’s websites. It is surprising how often they are not listed on aid agencies websites, or if they are, how hard it is to find this information

Areas for further attention

• Although the OIP refers to public sharing of information about “Feedback: A periodic presentation of our stakeholders’ views of us and our performance through annual reports and reports of external evaluations conducted by donors, in conjunction with AMREF”, this has not yet been included in the Implementation Plan
• The one conspicuous gap/omission in the Implementation Plan is project documentation. Possibly the most important of these would be project memorandums/agreements with donors, progress reports and evaluation reports. Given the number of these it would make sense for AMREF to do what they do in libraries when re-classifying books: start uploading this info on all new projects started this year and thereafter, and when time permits upload info on old projects (esp. those still in operation).
• Both the household and community baseline surveys recently carried out in Katine have generated sets of data that would be of interest and value to other parties outside AMREF, both within Uganda and outside. Unfortunately it is common practice for aid organisations to not routinely make this information public. Ideally the Katine survey data should routinely be made public. The only proviso being that in the case of household survey data respondents’ names should be replaced by ID numbers, to preserve anonymity. Or, where the survey is about politically or socially sensitive issues). The village profile data should be especially useful to other NGOs working in the Katine area (8 or more). It would also be useful to external evaluators, both myself and any mid-term review teams, as a continuing record of where AMREF is doing what.
• Given the volume of information that will be involved it will be essential that the new AMREF website has clear and usable ways that visitors can navigate around the webs site. There are at least three ways, all of which need to be offered: (a)a good key word search facility, such as that provided by Google, focused on AMREF’s own website, (b) a site map showing how the pages of the website are linked to each other, usually shown in a tree diagram form, (c) well structured pages that show what is available on other web pages e.g. by using tabbed menus at the top, etc.

January 2008 Visit Report by the external monitor

The January 2008 visit report is accessible here

• The main report
• The annexes

The Summary of Recommendations

1. The final objectives of the project need clarification and agreement, by AMREF, its donors and local stakeholders. This agreement should be evident in a smaller set of indicators that show changes in people’s lives, which reflect the impact of all five project components, and which can be easily be monitored by community groups.
2. Working with a range of local community groups, who are linked into government structures, is at the core of the KCPP strategy for achieving sustainable improvements to people’s lives in Katine. Ways of tracking overall improvements in the performance of these groups need to be developed, in consultation with them.
3. The scale of project investment varies substantially across components, but in at least three components there are not enough resources to provide all villages with at least some minimal level of assistance (however defined). The livelihoods component seems especially under-invested. A strategy needs to be developed to either obtain expanded funding, or to ensure replication of AMREF activities in unassisted villages by other parties.
4. The key to the long term sustainability of the achievements of the project is likely to be the work done on governance, both of the assisted community groups, and how they relate to the wider structures of government in the district. This is the area of the project most in need of technical back up, possibly by a third party, as is the case with Farm Africa and the livelihoods activities.
5. In the longer term, AMREF needs to be able to provide evidence of about two types of change processes, that ideally would then be replicable elsewhere • What type of packages of support can best lead to desired improvements2 in the functioning of particular community groups• How the functioning of specific community groups can lead to particular improvements in the quality of peoples lives.
6. Progress has been made with the development of an Open Information Policy but its implementation needs to be accelerated, if it is to be seen as anything more than a gesture. Disclosure of project documents to the external evaluator has improved, but will need to fit within the time schedule of future visits to Katine.
7. The Guardian website should refer to AMREF’s “Katine Community Partnerships Project” in order to emphasise that it is an AMREF project, not a Guardian project and that the project is working through partnerships with local community organisations, not directly aiding individual households.
8. AMREF’s participation in the Guardian website faces challenges. Cut-off time for comments on postings is too short for staff working with limited access to the internet and electrical power. The Guardian needs to adapt its standard procedures. On the other hand, AMREF’s policy of having their views expressed via one official spokesperson needs to be relaxed. It appears contrary to their objective in Katine of empowering people to speak up, rather than be spoken for.

Responses to the Farm Africa and Barclays 6th February Trip Report

1. This is a new section, not included in the draft version because the Farm Africa & Barclays report was not yet available. Their report raised a number of issues that relate to my Terms of Reference as the external evaluator.
2. They noted “a lack of clarity as to the overall goal and purpose of the project”, the reference to multiple types of outcomes but the lack of balance of attention to these within the baseline survey. Similar concerns have been expressed in this report. The problem is most evident when comparing AMREF’s views with Farm, Barclays and the Guardian, and less evident within AMREF.
3. They “estimated that there are at least a further twenty NGOs-CBOs in Soroti and Katine engaged in similar programmes employing a community group approach”. Associated with this is the risk “of placing an unnecessary burden on project beneficiaries to set up different groups to access the benefits of individual components”. My next visit will therefore need to pay attention to two questions:
   1. How many of the groups being used by AMREF staff are pre-existing, versus newly created? Ideally (but not always) AMREF will be building on existing structures.
   1. What do AMREF staff know about past groups that were set up and then failed? Ideally lessons from these local experiences will be feeding into the design of their work with current groups (both new and pre-existing)
4. Related to this point, their report noted “There are many other agencies working in Soroti District and Katine Village with several years of experience.  Some are trying to achieve the same goals as AMREF and FARM Africa and some have similar approaches.  We were concerned that Katine Project staff seem to have made little contact with these agencies to learn from their experience and some staff believed that they must seek formal approval from AMREF before they have dialogue with these agencies”. Relationships with such agencies will be monitored in the July  2008 visit and thereafter.
5. They noted “It is not clear how work in the 18 villages covered by the livelihood component relates to the other components and a map showing clearly the villages covered by the different components of the project would be useful”. This is  consistent with the discussion in this report about integration, and the potential usefulness of the village profiles database as a means of tracking how this is being realised at the village level.
6. They noted a difference in approach between the livelihoods and other components, with its less emphasis on quick wins and no use of material aid. These differences have the potential to create conflict and misunderstanding both within the Katine team and with the communities they are working with. They will be the subjects of attention in my next visit? in July 2008
7. Related to this are the inherent problems of the livelihoods component being subject to direction both from Farm Africa (technically) and AMREF (managerially). Further negotiations are clearly needed to establish shared expectations. This work also needs ongoing monitoring.

Comments on the AMREF M&E Plan for Katine.

Comments on the

KCPP Monitoring and Evaluation Plan Version 3. 29/11/2007

By Rick Davies, Tuesday, 25 December 2007 Version 2

1. It is good to see that the project has already developed a detailed M&E plan, and that has undergone a number of revisions.  My comments that follow below focus on areas where I think there are gaps, potential problems and potential opportunities.

2. As with the earlier Community Needs Assessment document, it is important to identify who is the author of this document. In London it has been suggested that this M&E plan was produced for AMREF by Accenture, but in the absence of author details we don’t know if this was the case or not.

3. An additional section on who participated in the development of this M&E plan would be very useful. There is a section on livelihoods (Objective 4) but it is not clear to what extent, and how, Farm Africa may have contributed to this section. Although Barclay’s plans for work on microfinance are still being developed ideally their plans for M&E of their interventions would be integrated into this plan. Even if the M&E of microfinance was discussed in a section of its own, it would still be useful to highlight how and where there would need to be some joint M&E activities with AMREF staff. Such as with the implementation of baseline surveys and mid-term reviews.

4. It is possible that this document is under-representing the participatory dimension of AMREF’s work in Katine. There is no reference to how Katine people have been involved in the design of this M&E plan. For example in the selection of appropriate indicators. Or, how they will be involved in monitoring and evaluation processes, other than survey respondents. It seems highly likely, and desirable, that AMREF will make extensive use of meetings with different stakeholders groups on a periodic and ad hoc basis, throughout the project period[1]. The plans for the use of these meetings should be described in this M&E plan.

5. The section on the Community Needs Assessment (p.3) does refer to “group discussions with a wide range of community residents in public meetings to ensure fair representations of concerns”. However, the one available report on the Community Needs Assessment provides no information on the consultation process, on the priorities that were expressed, and how priorities varied across stakeholder groups.

6. A brief mention has been made on page 5 of the M&E plan on the intention to use focus group discussions and community meetings during the mid-term and end-of-project reviews. Planning and documentation of these reviews will need to provide much more information than was provided in the summary report on the Community Needs Assessment.

7. Annex A lists the details of a very large number of indicators, on which data will be collected at various intervals. Approximately 136 in all. Of these, 41 will be collected quarterly, another 8 bi-annually, another 43 annually and another 32 at the beginning and end of the project.  My main concern here is how realistic is it for AMREF staff to aim to collect data on this many indicators?  Collecting all this data will take up valuable staff time, and analysing it will take up more time still, and so will communicating the results of this analysis.

8. There are three possible ways of rationalising this long list. The first is to ask how well the size of each set of indicators relates to the scale of AMREF’s planned investment in the objective concerned. Will AMREF be investing most effort (e.g. # of indicators) in monitoring its biggest investments, and the least in its smallest investments? The table below shows what appears to be an over-investment in monitoring the health objective, and under investment in monitoring the other objectives, especially the water and sanitation objective. Is there a good reason for this? Or, could the number of health indicators be reduced?

Objectives	% and # of indicators	Budget allocation
1. Improved community health	41% (62 )	26% (£299,000)
2. Improved access to quality education	23% (35)	26% (£295,000)
3. Improved access to safe water, sanitation and hygiene	10% (16)	18% (£205,000)
4. Improved income generating and food security opportunities	21% (32)	23% (£270,000)
5. Communities empowered to engage in local governance	5% (7)	7% (£76,000)
Total	100% (152)	100% (£1,145,000)

9. The second possibility is that the list of indicators proposed under each sub-objective could be presented to different stakeholder groups in the Katine community in the form of a menu. Those groups could then be asked to identify those changes (represented by the different listed indicators) they thought were most important to achieve. The percentages of groups choosing each indicator could then be used as an “importance” weighting for each indicator. These weightings would help in two ways. Firstly, to identify which indicators could be abandoned, because they were a low priority to most stakeholders. Secondly, to identify those indicator where analysis of progress needed to be done with most care and attention, because they were of widespread interest.  Thirdly, to guide what analysis needed to be communicated to what groups, according to their previously expressed priorities.

10. The third possibility is to look at the coverage of the indicators that have been listed in for each objective. Some focus on activities project staff will carry out, some relate to activities that involve both project staff and Katine peoples, and some focus on changes in Katine people’s lives. This corresponds approximately to a focus on Activities, Output and Purpose (outcome) levels in a Logical Framework. Ideally, with each project objective, AMREF would want to monitor a sequence of indicators that tell us about a sequence of changes, of how activities lead to outputs which then lead to outcomes (Purpose level changes). At present the mix of indicators under some of the objectives is unbalanced. Under the education objective there do not seem to be any education outcome measures like enrolment rates, attendance rates, drop out rates or graduation rates. Yet surely these are of concern to both the Katine community and AMREF. On the other hand, the health objective seems to have many outcome indicators, perhaps many more than the project could hope to significantly influence during its lifespan.

11. Annex B and C are referred to but not included. B refers to the data collection formats to be used for the Rapid Assessments, scheduled to be carried out in December 2007, and some of which have been completed. Annex C presumably refers to the survey instrument to be used for the Baseline Household Survey, scheduled for December 2007-January 2008. Copies of these instruments need to be provided not only to myself, as the external evaluator, but also to Farm Africa and Barclays, since they will need to use the data from these surveys. Ideally copies of these instruments would have been circulated to Farm Africa and Barclays before these surveys began.

12. On page 5 there is a reference to the household survey which will measure “change at the household level to see if targeted community members have benefited as a result of project activities” it is very important that the household survey also includes comparable households who have not participated in different project activities. This should not be difficult to do.

13. The M&E plan refers to an intended sample of 95 households for the household survey. The rationale for this sample size needs a clear explanation. Will the sample be intended to represent the 25,000 people of Katine sub-county, or only of Katine village itself? If the former, it looks like as very small sample[2]. If the latter, then why focus on Katine village alone. The sample size should be big enough to allow simple statistic tests of the significance of differences found between at least two categories of groups of people over two periods of time (participants and non-participants, before and after an intervention). For example, by using Chi Square test with four cells

14. It will be very useful if a modest plan could be drawn up for how the baseline survey results will be analysed. This will help inform decisions about appropriate sample sizes, and what indicators are most central to the baseline survey. This could include

• Frequency distributions: for which indicators, displayed in what form?
• Cross tabulations: what indicators against what indicators? There are at least 32 indicators that data will be collected on, which means more than 1000 possible cross tabulations of that data. In reality the most interesting cross-tabulations will be those that compare participation in specific project activities with changes in specific aspects of people’s lives. Differences in outcomes across different locations and groups will be especially important to analyse.

15. The M&E plan provides details on data sources for each of the indicators, but no information on who will be responsible for collecting and analysing what data. That would be a useful addition to the next version of this M&E plan. Also useful would information on what products of the M&E process will be made available to who and when. This could be presented in a Gantt chart form

16. A summary of the main recommendations made above:

• Consider reducing the number of indicators that will be used, and ensure that in each area of AMREF’s work the selected indicators cover a sequence of expected events (i.e. from activities>outputs>outcome>impact)
• Explain in detail the nature and rationale of the sampling strategy that will be used. Include a reference to the type of analyses that will be done with the survey results (because this affects the sample size required)
• Provide more information on who has and who will participate in the monitoring and evaluation processes (both partners such as Farm Africa and Barclays, and various groups within the Katine community). And in what way they will participate.
• Make this M&E Plan, of the next revised version, publicly available via the AMREF and Guardian websites

---

[1] In the September 25^th 2007 Conceptual Framework paper reference is made to plans to work with farmers groups, VHTs, PTA’s and other community committees

[2] Even the plans for the Rapid Assessment (village component) refer to visits to 66 villages. If this is the total number of villages in Katine sub-country (or less than the total) a household survey sample of 95 households will cover less than two households per village.