August 2008 Visit Report by the external evaluator

1 September, 2008

The August 2008 visit report is accessible here

1. Summary of issues and main recommendations

1.      Recommendations about changes almost by definition do not identify the good that is present and which needs to be maintained. With this in mind the following points should be noted:

  • The KCPP staff are committed to their work, and appear to work well together as a group. The management style is team oriented, which is essential if there is to be effective integration of the project components.
  • There is a breadth of experience and expertise within the team e.g. the Health component Project Officer’s experience with the Yellow Star programme, and the Project Manager’s experience with training in rights- based approaches.
  • There are good relationships with the district and sub-county authorities. Although the district authorities have made critical comparisons with NUSAF they are also openly appreciative of the good working relationship that they have with AMREF
  • Both the material aid and training provided by the different components are appreciated by the communities concerned within Katine. There have been no substantial criticisms of their quality or relevance.

2.      This section summarises some major recommendations. The main text of the report lists specific recommendations by number, where the issues arise. There are 14 sets of recommendations (28 in all). And a further seven recommendations relating to the work of CARE and UWESO in Annex E. Please read them all. Search for: Recommendations*

3.      There have been significant delays in the project implementation, with the possibility of more than a quarter of the planned activities not being completed by the end of the first year (and possibly a third of the year 1 budget remaining unspent). However what is not necessary is a rush to complete activities by the end of September, simply for the sake of completing as much of the original plan on time as possible. What is important is attention to the cause of these delays.  The results of this examination should be evident in more timely implementation in the October 2008 – March 2009 period. See section 3.

PS1: AMREF have since pointed out that the project workplans outline “a sequence of activities as opposed to specific deadlines” (through the workplans do assign specific activities to specific time periods). In their view delays are to be expected because AMREF is involved in community development, and thus AMREF does not set the pace on its own. However, as noted later in this report, delays have also arisen from inefficiencies of process within AMREF. The question now is how big do the delays need to be before the project donors can legitimately express concern?

PS2: Re reviewing the causes of delays, AMREF’s view is that their existing planning and review processes are adequate

4.      The importance of particular project activities depends on their place within a wider strategic view of how the project will achieve its ends. A project with many activities but little strategy could easily be accused of gap-filling, and little more. This is a serious criticism that can be made of many aid projects, and one that AMREF needs to be able to respond to KCPP. The issue of KCPP’s strategy is discussed in section 8, on the project’s theory-of-change (ToC).  One high level statement of KCPP’s ToC is that the Development of Models + Creation of Demand = Development impact. This report argues (in 3.5 and 8) that the project’s strategy for the encouragement of public demand (e.g. for access to quality health, education and water services) is underdeveloped and needs attention including greater use of communication channels outside of the government administration (e.g. via mass media and elected representatives at different levels).  The approach to the “development of models” also needs to be articulated more explicitly, because the options here are more complex than appear at first glance. In section 8.4 a number of different types of model creation have been identified. What would now be useful is clarification from AMREF on which of these are important in the KCPP. See section 8.

PS: AMREF’s view here is that its corporate (10 year) and Uganda country strategy provide a sufficient strategic guidance to the KCPP

5. Another high level statement of KCPP’s  ToC is that the Community Partnering + Capacity Building + Operations Research and Advocacy = Development Impact. There is ample evidence that Community Partnering and Capacity Building are already taking place. What is not yet present is sufficient investment in the analysis of the efficiency and effectiveness of the work being by the KCPP, which could then be used by staff and other stakeholders to advocate the use of certain development approaches (aka models). Doing so would be consistent with AMREF’s stated support for evidence-based advocacy (versus “assumption based advocacy”).  Progress with operations research and advocacy would be evident in (a) dedicated activities and budget allocations for this work within each of the components’ workplans, as well as for the M&E officer, and (b) documentation of progress made with advocacy efforts. See section 8.

PS: AMREF has commented that “AMREF is in the process of developing an advocacy strategy for the Katine project. The advocacy strategy will provide guidelines on how the project will gather the evidence and utilise it to influence policies and best practices at the local and national levels through the various foras that AMREF participates in such as the health policy action committee.”

6. Two important sources of risk for the project have been identified during this July visit. One is external, and one internal. The external risk is that the KCPP will face significant competition for attention from other development projects, such as NUSAF[1]. NUSAF is more decentralised in its management, and seen to be more cost-effective by Soroti district officials[2].  With the beginning of NUSAF Phase 2 District and sub-county officials may be less open to the adoption of KCPP models, unless the KCPP engages in serious evidence-based advocacy about the value of what it is doing.  There are also other development projects elsewhere in Uganda which will be competing for national and international attention along with the KCPP[3]. AMREF needs to identify how the KCPP approach is different and what difference these differences make (i.e. talking about its theory-of-change in practical terms) (See section 8.3). It needs to be able to show how it is adding value in ways that would not be possible if the same aid funds were to flow through different channels (either government or non-government).

PS: Re comparisons with other projects, AMREF have said “However we would like to desist from direct comparison with NUSAF because no formal comparable review has been done. ” Although AMREF may not wish to have comparisons made, other stakeholders are likely to persist in doing so, throughout the life of the project, and perhaps that reality needs to be acknowledged and responded to. This accommodation may have already begun, in that AMREF have also said that it “acknowledges issues raised by the district and sub-county authority and aims to take their comments on board. As part of the project steering committee agenda, AMREF will aim to do a comparative financial analysis of the project’s interventions vis a vis funds channelled through Katine sub-county and Soroti district.”

7. A significant internal risk to project success is the degree of centralised management within the Uganda country program.  Expenditures over UGX 100,000 (£33.50) .need to be approved by the Kampala office, whereas the Sub-county chief can authorise up to UGX 30 million. Quarterly workplans by all KCPP staff need to be approved by the Kampala office. Both processes take more time than more localised decision making. Strategies for communications and advocacy apparently need to be developed at the country level first. Country office staff represent the KCPP in both the monthly Project Management Committee and the quarterly Project Steering Committee meetings held in Katine and Soroti respectively. All these processes have implications for the efficiency, effectiveness and credibility of AMREF’s work in Katine.  It should be noted that concerns about the degree of centralisation have been made by members of the District Steering Committee, Sub-county Project Management Committee members, KCPP staff and Guardian visitors. There is, in my view, a strong argument in a favour of AMREF HQ undertaking a review of existing practices  Where it is decided that specific processes do need to remain centralised then there should be clear and transparent performance targets for their completion times. See section 3.7.

PS: AMREF’s response is that they use “standard management processes for Katine and all the other 23 projects in Uganda. These provide efficiency through scale as well as accountability that would be expensive and burdensome to replicate (sic) separately for every project. Moreover, these processes have been developed to support implementation across AMREF, based on our 51 years of work in Africa.” They also note that “The PMC and SC were established through participatory discussions with all the stakeholders and are subject to review according to local needs.

8. Within the project there are two areas of strategy which need particular attention. In the health component the investment in upgrading of health centre facilities and training of village health teams has involved a de facto assumption that the necessary drug supplies will be available, especially anti-malarials. Reports from the VHTs and Health Centre staff indicate otherwise, than in fact shortage of anti-malarials is a major problem. This is in the context where the AMREF’s 2007 needs assessment identified malaria as the single main cause of morbidity and mortality. This is clearly one area where a research and advocacy strategy needs to be developed as an integral part of the KCPP approach. See section 3.1.

PS: AMREF has commented that “Drug stock outs are a common problem in Uganda and AMREF has taken this as an advocacy issue.  In another project in Uganda, AMREF is undertaking an operations research into the issue of drug supply chain. The results of this research will be used as evidence for wider advocacy and policy influencing and will benefit KCCP. AMREF also has other malaria specific projects in Uganda that are addressing similar issues.”

9 The governance component also faces some challenges. On the ground in Katine the IEC[4] component has not yet started to address the issues of “the community’s rights to basic services…”, instead pre-existing conventional IEC materials are being re-used.. The piloting of community partnering “from a rights based approach” seems to be on hold. Discussions in Kampala suggested that this office favours a conservative interpretation of how issues of empowerment and rights can be dealt with. In Katine staff seem to have a more open view. AMREF should clarify its position by developing a strategy statement saying how its existing (or revised) governance activities in Katine will help achieve the governance objectives as stated in the project proposal. This strategy statement should be shared and discussed with its donors. See section 3.5.

PS: AMREF have confirmed that “Rights Based Approach (RBA) is prioritised as a core implementation strategy in the AMREF global and country strategic plans. The RBA model is being developed and rolled out across Africa since this is a relatively new area for AMREF.”…”Based on the agreed project design, the community empowerment component is broader than RBA. Progress on governance is based on the capacity of communities to engage with their decision and policy makers. AMREF’s role is to catalyze community action since taking on that role would disempower the community”.

10 Both of the two problems identified above may be reflections of what could be called a strategy gap in the current design of the KCPP. That is the absence of documented thinking about how activities in each component should connect causally to the project objectives of that component. The recent introduction of a new planning format for proposed activities may help fill this gap, if it asks the right questions. If not, they could be a form of micro-management.

11.  The project has its own objectives and indicators, and data on some of these indicators is now being collected, in addition to the work done on the baseline survey. However further steps are needed. The definition of specific levels of achievement on the various project indicators is essential, if the success of the project is to be evaluable. In the longer term, the sustainability of all project aided activities will be dependent of capable monitoring of those activities by the relevant community groups and government structures (e.g. PTAs, SMCs, PDCs, VSLAs, etc). This is more likely to happen if KCPP objectives (especially the associated targets) are closely aligned with those of its local government and community partners. Although some progress is being made with identifying targeted levels of achievement on indicators of shared concern, this work does need to be finalised soon, preferably by the end of this year. See section 4.

PS: AMREF believes the existing comprehensive M & E framework for the KCCP is adequate, but that “it will be updated periodically on the basis based on new project information and in response to community priorities”.

12. Regarding transparency, AMREF has been open in its dealings with local government, though local government would like to see more information on costs of activities. District and sub-county authorities have in turn expressed willingness to share information on their budgets, an offer worth pursuing. Although the importance of transparency seems to be well recognised in the KCPP office more attention does need to be given to improving transparency with the public at large in Katine, both about AMREF activities and the work of its partners. Simple measures could be useful, such as robust public notice boards at schools, health centres, markets and sub-county offices showing meeting minutes, survey results, annotated maps, and AMREF’s own plans. The use of talk-back radio has also been suggested[5]. Internationally, progress with greater transparency via the AMREF websites has been slow, but with a major improvement during my visit, hopefully now to be sustained. See section 7.

13. Transparency will not be painless.  It will almost inevitably bring with it some critical judgements of how well the KCPP is doing, when compared to others. For example, decentralised projects like NUSAF, and activities by smaller NGOs with lower overheads. Ideally AMREF will respond positively to these challenges, and provide in turn a demanding comparator for other development projects. This expectation is in effect the external evaluator’s theory-of-change: about the relationship between transparency, the value of comparisons and increased aid effectiveness.

PS: AMREF has replied that it is “confident that our development model is rigorous enough to enable us to engage in informed development debates with other likeminded organisations.”


[1] Northern Uganda Social Action Fund. NUSAF Phase 2 is about to start in January 2009

[2] But not without its own problems e.g. corruption. See Sh2.5b NUSAF funds missing New Vision, Thursday, 10th July, 2008

[3] Including those announced by President Museveni, in his recent visit to Soroti.

[4] Information, Education and Communication

[5] There are three radio stations in the area, and many households have radios.


One Response to “August 2008 Visit Report by the external evaluator”

  1. rick davies Says:

    Re paragraph 6 above, please read “Amref urged to cut back on training in Katine” article on the Guardian Katine website, posted on Thursday 5th March, at

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: