The August 2008 visit report is accessible here

1. Summary of issues and main recommendations

1.      Recommendations about changes almost by definition do not identify the good that is present and which needs to be maintained. With this in mind the following points should be noted:

  • The KCPP staff are committed to their work, and appear to work well together as a group. The management style is team oriented, which is essential if there is to be effective integration of the project components.
  • There is a breadth of experience and expertise within the team e.g. the Health component Project Officer’s experience with the Yellow Star programme, and the Project Manager’s experience with training in rights- based approaches.
  • There are good relationships with the district and sub-county authorities. Although the district authorities have made critical comparisons with NUSAF they are also openly appreciative of the good working relationship that they have with AMREF
  • Both the material aid and training provided by the different components are appreciated by the communities concerned within Katine. There have been no substantial criticisms of their quality or relevance.

2.      This section summarises some major recommendations. The main text of the report lists specific recommendations by number, where the issues arise. There are 14 sets of recommendations (28 in all). And a further seven recommendations relating to the work of CARE and UWESO in Annex E. Please read them all. Search for: Recommendations*

3.      There have been significant delays in the project implementation, with the possibility of more than a quarter of the planned activities not being completed by the end of the first year (and possibly a third of the year 1 budget remaining unspent). However what is not necessary is a rush to complete activities by the end of September, simply for the sake of completing as much of the original plan on time as possible. What is important is attention to the cause of these delays.  The results of this examination should be evident in more timely implementation in the October 2008 – March 2009 period. See section 3.

PS1: AMREF have since pointed out that the project workplans outline “a sequence of activities as opposed to specific deadlines” (through the workplans do assign specific activities to specific time periods). In their view delays are to be expected because AMREF is involved in community development, and thus AMREF does not set the pace on its own. However, as noted later in this report, delays have also arisen from inefficiencies of process within AMREF. The question now is how big do the delays need to be before the project donors can legitimately express concern?

PS2: Re reviewing the causes of delays, AMREF’s view is that their existing planning and review processes are adequate

4.      The importance of particular project activities depends on their place within a wider strategic view of how the project will achieve its ends. A project with many activities but little strategy could easily be accused of gap-filling, and little more. This is a serious criticism that can be made of many aid projects, and one that AMREF needs to be able to respond to KCPP. The issue of KCPP’s strategy is discussed in section 8, on the project’s theory-of-change (ToC).  One high level statement of KCPP’s ToC is that the Development of Models + Creation of Demand = Development impact. This report argues (in 3.5 and 8) that the project’s strategy for the encouragement of public demand (e.g. for access to quality health, education and water services) is underdeveloped and needs attention including greater use of communication channels outside of the government administration (e.g. via mass media and elected representatives at different levels).  The approach to the “development of models” also needs to be articulated more explicitly, because the options here are more complex than appear at first glance. In section 8.4 a number of different types of model creation have been identified. What would now be useful is clarification from AMREF on which of these are important in the KCPP. See section 8.

PS: AMREF’s view here is that its corporate (10 year) and Uganda country strategy provide a sufficient strategic guidance to the KCPP

5. Another high level statement of KCPP’s  ToC is that the Community Partnering + Capacity Building + Operations Research and Advocacy = Development Impact. There is ample evidence that Community Partnering and Capacity Building are already taking place. What is not yet present is sufficient investment in the analysis of the efficiency and effectiveness of the work being by the KCPP, which could then be used by staff and other stakeholders to advocate the use of certain development approaches (aka models). Doing so would be consistent with AMREF’s stated support for evidence-based advocacy (versus “assumption based advocacy”).  Progress with operations research and advocacy would be evident in (a) dedicated activities and budget allocations for this work within each of the components’ workplans, as well as for the M&E officer, and (b) documentation of progress made with advocacy efforts. See section 8.

PS: AMREF has commented that “AMREF is in the process of developing an advocacy strategy for the Katine project. The advocacy strategy will provide guidelines on how the project will gather the evidence and utilise it to influence policies and best practices at the local and national levels through the various foras that AMREF participates in such as the health policy action committee.”

6. Two important sources of risk for the project have been identified during this July visit. One is external, and one internal. The external risk is that the KCPP will face significant competition for attention from other development projects, such as NUSAF[1]. NUSAF is more decentralised in its management, and seen to be more cost-effective by Soroti district officials[2].  With the beginning of NUSAF Phase 2 District and sub-county officials may be less open to the adoption of KCPP models, unless the KCPP engages in serious evidence-based advocacy about the value of what it is doing.  There are also other development projects elsewhere in Uganda which will be competing for national and international attention along with the KCPP[3]. AMREF needs to identify how the KCPP approach is different and what difference these differences make (i.e. talking about its theory-of-change in practical terms) (See section 8.3). It needs to be able to show how it is adding value in ways that would not be possible if the same aid funds were to flow through different channels (either government or non-government).

PS: Re comparisons with other projects, AMREF have said “However we would like to desist from direct comparison with NUSAF because no formal comparable review has been done. ” Although AMREF may not wish to have comparisons made, other stakeholders are likely to persist in doing so, throughout the life of the project, and perhaps that reality needs to be acknowledged and responded to. This accommodation may have already begun, in that AMREF have also said that it “acknowledges issues raised by the district and sub-county authority and aims to take their comments on board. As part of the project steering committee agenda, AMREF will aim to do a comparative financial analysis of the project’s interventions vis a vis funds channelled through Katine sub-county and Soroti district.”

7. A significant internal risk to project success is the degree of centralised management within the Uganda country program.  Expenditures over UGX 100,000 (£33.50) .need to be approved by the Kampala office, whereas the Sub-county chief can authorise up to UGX 30 million. Quarterly workplans by all KCPP staff need to be approved by the Kampala office. Both processes take more time than more localised decision making. Strategies for communications and advocacy apparently need to be developed at the country level first. Country office staff represent the KCPP in both the monthly Project Management Committee and the quarterly Project Steering Committee meetings held in Katine and Soroti respectively. All these processes have implications for the efficiency, effectiveness and credibility of AMREF’s work in Katine.  It should be noted that concerns about the degree of centralisation have been made by members of the District Steering Committee, Sub-county Project Management Committee members, KCPP staff and Guardian visitors. There is, in my view, a strong argument in a favour of AMREF HQ undertaking a review of existing practices  Where it is decided that specific processes do need to remain centralised then there should be clear and transparent performance targets for their completion times. See section 3.7.

PS: AMREF’s response is that they use “standard management processes for Katine and all the other 23 projects in Uganda. These provide efficiency through scale as well as accountability that would be expensive and burdensome to replicate (sic) separately for every project. Moreover, these processes have been developed to support implementation across AMREF, based on our 51 years of work in Africa.” They also note that “The PMC and SC were established through participatory discussions with all the stakeholders and are subject to review according to local needs.

8. Within the project there are two areas of strategy which need particular attention. In the health component the investment in upgrading of health centre facilities and training of village health teams has involved a de facto assumption that the necessary drug supplies will be available, especially anti-malarials. Reports from the VHTs and Health Centre staff indicate otherwise, than in fact shortage of anti-malarials is a major problem. This is in the context where the AMREF’s 2007 needs assessment identified malaria as the single main cause of morbidity and mortality. This is clearly one area where a research and advocacy strategy needs to be developed as an integral part of the KCPP approach. See section 3.1.

PS: AMREF has commented that “Drug stock outs are a common problem in Uganda and AMREF has taken this as an advocacy issue.  In another project in Uganda, AMREF is undertaking an operations research into the issue of drug supply chain. The results of this research will be used as evidence for wider advocacy and policy influencing and will benefit KCCP. AMREF also has other malaria specific projects in Uganda that are addressing similar issues.”

9 The governance component also faces some challenges. On the ground in Katine the IEC[4] component has not yet started to address the issues of “the community’s rights to basic services…”, instead pre-existing conventional IEC materials are being re-used.. The piloting of community partnering “from a rights based approach” seems to be on hold. Discussions in Kampala suggested that this office favours a conservative interpretation of how issues of empowerment and rights can be dealt with. In Katine staff seem to have a more open view. AMREF should clarify its position by developing a strategy statement saying how its existing (or revised) governance activities in Katine will help achieve the governance objectives as stated in the project proposal. This strategy statement should be shared and discussed with its donors. See section 3.5.

PS: AMREF have confirmed that “Rights Based Approach (RBA) is prioritised as a core implementation strategy in the AMREF global and country strategic plans. The RBA model is being developed and rolled out across Africa since this is a relatively new area for AMREF.”…”Based on the agreed project design, the community empowerment component is broader than RBA. Progress on governance is based on the capacity of communities to engage with their decision and policy makers. AMREF’s role is to catalyze community action since taking on that role would disempower the community”.

10 Both of the two problems identified above may be reflections of what could be called a strategy gap in the current design of the KCPP. That is the absence of documented thinking about how activities in each component should connect causally to the project objectives of that component. The recent introduction of a new planning format for proposed activities may help fill this gap, if it asks the right questions. If not, they could be a form of micro-management.

11.  The project has its own objectives and indicators, and data on some of these indicators is now being collected, in addition to the work done on the baseline survey. However further steps are needed. The definition of specific levels of achievement on the various project indicators is essential, if the success of the project is to be evaluable. In the longer term, the sustainability of all project aided activities will be dependent of capable monitoring of those activities by the relevant community groups and government structures (e.g. PTAs, SMCs, PDCs, VSLAs, etc). This is more likely to happen if KCPP objectives (especially the associated targets) are closely aligned with those of its local government and community partners. Although some progress is being made with identifying targeted levels of achievement on indicators of shared concern, this work does need to be finalised soon, preferably by the end of this year. See section 4.

PS: AMREF believes the existing comprehensive M & E framework for the KCCP is adequate, but that “it will be updated periodically on the basis based on new project information and in response to community priorities”.

12. Regarding transparency, AMREF has been open in its dealings with local government, though local government would like to see more information on costs of activities. District and sub-county authorities have in turn expressed willingness to share information on their budgets, an offer worth pursuing. Although the importance of transparency seems to be well recognised in the KCPP office more attention does need to be given to improving transparency with the public at large in Katine, both about AMREF activities and the work of its partners. Simple measures could be useful, such as robust public notice boards at schools, health centres, markets and sub-county offices showing meeting minutes, survey results, annotated maps, and AMREF’s own plans. The use of talk-back radio has also been suggested[5]. Internationally, progress with greater transparency via the AMREF websites has been slow, but with a major improvement during my visit, hopefully now to be sustained. See section 7.

13. Transparency will not be painless.  It will almost inevitably bring with it some critical judgements of how well the KCPP is doing, when compared to others. For example, decentralised projects like NUSAF, and activities by smaller NGOs with lower overheads. Ideally AMREF will respond positively to these challenges, and provide in turn a demanding comparator for other development projects. This expectation is in effect the external evaluator’s theory-of-change: about the relationship between transparency, the value of comparisons and increased aid effectiveness.

PS: AMREF has replied that it is “confident that our development model is rigorous enough to enable us to engage in informed development debates with other likeminded organisations.”

Footnotes:

[1] Northern Uganda Social Action Fund. NUSAF Phase 2 is about to start in January 2009

[2] But not without its own problems e.g. corruption. See Sh2.5b NUSAF funds missing New Vision, Thursday, 10th July, 2008 http://www.newvision.co.ug/D/8/12/638475

[3] Including those announced by President Museveni, in his recent visit to Soroti.

[4] Information, Education and Communication

[5] There are three radio stations in the area, and many households have radios.

Advertisements

The January 2008 visit report is accessible here

The Summary of Recommendations

  1. The final objectives of the project need clarification and agreement, by AMREF, its donors and local stakeholders. This agreement should be evident in a smaller set of indicators that show changes in people’s lives, which reflect the impact of all five project components, and which can be easily be monitored by community groups.
  2. Working with a range of local community groups, who are linked into government structures, is at the core of the KCPP strategy for achieving sustainable improvements to people’s lives in Katine. Ways of tracking overall improvements in the performance of these groups need to be developed, in consultation with them.
  3. The scale of project investment varies substantially across components, but in at least three components there are not enough resources to provide all villages with at least some minimal level of assistance (however defined). The livelihoods component seems especially under-invested. A strategy needs to be developed to either obtain expanded funding, or to ensure replication of AMREF activities in unassisted villages by other parties.
  4. The key to the long term sustainability of the achievements of the project is likely to be the work done on governance, both of the assisted community groups, and how they relate to the wider structures of government in the district. This is the area of the project most in need of technical back up, possibly by a third party, as is the case with Farm Africa and the livelihoods activities.
  5. In the longer term, AMREF needs to be able to provide evidence of about two types of change processes, that ideally would then be replicable elsewhere • What type of packages of support can best lead to desired improvements2 in the functioning of particular community groups• How the functioning of specific community groups can lead to particular improvements in the quality of peoples lives.
  6. Progress has been made with the development of an Open Information Policy but its implementation needs to be accelerated, if it is to be seen as anything more than a gesture. Disclosure of project documents to the external evaluator has improved, but will need to fit within the time schedule of future visits to Katine.
  7. The Guardian website should refer to AMREF’s “Katine Community Partnerships Project” in order to emphasise that it is an AMREF project, not a Guardian project and that the project is working through partnerships with local community organisations, not directly aiding individual households.
  8. AMREF’s participation in the Guardian website faces challenges. Cut-off time for comments on postings is too short for staff working with limited access to the internet and electrical power. The Guardian needs to adapt its standard procedures. On the other hand, AMREF’s policy of having their views expressed via one official spokesperson needs to be relaxed. It appears contrary to their objective in Katine of empowering people to speak up, rather than be spoken for.

Responses to the Farm Africa and Barclays 6th February Trip Report

  1. This is a new section, not included in the draft version because the Farm Africa & Barclays report was not yet available. Their report raised a number of issues that relate to my Terms of Reference as the external evaluator.
  2. They noted “a lack of clarity as to the overall goal and purpose of the project”, the reference to multiple types of outcomes but the lack of balance of attention to these within the baseline survey. Similar concerns have been expressed in this report. The problem is most evident when comparing AMREF’s views with Farm, Barclays and the Guardian, and less evident within AMREF.
  3. They “estimated that there are at least a further twenty NGOs-CBOs in Soroti and Katine engaged in similar programmes employing a community group approach”. Associated with this is the risk “of placing an unnecessary burden on project beneficiaries to set up different groups to access the benefits of individual components”. My next visit will therefore need to pay attention to two questions:
    1. How many of the groups being used by AMREF staff are pre-existing, versus newly created? Ideally (but not always) AMREF will be building on existing structures.
    1. What do AMREF staff know about past groups that were set up and then failed? Ideally lessons from these local experiences will be feeding into the design of their work with current groups (both new and pre-existing)
  4. Related to this point, their report noted “There are many other agencies working in Soroti District and Katine Village with several years of experience.  Some are trying to achieve the same goals as AMREF and FARM Africa and some have similar approaches.  We were concerned that Katine Project staff seem to have made little contact with these agencies to learn from their experience and some staff believed that they must seek formal approval from AMREF before they have dialogue with these agencies”. Relationships with such agencies will be monitored in the July  2008 visit and thereafter.
  5. They noted “It is not clear how work in the 18 villages covered by the livelihood component relates to the other components and a map showing clearly the villages covered by the different components of the project would be useful”. This is  consistent with the discussion in this report about integration, and the potential usefulness of the village profiles database as a means of tracking how this is being realised at the village level.
  6. They noted a difference in approach between the livelihoods and other components, with its less emphasis on quick wins and no use of material aid. These differences have the potential to create conflict and misunderstanding both within the Katine team and with the communities they are working with. They will be the subjects of attention in my next visit? in July 2008
  7. Related to this are the inherent problems of the livelihoods component being subject to direction both from Farm Africa (technically) and AMREF (managerially). Further negotiations are clearly needed to establish shared expectations. This work also needs ongoing monitoring.