The evaluator evaluated

17 November, 2008

Here are…

AMREF REFLECTIONS ON 2ND EXTERNAL EVALUATION OF KCPP

tabled at the Quarterly Governance meeting, Barclays, 1 Churchill Place, 15th October 2.30-4.30 pm

along with my responses in red

Strengths

  • Mobilised different stakeholders in AMREF to critically reflect on the progress, key priorities over next 12 months – key development questions on what, when and why
  • Acknowledged the different pieces of work going on in Katine and the sheer volume of the work
  • Participatory and inclusive – got insights from different stakeholders and beneficiary groups
  • The process of responding to the report enabled the whole of AMREF to work together and crystallise and document our approach to development with a particular focus on KCPP
  • Useful recommendation and learning on how to improve our M&E processes

Areas for improvement

  • Very long/heavy report – really difficult to analyse especially when English is not a first language
    • There is a four page summary at the front, and a contents page, and separate annexes
    • The next report will aim to be 20 pages max (excluding Exec Summ and Annexes)
      • But this proposal is subject to discussions re plans for the AMREF Mid Term Revew (MTR)
  • Intensive process/time consuming for different stakeholders in KCPP (PIT, country office team, AMREF HQ, AMREF UK, Farm Africa).
    • Following discussions with Guardian in August my aim now is to progressively reduce the frequency of visits and, where possible, to integrate these with AMREF review processes.
  • The evaluation visits are too regular to enable staff on ground to learn, take action before participating in another evaluation (So far we have had 2 evaluations in 7 months). Most 3 year projects have 2 – midterm and end of project). KCPP will have 8 over the 3 years (6 + midterm and end of project).  This is not feasible if we want staff to value the evaluation  and deliver on the specific targets in their workplans
    • Following discussions with Guardian in August my aim now is to progressively reduce the frequency of visits and, where possible, to integrate these with AMREF review processes.
  • Analysis of KCPP did not fully take into account the wider AMREF context, its policies and procedures)
    • Examples of important missing contextual information would be useful
    • Not sure how this gap could be addressed while also reducing the size of the report
  • Analysis of KCPP did not fully take into account the wider development context of Katine, Soroti district and Uganda
    • Examples of important missing contextual information would be useful
    • Not sure how this gap could be addressed while also reducing the size of the report
  • The evaluation focussed a lot on the processes. To the staff, who are working under difficult circumstances, highlighting both the interim tangible and intangible outcomes could have been more motivating
    • In the early stages of a project it is the work processes that are most visible and important, and outcomes tend to be less visible. Future visits should focus progressively more on outcomes and impact.
  • In some instances the report  did not take into account key sensitivities about staff and impact on the relationships that AMREF has with different stakeholders
    • Details are needed here before I can respond. But on reflection it seemed as though  it was sections of AMREF who were the most sensitive, and sections of government who were quite robust (in wanting their views expressed).

Suggestions for the future

  • Make the report shorter and in simple, user-friendly language (maximum 10 pages)
    • 10 pages is too short. 20 pages is more realistic. But final agreement here will depend on the ambit of the next visit, which is under discussion.
  • Consider the implications of the evaluation recommendations on the capacity of the PIT and practical realities on the ground – deliver initial targets, address new issues within very tight deadlines
    • Noted: Recommendations should be limited in number
  • The report has multiple target audiences.  It would be good to have a summary which is user friendly and targeting our stakeholders in Uganda, especially the district officials (i.e beyond the UK donors, UK public through the website).  Otherwise we stand to be accused of using extractive evaluation processes
    • Noted: There is a need for an Executive Summary that can fulfil this function (in addition to the existing list of Recommendations). But I am not sure if I should produce customised versions for different stakeholders. This might be better done by AMREF.
  • Place analysis of findings and recommendations within the contextual realities of Katine sub-county, Soroti District and Uganda development context.  Co-evaluation with someone from Africa/Uganda would add a lot of value
    • Need explanation of what “contextual realities” are being referred to, and how they would be covered within a 10 page, or 20 page, report
    • Co-evaluation with “someone from Africa/Uganda” will happen if we integrate my next visit with the AMREF mid-term review
  • Extend the time and if possible limit the number of evaluations bearing in mind that we shall also have midterm and end of project
    • Agreed, as noted above
  • Next evaluation should focus more on interim outcomes and the foundations we are making for sustainability
    • Sounds appropriate at that point in time in the lifespan of the project. As part of this next review, I would like to see some systematic documentation of what has been done / happened in all the villages of Katine sub-country. A starting point would be a spreadsheet of villages x activities (including non-AMFREF activities).
  • Clarify roles and scope of the evaluation – technical and programmatic; operations and management.
    • The Terms of Reference (ToRs) for each visits should be where this is done. Draft ToRs were shared with AMREF, Barclays and the Guardian for comment prior to each of the two visits to Katine so far. The same will be the case with visits in the future.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: