The independent assessor for the Katine project, Rick Davies, answers questions from blog readers about his work.
My role in this project is to independently assess the work carried out by Amref in Katine, a process explained in detail elsewhere on the site.
Already, some readers have posted comments and questions on this blog about my work. Rather than posting a long reply myself, I thought it best to write a new entry.
One reader, Go4it2day, said:
Have you asked the people of Katine what they want? And I do not mean consultations between an NGO and some people called to a meeting. Believe it or not market research is available in Africa.
In my January 2008 visit, one of my tasks will be to examine the ways in which Amref staff have worked with people in Katine to identify their needs and priorities. My expectation is that they will use a variety of means, possibly including one to one surveys.
The same reader asked:
Why is the ‘independent auditor’ not African? Is it because his responsibility is to the Guardian and its readers, and not to the people of Katine?
I will not be the only person who will be monitoring and evaluating the performance of Amref’s work. In the first instance, Amref will have its own dedicated staff and procedures for monitoring and evaluating the implementation and outcomes of its work.There may also be a community level committee of local stakeholders, with similar responsibilities. My role will be more ‘meta-monitoring and evaluation’. One of my tasks will be to check on the adequacy of Amref’s systems and procedures for monitoring and evaluating its work, and to provide to them, where wanted, with help to build their monitoring and evaluation capacity.
My work will involve talking to Amref staff, reading their documents, and talking to people in the community. In the longer term, my role should become less and less important, as the Guardian and its readers feel increasingly confident in the abilities of Amref.
Go4it2day also asked:
Why are the KPIs [key performance indicators] of the project so vague? Why not use simple KPIs like average income, child survival rates, deaths per 1,000. Is it because nice and vague KPIs allow a whitewash of the project?
The objectives need to be agreed on first, through community consultations. Then, during the process, verifiable indicators of their achievement need to be identified. Again, this is best done through consultations with the groups/communities involved.
Having read Amref’s structured programme to tackle the issues of health, education, sanitation and safe water, income generation and good governance I am concerned to see there appears to be no specific acknowledgment of the centrality of women to the success or otherwise of the outcomes in each area.
One of the criteria that I will be using to assess the work of Amref will be how they address issues of equity. And one important dimension of equity is gender equity. I will examine not only how women benefit, versus men, but also how they are involved in planning, implementation and review processes.In a later comment, Java1930 said:
Also I urge Mr Davies when he is making his assessments on the project to ensure he is able to speak to women and girls in the right environment – they should have an opportunity to speak in the absence of their men folk whose presence may inhibit them discussing real issues. And when he feeds back he should be careful to ensure individual women are not identified as raising particular issues for fear of retribution – domestic violence is an issue there as it is anywhere else in the world.
In the first instance I will try to identify to what extent Amref staff have been able to do this, and if not, why not. Secondly, I will seek to follow this advice myself, when in Katine.
The same reader commented:
I would also like this project to be very clear and transparent on how much of the money (cash) raised is actually spent on the people in Katine ie how much pence of each pound raised. For the last 25/30 years Uganda’s been awash with government and non-governmental NGOs who receive millions of pounds and yet we see precious little reaching the people in whose name it is all being raised for. Sad to say but charity is a very big lucrative business.
I have suggested to Amref that the prepare a disclosure policy, which would state what types of information will, by default be publicly available, and what will not, but which may be available on request. Disclosure policies are used by major organisations such as the World Bank, and the IMF, but also by progressive NGOs such as ActionAid.Costs can be cut horizontally (eg the amounts spent in UK, spent in Kampala, and spent in Katine) or vertically (the total costs incurred at all levels for delivering a specific outcome on the ground eg building a new school or assisting a specific community).
The latter is more useful, if you want to compare costs against benefits. Analysing costs horizontally can involve mistaken assumptions: that all assistance will be in the form of things or money given directly to people in Katine, and that purchases made in Kampala or London will be of no benefit to them.
In another comment, Go4it2day said:
But this brings up the question in what will happen to Katine in 4 years, 10 years, 20 years after the Guardian campaign has ended. Africa is littered with ‘Aid Experiments’ that have failed. And to successfully develop somewhere takes decades not 3 years.
I will suggest that when the current ‘project’ comes to an end in 2010 that Amref tries to make a number of verifiable predictions about what will happen to the various achievements to date, over the next three years (ie by 2013).
This does not mean we expect Amref staff to be clairvoyant. Just that they try to make a reasonable best guess assessment of what is likely to happen.
I have asked the Guardian to consider committing myself (or another independent evaluator) to a follow-up visit and review of Katine three years after the end of the current project (ie in 2013). As well as looking for unexpected changes, that visit should also try to assess to what extent Amref’s predictions have been correct or not.