Comments on AMREF’s Baseline Household Survey of Katine
12 May, 2008
MS Word copy of this Comment, written by Rick Davies, Wednesday, 02 April 2008
1. On page 1 it is noted that “This same survey will be conducted again in 2010 in the same households to assess what improvements have occurred as a result of project interventions.” To come to such conclusions AMREF will need with-without and before-after measures of households’ wellbeing. Unfortunately, the sample does not include any “without” parishes that AMREF will not be working in. However, there will be internal differences in the extent to which parishes, villages and households participate in AMREF assisted activities. Before-after comparisons could be made of the wellbeing of these groups, contrasting those who had higher versus lower levels of participation. The household survey will enable comparisons of parishes and households (those re-surveyed), but only after some years have passed. Data on the participation of villages could be accumulated on a more continuous basis, through day to day contact by AMREF staff. Data on village level outcomes should also be available via the various groups concerned with water, sanitation, health, livelihoods and education. While this data may not be as accurate as household survey data, it will more continuously available and more immediately useful.
Differences between parishes
2. On page 5 the Summary states that “Overall baseline conditions in Katine sub county were uniformly poor with regard in access to safe water, latrine coverage, hygiene practices, prevalence of malaria and diarrhoea in young children, delivery in health facilities, food security, and use of livelihood support services” But this point seems to be contradicted later in the same summary section where differences between parishes are described in Table 2, and elsewhere noted in a number of areas, including access to drinking water, sources of treatment for malaria, and food security. As noted on in table 2 such differences have implications for “priority interventions in each parish“ If AMREF is to provide effective development aid that is relevant to people’s needs, sensitively to differences will be essential.
National – local differences
3. The statement that “A comparison of Katine indicators with that from the 2006 Uganda Demographic and Health Survey shows that overall baseline conditions for the Katine population are worse than the average for rural areas of the country” This important claim needs to be supported by a table comparing national and local figures. This has been done with water related indicators on page 23 and HIV/AIDS knowledge on page 26.
4. The behaviour of men and women has been differentiated in many areas of the report e.g. school participation, water collection, and diarrhoea prevalence. However, the gender of the respondents to the questions is less clear. Four different questionnaires were used in each respondent household, one addressed to “the household”, one to men, one to women and one to caretakers of children under five years of age. It would be useful to know where (on what questions) men responding on behalf of their “household” differed in their responses from women responding on behalf of their “household”. The answers could have practical implications for how AMREF staff engage with communities on specific issues.
Key program indicators
5. The one page summary of key program indicators is useful. Somewhat surprisingly it does not include any household level education indicators, although education data was collected. Children’s education outcomes are probably the closest thing to an aggregate measure of development impact that will be available to AMREF, since educational achievement is affected by health, water, sanitation, livelihoods and governance, and all within a reasonable span of time. This is in contrast to mortality rates which usually reflect changes over taking place over longer periods of time, and which are harder to document. There are also no household level governance / empowerment indicators in the table although there is at least one in the AMREF M&E framework
6. These 25 indicators are a selection from at least 34 other household level impact indicators in the Katine M&E framework, which relate to household level wellbeing. The baseline survey in turn included many other questions about household knowledge, activities and attitudes, covered by four separate questionnaires. It would be useful to know how these 25 indicators were selected as “key” indicators.
Community engagement with the results
7. The recommendations do not make any reference to what should or could happen next, with the survey results. Given this survey is part of a development project, rather than part of another national statistics gathering operation, some form of participatory process would seem appropriate. This could include:
- Checking the results against people’s expectations of the results. This could help highlight possible measurement problems, and identify changes that people were not aware of, and is a sure-fire way of engaging people in a discussion of the survey.
- Prioritisation of the problems evident in the various “key indicators”, as seen by different groups within the community (men/women, young/old, etc)
Securing the data
8. Baseline survey data can easily get lost, as time moves on and priorities and interests change. AMREF needs to make good provision for the storage and backup (in other locations) of both digital and hard copies of the survey results, the survey instruments and any associated documentation, such as that concerning the sampling process and the names of staff involved.
9. Survey data such as this should be seen as a public resource which should be made publicly available. So long as simple steps are taken to secure the anonymity of the respondents. It should be possible to make the survey data files available, at minimal cost, via AMREF’s website. And then publicise their availability via relevant email lists.
For more comments on aid organisations and public goods, such as these, please see my blog posting on “Aid organisations as self-interested businesses?”
 Though other factors would also need to be taken into account as well, such as local residents own priorities.