Tracking change and sharpening indicators

Published on: 18/10/2013

In 2011, I was involved in an assessment of the state of water service provision in three districts in Ghana. This assessment was done in order to test draft service delivery indicators developed by the Community Water and Sanitation Agency, in collaboration with IRC, under its Triple-S Project, as part of the development of a framework for monitoring water services in Ghana.

In line with what we expected, and in line with international experiences, about a third of handpumps were found to be not functional. Looking beyond functionality however, we found that shockingly few systems provided a basic level of service, in terms of quality, quantity, accessibility and reliability of services, as per the norms and standards set for rural water supply in Ghana. In the beginning of 2013, the second round of water service monitoring was done in the three districts, in order to see whether changes had occurred since the baseline assessment, and in order to further refine the indicators and data collection methods. This blog presents some of the main (preliminary) findings of this second round of service monitoring, particularly focusing on handpumps.

To our surprise, we found that about a fifth of new handpumps, constructed less than a year earlier, were not fully functional in round two of the assessment.

Confronted with the findings from the baseline assessment, local government in the three districts had set itself the task of improving functionality of handpumps, by repairing or rehabilitating some of these. This contributed to an improvement of functionality levels in a number of cases, indicated in dark green in Figure 1. However, we also found there are more handpumps with decreasing functionality levels (in orange and dark red), than handpumps with improving functionality levels, as shown by the bright red in Figure 1. A total of 107 new handpumps have been constructed since the baseline. Looking at the functionality of these new (less than a year old) handpumps, we found almost 20% to be either not or partially functioning.

Figure 1: Change in functionality status of existing handpumps from baseline to round 2

Overall, the proportion of fully functional handpumps has decreased. But so has the proportion of non functionaling handpumps…. A larger proportion was thus found to be “partially functioning, providing water, with not within 5 strokes. A reason for this could be the fact that the second monitoring round has taken place later in the year than the baseline data collection, more towards the end of the dry season. Water tables may have been lower during round 2, accounting for the increases proportion of handpumps not providing water within 5 strokes. Service level sub-indicators: The level of service provided by the handpumps was assessed against the benchmarks on water quality, quantity, accessibility and reliability, as set for the rural water sector by CWSA. Figure 2 gives an overview of the proportion of handpumps meeting the benchmarks on the different service level sub-indicators in the baseline and round 2 assessment. What we would like to see in the round 2 bars is:

• All new handpumps meeting the benchmarks (so round 2 bars with bright green and no bright red)
• An increase in handpumps which did not meet the benchmark in the baseline situation, but are meeting the benchmark now. And no existing handpumps which used to meet the benchmark, but are no longer meeting the benchmark (so round 2 bars with dark green and no dark red).

Figure 2: Proportion of handpumps meeting the service level sub-indicator benchmarks

However, the graph shows quite some dark and bright red in the round 2 bars. We found that almost half of newly constructed handpumps do not meet the benchmark on the distance, coverage and quantity indicator. Reliability of newly constructed handpumps is a bit better, with the majority of the new handpump entries being reliable. The proportion of existing handpumps not within 500 m from all users (so not meeting the distance benchmark) is higher in round 2 than in the baseline. A significant overall decrease in the percentage of handpumps meeting the benchmark on this indicator can be seen. If accurate, this could only be explained by population growth outside the maximum distance of 500 m from the handpump. However, we suspect that this difference is also, at least partially, caused by difficulties in giving an accurate assessment of whether or not the users are within 500 meters of the handpump.

The proportion of existing handpumps not within 500 m from all users (so not meeting the distance benchmark) is higher in round two than in the baseline.

The graph shows that a considerable proportion of handpumps that did not meet the benchmark on the ‘coverage’ indicator in the baseline (benchmark: serving not more than 300 people in the case of a borehole and 150 people in case of a hand dug well), did meet this benchmark in round 2. This cannot fully be explained by the increase in number of facilities. A more accurate way of assessing coverage was applied in round 2, as compared with the baseline. This has probably contributed to the increase in the percentage of existing handpumps meeting the coverage indicator. The proportion of handpumps meeting the quantity indicator, has increased slightly. This could also be (partially) due to the fact that the second round of monitoring data collection has taken place in the middle of the dry season, while baseline data collection has taken place in the beginning of the dry season. It could thus be that water use from the handpumps has indeed increased as compared to the baseline. So what has round 2 thought us about the service level sub-indicators?

• Overall, we found little change in reliability and quality between the baseline and round 2. This has strengthened our confidence in these indicators and the way data is collected.
• We suspect that the difference that we found in coverage is caused by better, more realistic assessment of this indicator in round 2. We are looking forward to the third data collection round, which will take place beginning of 2013, to confirm this suspicion.
• We think the differences in existing handpumps scoring differently on the distance and quantity indicator in round 2 than in the baseline, cannot be fully explained by observed changes. These changes may rather be caused by difficulties in assessing this indicator in an accurate way. Therefore, we would like to carefully re-assess these indicators and the way in which data is collected and analysied in order to score these indicators.

Service level

Taking the 5 service level indicators together, provides us with an overview of the proportion of handpumps providing a certain level of service in each of the three districts, in the baseline and round 2. See figure 3. It shows a slightly improved situation in the proportion of handpumps which meet all or all but one of the service level sub-indicator benchmarks. This is probably due to the increase in handpumps meeting the coverage benchmark, as discussed above.

Figure 3: Service levels

Conclusion

The second monitoring round has given us more insight in the accuracy of the functionality and service level indicators. It has made us realize that we have to revisit the way in which we collect data on distance, coverage and quantity. Overall, we did not find major improvements in functionality of existing handpumps over the past year. Actually, we found more existing handpumps with a decreased functionality status, than handpumps with an improved functionality status. However, the fact that round 2 monitoring data was collected towards the end of the dry season, while baseline data had been collected around the beginning of the dry season, could partly explain this. To our surprise (and shock), we found that about a fifth of new handpumps, constructed less than a year ago, were not fully functional in round 2. Also, a significant part of new handpumps did not meet the benchmark on the service level sub-indicators coverage, distance and quantity. We observed a slight improvement in the proportion of handpumps meeting the benchmark on 4 or 5 service level sub-indicators. This is most likely caused by an increase in proportion of handpumps meeting the coverage indicator. However, this seems to have been the result of a revised methodology for assessing coverage, rather than actual improvement of the situation on this indicator.The results of the next monitoring round will have to be used to confirm this.