Real movement, with more work needed: end of project evaluation of Triple-S

Published on: 03/04/2015

Sustainable Services at Scale (Triple-S) has been IRC's flagship project and, together with its partner WASHCost, has been instrumental in changing how IRC works. From December 2008 when the project officially started, till November 2014 when field activities formally stopped (a cost neutral extension of the project will maintain some activities until the end of 2016) the project has sought to challenge business as usual in rural water supply: driving a change agenda focused on the sustainable delivery of rural water services.

Between May and December 2014, an end of project evaluation was undertaken by a team from Hydroconseil and Tremolet consulting. As part of our commitment to transparency, and the use of data to adapt our actions, the final reports of that evaluation are shared here (see downloads section below). This blog post provides a short response to the evaluation's main findings – and an indication of priorities for further work.

Response to the evaluation
So, how did we do? The evaluators found that Triple-S largely or completely met its main goals. However, they also found that despite this success, final proof of concept of the Triple-S approach - in the form of measurable improvement in services in the five pilot districts where Triple-S worked - remains elusive.  Both of these findings are shown in the scorecard for our work in Ghana shown below.

Triple-S's most important – and arguably ambitious - overarching goal was to see a change in the paradigm by which rural water services are delivered: from hardware to services that last. And in this we are delighted that the evaluation found real evidence of movement – in the sector as a whole and particularly in our focus countries: Ghana and Uganda.

Were we the only ones driving this change? No. Indeed, as a project that self-consciously aimed to create a movement – it would be strange if we claimed this to be the case. But, as the evaluators say, we identified, supported and contributed to a tsunami of change that is currently sweeping the sector. Change that means that it is now difficult or impossible for any credible organization working in the sector to ignore the issue of sustainability. This is huge and positive change, and we're proud to have contributed to it.

Like any evaluation that serves as a mirror in which to critically observe yourself, the evaluation showed us things we liked – and things we liked less. In particular, it highlighted the fact that after three years of effort in our pilot districts our data does not show clear movement on quality of services delivered - this despite numerous anecdotes of increased user satisfaction and real efforts by our local government partners to bring more resources to bear to address issues of sustainability.

While we would have liked to see more and clearer improvement, we are not altogether surprised that we haven't. We've always been clear that the process of change required to make rural services truly sustainable will take longer than the three to four years we've been active in the districts. We remain committed to supporting the districts and continuing to monitor – and share – progress beyond the end of the project.

And again, as with any evaluation, there were areas where we simply disagree: where we feel the evaluators didn't "get" some of what we were trying to do. The most significant of these is around our decision not to provide districts with infusions of cash to fix broken facilities or fill funding gaps for service delivery. On the one hand, the evaluators are right in saying that had we spent money directly on service delivery or service improvement – we would have seen improvement in the levels of service delivery in our focus districts. On the other hand, we're not sure this would have proved much. After all, outsiders spending money on short-term fixes that deliver water for a few years is exactly what we were arguing against.

The fact that (in the preferred metaphor of Vida Duti, our country director in Ghana) having helped our national partners to peel several layers from the service delivery onion (including: service monitoring; asset inventories and life-cycle costing; and piloting several interventions around post-construction support) we are now hitting up against the huge challenge of overall insufficiency of funding is not, we think, a failure. Rather it means that we, together with the partnerships we have built in country, are getting to the heart of the challenge.

The next years will tell if we can break through this – mobilizing more resources into rural water: leading to genuinely scaled and genuinely sustainable solutions that don't rely on constant new injections of charitable or aid funding.

What next?
The evaluators made a number of recommendations that we are taking up and feeding into IRC's ongoing work.

They found that, while Triple-S content is of good (even excellent) quality, it is also sometimes dense and difficult to access. It would be much easier for those looking to adopt similar ways of working if there was a condensed and more accessible "toolkit" for them to draw on. We agree – and we're working on it. At IRC we constantly try to walk the line between too academic – and insufficiently rigorous. And we're the first to acknowledge that we don't always get it right.

Therefore, we're committed in 2015 to bringing out a comprehensive toolkit that deals with both the "what" of Triple-S vision – universal access to water services that last (everyone, forever) – and the "how" of getting to this vision: the approach we've been piloting for the last years to support deep, systemic and lasting sector change. We'd love to hear from anyone who has used Triple-S materials over the years. What worked for you? What didn't? What would you like more of?

Another important recommendation of the evaluation was that we should turn the focus of the project to sanitation – and maybe that we should have done so earlier. Again, we agree! Indeed, we have started to do this, with the recent publication of our think-piece on taking a whole system approach to urban sanitation. The service delivery approach that we tested in water is – we are convinced – just as applicable to sanitation and we are committed to extending our work into this critical area.

Triple-S was a large and complex project, and this is a long and exhaustive report. We can't do it full justice in a single blog – even one as long as this! The evaluation will serve as a basis for future work for months and years to come. And we'll address other recommendations – for example relating to presentation of results; support to learning processes; and refreshing of our baseline institutional analysis in subsequent blogs.

For now, it just remains to thank both the evaluators, and all of those who so generously shared their time with them for their efforts. And to assure you of the value that we give to the opinions that you have shared with us.