Small Non-Governmental Organizations (NGOs) in international development have a unique advantage when it comes to engaging communities, establishing partnerships, and advocating rights. However, studies suggest that they are disadvantaged when it comes to demonstrating evidence-based impact for several reasons: a) they lack expertise in development evaluation and tools that can enhance performance like logic models and benchmarks, b) they are under increased pressure to show accountability to their donors, and c) they are often forced to prioritize individual, short-term projects with potentially negligible outcomes due to time and budget constraints. Despite good intentions, small NGOs have the potential to worsen, rather than improve, health and wellbeing if evidence-based impact reporting remains unattainable.
Methodology
To explore the challenges small NGOs face in moving from a single-project-based entity to a vision-led professional entity, we documented the creation of low-cost evaluation tools using a case study of H2O 4 ALL. The NGO, founded in 2008, aims to bring clean water and sanitation to developing areas around the world by partnering with local organizations and communities to establish a vested interest and thus, achieve a lasting impact. Prior to this work, H2O 4 ALL had no formal evaluation experience. Their only standard practice for evaluating international projects was a short, situational analysis months before project implementation.
Photo caption: Magoggo, Uganda. Photo credit: Stephanie Lu.
In development evaluation, much of the focus is on establishing impact through experimental designs (e.g. randomized controlled trials) that are unrealistic for small NGOs. They typically focus on interventions or projects in isolation, rather than in the context of a larger organizational body. Thus, our case study is unique in its theory-driven approach to evaluating a small NGO from an organizational perspective. Additionally, by applying a utilization-focused evaluation framework to guide the process, we ensured that practical questions would lead to useful and actionable answers, and decision-making would be done under real-world constraints. In our previous work we conducted an evaluability assessment to lay the foundation for the development of appropriate indicators for performance measurement. The process helped establish the necessary trust between ourselves and H2O 4 ALL and provided a baseline of evidence for goal-setting and increased buy-in from stakeholders by understanding and prioritizing their evaluation needs.
Next, a focus group and further discussion questions involving H2O 4 ALL’s Board of Directors and Executive Director led to a logic model (Fig. 1) with a bottom-up design that reflected H2O 4 ALL’s grassroots approach to change. The logic model prompted the NGO to address questions such as “Is reasonable progress being made along the different paths to outcomes? What information is (or can be made) available to measure this progress?” In addition to the logic model, we surveyed co-operative education students who had worked at H2O 4 ALL and the NGO’s donors and volunteers. We also developed an observation-based checklist with the NGO that could be used for all of their projects, to keep them accountable to everyone a project serves, including community members and those who supported a project financially. The project implementation checklist was pilot-tested on a trip to Uganda in August 2016, where H2O 4 ALL installed a safe water system for a seniors outreach centre in a small rural town.
Figure 1: Developed logic model to measure impact.
Outcomes
Our evaluation provided a more comprehensive understanding of the organization’s strengths and weaknesses and helped H2O 4 ALL engage with this vital question -- “Are we an organization or are we a set of projects?” The developed logic model, online surveys, and project implementation checklist are examples of simple, low-cost tools that can be used to collect evidence that is easily understandable to someone with little or no formal evaluation training in a short period of time. From this evaluation work we also encountered a number of challenges that could be relevant to small NGOs including those outside of water-based development. For example, when the evaluation plan was first agreed upon, as evaluators, we felt confident that we had prioritized stakeholder needs. However, our experience taught us that having commitment at multiple levels of an organization is a must. Even though H2O 4 ALL’s Board of Directors was unanimous in their decision to support the evaluation, staff members were at risk of being overworked. Under these circumstances, full implementation of the created evaluation tools was limited. We learned that having an internal evaluator or staff member who champions evaluation would have been an asset to the project.
Conclusions
Based on our case study, we think it is imperative that NGOs invest in development evaluation in the expansion/growth stage, if not earlier in the start-up stage. Given the limitations of the case study approach, it is questionable if the findings can be applied to other small NGOs in international development. Approaches to evidence-based policy are often misguided when detailed knowledge of a local context and global knowledge of common behavioural relationships are not seen as complementary. Evaluations add value here because they help reveal why things happened the way they did (i.e. mechanism). Even if one NGO’s context differs from another, there is an evidence base for learning from the behaviour of others. Theory-driven approaches to evaluation are indeed valuable as they can inform practices in other contexts. Although developing evaluation tools using a utilization-focused evaluation approach does not guarantee uptake, by understanding the unique challenges of small NGOs, these tools can ultimately encourage evaluative thinking when other organizational needs are perceived as more pressing.
Lu, S.K., Elliott S. J., Majowicz, S. E., & Perlman, C. M. (2017). An evaluation toolkit for small NGOs in water-based development. Journal of International Development, 30(3), 457-473.
Contact: Stephanie Lu
For more information about WaterResearch, contact Amy Geddes.
