An impact audit is a short-term engagement with a nonprofit, during which an impact audit team collects information about the nonprofit’s intervention and operations and reviews the available internal and external evidence on the nonprofit’s impact, in order to generate a rating for the organization along four dimensions: Quality of Impact Evidence, Cost of Impact, Quality of Monitoring Systems and Learning and Iteration.
Even the best run nonprofits can benefit from an outside perspective on how to increase their accountability and impact. We provide an objective, independent appraisal of the nonprofit's program and operations, and then provide the nonprofit with specific, actionable guidance on how to maximize impact. In addition, the nonprofit, if does not withhold the final report, is issued a public rating to guide its donors and benefits from consumer engagement activities by us and our partners.
All nonprofits must agree to participate in an impact audit. The steps involved incldue:
- Document request and public information search: We request a set of documents and data from the nonprofit, focusing on program design, evidence collected, operations and costs, monitoring systems and major recent iterations to the model. In addition, we conduct a public search for information on the organization.
- Staff interviews: We interview senior management, some mid-level managers and some field staff.
- Field visit: On a case-by-case basis, we may conduct a field visit to observe the nonprofit's systems in the field. ImpactMatters covers its own travel costs.
- Key informant interviews: We often conduct interviews with key outside informants, including both sector experts and individuals with knowledge of the nonprofit. The interviews are selected collaboratively with the nonprofit.
- Follow-up document request: Based on our initial document review and interviews, we may request additional documents and data from the nonprofit.
- Review and analysis: We review and analyze the evidence base for the nonprofit's program and all public and private information we have collected.
- Prepare and share draft impact report and rating: We prepare and share a draft of the impact report and give the nonprofit an opportunity to provide comments or corrections.
- Prepare and share final impact report and rating: Based on the nonprofit's comments, we complete the final impact report and share with the nonprofit. The nonprofit has an opportunity to ask that the report be withheld or to provide a comment to be included in the final report.
- Prepare and share management letter: We prepare and share a letter detailing recommendations to improve the nonprofit's program and operations and provide access to relevant resources.
- Publish impact report and rating: If the nonprofit does not request the results be withheld, we publish the report and work with our partners to disseminate it to donors.
No. Although randomized controlled trials (or RCTs) are often called the gold standard in evidence, we believe that is a misleading label. First, for many nonprofits, a randomized trial is not feasible (perhaps due to program size, or the nature of the intervention, such as advocacy). Second, if an idea is fully vetted, with RCT evidence from elsewhere, a nonprofit likely ought not conduct its own RCT. Instead, it should implement the program well, and produce evidence that it is doing what it said it would do. Third, innovation should be rewarded, and if an idea is too early for a randomized trial, the nonprofit ought not be discredited for being too early. Having said that, if a randomized trial is feasible, and there is no evidence from elsewhere on the core idea being implemented, we believe that nonprofits ought to be conducting appropriate research to measure impact, which often means a randomized trial, and should share the results of such research with the public.
We strongly encourage organizations to agree to publish their impact audit report. However, nonprofits can request that the report be published. In that case, we will not publish the report. However, we may still privately share information from the report. In addition, in order to conduct the audit, we interview key informants, who will therefore know of the impact audit engagement.
When a donor has commissioned an impact audit of a specific nonprofit, we will share the final impact audit report with the donor. We strongly encourage the donor and nonprofit to agree up front to the confidentiality protocols for the report.
ImpactMatters staff use documents and other information provided by the nonprofit in order to conduct the impact audit. The final impact report includes abstractions of some of the information provided by the nonprofit. For instance, it may provide summary numbers on total beneficiaries reached by the program in a given year. The final document will not be released publicly at the nonprofit’s request. We safeguard documents and data shared by the nonprofit request permission before publishing or sharing copies.
Although missions vary, nonprofits all follow a basic model: a nonprofit conducts activities in order to achieve outcomes that advance a stated mission. Our objective is to help ensure that there is evidence to support that the model achieves the intended outcomes. Some nonprofits are engaged in unique activities, whereas others are engaged in activities more comparable to others. We recognize that, and work to appropriately incorporate that into the analysis.
How can ImpactMatters assess nonprofits that conduct advocacy, research, or a similar long-term or difficult to measure activity?
Long-term or difficult to measure activities pose particular problems for assessing impact. We recognize both these difficulties and the importance of these activities, and do not penalize nonprofits that conduct such activities. Where possible, we will use evidence to establish the individual steps in the theory of change. “Evidence” is broader than “evidence of impact.” If “evidence of activities” is the appropriate metric for a nonprofit, then that is what we will follow. We will not penalize nonprofits if impact evidence cannot be generated or it is cost prohibitive to do so.
Can a nonprofit that is developing or piloting a new program that is not evidence backed yet do well?
Yes. We strongly support innovation in the nonprofit field, and will not penalize nonprofits for developing new programs or approaches to solve the world’s problems. Indeed, the opposite is true. Our process will reward innovation, but only if the nonprofit produces appropriate evidence on the innovation both for their own stakeholders and others in the world to learn from what they have done. Often this is a multistep process, and we do not expect nonprofits to jump directly into large impact evaluations. However, programs should be planned and designed with the intent to eventually generate credible evidence on the program’s impact.
We understand that staff time is not free, and as a result each audit imposes a cost on the nonprofit. Generally, we estimate 3-5 hours of time from senior management, 2-3 hours from a few other key staff, and 20-30 hours of time from a designated audit lead within the nonprofit. We believe the end product – a rating with which to fundraise if the nonprofit passes and private guidance to management on how to strengthen programs and systems for all nonprofits – justifies the cost. We do everything possible to minimize the burden on the audited nonprofit.
Still have questions?