Impact audits are short-term engagements that draw from internal and external data to generate the best estimate of philanthropic impact. The audit also recommends ways to marshal data to create evidence of effectiveness.
Even the best run nonprofits can benefit from an outside perspective on how to increase their accountability and impact. We provide an objective, independent estimate of the nonprofit's impact and cost, and then provide the nonprofit with specific, actionable guidance on how to improve evaluation and data systems. If the nonprofit agrees to publication, ImpactMatters issues a report, which includes estimates of impact and cost, ratings of the quality of evidence undergirding those estimates and ratings of monitoring systems and data protocols. The impact audit team then issues a private management letter outlining ways that the auditee can improve its effectiveness.
All nonprofits must agree to participate in an impact audit. The steps involved include:
- Document request: We request a set of documents and data from the nonprofit, focusing on program design, impact data and cost data.
- Staff interviews: We interview senior management, some mid-level managers and some field staff.
- Key informant interviews: We often conduct interviews with experts in the nonprofit's sector and individuals with knowledge of the nonprofit.
- Review and analysis: We analyze the nonprofit's impact and cost and assess the evidence base substantiating the nonprofit's impact.
- Prepare and share draft impact report We prepare and share a draft of the impact report and give the nonprofit an opportunity to provide comments or corrections.
- Prepare and share final impact report: Based on the nonprofit's comments, we complete the final impact report and share with the nonprofit. The nonprofit has an opportunity to ask that the report be withheld or to provide a comment to be included in the final report.
- Prepare and share management letter: We prepare and share a letter with recommendations on how to improve the nonprofit's evaluation and data systems.
- Publish impact report: If the nonprofit does not request the results be withheld, we publish the impact audit report.
No. Although randomized controlled trials (or RCTs) are often called the gold standard in evidence, we believe that is a misleading label. First, for many nonprofits, a randomized trial is not feasible (perhaps due to program size, or the nature of the intervention, such as advocacy). Second, if an idea is fully vetted, with RCT evidence from elsewhere, a nonprofit likely ought not conduct its own RCT. Instead, it should implement the program well, and produce evidence that it is doing what it said it would do. Third, innovation should be rewarded, and if an idea is too early for a randomized trial, the nonprofit ought not be discredited for being too early. Having said that, if a randomized trial is feasible, and there is no evidence from elsewhere on the core idea being implemented, we believe that nonprofits ought to be conducting appropriate research to measure impact, which often means a randomized trial, and should share the results of such research with the public.
We encourage nonprofits to agree to publish their impact audit report. However, nonprofits can request that the report be kept private. In that event, we do not publish the report or share with third-parties. In order to conduct the audit, we interview key informants who will therefore know an impact audit is occurring; however we do not share the final results with those key informants.
When a donor has commissioned an impact audit of a specific nonprofit, we will share the final impact audit report with the donor. We strongly encourage the donor and nonprofit to agree up front to the confidentiality protocols for the report.
ImpactMatters staff use documents and other information provided by the nonprofit to conduct the impact audit. We safeguard documents and data shared by the nonprofit and do not publish primary source documents. The final impact report will include references to primary sources provided by the nonprofit provided by the nonprofit, such as the number of beneficiaries served in a particular year.
Although missions vary, nonprofits all follow a basic model: a nonprofit conducts activities in order to achieve outcomes that advance a stated mission. Our objective is to analyze the nonprofit's impact on those outcomes. Some nonprofits are engaged in unique activities, whereas others are engaged in activities more comparable to others. We recognize that, and work to appropriately incorporate that into the analysis.
How can ImpactMatters assess nonprofits that conduct advocacy, research, or a similar long-term or difficult to measure activity?
Long-term or difficult to measure activities pose particular problems for assessing impact. We recognize both these difficulties and the importance of these activities. At the moment, we conduct impact audits on direct service delivery programs. We are working to build a standard for assessing advocacy organizations. If you are interested in contributing, please get in touch.
Can a nonprofit that is developing or piloting a new program that is not evidence backed yet do well?
Yes. We strongly support innovation in the nonprofit field. Our impact audits are best used for nonprofits with some track record of activities. For nonprofits that are innovating, we provide advice on setting up appropriate systems for evaluation from the start. If you are interested, please get in touch.
We understand that staff time is not free, and as a result each audit imposes a cost on the nonprofit. Generally, we estimate 3-5 hours of time from senior management, 5-8 hours from a few other key staff, and 20-30 hours of time from a designated audit lead within the nonprofit. We believe the end product – a rigorous impact estimate with which to fundraise and private guidance to management on how to strengthen data systems – justifies the time. We strive to minimize the burden on the audited nonprofit.
Still have questions?