Village Enterprise

Executive Summary

Village Enterprise’s mission is to increase income for the extreme poor in Sub-Saharan Africa. The theory behind Village Enterprise’s program is that people living in extreme poverty suffer from multiple, reinforcing barriers that prevent them from making productive investments that could raise their standard of living. These reinforcing barriers, commonly referred to as a poverty trap, include lack of access to credit or savings, lack of information and behavioral biases.

To address these barriers, Village Enterprise provides a combination of cash for working capital, technical skills, financial services and mentorship to the rural poor in East Africa, who create new small businesses. The intervention, similar to what is often called a Graduation program, addresses these many simultaneous causes of poverty. This package of activities is intended to provide a “big push” that moves participants out of a poverty trap and onto a trajectory of increasing productivity and income.

Village Enterprise monitors delivery through process metrics, most importantly whether the businesses that participants form survive. Internal reports indicate 75% of all businesses are still in operation after four years. Village Enterprise’s primary outcome is household income. This primary outcome could lead to additional positive downstream outcomes, including increased consumption, productive assets and savings.

Village Enterprise implements in Uganda and Kenya. In 2015, Village Enterprise directly enrolled 8,148 participants in its program, and expects to have 9,705 participants in 2016. In addition, Village Enterprise is in the process of launching the Village Enterprise Accelerator, which will conduct research and work with other partners to implement and scale the Village Enterprise model.

Village Enterprise is operating at the scale stage. Founded in 1987, Village Enterprise underwent a major shift in organizational strategy and rebranding in 2011, when its first CEO was hired and first five-year strategic plan was developed. The program model rolled out in 2011 most closely resembles the program in its current form. For an organization at this programmatic stage, the quality of impact evidence is assessed along two dimensions: the quality of the internal evidence is has produced and the applicability of external evidence on similar programs.

The theory behind Village Enterprise’s model is supported by rigorous, counterfactual evaluations of similar programs. Four academic papers draw on nine randomized controlled trials, estimating statistically significant and economically important impacts on consumption, expenditure, savings and earnings. The four studies are moderately applicable to Village Enterprise. The interventions in those studies were variations on the same Graduation program approach, situated in low- and middle-income countries. We will discuss potentially important differences between those settings and the Village Enterprise program below.

Village Enterprise is currently participating in a randomized controlled trial on its program. However, as the results are not available yet, this study does not affect Village Enterprise’s Quality of Impact Evidence rating. ImpactMatters will update this impact audit once results are available in mid-2017.

Using evidence from randomized trials of similar programs, we predict the Village Enterprise program increases average household income by $460, from a base of $365 in annual household income. The cost of the program is $250 per participant, meaning that a dollar invested yields about 1.8 dollars in additional household income. The predicted impact in the first year is a $22 increase in household income. The available empirical data provide strong evidence to support the assumption that benefits are perpetual. Although the program is too young to have lifetime data for participants, Village Enterprise’s internal data on business survival shows a 75% survival rate at four years. Long-term follow up data from elsewhere similarly supports that Graduation programs may lead to perpetual benefits. One randomized controlled trial in India has preliminary long-term follow up results, which show the difference between annual impact in year one and year seven is statistically the same (and if anything, income rises slightly in year seven compared to year one). However, because the longest follow-up data available is at seven years, a prediction of $130 using a seven-year horizon is also considered, for a cost of impact ratio of $0.52 per $1 spent.  The cost of impact calculations described used discounted cash flows, adjusted for purchasing power parity and inflation.

Village Enterprise collects high quality monitoring data, which generates useful knowledge for the nonprofit and external stakeholders. The principal monitoring systems reviewed in this impact audit are the consumption and expenditure surveys, entry and exit survey, business tracking report, mentor and training report and the savings group constitution and exit report. The audit evaluates how well Village Enterprise collects five types of monitoring data: what the nonprofit does (activities), who it reaches (targeting), how well those in the program participate (engagement), what participants and other stakeholders think about the program (feedback) and the results of the intervention for participants and other beneficiaries (outcomes).

Most of Village Enterprise monitoring data are collected with a credible methodology, meaning that the information is trustworthy to personnel and outsiders alike. The data are circulated to the appropriate personnel throughout the organization in a timely manner, enabling managers and executives to understand and respond to current problems. Monitoring and evaluation staff demonstrate a high degree of concern for respondents’ burden in data collection and also for respondents’ confidentiality. Monitoring data uses well-known, recognized and calibrated surveys to assess poverty in addition to idiosyncratic geographic filters and participatory methodology. The audit found some room for improvement in monitoring systems. For instance, the focus of the monitoring data could be expanded to include the assumptions embedded in the theory of change, and Village Enterprise could provide the public with more guidance for interpreting the results of before-and-after comparisons while refraining from presenting those results as evidence of impact. However, overall, Village Enterprise has excellent monitoring systems.

Village Enterprise iterates its model periodically and systematically on the basis of high-quality data. It has evolved considerably from its founding as a volunteer-run nonprofit in 1987. Personnel systematically develop and test new ideas. Stakeholders generate hypotheses about viable business plans, test the proposed technologies, study the financial returns and the experiences of innovation against the existing technology, and present their findings across the enterprise through the Innovation Summit.

Mostly significantly, Village Enterprise chose to participate in a large-scale multiple treatment arm randomized controlled trial to test the impact of different variations of its model. Village Enterprise has demonstrated a commitment to iterate its model based on the findings from the study. This study is an exemplary model for how nonprofits should make decisions about their program.

Nonprofit Comment

In this section, the nonprofit is given the opportunity to respond to the impact audit. This statement has not been edited by ImpactMatters.

Village Enterprise is extremely grateful to have participated in ImpactMatters’ audit and to receive expert review of our monitoring, evaluation, research and learning systems along with feedback on how to further strengthen our ability to use and produce appropriate evidence. We are honored to have received three stars out of three in all but one of the categories, demonstrating our commitment to excellence and delivery of a high quality Graduation program.

We are committed to implementing Monitoring and Evaluation programming that meets the CART principles: credible, actionable, responsible and transportable. Through efforts such as employing quasi-independent data collection staff and investing in mobile-based data collection Village Enterprise has already made significant strides on this front. Moving into the future we will continue building on the progress we have already made. For example, to increase the transportability of our work, Village Enterprise plans to update our public facing materials to ensure our M&E systems are clearly defined and that feedback gathered from stakeholders through these systems is also shared.

In spring 2017 we expect the results of our first large scale, five-treatment arm and independent randomized controlled trial (RCT) to be released. In order to end extreme poverty in rural Africa we believe continuous data-driven innovation must be prioritized. By this principle, we are committed to sharing our complete RCT results publicly. It will contribute to further build the current body of evidence of the impact of the Graduation model capturing the marginal effects of some of the unique features of the Village Enterprise program to increase knowledge of the relative impact of each component (cash, training/mentorship, and savings) of our Graduation program, and we are committed to adjusting our programming based on the findings. Village Enterprise is also dedicated to continue innovating through research and pilots, using credible counterfactual evidence whenever possible.

ImpactMatters’ in-depth assessment of our organization highlighting the quality of our monitoring systems, our commitment to learning and iteration, our innovation, and our cost effectiveness, is an exceptional source of pride, inspiration and motivation for our whole staff, board and program participants. And we are extremely thankful for the collaboration, the hard work, dedication and rigor of ImpactMatters’ team.

Nonprofit Program Description

In this section, ImpactMatters summarizes the essence of the nonprofit’s mission and constructs a theory of change for the nonprofit that describes the problem, the nonprofit’s intervention and the appropriate process metrics and outcome metrics for tracking success.

Mission

To increase income for the extreme rural poor in Sub-Saharan Africa.

Theory of Change

Problem

Village Enterprise addresses a problem known in economic theory as a poverty trap. Participants face mutually reinforcing barriers to investment, asset accumulation, education and credit. The intervention Village Enterprise runs is designed to address specific facets of the poverty trap, providing participants with education, training, mentorship, working capital and consumption support.

A poverty trap refers to a situation where poverty is self-reinforcing. Out of the poverty trap, households can productively invest and raise productivity. But for those in a poverty trap, long-term investments are not viable, and all funds go to subsistence needs. Poverty traps can persist for a myriad of reasons, including missing credit and savings markets, poor information, behavioral constraints that hinder investment and missing insurance markets.1–11

Missing Credit and Savings Markets

The rural poor in sub-Saharan Africa have limited access to financial services. Bank lending is scarce and expensive.12 High borrowing costs can result from poor information about creditworthiness, poor access to collateral, and high transaction costs to the lender. Savings are also scarce and expensive. Nonfinancial assets, when they do exist, may be tied up in illiquid forms such as livestock, poorly suited to collateral. The absence of insurance in agriculture across low-income countries is a drag on investment, and agriculture insurance remains an area of active policy innovation.

Poor information

An information failure occurs when a market actor has imperfect knowledge. The rural poor systematically lack information about how to productively and efficiently manage their assets.13 Village Enterprise participants lack in several domains: business management, financial literacy and knowledge of new productivity-enhancing technologies that would, for instance, enhance crop yields and thereby lead to increased incomes.14

Behavioral biases and Missing Insurance Markets

The extreme poor may suffer from low productivity because of lowered expectations for the future and low levels of hope and aspiration, leading to lower investment and income. Some economic research suggests that “temptations” and social obligations cause the poor to fritter away savings or borrow and thus incur interest costs, which makes poverty self-reinforcing.15 Savings accounts with reminders and commitment devices, at even very low balances, can reverse the endogenous deterioration of self-control due to poverty.16,17 Even the normal distractions of life can cause disproportionate interference with the work of the poor, although there remains little empirical work on inattention and poverty traps.18,19

Furthermore, the extreme poor may be highly risk averse. Where insurance markets are missing, this may lead to underinvestment in productive activities.

Other factors

There are a number of other factors that may play a role in sustaining the poverty trap faced by Village Enterprise’s clients. These include:

  • Land policy: Because of land subdivision policies, many ultra-poor individuals in Kenya have very little or no land and cannot start an agricultural business.20
  • Conflict: Village Enterprises implements in Northern Uganda. The legacy of civil war may have an ongoing impact on clients, due to ongoing displacement or instability, or low levels of investment in human capital and productive assets.21
  • Poor nutrition: The extreme poor may not consume enough food to make them productive.22 Recent research suggests that malnutrition is seldom the primary cause of low productivity in a poverty trap.23
  • Tragedy of the Commons: Village Enterprise clients that live near nature preserves (the Kisere Forest in Kenya and the Budongo Forest in Uganda) may use limited natural resources in an unsustainable way, leading to a negative impact on livelihoods and the environment.24

Local Context

Village Enterprise targets the rural poor in Kenya and Uganda, focusing on specific districts and villages with the highest rates of poverty. Within those villages, Village Enterprise conducts a participatory wealth ranking exercise, wherein local opinion leaders are invited to a focus group to define categories of relative wealth for their village and then rank households in the village according to those categories. Of the poorest households identified, Village Enterprise uses the Progress out of Poverty Index, a country-specific poverty measurement tool, to select only those with a 95% likelihood of falling under a poverty line of $1.90 in daily income.25 This multi-step targeting process results in a participant population that has no current employment apart from farm income and no history of business ownership. 81% of participants are women, with an average age of 36 and family size of 6.5 dependents.26

Credit and savings: In rural Uganda, 16% of the population borrowed from a bank in 2014, but 81% borrowed money.27 The most common reasons to borrow money are for school and health care, and 22% borrowed money to start, operate, or expand a business (including farms). For rural Kenya, 16% of the population borrowed from a bank in the past year, but 79% borrowed money, and 24% borrowed to start, operate, or expand a business. Just 3% of Kenyans and 9% of Ugandans working in agriculture purchased insurance in 2011 (the most recent year available).

Business training: Poor villagers often have no formal training in finance, accounting, operations, marketing, or management, which exacerbates the risk of running a small business. They may also lack technical skills in agriculture, livestock, commerce, and various other small businesses.

Missing market for insurance: The rural poor have limited or no access to insurance against catastrophic losses in agriculture and livestock, leading them to minimize the risk of losses rather than maximize productivity. Insurance would encourage higher investment in improved farm inputs, such as seed, fertilizer, feed, and veterinary care, by protecting against catastrophic losses.

Villagers remain poor because these interlocking problems impede investment, asset accumulation, and long-term productivity gains. The resulting social failure is the low level of per capita consumption expenditure. 

Activities

Village Enterprise helps the rural poor start small businesses. The Village Enterprise program is delivered over one year. First, Village Enterprise uses a multi-step process to identify people living in extreme poverty to participate in the program. Participants then begin a sequence of trainings led by Business Mentors and are formed into groups of three. Each business group chooses a business to start and develops a business plan. Village Enterprise then transfers an initial grant of $100 in seed capital to the business group. Village Enterprise forms Business Savings Groups consisting of ten business groups each (i.e. 30 business owners in one Business Savings Group). Business Savings Groups provide a mechanism for participants to save and loan out their savings. Business Mentors continue to conduct training and coaching for participants. Six months following the initial grant, if business groups satisfactorily meet business goals, they receive a second grant of $50.

Business Mentors

Business Mentors assist groups in selecting an enterprise and provide support to each group throughout the training process.25 Village Enterprise Business Mentors are hired from within the communities in which Village Enterprise works. The process starts with advertising for the position on local notice boards, local radio stations, the Village Enterprise website and through appropriate listservs. Applicants are shortlisted and invited to an induction exercise. Village Enterprise takes potential Business Mentors through a week-long hiring and induction exercise that involves training on the modules and the delivery methodology, breakout sessions, individual training practice sessions, testing of understanding and a field visit to the communities. Village Enterprise invites a larger number of potential Business Mentors to the induction week than is required. Throughout the week, the potential Business Mentors are asked to market themselves for the position.  On a daily basis, recruits are assessed and only those who demonstrate ability to articulate the training objectives, methodology, ability to train, and demonstrate a clear understanding of the clientele, organization and the expected outcomes are invited to continue on. Only those who make it to the last hour get hired as Business Mentors.

Targeting and Selection

Village Enterprise conducts a multi-step process to identify participants who are living in extreme poverty. Village Enterprise starts by selecting poor districts and then working with district officials to identify poor villages within those districts. At the village level, Village Enterprise conducts a Participatory Wealth Ranking (PWR) Exercise, during which local opinion leaders define poverty in the local context and rank households from “very poor” to “rich.” Finally, Village Enterprise uses a standard checklist, the Progress out of Poverty Index (PPI), to verify that participants are poor. Participants who are identified through the PWR and PPI are then invited to join the program.

Training

With the assistance of Business Mentors,25 individual entrepreneurs form groups of three, called business groups, who co-own and operate a business. Business groups follow a year-long business and financial skills curriculum. The curriculum includes modules that cover business planning, profit analysis, marketing, savings and bookkeeping.28 Each group must create and submit a business plan to Village Enterprise.

Seed Funding

Village Enterprise provides a $150 grant to business groups in two installments: $100 after four months of training and, dependent on satisfactory achievement of business goals, the remaining $50 after another six months. The organization does not provide loans, with the twin rationales that entrepreneurs (1) need to be protected from the risk of default and (2) need time to launch their businesses before cash is available for debt service.25

Business Savings Groups

During the year-long training process, business groups form 30-member Business Savings Groups (BSGs). BSG members pool savings and make loans to other members, thereby easing future credit constraints faced by member businesses. As a part of the overall training process, participants are trained in bookkeeping and savings and loans strategies, and each BSG must create a constitution and elect its own executive committee.29 Business Savings Groups are the exit strategy for Village Enterprise – the means by which the entrepreneurs it trains can continue to access support and capital from fellow entrepreneurs.

Assumptions

In order for Village Enterprise’s service model to be effective, a number of assumptions must be met. Assumptions describe inputs provided to the intervention by stakeholders other than the project’s own personnel, funds and operations. There are a nearly endless number of assumptions in any organization about the quality of employees and operations; and here we specifically focus on things other stakeholders provide to the project.

  1. Profitable business opportunities: Village Enterprise participants have profitable business opportunities that can be accessed with a small grant of working capital. Due to a lack of financial services, they can neither accumulate assets at reasonable cost, nor borrow working capital. In other words, the grant of working capital is more effective than a simple, unconditional cash transfer if it permits participants to start a sustainable business.
  2. Business training raises productivity: Financial literacy and business training will make business owners more productive. The theory of change assumes that financial literacy and business training are effective in raising returns to small businesses.
  3. Desire: Participants want to own and operate a small business. They have whatever physical strength, intelligence, social networks and grit is required to operate a sustainable business. Their homes and lifestyles are compatible with the demands of the businesses they choose.
  4. Ability to motivate: Business Mentors will develop rapport with business owners, so that they can advise and motivate them.
  5. Access to inputs: Agriculture and livestock extension services can provide locally appropriate inputs for farm businesses.
  6. Market access: Markets for selling goods from the small business (typically in agriculture and livestock) are accessible and predictable. Business owners know how and when to sell their goods.

Risks

A number of risks could potentially undermine the impact of Village Enterprise’s program, even if Village Enterprise is successful in providing the promised services to participants.

  1. Social pressure: Female business owners may face social pressure to share ownership of the business and profits with male members of the household or family. The business owners may run the risk of losing control of their assets, whether productive assets (such as livestock and agriculture inputs) or liquid assets. Household members could also make additional demands on the business owners’ time outside of work, which may interfere with their productivity.
  2. Climactic and health factors: Participants primarily purchase livestock or agricultural inputs with seed capital provided by Village Enterprise. Climactic factors, such as drought, or health factors, such as outbreaks of livestock diseases, may reduce or destroy participant’s assets.
  3. Insufficient capital: Seed capital may be insufficient to begin or sustain a small business.
  4. Conflict: Interpersonal conflicts could threaten the survival of business groups and Business Savings Groups. In addition, Village Enterprise operates in areas of Uganda that have experienced significant recent violence that heavily disrupted local economies. There is a risk that violence could resume, jeopardizing the participants’ businesses.
  5. Appropriation: Participants are endowed with new wealth through the program in such a way that it is widely known throughout the community. Village Enterprise operates in areas with underdeveloped local legal systems, increasing the risk that new assets could be appropriated. Similarly, local cultural and political factors could prevent the efficient operation of participants’ businesses, even if no direct appropriation occurs.
  6. Local price effects: Business owners run the risk of crowding into the same markets. If business owners produce too many similar goods and services, prices and profits could fall. They are exposed to scarcities of inputs and gluts in output.

Process Metrics

Village Enterprise tracks the participation rates in its program through the following primary metrics:

  • Number of individuals that enter the program
  • Poverty status of participants and their families at the selection phase (Progress out of Poverty Index)
  • Number of businesses formed
  • Number of savings groups formed
  • Survival of businesses at end of the program (12 months)
  • Survival of savings groups at end of the program (12 months)

Outcome Metrics

Primary (household): Income

Secondary (household):

  • Savings30
  • Assets
  • Consumption and expenditure
  • Food security
  • School attendance
  • Nutrition31

Program Details

Geography

Village Enterprise works in Kenya and Uganda, specifically targeting sub-counties or districts and villages with the highest rates of poverty. In Kenya, those sub-counties are Cherangany, Kwanza and Soy; in Uganda, Village Enterprise works in the districts of Amuria, Dokolo, Hoima, Katakwi, Kole, Kiryandongo, Kumi, Lira, Masindi, Ngora, Nwoya and Soroti.

Figure 1. Locations of Village Enterprise Operations in Kenya

Locations of Village Enterprise Operations
Locations of Village Enterprise Operations

Figure 2. Locations of Village Enterprise Operations in Uganda

Locations of Village Enterprise Operations
Locations of Village Enterprise Operations

Stage

With a history of more than 20 years, Village Enterprise is at the scaling stage. Village Enterprise has had a professional CEO since 2010 and explicitly describes ending extreme poverty across Africa as its strategic goal. Its current approach to partnerships focuses on expanding the target population rapidly, not testing how or where the intervention works, as we might expect from a validation stage organization.

Scale and Age

In 2016, Village Enterprise trained 9,705 business owners who opened 3,235 businesses. Since its founding in 1987, Village Enterprise has trained 146,000 owners that opened 36,000 businesses.32

Figure 3. Businesses Opened Annually and Number of New Business Owners

Village Enterprise Scale

* Based on Annual Reports for 2012 and 2013
** Based on review of raw monitoring data provided by Village Enterprise
*** Based on projections provided by Village Enterprise

Scaling Strategy

Village Enterprise Accelerator

In 2016, Village Enterprise launched the Village Enterprise Accelerator, a revenue-generating division of the organization that scales up cost-effective, evidence-based approaches to poverty alleviation in rural Sub-Saharan Africa.

The Accelerator performs three key functions:

  1. Provides technical assistance to larger NGO and government partners interested in replicating the Village Enterprise model in other countries.
  2. Partners with research institutions and universities to identify best practices and serve as thought leaders in the sector.
  3. Pilots and incubates new innovations that will contribute to ending extreme poverty in rural Sub-Saharan Africa.

Iterations that have been tested at the Village Enterprise Accelerator and adopted thereafter include the Smarter Market Risk Analysis Tool (SMART) and Savings With A Purpose (SWAP). SMART is a mobile technology tool that provides farmers with up-to-date market information, allowing them to identify top-performing crops and make informed business decisions. SWAP aims to increase savings in BSGs by encouraging BSG members to set personal savings goals and holding them accountable to achieving their respective goals.33 An assessment of the iterations that have been tested and adopted within the past three years is included in Learning and Iteration.

Partnerships

Scaling Partnerships

Village Enterprise forms partnerships to integrate and deliver its microenterprise development program as part of other organizations’ programmatic offerings. For example, the Community Connector Project managed by FHI 360 and funded by USAID employs the Village Enterprise model as part of its broader program to improve the nutrition and livelihoods of vulnerable populations in Uganda, with a focus on youth poverty.34

Technical Assistance Partnerships

Village Enterprise also partners with other organizations to train them in implementing specific components of the core microenterprise development program. For example, Village Enterprise has formed a partnership with the Wildlife Conservation Society to train Forest Monitors, who are akin to Business Mentors in that they in turn train private forest owners in Uganda using a shortened and modified version of the standard Village Enterprise BSG program and assist them in forming BSGs.

How Donations Are Used

Village Enterprise is investing new donations into strengthening the foundations for rapid scaling and implementing its five-year strategic plan. Village Enterprise’s five-year plan, adopted under the advice of Bain & Company in late 2015, outlines a two-pronged growth strategy:

  1. To continue to expand organically in Kenya and Uganda, with a stated goal of continuing to scale the number of businesses started directly by Village Enterprise by at least 15% annually;
  2. To scale up in other countries in Africa by providing technical assistance to other NGOs and governments via the Village Enterprise Accelerator.

Village Enterprise anticipates the Accelerator will eventually be financially self-sustainable, as it will earn revenue from licensing the Village Enterprise model and from providing technical assistance to other entities.

Quality of Impact Evidence

Rating

Village Enterprise is operating at the scaling stage. Though founded in 1987, Village Enterprise underwent a major shift in organizational strategy and rebranding in 2010, when its first CEO was hired and first five-year strategic plan was developed. The programmatic model rolled out in 2010 most closely resembles the program in its current form. Village Enterprise has some evidence that its own program works, and good evidence that similar programs have worked in comparable contexts elsewhere. For an organization at the scaling stage, the quality of impact evidence is assessed along two dimensions: the quality of the internal evidence it has produced and the applicability of external evidence on similar programs. Village Enterprise has medium-applicability external evidence. While Village Enterprise is currently producing high-quality internal evidence, the study is not completed. As a result, Village Enterprise earns two stars for Quality of Impact Evidence.

Note: Village Enterprise is currently conducting an independent validation of its program in conjunction with Innovations for Poverty Action. Endline results from the study are expected in mid-2017. ImpactMatters expects these findings will likely increase Village Enterprise’s rating for Quality of Impact Evidence.

Table 1. Findings on Quality of Impact Evidence

Evidence Source Finding
Internal Evaluation Medium Quality
Independent Validation In Progress
Evidence from Elsewhere Medium Applicability

Internal Evaluation Medium Quality

Village Enterprise collects comprehensive internal monitoring data on the effectiveness of its targeting processes, participants’ take-up and engagement in activities, and non-counterfactual outcomes in savings, consumption and expenditure and income (as predicted by Progress out of Poverty Index (PPI) scores). Key monitoring data excerpted here show Village Enterprise is highly effective at reaching the extreme poor for inclusion in its program and has low program attrition, high attendance at trainings, strong business group survival and strong business survival, but high attrition from BSGs. Pre-post comparisons – which lack a counterfactual and therefore are a poor measure of impact – show that participants’ PPI scores and personal savings increased considerably after the program, but that consumption and expenditure increased modestly in some areas and decreased in others. Quality of Monitoring Systems provides a full discussion of how these data are collected, validated and used.

Targeting Effectiveness

Village Enterprise targets individuals who live on under $1.90 per day, have no experience operating a business and are unable to provide for their family’s wellbeing. The organization uses a multi-step targeting process to identify individuals who meet these criteria: first, they target the poorest geographies. Then, within the poorest areas, they conduct participatory wealth ranking exercises to identify the poorest households. Finally, they verify the results of the exercise with a Progress Out of Poverty Index survey, as well as an assessment against several exclusion and inclusion criteria.

Participatory wealth ranking is an exercise through which individuals identify the relative wealth or poverty of members of their community, thereby identifying the poorest households for participation in an intervention. Implementation varies by location, but generally involves facilitation of a meeting in which a group of community members name and sort all households into agreed-upon categories of poverty. Village Enterprise conducts the exercise with community leaders, who sort all households into four categories: very poor, poor, moderate, and wealthy. Households identified as very poor and poor are eligible for participation.

The Progress Out of Poverty Index (PPI) is a tool that calculates the likelihood that a household in a given country is impoverished. Interviewers administer a ten-question survey to households, asking about household features (for example, household size, school enrollment, amenities and structural features). The answers are scored and a composite index is computed to indicate the likelihood that a household falls under international poverty lines, where a lower index corresponds to a higher likelihood of poverty. The tool is often used to target households that would benefit from poverty-alleviation programs (and to exclude those who would not). Functionally, this means that an organization sets a cut-off score, and households below that score qualify for participation in the program. Village Enterprise’s current qualification cut-off scores are 39 in Kenya and 29 in Uganda, which translate to a 95% chance that a given household is living below $2.50 a day (2005 PPP).

Village Enterprise may include participants who score above the PPI threshold if they meet two of the following criteria: the household head is unemployed or un-pensioned, there are eight or more children living in the household, the household head is widowed, disabled, or an orphan under the age of 18, the household suffered from a natural catastrophe and the household head suffers from HIV. Likewise, participants who score below the threshold may be excluded if they meet any one of the following criteria: teacher or salaried worker in the household, ownership of three or more cows and a structurally complete house (cement floor, brick, metal roof).

Participatory wealth ranking is an effective means to stratify a community into broad categories of relative wealth, as defined by measures of health, food security, income and landholding status.35–37 One study found it to be the least accurate in terms of measuring absolute poverty, compared to three other poverty measurement tools (Poverty Assessment Tool, Visual Impression tool, and the Ladder of Life). In that study, it accurately identified between 70% and 80% of households living on less than $1.00 per day.36

There is limited but positive evidence for the validity of the PPI as a means to identify poor households. One study, examining Rwanda’s PPI, found that it correctly distinguished poor and non-poor households, such that the percentage of impoverished households increased as the scores decreased.

There is some evidence that combining targeting methods is an effective way to identify poor households. One study found that a participatory wealth ranking exercise followed by verification performed better than random selection, benchmarked against five poverty metrics. However, there was not strong evidence that the combined approach was more accurate than other methods, such as a housing index or the PPI alone, and the results of the study were heterogeneous by country of intervention and poverty benchmark.38 Another study found that a combined approach, as described above, was more accurate than a census, according to measures of land holdings, assets and credit access, but not consumption.39

Based on the baseline results of Village Enterprise’s randomized controlled trial, there is evidence that the organization is targeting some of the poorest households in Uganda. The investigators compared a sample of Village Enterprise households to both the overall results of the Uganda National Household Survey (UNHS) 2012-13 and a subsample of the survey from the districts in which Village Enterprises operates. They then compared the samples using eleven welfare indicators, such as the percentage of households that have a thatched roof or those in which every household member owns a pair of shoes. On seven of the eleven indicators analyzed, the districts in which Village Enterprise operates were relatively poorer than the national sample, and the Village Enterprise sample was relatively poorer than the survey district samples on all eleven indicators.

Over 97% of targeted households in the baseline sample scored below 39 on the Progress out of Poverty Index, which is equivalent to a 55.3% likelihood of living below the $1.25 (PPP) per day poverty line and a 93% likelihood of living below the $2.50 per day poverty line. Based on these results, it is highly likely that Village Enterprise is effectively targeting the ultra-poor for its program.

The investigators also calculated a multi-dimensional poverty index to represent the relationship between observable characteristics, such as having a thatched roof, and unobservable characteristics, such as household wellbeing. In comparing the distribution of scores from the three samples, the study found that the median score of Village Enterprise households was one standard deviation lower than the median score in the national sample. Three quarters of the Village Enterprise households fall below the median score of the district sample. Segmenting the Village Enterprise sample by district and national quartile scores reveals that over half of the households in the Village Enterprise sample would fall into the poorest quartile of the other samples.

There are two limitations to this analysis. First, it does not indicate whether Village Enterprise is targeting the poorest households within villages, as the smallest geographic unit is the district. Second, the district subsample comprises just 171 households. The survey is designed to be valid at the national level, rather than the district level, so it is not clear that the district subsample is large enough to accurately represent the underlying population.

Activity Take-Up and Engagement

Monitoring data generally demonstrate that participants have high rates of take-up and engagement in Village Enterprise’s program activities. Over Village Enterprise’s last three cycles, attrition from the program across all sites remained within the 2-6% range and decreased over time. The interim report from Village Enterprise’s randomized controlled trial in Uganda corroborates this result. Attrition from BSGs is higher: Village Enterprise’s most recent available internal performance report from 2013 to 2014 shows 75% of BSGs in Hoima, Uganda experienced dropouts; 27% in Soroti, Uganda; 80% in Kakamega, Kenya; 63% in Kisere, Kenya; and 70% in Eldoret, Kenya. However, of the BSGs that experienced dropouts, the average number of participants that left was not of great concern, ranging from just one in Soroti to 11 in Eldoret, out of 30-member BSGs.

A sample of recent spot check data shows high rates of business group survival, low numbers of business owners missing trainings and acceptable attendance rates at BSG meetings. Village Enterprise’s internal data on business survival shows a 75% survival rate at four years. An internal presentation on business spot checks conducted during the program reports that of the 85% of total businesses that had been spot-checked so far, only 1% had failed.

Outcomes

Importantly, all of the following data are based on before and after comparisons, which have a weak counterfactual and therefore are poor measures of impact. Village Enterprise’s most recent end-of-cycle internal performance report indicates large increases in PPI scores in all sites: 79% in Uganda and 45% in Kenya. Consumption and Expenditure Survey data from 15% and 5% of business owners in Uganda and Kenya, respectively, showed modest gains, which were complicated by some decreases: in Uganda, 2% and 11% in Hoima and Soroti, respectively; in Kenya, -20%, -11% and 15% in Kakamega, Eldoret and Kisere, respectively. The number of business owners with personal savings increased from 50% to 76% from baseline to endline in 2014 in Uganda, and the average savings value increased by 96%. Information on individual business owner savings was not available for Kenya.

Displacement Effects

Independent Validation In Progress

In 2013, Village Enterprise launched a three-year randomized controlled trial conducted in partnership with an independent evaluator, Innovations for Poverty Action, to evaluate its model. Baseline and midline reports have been completed, with endline results forthcoming in 2017.

In order to obtain a three-star rating in Quality of Impact Evidence, organizations that are operating in the scaling stage must have highly applicable internal evidence (from internal evaluation or independent validation) or medium applicable internal evidence and highly applicable evidence from elsewhere. Because the Village Enterprise independent randomized controlled trial is still ongoing, it does not yet constitute evidence to support scaling up the program. Though Village Enterprise is producing highly applicable internal evidence, it does not at present already have highly applicable internal evidence. However, ImpactMatters expects Village Enterprise’s ratings to change in the near future upon completion of the randomized controlled trial.

Note: Village Enterprise is currently conducting an independent validation of its program in conjunction with Innovations for Poverty Action. Endline results from the study are expected in mid-2017. ImpactMatters expects these findings will likely increase Village Enterprise’s rating for Quality of Impact Evidence.

*Note to the reader: ImpactMatters is currently being incubated by Innovations for Poverty Action (IPA). However, IPA had no role in or decision-making power over the contents of this report.

Table 2. Findings on Independent Validation Studies

StudyQualityRelevance
Proefke et al. (forthcoming)In ProgressIn Progress
ConclusionUnknownUnknown

Proefke et al. (in progress)

Table 3. Details of Proefke et al. (in progress)

Timeframe2013-2017
InterventionVillage Enterprise model
MethodCluster-randomized controlled trial. Villages in the RCT are randomized into five groups: control; cash transfers only; full Village Enterprise model; Village Enterprise model without Business Savings Groups (BSGs); and Business-in-a-Box (BIAB) model
Sample
  • Control: 2,520 households selected from 36 villages
  • Full Village Enterprise model: 2,520 households selected from 36 villages
  • Village Enterprise model, no BSG: 1,680 households selected from 24 villages
  • Cash transfer: 2,520 households selected from 36 villages
  • BIAB model: 420 households selected from 6 villages
  • Total: 6,678 households from 138 villages randomized into above 5 groups
GeographyUganda
EvaluatorInnovations for Poverty Action
InvestigatorsRachel Proefke, Richard Sedlmayr, Munshi Sulaiman
StatusBaseline and midline reports completed; expected publication: 2017

Quality of Findings In Progress

ImpactMatters has not reviewed the study for bias and other issues, as the final report is not available. However, based on the study design, study pre-registration, baseline and midline results that were made available confidentially to ImpactMatters, and interviews with some members of the study research team, we anticipate that the study will find high-quality evidence for whether Village Enterprise is having an impact and the size of that impact.

Relevance of Findings In Progress

The independent randomized controlled trial includes multiple treatment arms that test the Village Enterprise model currently in practice as well as the model modified to include or exclude key components, such as BSGs. The trial is being conducted directly on Village Enterprise’s operations in regions in which it typically operates. The trial used the same selection criteria used by Village Enterprise in standard operations. As a result, the relevance of the findings to Village Enterprise’s program is likely high.         

Evidence from Elsewhere Medium Applicability

The impact of the Village Enterprise model is supported by high-quality counterfactual evaluation of the impact of similar programs. Nine high-quality randomized controlled trials, reported in four academic papers, of similar interventions were identified. The studies found statistically significant impacts across several measures of economic outcomes, including consumption, expenditure, savings and earnings. The four studies are moderately applicable to Village Enterprise. All four studies were variations on a multi-faceted livelihoods intervention targeted at the extreme poor in low income and lower-middle income countries, with the most common components of the intervention being a cash or productive asset transfer, micro-entrepreneurship training and a savings mechanism.

However, there were important differences in the design of each study and Village Enterprise. Two of the studies were specifically targeted at young adults, in contrast to Village Enterprise’s model, which does not have an age focus. In addition, some studies included a health component and a consumption stipend for participants, which Village Enterprise’s program does not include. The grant size, intensity of training and mentoring and length of the program varied between studies and between the studies and Village Enterprise. In general, the per-person grant size in the Village Enterprise program ($50) is significantly lower than the studies.

In addition, the Graduation intervention is particularly complex and several elements, such as participant targeting and training, are highly dependent on the quality of implementation. Given the complexity of the program, the relevance of these findings to Village Enterprise’s program are lower than for a simpler intervention, such as a cash transfer.

As a result, although a large, high-quality body of studies shows impact for a similar intervention to Village Enterprise’s program, the external evidence is of medium relevance, leading to overall medium applicability evidence from elsewhere.

Blattman et al. (AEJ-AE 2015)

Table 5. Details of Blattman et al. (AEJ-AE 2015)

InterventionWomen's Income Generating Support (WINGS) program
Timeframe2002-2005
EvaluatorInnovations for Poverty Action
MethodCluster-randomized controlled trial
Sample120 villages randomized into treatment and control groups
Treatment: 60 villages (896 participants)
Control 60 villages (904 participants)
GeographyTwo war-affected districts in northern Uganda, Kitgum and Gulu

Study Abstract

We show that extremely poor, war-affected women in northern Uganda have high returns to a package of $150 cash, five days of business skills training, and ongoing supervision. Sixteen months after grants, participants doubled their microenterprise ownership and incomes, mainly from petty trading. We also show these ultrapoor have too little social capital, but that group bonds, informal insurance, and cooperative activities could be induced and had positive returns. When the control group received cash and training 20 months later, we varied supervision, which represented half of the program costs. A year later, supervision increased business survival but not consumption.

Findings (at 16-month endline)

151% increase in participant monthly cash earnings***

(control baseline mean: $11.93.03 PPP per month, or $0.39 PPP per day)

29% increase in household non-durable consumption***

(control endline mean: $125.13 PPP per month, or $4.11 PPP per day)

309% increase in household savings***

(control baseline mean: $7.00 PPP)

*** (**) (*) Significant at 1% (5%) (10%) level. All percentage changes compared to control means at endline. Estimates are intent to treat. We report effects from group training (as opposed to no group training), as this is more comparable to Village Enterprise’s model. Additional economically and statistically significant effects found on durable asset consumption.

Quality Of Evidence High

No major threats to internal validity identified. Our analysis is based on the Cochrane criteria for assessing risk of bias and the Coalition for Evidence-Based Policy’s checklist for reviewing randomized controlled trials.

Risk of bias associated with 
Random sequence generation:Low
Allocation concealment:Low
Blinding of participants and personnel:Low
Blinding of outcome assessment:Uncertain
Incomplete outcome data:Low
Selective reporting:Low
Study design uses 
Appropriate level of random assignment:Yes
Adequate sample size:Yes
Validated measure of the outcome:Yes
Appropriate follow-up period:Yes

Relevance Medium

Complexity of intervention

The intervention implemented both in Blattman et al. and by Village Enterprise is highly complex. For several components, the quality of implementation will likely heavily influence the success of the program:

  • Targeting participants through participatory wealth ranking
  • Selection of livelihood activities
  • Business skills training
  • Coaching and mentoring

Furthermore, there are components where variations in specific design could contribute to substantially different outcomes, such as the size of the grant transferred. Finally, external factors, such as weather patterns, local attitudes toward entrepreneurship and the structure of the local economy could substantially affect outcomes.

Similarity of models

The Blattman et al. intervention includes targeting and three components, a cash grant, training and ongoing supervision, all of which are also part of the Village Enterprise model. The main differences in implementation of these common components are as follows:

Cash grant: Participants in the Blattman et al. program received $150 in cash each, while Village Enterprise participants receive $50 each. In addition, Village Enterprise’s disbursement timeline is longer: the Blattman et al. grant is divided into two equal installments disbursed two and six weeks post-training, whereas Village Enterprise business groups (three participants each) receive $100 four months into training and, contingent on their success, receive $50 ten months into training.

Training: WINGS participants in Blattman et al. received five days of business skills training, culminating in the creation of a business plan. Village Enterprise participants are also trained to develop business plans, but they receive 16 training modules over the course of 12 months.

Ongoing supervision: WINGS participants in Blattman et al. received up to five one-on-one supervisory visits over the six months following grant disbursal. Village Enterprise participants receive mentoring visits every month for 12 months.

Finally, program duration differed between the two models. Blattman et al.’s WINGS program lasted approximately eight months (six months of supervisory visits following grant disbursal), while Village Enterprise’s program lasts one year.

Targeted population

Blattman et al. targeted extremely poor young women in Uganda. Village Enterprise’s program targets a similar population in Uganda and Kenya, but does not focus on youth. Blattman et al. argue that young adults may have "unobserved initiative, connections or affinity for entrepreneurship" that increase their likelihood of success in such a program, and this may therefore limit the relevance of Blattman et al.’s findings to the Village Enterprise population.

Local context

There is considerable overlap between the implementation contexts of Village Enterprise and Blattman et al.’s WINGS. Village Enterprise implements in Uganda and Kenya, and Blattman et al.’s study also took place in Uganda. Most notably, WINGS included participants from Kitgum and Gulu, two areas affected by armed conflict in Northern Uganda. Village Enterprise also works in war-affected and newly resettled areas in Northern Uganda.

Bandiera et al. (QJE 2016)

Table 6. Details of Bandiera et al. (QJE 2016)

InterventionTargeted Ultra-Poor (TUP) program
Timeframe2007-2011
EvaluatorImproving Institutions for Growth (iiG); BRAC
MethodCluster-randomized controlled trial
SampleTreatment participants: 4,045
Control participants: 2,687

Study Abstract

The world's poorest people lack capital and skills and toil for others in occupations that others shun. Using a large-scale and long-term randomized control trial in Bangladesh this paper demonstrates that sizable transfers of assets and skills enable the poorest women to shift out of agricultural labor and into running small businesses. This shift, which persists and strengthens after assistance is withdrawn, leads to a 38% increase in earnings. Inculcating basic entrepreneurship, where severely disadvantaged women take on occupations that were the preserve of non-poor women, is shown to be a powerful means of transforming the economic lives of the poor.

Findings (at four-year endline)

38% increase in participant annual earnings***

(treatment baseline mean: $240.98 PPP per year, or $0.66 PPP per day)

4% increase in household per-capita food expenditures*

(treatment baseline mean: $463.43 PPP per year, or $1.27 PPP per day)

866% increase in household savings***

(treatment baseline mean: $6.35 PPP)

*** (**) (*) Significant at 1% (5%) (10%) level. Note: no control baseline mean data for selected outcomes provided for comparison. Baseline treatment mean inflated at 5% annual social discount rate. Additional economically and statistically significant effects on asset ownership, food security, non food per-capita expenditures and well-being.

Quality of Evidence High

No major threats to internal validity identified. Our analysis is based on the Cochrane criteria for assessing risk of bias and the Coalition for Evidence-Based Policy’s checklist for reviewing randomized controlled trials.

Risk of bias associated with 
Random sequence generation:Uncertain
Allocation concealment:Low
Blinding of participants and personnel:Low
Blinding of outcome assessment:Uncertain
Incomplete outcome data:Low
Selective reporting:Low
Study design uses 
Appropriate level of random assignment:Yes
Adequate sample size:Uncertain
Validated measure of the outcome:Yes
Appropriate follow-up period:Yes

Relevance Medium

Complexity of intervention

As with other Graduation programs, the intervention implemented by Bandiera et al. is highly complex and success is therefore contingent on quality of execution and details of intervention design.

Similarity of models

The Bandiera et al. intervention includes targeting and two of the three Village Enterprise components, a productive asset transfer and technical skills training. Village Enterprise includes both components, though with slight differences in implementation:

Transfer of a productive asset or grant: In its core model, Village Enterprise strictly distributes cash grants only, whereas the Bandiera et al. intervention transferred livestock based on an assessment of the local market context. However, the majority of Village Enterprise businesses do end up spending their cash grants on inputs for livestock-based businesses: 56% of businesses in Kenya and 76% in Uganda are livestock-based. The value of the asset transfer was also $140 per participant in Bandiera et al. compared to $50 per participant in Village Enterprise’s program.

Training: Skills training is described in Bandiera et al. as “intensive training in running [the participant’s] chosen business [over two years].” There is not enough information to compare this to Village Enterprise’s program, in which participants receive 16 modules of training over the course of 12 months.

The Bandiera et al. program lasts twice as long as Village Enterprise’s year-long program.

Targeted population

Bandiera et al. targeted extremely poor households in Bangladesh. Village Enterprise’s program targets a similar population. Given that there were no significant issues with sampling methodology or differential attrition, the study’s findings are likely of high external validity to its intended sample universe, which is comparable to the population Village Enterprise selects for its program.

Local context

Village Enterprise and the Bandiera et al. study cover comparably resource-constrained contexts. Village Enterprise implements in Uganda and Kenya, which are, respectively, low income and lower-middle income countries by World Bank classifications. Bangladesh, the site of Bandiera et al.’s study, is classified as a lower-middle income country. However, Bangladesh differs markedly from Village Enterprise’s East Africa sites in several ways, including having far more arable land and yet a far smaller proportion of the labor force engaging in agriculture.

Blattman et al. (QJE 2013)

Table 7. Details of Blattman et al. (QJE 2013)

InterventionYouth Opportunities Program (YOP)
Timeframe2008-2012
EvaluatorInnovations for Poverty Action
MethodRandomized controlled trial
Sample535 groups randomly assigned to treatment and control.

Study Abstract

We study a government program in Uganda designed to help the poor and unemployed become self-employed artisans, increase incomes, and thus promote social stability. Young adults in Uganda’s conflict-affected north were invited to form groups and submit grant proposals for vocational training and business start-up. Funding was randomly assigned among screened and eligible groups. Treatment groups received unsupervised grants of $382 per member. Grant recipients invest some in skills training but most in tools and materials. After four years, half practice a skilled trade. Relative to the control group, the program increases business assets by 57%, work hours by 17%, and earnings by 38%. Many also formalize their enterprises and hire labor. We see no effect, however, on social cohesion, antisocial behavior, or protest. Effects are similar by gender but are qualitatively different for women because they begin poorer (meaning the impact is larger relative to their starting point) and because women’s work and earnings stagnate without the program but take off with it. The patterns we observe are consistent with credit constraints.

Findings (Results at four-year endline)

38% increase in participant monthly cash earnings***

(control baseline mean: $83.63 PPP per month, or $2.75 PPP per day)

57% increase in business asset ownership***

(control baseline mean: $434.48 PPP)

*** (**) (*) Significant at 1% (5%) (10%) level. All percentage changes compared to control means at endline. Estimates are intent to treat. Economically and statistically significant effects found on durable and non-durable asset consumption.

Quality of EvidenceHigh

No major threats to internal validity identified. Our analysis is based on the Cochrane criteria for assessing risk of bias and the Coalition for Evidence-Based Policy’s checklist for reviewing randomized controlled trials.

Risk of bias associated with 
Random sequence generation:Uncertain
Allocation concealment:Uncertain
Blinding of participants and personnel:Low
Blinding of outcome assessment:Low
Incomplete outcome data:High
Selective reporting:Low
Study design uses 
Appropriate level of random assignment:Yes
Adequate sample size:Uncertain
Validated measure of the outcome:Yes
Appropriate follow-up period:Yes

Relevance Medium

Complexity of intervention

As with other Graduation programs, the intervention implemented in Blattman et al. is highly complex and success is therefore contingent on quality of execution and details of intervention design.

Similarity of models

The Blattman et al. intervention included village-level targeting and one core component: inviting groups of young adults from targeted villages to apply for cash grants. Groups were to submit proposals describing how they would use the grant for “non-agricultural skills training and enterprise start-up costs.” The main differences between Village Enterprise’s microenterprise development program and Blattman et al.’s YOP are as follows:

Targeting: By screening groups based on written proposals, YOP specifically targets “the motivated poor.” Blattman et al. acknowledge YOP participants likely have unobserved characteristics, such as higher initiative and ability, which positively influence their outcomes. Furthermore, applications had to go through village and district officials and it is unclear whether their screening criteria were fair, consistent across officials and enforced well, which could introduce the risk that the Blattman et al. sample over-represents people with the most need or connections. Meanwhile, Village Enterprise targets the ultra-poor more generally, only introducing conditionality in the disbursement of the second grant installment.

Cash grant: As with Village Enterprise, cash grants in YOP were disbursed to groups rather than to individuals or households. However, at $382, the average per capita grant size in Blattman et al. was nearly eight times as large as Village Enterprise’s grant, and was distributed in one lump sum rather than in two installments. Moreover, the YOP grants were only to be used for non-agricultural skills training and microenterprise development, whereas Village Enterprise participants are free to spend on developing any microenterprise of their choice. 97% of Village Enterprise businesses in Uganda are related to livestock or agriculture.

Training: In YOP, groups selected their own trainers, who were typically local artisans or from small institutes for skilled trades. It is likely that group-based training in YOP had a greater focus on the trades themselves, whereas Village Enterprise’s training is more comprehensive, including business planning, record-keeping and marketing training. There is not enough information on the frequency and duration of training in YOP to allow a detailed comparison to Village Enterprise.

Coaching and mentoring: The program in Blattman et al. is described as a “one-time unsupervised grant,” whereas the Village Enterprise approach is far more high touch, with business owners receiving monthly mentoring visits for a year.

Targeted population

Blattman et al. targeted extremely poor young adults in Uganda. Village Enterprise’s program targets a similar population in Uganda and Kenya, but does not focus on youth. In order to calculate population average treatment effects for the full age range of individuals, Blattman et al. merged their study sample with the 2008 Northern Uganda Survey, a clustered population-based household survey. They found some comparable employment and durable asset impacts, but noted that the population average treatment effect may overstate true effects “since young adults are selected into our sample because of unobserved initiative, connections or affinity for entrepreneurship.” The target population age in Blattman et al. may therefore limit the relevance of the study’s findings to Village Enterprise.

Local context

There is considerable overlap between the implementation contexts of Village Enterprise and Blattman et al.’s YOP. Village Enterprise implements in Uganda and Kenya; Blattman et al.’s study also took place in Uganda.

Banerjee et al. (Science 2015)

Table 8. Details of Banerjee et al. (Science 2015)

InterventionMulti-faceted livelihoods intervention for the ultra-poor
Timeframe2007-2014
EvaluatorInnovations for Poverty Action
MethodSix sites: three cluster-randomized controlled trials; three randomized controlled trials
Sample21,036 adults in 10,495 households
GeographyEthiopia, Ghana, Honduras, India, Pakistan, Peru

Study Abstract

We present results from six randomized control trials of an integrated approach to improve livelihoods among the very poor. The approach combines the transfer of a productive asset with consumption support, training, and coaching plus savings encouragement and health education and/or services. Results from the implementation of the same basic program, adapted to a wide variety of geographic and institutional contexts and with multiple implementing partners, show statistically significant cost-effective impacts on consumption (fueled mostly by increases in self-employment income) and psychosocial status of the targeted households. The impact on the poor households lasted at least a year after all implementation ended. It is possible to make sustainable improvements in the economic status of the poor with a relatively short-term intervention.

Findings (at three-year endline)

37.5% increase in livestock income***

(control endline mean: $80.60 PPP per month, or $2.65 PPP per day)

5% increase in per capita consumption***

(control baseline mean: $69.13 PPP per month, or $2.27 PPP per day)

6% increase in per capita food consumption***

(control endline mean: $41.20 PPP per month, or $1.35 PPP per day)

96% increase in household savings***

(control endline mean: $78.40 PPP)

*** Significant at 1% level. All percentage changes compared to control means at endline. Estimates are intent to treat. Banerjee et al. find large, statistically significant shifts in incomes and asset values, as well as statistically significant effects on food security, financial inclusion, time use, perception of health status, and mental health.

Quality of Evidence High

No major threats to internal validity identified. Our analysis is based on the Cochrane criteria for assessing risk of bias and the Coalition for Evidence-Based Policy’s checklist for reviewing randomized controlled trials.

Risk of bias associated with 
Random sequence generation:Low
Allocation concealment:Low
Blinding of participants and personnel:Low
Blinding of outcome assessment:Uncertain
Incomplete outcome data:Low
Selective reporting:Low
Study design uses 
Appropriate level of random assignment:Yes
Adequate sample size:Yes
Validated measure of the outcome:Yes
Appropriate follow-up period:Yes

Relevance of Findings Medium

Complexity of intervention

As with other Graduation programs, the intervention implemented in Banerjee et al. is highly complex and success is therefore contingent on quality of execution and details of intervention design. The heterogeneity of outcomes across the six Banerjee et al. sites (cost-benefit ratios ranged from -198% to 433%) demonstrates that success is sensitive to the specific ways in which the intervention is implemented.

Similarity of models

The Banerjee et al. intervention includes targeting and six components: transfer of a productive asset or grant, training, life skills coaching, consumption support, access to savings mechanisms and health information or services. The Village Enterprise model also includes targeting, but only three of the six components: transfer of a productive asset or grant, training and access to savings mechanisms. There are also differences between Village Enterprise and Banerjee et al. in the implementation of these common components:

Transfer of a productive asset or grant: A crucial difference between the two models is the value of the asset transfer: the assets were worth $437-1,228 per household in the Banerjee et al. experiments, whereas Village Enterprise’s cash grant is worth $50 per business owner. Furthermore, in its core model, Village Enterprise strictly distributes cash grants only, whereas the Banerjee et al. intervention transfers productive assets in the form of locally appropriate livestock. However, the majority of Village Enterprise businesses do choose to spend their cash grants on inputs for livestock-based businesses: 56% of businesses in Kenya and 76% in Uganda are livestock-based.

Training: In Banerjee et al., participants received business skills training before receiving their assets, whereas Village Enterprise participants received far more extensive group training throughout the intervention. Village Enterprise participants had access to 16 training modules spread out over the course of 12 months.

Savings: The savings component in Banerjee et al. varied from providing participants access to savings accounts with microfinance institutions and financial institutions to encouraging participants to form savings groups, whereas in Village Enterprise’s model, participants only form savings groups.

Finally, program duration differed between the two models. The programs studied in Banerjee et al. were two years long, while Village Enterprise’s program lasts one year.

Targeted population

Banerjee et al. targeted extremely poor households in developing countries (Ethiopia, Ghana, Honduras, India, Pakistan and Peru). Village Enterprise’s program targets a similar population. Given that there were no significant issues with sampling methodology or differential attrition, the study’s findings are likely of high external validity to its intended sample universe, which is comparable to the population Village Enterprise selects for its program.

Local context

Village Enterprise and the Banerjee et al. study cover comparably resource-constrained contexts. Village Enterprise implements in Uganda and Kenya, which are, respectively, low income and lower-middle income countries by World Bank classifications. One of the Banerjee et al. countries is classified as low income (Ethiopia), four are lower-middle income (Ghana, Honduras, India, Pakistan) and one is upper-middle income (Peru). In particular, Banerjee et al.’s Ethiopia site bears many resemblances to Uganda and especially Kenya, with whom Ethiopia shares similar climactic conditions, rural-to-urban population characteristics and arable land availability.

Cost of Impact

Prediction

Village Enterprise generates an impact of approximately $460 in additional household income at a cost of $250 per participant. The prediction of impact is based on applying the median estimated return on investment from nine high-quality randomized trials conducted on similar programs to the baseline income of participants in Village Enterprise’ program. This prediction assumes that benefits are perpetual, with future benefits discounted by 5% per year. This assumption is based on both the theory of the program (moving the extreme poor out of poverty traps) and the available data (which shows benefits persist and increase slightly at 7 years, suggesting benefits likely continue in year 8 and beyond). The estimate of cost is based on audited financials from Village Enterprise and estimated societal costs.

Village Enterprise has a Cost of Impact ratio of 1.8 dollars of additional household income for every dollar of cost incurred by nonprofit, partners and beneficiaries. This implies a social rate of return of 8.6%, which means that benefits will outweigh costs at any discount rate up to 8.6%. For context, the World Bank’s standard discount rate is 5%. The payback period for benefits is seventeen years. The impact figures are counterfactual, discounted and based on total rather than marginal cost. The assumptions behind the numbers are presented first, followed by sensitivity analyses and a discussion of externalities and displacement that could affect these estimates.

Table 9. Findings on Cost of Impact

SpecificationPrediction
Average Increase in Annual Income (1 Year)$22
Average Increase in Annual Income (Total, perpetuity)$460
Average Cost per Participant$250
Important Note: Village Enterprise is currently conducting an independent validation of its program in conjunction with Innovations for Poverty Action. Endline results from the study are expected in mid-2017. ImpactMatters expects these findings will change the program’s impact figures, although we do not have an expectation about whether impact will rise or fall.

Assumptions

Impact per participant ($460). The impact per participant is predicted based on rigorous counterfactual evidence from studies conducted on similar interventions in similar populations. The median estimate of impact in those studies is a 38% annual increase in household income, which translates to a median rate of return of 8%.

Duration of benefits (perpetuity). The duration of benefits is dependent on the longevity of the business, the returns that participants can earn from assets and savings they acquire through increased income and the long-run returns to higher skills and motivation. The theory behind Graduation programs is that they move participants permanently out of poverty traps and onto a higher income plane.

The available empirical data provide strong evidence to support the assumption that benefits are perpetual. The program is too young to have lifetime data for participants. However, Village Enterprise’s internal data on business survival shows a 95% survival rate at three years, or 98% annual survival of businesses. If the business survival rate remains 98% (a reasonable assumption, as businesses should be more likely to fail in earlier years) and businesses generate similar returns to those observed in the randomized trial literature, the perpetuity at a 5% discount rate is underestimating benefits.

Long-term follow up data from elsewhere similarly supports that Graduation programs may lead to perpetual benefits. One randomized control trial in India has preliminary long-term follow up results, which show the difference between annual impact in year one and year seven is statistically the same (and if anything, income rises slightly in year seven compared to year one).

Partner organizations. The cost estimates include all resources invested, including contributions from partner nonprofits and official development assistance.

Two-year average. The costs of all participants over two years of the program have been averaged together, as if the average cost per participant were constant over that period of time. Benefits for the entire two-year population of participants and other beneficiaries are calculated as if they began in the same year, rather than staggered between two different years.

Total costs. The ImpactMatters approach uses total costs rather than marginal costs. It includes the management, fundraising and indirect costs as well as the direct program costs.

Table 10. Metadata and Sources for Cost of Impact

OrganizationVillage Enterprise
OutcomeHousehold income
Year of analysis2016
Base year (first year of cost data)2013
Timeframe for cost data2013, 2014 (July 1, 2012 through June 30, 2014)
Participant data includedUsing documents shared privately by Village Enterprise, we include data for participants in Village Enterprise's program in 2013 and 2014.
Source of impact predictionBanerjee et al. (Science 2015)
Blattman et al. (AEJ-AE 2015)
Bandiera et al. (QEJ 2013)
Blattman et al. (QJE 2013)
Source for cost dataVillage Enterprise’s audited financial statements for 2013 and 2014, shared privately with ImpactMatters

Cost of Delivery

Village Enterprise’s cost of delivery includes costs to the nonprofit, costs to partners and costs to participants and other beneficiaries. The nonprofit’s costs include both direct costs and overhead costs. Overhead costs are fully allocated to the program, as Village Enterprise does not devote substantial resources to advocacy and other programs. See the Cost of Impact spreadsheet attached to this report for details of the analysis in Table 11.

Table 11. Summary of Village Enterprise’s Costs

ItemYearUSD (MM)Share
A. Total organization costs, 201320131.3136%
Cost breakdown   
B. Education and advocacy costs201300%
C. Youth Programming201300%
D. Conservation Programming201300%
E. Total management and fundraising costs20130.267%
F. Proportion attributable to B, C and D201300%
G. Management and fundraising costs attributable to B, C and D201300%
H. Total costs attributable to B, C and D201300%
I. Organization costs for average cost calculation20131.3136%
J. Partner Costs (See Summary of Partner Costs)20130.257%
K. Participant Costs (See Summary of Participant Costs)20130.031%
M. Total organization costs, 201420141.9554%
Cost breakdown   
N. Education and advocacy costs201400%
O. Youth Programming201400%
P. Conservation Programming201400%
Q. Total management and fundraising costs20140.318%
R. Proportion attributable to O, P and Q201400%
S. Management and fundraising costs attributable to O, P and Q201400%
T. Total costs attributable to O, P and Q201400%
U. Organization costs for average cost calculation20141.9554%
V. Partner Costs (See Summary of Partner Costs)20140.072%
W. See Summary of Participant Costs20140.041%
X. Total costs, 201420142.0657%

Other Contributions

Other organizations contributed less than 10% of the two-year costs of the program. Other organizations incorporated the Village Enterprise intervention into existing programs.

Table 12. Summary of Partner Organization Costs

ItemYearUSD (MM)Share
J. Costs incurred by others in executing intervention 20130.257%
V. Costs incurred by others in executing intervention20140.072%

Participant costs are a small proportion of the cost of delivery. Participants spend substantial time attending meetings, trainings and reporting on progress to the nonprofit. However, due to the low opportunity cost of participants’ time, the investment of time has a very small effect on the overall cost structure of the program.

Table 13. Summary of Participant’s Costs

ItemYearUSD (MM)Share
K. Estimated opportunity cost of participant time20130.031%
W. Estimated opportunity cost of participant time20140.041%

Prediction of Impact

The prediction of impact is estimated using a simple linear function. The total impact, I, is equal to the per-capita rate of return, β, times the resources invested per participant, C, and the total number of participants, P.

I = β * C * P

The number of participants is the total flow of participants through the program on a two-year basis.  In the two-year frame of analysis, two classes of participants would complete the program.

Table 14. Summary of Impact

20132014TotalUnits
Number of Participants6250814814398Individuals
Per Household Annual Impact$22 $22  USD
Annual Impact$138 $179 $317 USD (K)
Per Household Lifetime Impact$460 $460  USD
Lifetime Impact$2.90 $3.70 $6.60 USD (MM)

Sensitivity Analysis

The Cost of Impact is a ratio of impact per dollar spent by the nonprofit. This estimate provides the best guidance to donors about what a contribution to the nonprofit could achieve. The Cost of Impact framework has the additional benefit of being applicable when benefits are not measured in dollars (for instance, lives saved or additional years of education).

Table 17 shows the sensitivity of the Cost of Impact ratio to a range of different assumptions. We vary two assumptions in the analysis:

  • Discount rate: The discount rate captures both the time value of money (we prefer money today over money tomorrow) and the uncertainty of the future. A higher discount rate means more future uncertainty and present value of money.
  • Length of benefits:There is no good data about how long the benefits from Village Enterprise’ program persist. Assuming longer benefits suggests that participants gain lasting skills and move permanently out of a poverty trap, leading to sustained gains in living standards. Assuming shorter benefits suggest that program does not move participants permanently out of a poverty trap, and the benefits wear off over time. Experimental evidence suggests the benefits of a Graduation program last at least seven years. Data are still being collected on benefits beyond the seventh year. Table 17 therefore presents the seven-year case, which is the maximum duration of benefits currently supported by experimental evidence.

In addition to ImpactMatters’ standard Cost of Impact ratio, Table 17 provides different specifications for the cost-effectiveness of the nonprofit, including:

  • Social rate of return: The Social Rate of Return is the future discount rate at which benefits equal costs.  A Social Rate of Return of 100% implies that benefits will equal costs when all future benefits are discounted 100%.
  • Donor’s Cost of Net Impact: The Donors Cost of Impact is the ratio of net impact (gross impact less beneficiary costs) to the donor’s cost. Importantly, the Donor’s Cost of Impact, unlike the Cost of Impact, does not capture societal costs not paid by the donor. For instance, if a program is co-funded by a government grant, the full net impact of the program is compared to just the donor’s costs, yielding a higher ratio than the Cost of Impact, which would include the donor’s and government’s costs.
  • Discounted payback period: The discounted payback period is the number of years of impact that equal the up-front costs.  The lower the number of years, the sooner impact exceeds costs.

Table 15. Cost of Impact Base Case Assumptions

Assumptions
Discount Rate5%
Length of BenefitsPerpetual

Table 16. Cost of Impact Base Case Analysis

Specifications
Cost of Impact1.81
Simple Cost of Impact2.03
Donor's Cost of Impact2.03
Discounted Payback Period17
Social Return of Return8.6%

Table 17. Cost of Impact Sensitivity Analysis

Discount rate (Base Case with sensitivity to alternative discount rates)

Assumed discount rate:0%5%*10%20%40%
Cost of Impact..1.810.950.660.26
Simple Cost of Impact..2.031.060.740.29
Donor’s Cost of Net Impact..2.011.040.720.27
Discounted Payback Period1217......

Length of Benefits (Base Case with sensitivity to alternative length of benefits)

Assumed length (years)371020Perpetuity*
Cost of Impact0.250.520.701.391.81

Simple Cost of Impact0.280.590.781.562.03
Donor’s Cost of Net Impact0.230.510.681.382.01
Social Rate of Return-64.2%-14.1%-3.1%8.1%8.6%

Scenarios

Externalities and Displacement

Negative Externalities Low Importance
Positive Externalities Low Importance

The Village Enterprise program is potentially displacing other businesses and generating community ill will, though both effects are likely of low importance. The program may be having an ambiguous price effect (both positive and negative), which is of low importance. Finally, the program is possibly leading to local economic growth, which is of low importance. In three Graduation studies where spillovers were measured, researchers are still working to understand the effects.38

Displacement Negative, low importance

Village Enterprise participants, at an advantage because of their free capital, produce or sell goods, which may drive other sellers or producers out of the market and potentially reduce those producers’ or sellers’ incomes. The likelihood and significance of this is low. Some sellers in the market before Village Enterprise arrives may be extracting rents due to their pseudo-monopolistic position in the market, and they may be harmed, but increased competition in this scenario benefits the larger community. Given the shallow nature of markets in these communities, these risks are minimal.

Community Ill Will Negative, low importance

Community members who do not receive the program may feel ill will towards those that do. One technique that may reduce ill will is participatory wealth ranking, which by creating a transparent selection process for participants and involving community leaders may increase acceptance of the program. Village Enterprise’s program still likely generates some amount of community ill will. However, the concern compared to improving living standards for extremely poor women is low.

Price Effects Ambiguous, moderate importance

By supporting the creation of new businesses, Village Enterprise may increase market competition, potentially leading to a drop in price for the mostly poor consumers in the area. However, as participants tend to select income-generating activities that produce goods (as opposed to selling goods), Village Enterprise participants may not have a significant economic effect on basic consumer good prices. On the other hand, Village Enterprise, by introducing money into the community, may lead to temporary, localized inflation. Village Enterprise has anecdotally observed such inflation.

Economic Growth Positive, moderate importance

Participant businesses likely increase local economic growth, benefiting the community. Bandiera et al. (2013) finds evidence to support this. In addition, participants can make Business Savings Groups open to the broader public. Village Enterprise has anecdotally observe that some savings groups do indeed include members from the broader community.

Context and Analysis

Extreme poverty of the population

Village Enterprise’s program targets the poorest of the poor. Within poor villages, they are further targeting individuals who are very poor within that village. Baseline data from the randomized trial they are currently conducting found household income was $365. The average family has six members.  This translates to $61 in annual income per person, or roughly 17 cents per day.

Therefore, the marginal benefit of small increases in income is high. This is particularly important, as where inequality is significant, Village Enterprise may be more effective at reaching a poorer population than programs that are targeted at the village, district or regional level.

Program stage

Village Enterprise has a stated commitment to scale its program, and is actively seeking funding and partners to do so. However, Village Enterprise is also currently conducting a well-design randomized trial, which may be leading to short-term increases in costs.

Operational and environmental risks

Village Enterprise’s program is technically and operationally complex. Program success relies on several factors, some of which are out of Village Enterprise’s control. For example, the majority of participants choose to start livestock-based businesses, which can be susceptible to disease and drought. Village Enterprise is careful and thoughtful about designing and monitoring implementation and conducts substantial work to understand appropriate strategies for different contexts and to identify and correct problems. For a longer discussion, see Risks above.

Comparison to similar programs

ImpactMatters has not conducted sufficient comparative analysis to make a judgment about how this Cost of Impact compares to other programs. However, Graduation programs are commonly compared to unconditional cash transfers. Evidence from a randomized trial on the GiveDirectly unconditional cash transfer program program in Kenya found positive impacts on consumption, food security and asset holdings in the short run. Because it is cheaper to deliver cash than a more complex program such as Graduation, the short-run benefit-to-cost ratio may be higher. However, little long-term data exists for cash transfer impacts. Long-term data on Graduation suggests that program benefits persist. A study of comparable Graduation programs in six countries observed no decline in consumption after three years, and in the one site where seven-year data are available, benefits persisted and may have increased slightly. If benefits are permanent for Graduation programs but not unconditional cash transfers, this substantially affects how the two programs are compared.

Quality of Monitoring Systems

Rating

Village Enterprise has high-quality systems for monitoring activities, targeting, engagement, feedback and outcomes. Although there is small room for improvement, overall Village Enterprise’s monitoring systems are highly robust, indicating that Village Enterprise is consistently delivering a high-quality program to its participants.

The main components of the monitoring systems used by Village Enterprise are:

  1. Targeting: Tracks village selection, Participant Wealth Ranking and Progress out of Poverty Index used to select participants for the program.
  2. Mentoring and Training: Tracks implementation of mentoring and training process, ensuring that Business Mentors are performing their duties and clients are attending and understanding trainings.
  3. Business Tracking: Tracks progress toward business implementation, key operating goals (which determines either disbursal of second portion of grant or additional support) and business status at completion of program.
  4. Savings Group Constitution and Exit Report: Monitors the structure and rules and tracks the sustainability of Business Savings Groups.
  5. Entry and Exit Survey: Tracks client poverty status.
  6. Consumption and expenditure surveys: Captures data on food, semi-durable and durable goods consumption.31
  7. Progress out of Poverty Index pre-post comparison: Participants are scored using the Progress out of Poverty Index not only for targeting purposes at baseline, but also after program completion to compare their poverty likelihood before and after the program.

In order to conduct this analysis, ImpactMatters interviewed senior management and mid-level managers and reviewed 286 documents and datasets provided by Village Enterprise.

Data Type Credible Actionable Responsible Transportable
Activities
Targeting
Engagement
Feedback
Outcomes

Activities

Credible
Actionable
Responsible
Transportable

Village Enterprise collects data on its own activities in a credible and responsible manner. Managers have timely access to relevant data. Reports on monitoring data are structured so as to identify participants (business owners) and staff (Business Mentors) that have encountered challenges. Managers and executives have demonstrated commitment to address problems as they arise.  Managers regularly address common problems facing participants, such as use of business funds for consumption, discord between business co-owners and gender-related issues with business ownership. Reviews are conducted biweekly, quarterly and annually and include discussion of lessons learned from monitoring data.

Business owners’ earnings are closely related to the productivity of their chosen business and local market conditions. One of the tools used by Village Enterprise, the Smarter Market Analysis Risk Tool (SMART), identifies high-performing business lines in order to steer participants to choosing investments in the most profitable sectors. Village Enterprise could improve this tool by monitoring the climactic and market conditions for its businesses on a routine basis, such as quarterly or annually, and updating SMART with new data.

Systems

Village Enterprise’s core model includes following activities: training of Business Mentors, targeting of participants, training for business owners, business group formation, seed funding in two installments, business owner mentoring and Business Savings Group (BSG) formation. Village Enterprise monitors these activities using the following systems:

  • Knowledge test scores and other documentation to track the recruitment and induction training of Business Mentors
  • Progress Out of Poverty Index reports and Participatory Wealth Ranking reports to track components of targeting
  • BSG training spot checks to track training of business owners
  • BSG attendance sheets to track business owners’ attendance at trainings
  • Application forms to approve the first and second seed capital grants and receipts to track the disbursal of the grants
  • Business owner registration forms to track all business owners entering the program
  • Business spot checks to track business activity
  • A mentoring checklist to track Business Mentors’ visits to business owners
  • BSG spot checks and exit surveys to track the saving, lending and meeting activity of BSGs

Credible

Village Enterprise’s systems for verifying the completion of activities and counting activities and outputs capture each important component of its model. Village Enterprise has provided ImpactMatters with evidence verifying activities, through data and reports related to targeting, Business Mentor recruiting and induction training, business owner training, seed capital grant disbursal, business owner registration,  business activity, mentoring visits and BSG activity.

Village Enterprise uses standardized data collection instruments and data collection manuals to collect activity data. In addition, supervision at several stages of the data collection process helps ensure data quality. For example, one Village Enterprise country director described how Monitoring and Evaluation (M&E) associates who find unlikely survey timestamps during the data cleaning process can alert Field Coordinators to investigate the issue and ultimately obtain explanations from Business Mentors for the detected data discrepancies.

Enumerator induction training sessions on data quality help ensure enumerators understand the importance of and process for collecting unbiased data. Village Enterprise’s standardized data collection instruments contain questions that have been worded thoughtfully and neutrally.

Village Enterprise’s sampling strategies for spot checks are appropriate and do not introduce significant bias. Business spot checks after the first grant are conducted on every business, while business spot checks following the second grant are conducted on a random sample comprising at least a third of all businesses. Every BSG receives at least one BSG spot check. There are at least two training spot checks per Business Mentor, or three spot checks per training module per area.

Actionable

Data collected are almost immediately available to M&E staff, for use in troubleshooting data quality issues, and to country office management staff, to enable corrective managerial action. Interviews with multiple key decision-makers in the organization indicate operational and managerial decisions are largely made on the basis of M&E reports. For example, at the conclusion of each of the three cycles of participant enrolment per calendar year, end-of-cycle reports are jointly reviewed by the M&E team, CEO and Senior Director of Institutional Giving. Adjustments are made such that, within four months, when a new cycle starts, Village Enterprise has either set up new trainings or processes to resolve those issues, or has plans to further research those issues.

Village Enterprise’s theory of change rests on the assumption that the seed capital provided to business owners is of a size that will lead to changes in outcomes. Village Enterprise has made efforts to test this assumption, but that testing has been irregular.  As part of the annual Innovation Summit in 2016, a group of Village Enterprise staff presented to their peers "considerations for determining a better grant value based on a specific set of business inputs and long term profits." Summit attendees voted on whether the grant size should increase and this will vote was to be factored into next year's budget. However, Village Enterprise does not continually monitor prices of the most commonly used business inputs or local market and climatic conditions. The 2016 Innovation Summit recalibration of the seed capital amount appears to be an isolated instance rather than part of a continual monitoring effort to ensure the seed capital amount is appropriate given changing market conditions.  

Village Enterprise’s strategic plan, M&E descriptive overview and Innovation Summit reports all demonstrate the senior management team takes action based on monitoring reports.

Responsible

Standard data collection protocols provide enumerators with specific instructions that help increase the efficiency of their interactions with program participants. In addition, the use of digital and cloud-based platforms such as Salesforce and TaroWorks streamlines Village Enterprise’s monitoring processes and minimizes the burden of data collection on Village Enterprise’s own financial and human resources as well as on participants’ time.

Village Enterprise uses standard data collection protocols and enumerator training sessions to convey the importance of collecting data ethically, in terms of confidentiality, courtesy shown to participants and full disclosure of the purpose of Village Enterprise’s surveys.

Transportable

Village Enterprise tracks activity data that are closely aligned to its theory of change, including detailed data related to Business Mentor recruiting and induction training, targeting activities, business owner training, seed capital grant disbursal, business owner registration, business activity, mentoring visits and BSG activity. Village Enterprise is not collecting substantial data that does not relate to its theory of change.

Village Enterprise shared extensive documentation of its activity monitoring with ImpactMatters during the course of the impact audit, including standardized data collection instruments, data collection manuals and protocols, and raw data and data reports related to the activities it carries out. Village Enterprise’s website and annual reports describe its activities and indicate the scale of its activities, such as by reporting the number of businesses started, the number of business owners trained, the amount saved per BSG and the percentage of businesses still in operation after four years.47

Targeting

Credible
Actionable
Responsible
Transportable

Village Enterprise collects high-quality data on individuals that are targeted for the program. It uses a combination of geographic filters, survey data and community input to identify individuals that meet its poorest-of-the-poor criteria.

Village Enterprise reconciles the findings of the different surveys to minimize the number of non-poor participants who are accepted into the program. Appropriate and frequent reports are provided to managers and executives, who have a demonstrated commitment to ensure that participants are appropriately targeted.

Village Enterprise partially tests its assumption that participants lack financial literacy and business education. Village Enterprise could improve its monitoring of the assumptions in its theory of change by regularly surveying the availability of financial services, both at the level of the districts targeted and also at the level of the beneficiary households.

Village Enterprise’s targeting data use a well-accepted standard for proxy assessment of poverty, the Progress out of Poverty Index® (PPI®). Widely used proxy measures such as PPI make it simpler for third parties to understand and assess Village Enterprise’s targeting data.

Village Enterprise should be more explicit in it’s targeting to ascertain the prior business training of participants. In order for Village Enterprise’s training to be effective, it must address gaps in the knowledge of participants. While the targeting criteria for poverty are satisfactory, the targeting criteria for business knowledge, financial access and financial literacy are more assumed than tested.

Systems

Village Enterprise targets participants through a multi-step process:

  • District or sub-county selection and approval: Village Enterprises selects districts or sub-counties based on national poverty data and conducts courtesy visit with leaders at the district or sub-county level to both introduce the program and collect population and vulnerability information on villages in the district or sub-county.
  • Village selection: Villages are shortlisted based on information provided by district or sub-county leaders and Business Mentors, Field Coordinators and Assistant Country Directors verify the shortlist.
  • Transect walk: Business Mentors and Field Coordinators meet with leaders of the shortlisted villages and conduct a walk-through of each village to identify opinion leaders to be invited to the Participatory Wealth Ranking focus group.
  • Participatory Wealth Ranking (PWR): Opinion leaders define poverty in the context of their village and then rank households from “Very Poor” to “Rich.”
  • Village introductory meeting: Village Enterprise conducts an open meeting to introduce the program to any member of the village.
  • Progress out of Poverty Index (PPI): Households identified as “Poor” and “Very Poor” in PWR are surveyed using the PPI to determine program eligibility. Village Enterprise also has a set of inclusion and exclusion criteria appended to the PPI survey.
  • Community interest meeting: Eligible households are invited to a meeting where details of Village Enterprise’s program are provided and expectations of participants and of Village Enterprise are made clear.

Village Enterprise collects raw data from PWR sessions and PPI surveys, analyzes and summarizes this data into targeting reports. Village Enterprise also spot-checks PPI scores in order to measure targeting accuracy.

Credible

Village Enterprise collects sufficient data to track the PWR and PPI stages of the targeting process, including raw data collected by PWR facilitators and PPI surveyors, reports generated from that data and data from spot checks.

Standardized data collection instruments and data collection manuals help ensure targeting data are collected reliably. The village selection form gives Business Mentors a defined sequence of questions and standardized phrasing of questions to use when collecting information from community representatives at village introductory meetings. PWR data collection instruments and facilitation guidelines are consistent across all implementation sites and PPI survey instruments and guidelines are consistent within countries. There is minimal written guidance for the less significant stages of the targeting process, such as how to conduct the transect walk, but interviews indicate they have produced reliable results in practice.

Village Enterprise provides detailed guidance to targeting staff on how to create a setting where complete and honest responses can be expected from participants. The PPI and PWR surveys contain questions that are worded thoughtfully and neutrally. In addition, the local leaders Village Enterprise consults on transect walks are recommended by government-appointed chiefs; an interview with Village Enterprise suggests that government involvement increases perceived accountability and that chiefs are trusted individuals in the community, thereby reducing the risk that PWR participants are included for reasons other than genuine opinion leadership.

Actionable

PPI and PWR data collected are almost immediately available to M&E staff and country office management staff. PPI spot check reports and overall targeting reports are produced every cycle (three times a year), enabling M&E staff to identify and resolve ineffective targeting in a timely manner. For example, routine spot check reports brought to attention high variances in regular participant targeting conducted by Business Mentors versus spot check targeting conducted by M&E enumerators. This led to a re-assessment of targeting systems, including providing refresher trainings to Business Mentors on standardized usage of the PPI instrument. Village Enterprise has also produced a Targeting Effectiveness Report based on baseline data from the RCT currently underway in Uganda. In producing the report, Village Enterprise demonstrated the ability to translate dense technical information presented by the RCT research team into a digestible summary tailored for key decision-makers in the organization.

In carrying out the PWR, PPI and the community interest meeting stages of its targeting process, Village Enterprise collects targeting data that address a number of assumptions that undergird its theory of change. This targeting data substantiate the assumptions that participants face credit and savings constraints and lack business training and financial literacy; that participants want to start businesses and want to start them in groups with two other business owners; and that participants want to save money.

Village Enterprise’s PPI spot check reports and overall targeting reports demonstrate that managers have a record of acting upon the contents of the reports.

Responsible

PPI survey respondents concerned about the duration of the survey are informed of the time burden and made aware of the option to complete the survey at a more convenient time. The PPI and PWR survey instruments are programmed into TaroWorks, reducing the data collection burden on Village Enterprise's financial and human resources, as well as on survey respondents. There is evidence that Village Enterprise has weighed the costs and benefits of PWR against other methods of targeting data, and has found that the benefits of PWR, including fostering community buy-in, justify the four-hour duration.

Village Enterprise trains staff to collect PPI data, conduct courtesy visits and conduct community interest meetings ethically, in terms of confidentiality, courtesy shown to participants, and full disclosure of the purposes of surveys.

Transportable

Village Enterprise’s targeting data do not directly validate that the targeted population in fact demonstrates the expected characteristics described in Village Enterprise’s theory of change: namely, that the population faces credit and savings constraints, lacks business training and financial literacy and takes on inefficient levels of risk due to behavioral biases.

Village Enterprise has shared with ImpactMatters standardized data collection instruments, data collection manuals, and raw data and data reports for its targeting activities. Village Enterprise’s website describes its targeting strategy and the general profile of its target population.25,26

Engagement

Credible
Actionable
Responsible
Transportable

Village Enterprise does an outstanding job of gathering high-quality data on participants’ engagement. Using a combination of its own personnel and quasi-independent enumerators, it surveys participants both regularly and through random spot-checks. The findings of these surveys are directly tied to key process metrics in the theory of change. The findings also directly measure the key risks of the project, such as whether participants are able to direct financial assets toward business purposes; whether participants can maintain harmony in small partnerships and large savings groups; whether businesses are profitable; and the threat of family members interfering in the operation of the business.

The organization uses timely engagement data to manage at several levels of the organization, including partners (Business Mentors), personnel (field managers), and executives. Spot checks and surveys are conducted using computer assisted personal interviews (CAPI) and databases that streamline the data collection and quality assurance processes. Minimizing the reporting burden on participants and protecting confidentiality are two of the priorities described by executives in their discussion of engagement data systems.

Systems

Village Enterprise collects the following take-up and engagement data:

  • Participant business tracking: Business groups’ application forms for seed capital grants and business spot checks track business group formation and subsequent business activity
  • Grant tracking: Seed capital receipts and business spot checks track whether grants are collected and used for business activity
  • Mentoring: Mentoring checklists track business owners’ participation in mentoring visits from their Business Mentors
  • BSG engagement: BSG attendance sheets, BSG training spot checks and BSG exit surveys track business owners’ attendance and participation in trainings
  • BSG activities: BSG spot checks and BSG exit surveys track saving activity

Credible

Village Enterprise has provided raw data and data reports that track participant take-up and engagement at every touch-point of its activities, including trainings, BSGs, business groups, receiving and spending seed capital, business activity and mentoring visits.

Standardized data collection instruments and data collection manuals are used to ensure participant take-up and engagement data are collected in a reliable manner.

Data collection protocols and training sessions ensure enumerators and Business Mentors understand the importance of and process for collecting unbiased engagement data. The standardized instruments for collecting engagement data contain questions that have been worded thoughtfully and neutrally. In addition, the sampling strategies for engagement-related spot checks are appropriate and do not introduce significant bias.

Actionable

Reports on take-up and engagement data are produced at appropriate and regular intervals, monthly and per cycle (three times a year). Country office management staff members are the primary consumers of these reports. Spot check reports are also often used for internal research presented at Innovation Summits, which feeds into Village Enterprise’s strategic planning.

Engagement data help Village Enterprise substantiate the assumptions that the business selection process is effective and that businesses selected are viable for the market by tracking business survival rates, reasons for business failure and the industries in which business groups are operating. Engagement data complement outcomes data by allowing Village Enterprise to determine which industries are correlated with successful business outcomes. In addition, Village Enterprise’s investment in developing the Smart Market Analysis Risk Tool and carrying out its Business Decisions Study demonstrate an organizational commitment to substantiating the assumption that participants are selecting effective businesses.

Village Enterprise also faces risks surrounding the ways in which participants relate to each other while engaging in Village Enterprise’s activities. Business spot checks identify which of the original members of the business group are still active in running the business and ask for reasons why members are no longer active. BSG exit surveys ask about the gender split in BSG membership, the composition of the executive committee and reasons why members left the BSG, allowing Village Enterprise to track trends of female members voluntarily or involuntarily leaving, in light of concerns that women are prevented from engaging in activities due to cultural norms. A group of Village Enterprise staff analyzed these data, supplemented by other monitoring data such as training attendance, and presented their findings at the 2015 Innovation Summit. To track the risk that BSGs break down due to interpersonal conflicts, Village Enterprise is able to refer to BSG spot check and exit survey data, which measure BSG meeting attendance, BSG membership, the quality of leadership at the BSG meeting observed, membership attrition and reasons for leaving the BSG.

The Village Enterprise senior management team demonstrates commitment to take action based on engagement data.

Responsible

Attendance sheets, seed capital receipts and mentoring checklists all pose negligible data collection burdens to program participants, while spot checks, the BSG exit survey and seed capital surveys are generally brief and pose low data collection burdens. The use of Salesforce and TaroWorks also minimizes the data collection burden on Village Enterprise’s own human and financial resources.

Data collection manuals for collecting the various forms of take-up and engagement data described do not contain explicit instructions for ensuring data collection is ethical. However, based on Village Enterprise's close observation of data misuse policies and other policies to protect the interests of participants in its other data collection activities, as evidenced by data collection manuals for other survey instruments and key informant interviews, it is likely that Village Enterprise's staff collect take-up and engagement data with the same attention to confidentiality.

Transportable

The engagement data collected enables Village Enterprise to track whether participants are taking up and engaging with activities as expected in its theory of change. Village Enterprise is not collecting substantial engagement data that does not relate to its theory of change.

Village Enterprise has shared with ImpactMatters standardized data collection instruments, data collection manuals and raw data and data reports related to participant engagement. Village Enterprise’s website and annual reports describe in sufficient detail the specific ways in which program participants are expected to engage with Village Enterprise activities, and disclose approximate numbers of business started and business owners actively participating in the program.25,26

Feedback

Credible
Actionable
Responsible
Transportable

Feedback from line level staff (including Business Mentors) and participants (business owners) is gathered at many stages of the project and throughout the year. Participant surveys include ratings of different experiences with the program and free-text comment boxes. Participants are encouraged to share their views of the project with staff, including Business Mentors and field managers. Independent enumerators, who do not have operational responsibilities or share the same chain of command as the project staff, also survey a percentage of participants.

Managers use feedback in their biweekly and quarterly meetings to assess whether the project is on track. Executives provide regular forums for review and response to participants’ concerns. The annual Innovation Summit includes a panel based on the ideas and concerns of participants.

Systems

Program participants have substantial opportunities to provide feedback on the program. Business owners’ feedback is channeled through their respective Business Mentors to the country management team and other Business Mentors at bi-weekly meetings. Business participants are interviewed during both exit surveys and business spot checks.

Several Village Enterprise Accelerator pilot and research studies are dedicated to developing new programmatic services based on business owner feedback. Examples include the Business Decisions Study, which seeks to understand how business owners rank different business types according to criteria such as risk and profitability; the Risk Study, in which farmers rank their crops in order of risk and profitability; and the Household Portfolio pilot, which gauges business owners’ understanding of nutrition, farming practices and their attitudes about working in a group.

Staff members also provide feedback. At bi-weekly meetings, Business Mentors and country office management staff provide feedback on the design and implementation of each BSG training session.

Annual Innovation Summits give Village Enterprise staff members at most levels of the organization the opportunity to participate in Village Enterprise's strategic planning process. About six months prior to the Summit, staff members submit ideas for Summit topics, which the Field Management Team discusses and consolidates into a final list of topics to be shared organization-wide. Teams of Village Enterprise staff are assigned to topics. Topic teams research and prepare to present at the Summit. Presentations made over the first four days of the Summit are used as the basis of strategic planning (performed with the group at large) on the fifth day, when staff members vote on the next steps for pilots, research and changes to programmatic elements.

Village Enterprise presents feedback from participants on its website under the heading “Most Significant Changes”. However, Village Enterprise does not provide supporting documentation for how or why these stories were chosen. Since the context of these communications is on the public-facing website, it is understandable that an incomplete methodology is presented.

Feedback mechanisms for partner organizations vary from partner to partner. Examples include surveying Community Connector Officers (USAID/FHI360 staff trained by Village Enterprise) for feedback after the program and holding review meetings with Forest Monitors (Wildlife Conservation Society staff trained by Village Enterprise).

Credible

Interviews with Village Enterprise staff indicate that the Business Mentors’ bi-weekly meetings effectively channel feedback from business owners to country management staff. Raw data and data reports made available verify that the exit survey, BSG exit survey and business spot checks collect valuable feedback data from participants on a wide range of programmatic elements, including which aspects of the Village Enterprise program as a whole were most and least valuable; how satisfied they were with the BSG experience and how useful they found savings practices like using a passbook; suggestions for refresher training topics for business owners; and open-ended suggestions for improving the Village Enterprise program. Village Enterprise also provides opportunities for staff at all levels of the organization to voice their feedback.

Standardized data collection instruments for the exit survey, BSG exit survey and business spot checks help ensure the reliability of feedback data collected from program participants. Village Enterprise also discourages interaction between Business Mentors and M&E enumerators conducting spot checks to ensure the independence of enumerators. Village Enterprise’s other methods for collecting feedback tend to be more unstructured and discussion-based, such as Business Mentor bi-weekly meetings and review meetings with staff from partner organizations. However, this is appropriate for the oftentimes idiosyncratic and potentially sensitive nature of feedback data.

The feedback questions included in surveys are framed in a neutral way. However, there is a risk of response bias if business owners believe their answers with influence their chances of receiving the second seed capital grant, future training or other Village Enterprise services. Interviews indicate Village Enterprise takes steps to mitigate this: surveyors are instructed to assure business owners that their feedback will have no impact on their inclusion in the program and that they should feel comfortable answering candidly. Moreover, some feedback questions are open-ended, encouraging flexibility and freedom in responses. Village Enterprise also supplements feedback from current business owners with feedback from business owners who have already graduated from the program, thereby minimizing the effects of response bias on the overall set of data collected. Lastly, Village Enterprise has created an organizational culture that prizes input from staff at all levels of the organization, creating an environment where staff members likely feel safe to provide honest feedback.

Actionable

Participant feedback shared at Business Mentor bi-weekly meetings are recorded in meeting minutes and raised at higher-level meetings, such as Field Management Team meetings. Feedback data from participant surveys are collated in monthly and end-of-cycle reports for use by country management teams, M&E staff and senior management. Feedback data from ex-participants are collected by enumerators and shared with program associates, who raise issues to the Director of Monitoring, Evaluation and Learning at weekly meetings. These data feedback loops allow Village Enterprise staff to address problems as they arise.

Risks and assumptions are largely addressed by monitoring data other than feedback data. There are no unaddressed risks and assumptions that ought to have been studied using feedback data.

Interviews with Village Enterprise management and summaries of the findings and next steps of Village Enterprise Accelerator pilot and research studies consistently show that managerial staff members demonstrate a strong commitment to taking action based on feedback data.

Responsible

The bulk of the feedback data collection burden falls on Business Mentors rather than program participants, who are only asked to answer feedback questions within other surveys, such as business spot checks and exit surveys. The burden on Business Mentors is justified by how valuable the data have been in informing programming and operational decisions.

Data collection manuals train enumerators and Business Mentors in ethical data collection.

Transportable

Village Enterprise collects participant feedback on essential components of its program to identify problem areas where operational and programmatic improvements can be made. Village Enterprise is not collecting substantial data that does not relate to its theory of change.

Village Enterprise has shared with ImpactMatters standardized data collection instruments, data collection manuals, raw data and data reports for surveys that contain feedback questions. Village Enterprise has also made available information related to the Village Enterprise Accelerator pilot and research studies that are designed to capture program participants’ detailed feedback, as well as summit proceedings and examples of staff presentations from the last two annual Innovation Summits. Publicly, Village Enterprise does not describe what feedback data it collects and how they are collected. However, Village Enterprise regularly publishes Most Significant Change stories of randomly selected business owners. Most Significant Change is a technique for evaluating complex interventions and a way of holding organizations accountable to program participants.31,48 Each Business Mentor interviews three business owners at the start and end of every programmatic cycle, using written notes or audio recordings to capture their stories, which are then published on the Village Enterprise blog.49

Outcomes

Credible
Actionable
Responsible
Transportable

Village Enterprise uses an appropriate methodology to monitor the survival of businesses and Business Savings Groups. Rapid poverty assessments are used in tandem with random, independent, in-depth Consumption and Expenditure surveys that track key outcomes in detail.

Field managers and executives closely watch the findings from biweekly, quarterly, and annual reports. Reports that indicate trouble receive appropriate attention to remedy problems within days or weeks. Monitoring, learning and evaluation (MLE) staff have invested in an enterprise database for rapid reports on monitoring data. Quality assurance processes are reasonable and presumed to be effective, although ImpactMatters did not review data cleaning procedures.

Village Enterprise considers and mitigates the burden of survey response on participants. It uses computer-assisted personal interviews (CAPI) and a cloud database in order to streamline data collection and quality assurance. Length of interview and confidentiality are considered in the design of monitoring systems.

Though Village Enterprise’s public communications are transparent in terms of publishing quantitative changes in outcomes as measured by pre-post comparisons, they also tend to misrepresent these changes in outcomes as constituting evidence of impact. Pre-post comparisons use a poor-quality counterfactual and therefore cannot be the basis for claims of program impact. While consumers of Village Enterprise’s public communications likely appreciate the outcomes data published, they risk being misled if such data are not presented with greater care.

Systems

Village Enterprise’s outcomes data come from PPI surveys, Consumption and Expenditure surveys, Entry/Exit surveys (each collected at the start and end of every cycle; there are three cycles per year) and BSG exit surveys (collected at the end of each cycle). Village Enterprise uses these survey instruments to track its process metrics (number of participants enrolled, business formation and survival rates, BSG formation and survival rates) and to construct non-counterfactual, pre-post comparisons of outcome metrics (consumption, expenditure and savings).

Credible

Village Enterprise’s detailed Consumption and Expenditure and Exit/Entry surveys capture the full scope of its intended outcomes. The PPI is a validated country-specific predictive tool for targeting, but its use in longitudinal outcomes assessment may not be as suitable. However, because pre-post PPI data are supplemented by Consumption and Expenditure and Entry/Exit surveys, overall outcomes data collected by the three instruments are considered sufficiently comprehensive.

Village Enterprise ensures outcomes data are reliable by using standardized data collection instruments and manuals for the PPI, Consumption and Expenditure survey, Entry/Exit survey and BSG exit survey.

Village Enterprise takes appropriate measures to ensure the surveying environment promotes unbiased answers, such as training enumerators in neutral probing techniques, uniformity of question administration and being critical of questionable responses from participants if there is obvious evidence to the contrary.

Actionable

Outcomes data are reported each programmatic cycle in country-specific as well as organization-wide performance reports, which are jointly reviewed by the M&E team, CEO and Senior Director of Institutional Giving. These periodic reviews enable senior management to be responsive and take corrective action before the next cycle begins.

Outcomes data provide substantiation for the assumptions that undergird the theory of change; namely, that program participants face credit and savings constraints, a lack of business training and financial literacy and undertake inefficient levels of risk due to behavioral biases. To gauge credit and savings constraints, the Entry/Exit survey asks whether the household currently saves money, the value of those savings and how much money certain household assets would be worth if liquidated on the market today; to gauge levels of business training and financial literacy, it asks which types of business participants personally have knowledge about and whether they have had any formal business training; and to gauge risk aversion due to behavioral factors, it examines whether business owners feel they have the power to change the course of their lives and whether they have future opportunities they feel are worth saving for. As previously mentioned, a combination of outcomes data and engagement data are used to validate the assumed market viability of businesses selected by business owners. Outcomes data also address the risk that there are cultural norms that prevent women from participating and that BSGs break down due to interpersonal conflict.

According to interviews with Village Enterprise’s senior leadership, Village Enterprise is pursuing outcomes-based funding to tie its outcomes directly to additional funding. Explicitly subjecting outcomes to funders’ scrutiny and decision-making demonstrates Village Enterprise’s commitment to take action based on outcomes data.

Responsible

The reporting burden associated with the three major outcomes surveys is approximately six hours per participant. Though this is not an insignificant amount of time, there is consistent evidence from interviews that staff have been continuously making efforts to condense the approximately two-hour-long Consumption and Expenditure survey, which was previously three hours long. Interviews also indicate that enumerators administering surveys are cognizant and respectful of participants’ time, offering the opportunity to postpone surveys to more convenient times.

Transportable

Village Enterprise collects data that directly measure its intended outcomes as per its theory of change.

Village Enterprise shared with ImpactMatters standardized data collection instruments, data collection manuals, raw data and data reports for the Consumption and Expenditure survey, Entry/Exit survey and the PPI. Village Enterprise's Annual Report and Our Impact webpage publicly state the quantitative change in household consumption, expenditure and savings, based on cumulative 1987-2015 data and collected by conducting pre-post surveys on 20% of business owners.26,47 One remaining concern is that Village Enterprise’s public communications consistently present pre-post changes in outcomes as evidence of impact. Claiming impact based on changes in outcomes using a poor-quality counterfactual, as in a pre-post study or a study of self-selected comparison groups, is discouraged. Such data are suggestive rather than conclusive, and it is recommended that Village Enterprise guide its audience in interpreting that data carefully and realistically.

Learning and Iteration

Rating

Village Enterprise makes systematic and continuous change on the basis of high-quality data. Iterations to the Village Enterprise model are routinely subjected to testing using a counterfactual model, and the iterations that are scaled are supported by reasonable evidence. Each of the iterations discussed below had either a counterfactual test of impact or a strong effect sizes based on a large pilot test. Most of the successful iterations occurred within a systematic innovation framework, with recognizable components of Plan-Do-Study-Act cycle.50 Iterations take place within a well-defined pipeline of tests, with a calendar of tests and results prior to scaling up. The iteration testing function is distinct from other monitoring and evaluation processes.

Criteria Finding
Iteration is based on data Yes
Data are of high quality Yes
Iteration is systematic and periodic Yes

Iterations

Village Enterprise is in the process of conducting a randomized controlled with multiple treatment arms that compare Village Enterprise’s standard program to variations of the standard program, including a variant that excludes BSGs, a Business-in-a-Box variant that provides business assets and inputs instead of or in addition to cash, an unconditional cash transfer variant, and an unconditional cash transfer modified with a behavioral intervention. This randomized trial is ongoing and no iterations have been made to Village Enterprise’s program based on the findings, but Village Enterprise worked closely with the researchers to design the study and have expressed a commitment to make iterations to its core program if they are suggested by the findings of the study.

Over the past three years, Village Enterprise adopted six iterations, three of which are considered “minor” iterations and three of which are considered “additive” iterations, in that they are additional components to the intervention rather than modifications.

All six iterations implemented over the past three years were reviewed in assessing VE’s Learning and Iteration. However, additive iterations are considered to have less of a burden of proof and are not included in the written analysis presented here. The three minor iterations reviewed are the Savings With A Purpose component, the Family Support Training module and the Leadership training modules.

  • Savings With A Purpose (SWAP): Through SWAP, Village Enterprise trains business owners to save money toward a set goal with a set time frame. Pooled funds in the Business Savings Group (BSG) are organized into primary account, emergency fund, and SWAP account savings, and one business owner per BSG is elected into the role of SWAP Officer.
  • Monitoring for Sustainability (M4S): The M4S training module is intended to enable business owners to lead their own BSGs more effectively, for instance by making informed decisions during BSG executive committee elections, having executive committees that monitor businesses, and sharing business knowledge and experiences with each other during BSG meetings. The ultimate goal is to ensure the sustainability of the BSG and businesses after business owners exit the Village Enterprise program.
  • Family Support Training Module: This module aims to educate business owners’ family members about the Village Enterprise program and the benefits of the program that accrue to the entire family, thereby minimizing instances of unsupportive family members spurring program attrition and poorer outcomes.

The three additive iterations by Village Enterprise were Passbooks (Record Keeping for the Illiterate), Smarter Market Analysis Risk Tool (SMART) and Visual Year Long Plan.

  • Passbooks (Record Keeping for the Illiterate): Village Enterprise provided stamps and inkpads to enable illiterate business owners to track their savings.
  • Smarter Market Analysis Risk Tool (SMART): SMART guides business owners through the business selection process by providing business owners with information on profit, demand, risk, price variability and sustainability. Information comes from several data sources: focus groups with “good” farmers, agronomists from local universities, village and urban point-of-sale surveys and agricultural-veterinary shop surveys.
  • Visual Year Long Plan: In 2013, Village Enterprise introduced a new system to enable illiterate business owners to draw pictorial business plans. In 2016, Village Enterprise aims to make it easier to create, use and interpret the business plans by providing business owners with pre-drawn images and tables.

One of the primary methods through which Village Enterprise determines which iterations to adopt is an annual weeklong Innovation Summit for staff to present their ideas for, and the outcomes of, iteration testing. Summit proceedings and presentation slides from the 2015 and 2016 Innovation Summits were reviewed to assess VE’s Learning and Iteration. M&E plans and internal reports for each iteration were also reviewed. In addition, the following analysis draws heavily from a summary document, managed by VE’s Innovations Team, that tracks the iteration pipeline and decisions about implementation for all iterations over the past three years.

Data Quality High

The randomized controlled trial Village Enterprise is currently conducting is a high-quality test of different iterations of the Village Enterprise trial. Such a high-quality test to specifically benchmark different program designs and determine which components of the program are most effective is exceptionally rare and Village Enterprise deserves strong credit for designing, funding and participating in the study. Two of the three minor iterations over the past three years to the Village Enterprise model were adopted based on high-quality data. The third was adopted on the basis of some data, although the data used could likely have been of higher quality. However, Village Enterprise has demonstrated that it typically uses high-quality data to make decisions.

Village Enterprise conducted an A/B test on Savings With A Purpose (SWAP). Four Business Mentors representing three Village Enterprise implementation geographies were selected to pilot SWAP. Business Mentors chose one of their BSGs to serve as the treatment group, using their other BSG as a control group. Baseline data was collected using a separate SWAP survey, endline data was collected six months into saving and monthly saving activity was continuously monitored through spot checks. Comparing BSGs with the SWAP iteration against those without, Village Enterprise found 31% higher total savings in Soroti, Uganda, 73% higher total savings in Hoima, Uganda, and no data from Kenya at the time of analysis. Despite a small sample size of only eight BSGs, the effect on savings was large and positive, justifying Village Enterprise’s adoption of the SWAP iteration.

Village Enterprise also tested a new training module known as Monitoring for Sustainability. M4S was pilot tested with 14 BSGs across Village Enterprise’s three areas of operation at the time: Hoima and Soroti (Uganda), and Eldoret (Kenya). Treatment group BSGs were equipped with leadership training and a self-monitoring system based on spot checks, while control groups received the program in its original form. Village Enterprise used routine monitoring systems (exit surveys and spot checks) modified with M4S-specific survey questions to evaluate the program. At endline, Village Enterprise found a higher percentage of treatment group BSGs reviewed their BSG constitutions every meeting and a consistently higher percentage of treatment group business owners reported perfect attendance at BSG meetings, but there were no clear trends in business owners’ ratings of BSG leadership quality, in executive committee members’ attendance and in the percentage of BSGs that discussed business challenges during meetings. While these are modest results, Village Enterprise used a reasonable (albeit based on a small sample size) estimate of the counterfactual, giving it strong enough grounds to adopt the iteration.

To test Family Support Training, Village Enterprise conducted interviews with business owners and their family members who participated in a pilot Family Support Training session. Family members reported having a better understanding of the program and of the importance of their support. However, the test collected only feedback data and did not observe changes in process metrics. As a result, this test did not provide high-quality data to support the iteration, particularly as this form of self-reporting can lead participants to provide socially desirable responses. After rolling it out, Business Mentors were instructed to measure attendance at trainings and provide feedback during bi-weekly team meetings to better evaluate the new training module.

Systematic Change Yes

All three of the minor iterations over the past three years to the Village Enterprise model were adopted through a systematic process.

The SWAP iteration was adopted after undergoing a Plan-Do-Study-Act learning cycle, wherein the iteration idea was sourced; tested; analyzed and summarized for decision-makers; and then accepted and implemented systematically. Village Enterprise sourced the idea for SWAP from monitoring data that revealed many participants were not fully utilizing BSGs: participants were saving, but not concurrently taking out useful loans. After conducting an A/B test of SWAP, Village Enterprise staff analyzed test findings in an internal report for decision-makers. The decision to adopt the new component was based on positive results was documented in a collaborative spreadsheet managed by the Innovations Team and presented at the 2016 Innovation Summit to all staff.

Village Enterprise reports that many program participants express a desire for continued Village Enterprise staff presence even after exiting the program and that past training modules, though valuable for financial literacy and business management, provide less guidance on leadership. Village Enterprise documented the design and findings of iteration tests for two new training modules, Leadership I and II, and presented the modest, but high-quality, results in internal reports as well as at the 2015 and 2016 Innovation Summits. As with other Village Enterprise iterations, the decision-making process was centrally documented by the Innovations Team.

Need for the Family Support Training module surfaced when Village Enterprise staff and Business Mentors observed that female participants were not receiving adequate support from family members with regard to their new roles as business owners. Village Enterprise ran a pilot Family Support Training session with positive qualitative results, which were cited in the Innovations Team’s documentation of its decision-making process and presented to all staff at the 2015 Innovation Summit.

Village Enterprise has an Innovations Team, composed of Village Enterprise staff from various country offices and levels of the organization, that manages current and planned projects, pilots and studies.

Periodic Change Yes

Village Enterprise has systems for periodically considering and adopting iterations to its core model.

The primary mechanism for sourcing and testing iterations is based around the annual Innovation Summit. Each year, Village Enterprise staff members submit their ideas for potential iterations about six months in advance of the Summit. Based on proposed ideas, the Field Management Team decides on a final list of iterations and staff members are assigned to small teams responsible for researching, pilot testing and presenting those iterations at the Summit. The Innovations Team also presents the findings of its pilot studies and research conducted throughout the year.

The Summit brings together staff at almost all levels of the organization. After four days dedicated to iteration testing presentations, Village Enterprise convenes a strategic planning day, during which all staff vote on next steps for pilots, research priorities and programmatic changes. Specific takeaways and resultant changes to the Village Enterprise model are recorded and disseminated to staff after the Summit. Village Enterprise has thus developed a system for routinely “crowd-sourcing” ideas for iterations from those who know the program and program participants most intimately.

Documents shared and interviews conducted with Village Enterprise team members demonstrate that staff members, at multiple levels of the organization, are able to articulate which iterations are under consideration and which are under testing. In particular, the Innovations Team centrally manages a collaborative document that lists all iteration studies conducted during the 2014-2016 period, along with their respective goals, findings, next steps, reasons for not pursuing, and a red, yellow or green light indicator representing whether the iteration will be adopted.

With a focus on innovation, Village Enterprise has conducted an impressive number of iteration tests in the past three years. The periodicity of iteration testing will likely only increase as Village Enterprise launches its Accelerator program to formalize internal R&D efforts and leverage partnerships with external research institutions.

Backmatter

Metadata

Nonprofit Details

Legal Name Village Enterprise
EIN 22-2852248
Founded 1987
Website villageenterprise.org
Chief Executive Dianne Calvi
Revenue $1,958,459 (2014-15)
Contact email info@villageenterprise.org
Addresses

Mailing:
751 Laurel Street, PMB 222
San Carlos, CA 94070

Physical:
1161 Cherry Street, Suite M
San Carlos, CA 94070

Note to donors

Impact Audit Detail

Activities

Evidence review, document and data review, headquarters visit, senior management interviews, field staff interviews and key informant interviews.

Completed 2016-11-21
Published 2016-11-21
Valid through 2018-12-31
Impact audit team

Tamsin Chen, Elijah Goldberg, Dean Karlan and Ben Mazzotta

Conflict disclosures

Innovations for Poverty Action (IPA) is currently conducting a randomized controlled trial of Village Enterprise's program in Uganda. Dean Karlan is President of both ImpactMatters and IPA, and IPA is currently incubating ImpactMatters.

Glossary

A/B Test

An A/B test compares the current version of the program to a modified version in order to test which version is more effective at changing engagement, outcomes or some other metric of interest. A/B tests do not have a pure control group and are not designed to test the overall impact of a program. Instead, they are intended to improve the design of a program by determining whether a nonprofit should modify its program or keep it as is.

Activity Data

Activity data is a form of monitoring data that tracks program activities completed and outputs delivered. Activities are the day-to-day tasks an organization must undertake in order to provide a product or service. Each program activity has at least one output associated with it. Outputs are the products or services produced by the nonprofit.

Additive Iteration

An additive iteration is a change to a nonprofit’s program that adds a new component, as opposed to modifying an existing component or removing a component. When assessing how a nonprofit learns and iterates, an additive iteration has a lower burden to justify adoption if it meets three conditions: (1) it is unlikely to have a negative impact (but may have no impact), (2) is unlikely to reduce the impact of other components of the program and (3) does not significantly increase program costs.

Applicability

Applicability of evidence to a nonprofit’s program includes two distinct concepts: quality and relevance. Quality captures the internal validity of the evidence: is the evidence free of factors that may bias the reported findings? Relevance captures the external validity of the study to the nonprofit’s intervention: to what extent do we expect the intervention to generate similar impact as the findings observed in the study?

Attrition

Attrition refers to cases where members of a sample drop out between rounds of data collection. For instance, if a 100 people are surveyed at the beginning of the program but only 90 can be surveyed at the end of the program, the attrition rate is 10%. Attrition can be problematic if attrition from the sample is correlated with outcomes. For instance, when following up on a health intervention, those who are sick may be more difficult to find than those who are healthy. As a result, the reported results may be biased because they include outcomes for fewer sick individuals.

Average Costs

Average costs are the total amount of money spent by the nonprofit divided by some unit of output or outcome. Average costs include costs that are fixed and not expected to increase as outputs or outcomes grow, such as salaries of senior managers. See also Marginal Costs.

Behavioral Bias

A behavioral bias is any tendency that leads people to not make rational decisions given available information and their own preferences. Common behavioral biases include confirmation bias, where people weight facts that confirm their opinions higher than those that do not; loss aversion, where people value losses greater than equivalent gains; and availability bias, where people overestimate the likelihood of events based on how “available” they are in their memory. Many charitable interventions are designed (sometimes unintentionally) to help individuals correct these biases.

Bias

Bias is a non-random error in a statistical estimate. Whenever estimates are based on a sample from a larger population, there will be random error in that estimate: no two samples will produce exactly the same estimates. An estimate is biased when those errors lead it to be consistently above or below the true value that is being estimated.

Business Group

In the Village Enterprise model, a business group is made up of three business owners who come together to start a microenterprise.

Business Mentors

Business Mentors are Village Enterprise’s primary field staff. Business Mentors assist business groups in selecting an enterprise and provide training and support to each group throughout the program.

Business Savings Groups (BSG)

In the Village Enterprise model, Business Savings Groups are 30-member groups that pool savings and make loans to members. Each BSG comprises ten three-member business groups.

CART (Credible, Actionable, Responsible, Transportable)

The CART standard is a method for understanding the quality of monitoring systems. CART stands for:

  • Credible: Monitoring systems are credible if they collect high-quality data that is analyzed accurately.
  • Actionable: Monitoring systems are actionable if the nonprofit commits to act on the data that it collects.
  • Responsible: Monitoring systems are responsible if the nonprofit minimizes the burden of data collection and collects data ethically.
  • Transportable: Monitoring systems are transportable if the data collected is tied to the nonprofit’s theory of change and is shared appropriately.
Cluster-randomized

A study is cluster-randomized if the randomization was performed at the group (or cluster) level, instead of the individual participant level. Types of clusters include, but are not limited to, villages, schools and districts. See also Randomized Controlled Trial.

Consumption

Definitions of consumption vary, but it tends to be defined as those goods and services consumed by individuals. In economic development, there are particular measures of consumption that are important, including food consumption, non-durable consumption (items that have a short lifespan, such as clothing) and durable consumption (items that have longer lifespans, such as appliances).

Control Group

A control group is a group of participants who did not receive the intervention. Control groups enable nonprofits and researchers to compare what happened to beneficiaries in their program to what might have happened if they were not in the program.  See also Treatment Group.

Counterfactual; Counterfactual Evidence

The counterfactual is what would have happened in the absence of a program or other event. Understanding the counterfactual is essential to understanding the impact of a program. Participant outcomes may change over time for many different reasons not related to the program. By comparing the difference between participant outcomes and counterfactual outcomes, the impact of a program can be estimated.

The counterfactual cannot be directly measured, as researchers cannot observe the same participant both participating and not participating in the program. However it can be approximated by randomizing participants into an intervention group and a control group, and then comparing outcomes across the two different groups.

Cost of Impact

The Cost of Impact is a ratio of impact per dollar spent by the nonprofit. This estimate provides the best guidance to donors about what a contribution to the nonprofit could achieve. The Cost of Impact framework has the additional benefit of being applicable when benefits are not measured in dollars (for instance, lives saved or additional years of education).

Design

A nonprofit at the design stage has a program model that is undergoing change.

Discount Rate

People tend to value benefits in the future less than benefits in the present, for three primary reasons. First, benefits today can be reinvested and generate some return. Second, the future is uncertain, and we are often uncertain if future benefits will actually materialize. Third, most people are impatient, and prefer immediate gratification over future gratification. A discount rate captures this by discounting or reducing future benefits compared to current benefits.

Donor’s Cost of Impact

The Donor’s Cost of Impact is the ratio of net impact (gross impact less beneficiary costs) to the donor’s cost. Importantly, the Donor’s Cost of Impact, unlike the Cost of Impact, does not capture societal costs not paid by the donor. For instance, if a program is co-funded by a government grant, the net impact of the program is compared to just the donor’s costs, yielding a higher ratio than the Cost of Impact, which would include the donor’s and government’s costs.

Economic Significance

“Economically significant” results means the study found an effect of an intervention (say increased literacy) that is not only statistically significant (i.e. unlikely to arise by chance), but also is of a size that is “meaningful.” For instance, a 1% change in income may not be meaningful enough to invest in the program, but a 1% change in temperature may be. Economic significance combines the effect size, the statistical significance and the context to make a statement about whether that particular intervention achieves something that is “worth it.”  Economically significant results are also commonly referred to as “important results” (in contrast to “significant results”, which implies statistical significance).

Effect Size

How large the measured impact was on outcomes in the group receiving the program compared to a similar group that did not receive the intervention.

Engagement Data

Engagement data is a form of monitoring data that tracks initial take-up of the program and how people interact with the product or service. For instance, if individuals are offered a savings account, engagement data might include how many people accept the offer and open a savings account, how many times people deposit and withdraw, how many times people check their balance and similar measures of how people interact with the product.

Enumerator

A person employed to collect data. Enumerators are often hired by survey firms to collect data on behalf of a study or nonprofit. Enumerators are often, but not always, independent of the program delivery staff.

Externality

An externality is a consequence or effect of an activity that is not reflected in the cost of the goods or services exchanged. Externalities affect third parties, and those effects can either be positive or negative. Nonprofits often exist to correct externalities, such as pollution. Nonprofits can also themselves generate externalities, such as positive economic growth in a community when they provide services to some community members.

Evidence from Elsewhere

In an impact audit, evidence from elsewhere includes studies – such as randomized controlled trials, quasi-experimental studies, laboratory results and systematic reviews – on interventions that are similar to the nonprofit’s intervention. The motivating theory behind using evidence from elsewhere is that there exists some true effect size for a specific intervention (or more realistically, a range of true effect sizes). If the same intervention has been measured elsewhere and shown to produce a particular effect – and that intervention has some true effect size – one should expect the same intervention, given a similar context and quality of implementation, to have a similar effect size (after accounting for random noise).

Extreme Poor

Extreme poverty is defined by the United Nations as “a condition characterized by severe deprivation of basic human needs, including food, safe drinking water, sanitation facilities, health, shelter, education and information.” One feature of the extreme poor is that they often cannot be defined in terms of simply geography – for instance, within a poor village, there may be a stark difference between the living conditions of the poor and those of the extreme poor.

External Validity

See Relevance.

Feedback Data

Feedback data is a form of monitoring data that gives information about the strengths and weaknesses of the program from participant or other stakeholder perspectives. Feedback data can provide valuable information about how to improve program design.

Food Security

Food security is defined as having consistent and reliable access to a sufficient quantity of affordable and nutritious food.

Graduation Program

A Graduation program is a multi-component intervention designed to help the extreme poor start a livelihood activity. Graduation programs often include initial targeting of the extreme poor, followed by training, selection of a livelihood activity, transfer of cash or a productive asset, and supporting services, including regular coaching and mentoring visits, access to savings or other financial products, and sometimes health or consumption support.

Human Capital

Human capital is all the knowledge, skills, attitudes and experiences that enable people to produce value for themselves or other people or organizations.

Impact

Impact is a change in beneficiary outcomes attributable to a nonprofit’s activities and outputs. See also Outcome Metrics; Outcomes.

Independent Evaluator

An independent evaluator can include a research organization or academics engaged to analyze the impact of a program. Independent evaluators are not directly employed by the program, although they may be paid through program resources.

Independent Validation

Independent validation includes all evaluation efforts that include a substantial role for a third-party in the design and analysis of the evaluation. Independent validations do not necessarily need to be conducted at an arm’s length; the nonprofit is often involved in the design and analysis phase, and will be involved in executing the actual program itself and often in collecting data. However, to qualify as an independent validation, a third-party must have a substantial decision-making role in design and overall control over analysis of the evaluation.

Internal Evaluation

Internal evaluation includes all efforts by the nonprofit itself to evaluate the impact of its work. Internal evaluation can include anything from collecting outcomes before and after implementation to conducting a randomized controlled trial.

Intervention

An “intervention” is what researchers study and nonprofits do. An intervention includes anything from a medical procedure to a conditional cash grant. ImpactMatters studies the intervention that a nonprofit implements, mapping that intervention to the evidence base out there on that particular intervention.

Learning and Iteration

Learning and Iteration is the section in the impact audit that assesses and provides a rating for the historical processes the nonprofit has used to determine changes to the design of its intervention. We rate how well the nonprofit uses data to learn what does and does not work, and then appropriately iterates on its model.

Liquidity

Liquidity is both the extent to which an asset can be converted into cash and the extent to which a person has readily convertible assets to meet short-term financial obligations.

Marginal Costs

The incremental change in total cost due to increasing the quantity produced by one unit. In an impact audit, for example, marginal cost refers to the change in total cost incurred when one more participant is served in the nonprofit’s program. See also Average Costs.

Market Failure

A market failure is a situation in which the allocation of goods and services is not efficient. There exists another conceivable outcome where individuals may be better off without making anyone else worse off.

Missing Market

A missing market exists when there is demand for a good or service, but there is no available supply of this sought-after product. For example, the extreme poor face missing markets for financial services, such as microcredit, microsavings and microinsurance.

Multiple Treatment Arm Randomized Controlled Trial

A randomized controlled trial that uses multiple treatment groups to simultaneously test variations of an intervention or disentangle effects of multi-component interventions. See also Randomized Controlled Trial.

Outcome Metrics; Outcomes

Outcome metrics are a direct measure of the success of the program in addressing the underlying problem. For example, in a malaria control program, the number of households with sufficient insecticide-treated bednets would be a process metric and the rate of malaria infections in the zone would be a measure of outcomes. See also Process Metrics.

It is important to emphasize that change in outcome metrics is still not sufficient to document impact, since there is no counterfactual comparison. But the unit of measure of the outcome (malaria prevalence) is the same as the measure of impact, since the measure of impact is a simple arithmetic difference between the observed outcome and the estimated counterfactual outcome

Payback Period

The length of time required to recover the cost of an investment. In an impact audit, the payback period is the number of years that must elapse before cumulative benefits exceed the costs of the intervention.

Participatory Wealth Ranking

Participatory wealth ranking is an exercise through which individuals identify the relative wealth or poverty of members of their community, thereby identifying the poorest households for participation in an intervention. Implementation varies by location, but generally involves facilitation of a meeting in which a group of community members name and sort all households into agreed-upon categories of poverty.

Plan-Do-Study-Act

The Plan-Do-Study-Act cycle is a repetitive four-step model for carrying out change in an organization. In an impact audit, nonprofits are assessed on whether iteration ideas were sourced; tested; analyzed and summarized for decision-makers; and then accepted and implemented systematically. Also known as the Plan-Do-Check-Act cycle, Deming cycle and Shewhart cycle.

Poverty Trap

A self-reinforcing barrier that prevents people in poverty from making productive investments that could raise their standard of living. Examples include lack of access to credit or savings, lack of information and behavioral biases.

Poor Information

Poor information refers to a market failure wherein one or more parties has imperfect knowledge when transacting, investing, or establishing behavioral norms.

Problem

The problem comprises a target population that suffers from an underlying market or government failure (referred to as the source of the problem), leading to a social inefficiency. See also Social Inefficiency.

Process Metrics

Process metrics describe delivery of goods and services and observable behavior changes in the target population. See Outcome Metrics; Outcomes.

Progress out of Poverty Index (PPI)

The PPI is a country-specific poverty measurement tool. A household’s answers to a ten-question survey are used to compute the likelihood that it is living below the national poverty line and other internationally recognized poverty lines. The PPI enables organizations to identify and target poor populations.

Purchasing Power Parity (PPP)

The purchasing power of a currency is the quantity of the currency needed to purchase a common basket of consumer goods and services. PPP equalizes the purchasing power of two given currencies by accounting for differences in the cost of living and inflation in the two countries.

Quality of Impact Evidence

Internal validity is the extent to which we are able to say that no other variables except the one under study caused the result. In other words, high internal validity denotes a degree of confidence that we can attribute causation (in some ways, another way of saying “impact”) to the intervention.

Quality of Monitoring Systems; Monitoring Systems

Quality of Monitoring Systems is the section in the impact audit that assesses how well the nonprofit produces and uses data to ensure it is consistently delivering its program at high quality. Monitoring systems track every step required in the delivery of the intervention using five types of data: activity, targeting, engagement, feedback and outcomes data. In an impact audit, monitoring systems are assessed to determine if they fulfill the CART standard. See also CART.

Randomized Controlled Trial

A randomized control trial is an evaluation design by which individuals (or groups) are randomly allocated into treatment and control groups, where the treatment group receives the program. The outcomes of the two groups are then compared in order to estimate effect size. See also Effect Size.

Relevance

External validity has two meanings. In the more general sense, it means, how sensitive is this program to context? In other words, if we tried the same thing elsewhere, how confident are we that we would find the same results?

Within the context of this impact audit, we use a more narrow definition: “external validity” compares the findings of a particular study to the nonprofit’s program to determine whether the conditions under which that study were implemented are similar enough to believe they would hold for the nonprofit’s program instead.

In general, we consider four dimensions of comparability:

  • Intervention design: What components were included in the intervention? No two interventions will be exactly the same, and here theory places a valuable role in understanding whether any differences in design are likely change the “mechanism” through which the program works.
  • Intervention fidelity: How “well” was the intervention implemented? The same design can be carried out well or poorly. If you held a training on the exact same material, but one was carried out by a native speaker and the other by only a proficient speaker, we would consider the latter to potentially have lower “intervention fidelity”.
  • Local context: How similar are the geographic areas, and the accompanying social, cultural and political structures of those areas? This is challenging to assess, given the complexity of human nature. One approach here is to replicate across different settings and examine differences in effect size. Another is to look at the mechanism through which a program works – for instance, providing a woman with a grant to start small shops – and see if the market failure (credit constraints) applies elsewhere. If it does, an intervention adjusted for that context that does a similar thing – for instance, providing a woman with a grant to purchase livestock – is likely to work as well.
  • Targeted population: Does the intervention target generally the same group of people? This is challenging as well. However, looking for similarities in economic situation (such as credit constraints) or in other concrete similarities that motivate a program (such as too poor to afford health care services) is one approach to mapping population external validity.
Restricted Donations

A nonprofit’s use of restricted donations is limited to particular purposes by the donor. See also Unrestricted Donations.

Sample; Sample Size

The sample is the portion drawn from a population for testing or analysis that is intended to enable statistical estimates of the behavior or attributes of the whole population. The sample size is the number of units that comprise the sample; a large enough sample size allows inferences about the whole population to be made.

Savings and Credit Constraints

Savings and credit constraints exist when people are limited by a lack of resources saved and a lack of borrowable resources, and are therefore unable to make productive investments that could raise their standard of living. See also Poverty Trap.

Scaling

A nonprofit at the scaling stage is in the process of expanding its program.

Social Inefficiency

The social inefficiency is the result of the underlying market and government failures. It is the primary reason that the nonprofit’s intervention is socially beneficial. It effectively answers the so-what question: if a skeptic is willing to grant that the underlying market or government failure exists, then, “So what?”

Social Rate of Return (SRR)

The SRR is the future discount rate at which benefits equal costs. An SRR of 100% implies that benefits will equal costs when all future benefits are discounted 100%. The SRR accounts for extra-financial benefits and costs – that is, all social benefits and costs not included in conventional financial accounts.

Spot checks

Spot checks are tests conducted without prior notice on a randomly selected subject. Spot checks are a common component of monitoring systems, for example to ensure that the intervention is being delivered at the expected quantity and quality.

Statistical Significance

A statistically significant result (often a difference of means of the main outcome of interest) is a result that is unlikely to arise as a result of chance. This doesn’t mean the finding cannot be due to chance – just that it is very unlikely.

Targeting Data

Targeting data are one form of monitoring data that tracks the identification of beneficiaries to receive the program.

Theory of Change

A theory of change connects the problem to the intervention the nonprofit runs to expected process and outcome metrics. The objective of a theory of change is to provide a testable hypothesis for why the intervention is solving some problem that will lead to positive changes for the targeted beneficiaries. In an impact audit, ImpactMatters requires that the problem be framed in terms of a market failure or government failure.

Tragedy of the Commons

A shared-resource system where individual users acting independently and rationally according to their own self-interest behave contrary to the common good of all users by depleting the resource. For example, the mismanagement of common water and livestock resources suffers from the tragedy of the commons.

Treatment Group

In an experiment, the treatment group is comprised of experimental subjects that receive the treatment being evaluated. Also known as an intervention group. See also Control Group.

Unrestricted Donations

A nonprofit’s use of unrestricted donations is not limited to any particular purposes by the donor and may be used as the nonprofit sees fit. See also Restricted Donations.

Unconditional Cash Transfer

An unconditional cash transfer is a cash grant to a recipient whose use of the grant is not limited to any particular purpose or tied to the recipient’s fulfillment of any conditions.

Validation

A nonprofit at the validation stage is testing its program’s impact.

Reference List

  1. Goldstein M, Udry C. The Profits of Power: Land Rights and Agricultural Investment in Ghana. J Polit Econ. 2008;116(6):981–1022.
  2. Bowles S, Durlauf SN, Hoff K, Russell Sage Foundation. Poverty traps. PRINCETON; OXFORD: Russell Sage Foundation; 2006. 241 p.
  3. Jalan J, Ravallion M. Household Income Dynamics in Rural China. In Oxford: Oxford University Press; 2005. p. 107–23.
  4. Bandiera O, Rasul I. Social networks and technology adoption in Northern Mozambique. Econ J. 2006;116(514):869–902.
  5. Conley TG, Udry CR. Learning about a new technology: Pineapple in Ghana. Am Econ Rev. 2010;100(1):35–69.
  6. Barrett CB, Bezuneh M, Aboud A. Income diversification, poverty traps and policy shocks in Côte d’Ivoire and Kenya. Food Policy. 2001;26(4):367–84.
  7. Morduch J. Poverty and vulnerability. Vol. 84, American Economic Review. 1994. p. 221–5.
  8. Dercon S. Wealth, risk and activity choice: cattle in Western Tanzania. J Dev Econ [Internet]. 1998;55(1):1–42. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0304387897000540\npapers3://publication/doi/10.1016/S0304-3878(97)00054-0
  9. Dercon S. Growth and shocks: Evidence from rural Ethiopia. J Dev Econ. 2004;74(2):309–29.
  10. Dercon S, Christiaensen L. Consumption risk, technology adoption and poverty traps: Evidence from Ethiopia. J Dev Econ. 2011;96(2):159–73.
  11. Kraay A, Mckenzie D. Do Poverty Traps Exist? J Econ Perspect. 2014;28(3):127–48.
  12. Banerjee A V. Contracting constraints, credit markets and economic development. 2001;
  13. Banerjee A V, Duflo E. The Economic Lives of the Poor. J Econ Perspect [Internet]. 2007;21(1):141–68. Available from: http://economics.mit.edu/files/530
  14. Bunthuwong K. Leading the Way in Being S.M.A.R.T.: Introducing the Smarter Market Analysis Risk Tool! - Village Enterprise [Internet]. 2013 [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/leading-the-way-in-being-s-m-a-r-t-introducing-the-smarter-market-analysis-risk-tool/
  15. Banerjee A, Mullainathan S. The Shape of Temptation: Implications for the Economic Lives of the Poor. 2010 May. (Working Paper Series).
  16. Bernheim BD, Ray D, Yeltekin Ş. Poverty and Self-Control. Econometrica. 2015;83(5):1877–911.
  17. Karlan D, McConnell M, Mullainathan S, Zinman J. Getting to the top of mind: How reminders increase saving. Manage Sci. 2016;
  18. Banerjee A V, Mullainathan S. Limited attention and income distribution. Am Econ Rev. 2008;98(2):489–93.
  19. Beaman L, Magruder J, Robinson J. Minding small change: Limited attention among small firms in kenya. 2013.
  20. Graves H. Through Heidi’s Lens: Vocational Skills Training - Village Enterprise [Internet]. 2015 [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/through-heidis-lens-vocational-skills-training/
  21. Graves H. Through Heidi’s Lens: Peace and Business — A Story of Reconciliation in Adekwok, Uganda - Village Enterprise [Internet]. 2016 [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/through-heidis-lens-peace-and-business-a-story-of-reconciliation-in-adekwok-uganda/
  22. Dasgupta P, Ray D. Inequality as a Determinant of Malnutrition and Unemployment: Theory. Econ J [Internet]. 1986;96(384):1011–34. Available from: http://www.jstor.org/stable/2233329?origin=crossref\npapers3://publication/doi/10.2307/2233329
  23. Banerjee A V., Duflo E. Poor Economics. Poor Econ. 2012;303.
  24. Conservation Education- Changing Practices in The Kisere Forest - Village Enterprise [Internet]. 2012 [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/conservation-education-changing-practices-in-the-kisere-forest/
  25. What We Do - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/what-we-do/
  26. Our Impact - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/our-impact/
  27. Singer D, Klapper L, Demirguc-Kunt A, Van Oudheusden P. The Global Findex Database 2014 : measuring financial inclusion around the world. 2015. p. 1–97.
  28. From the Field: Smiles and Grant Disbursements - Village Enterprise [Internet]. 2012. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/from-the-field-smiles-and-grant-disbursements/
  29. Graves H. Through Heidi’s Lens: Business Savings Groups - Village Enterprise [Internet]. 2015. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/through-heidis-lens-business-savings-groups-2/
  30. Strategy - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/about-us/strategy/
  31. Monitoring and Evaluation - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/our-impact/monitoring-and-evaluation/
  32. Home - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/
  33. The Village Enterprise Incubator - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/what-we-do/village-enterprise-incubator/
  34. USAID/Uganda Community Connector (CC) Project | FHI 360 [Internet]. [cited 2016 Nov 21]. Available from: https://www.fhi360.org/projects/usaiduganda-community-connector-cc-project
  35. Adams AM, Evans TG, Mohammed R, Farnsworth J. Socioeconomic stratification by wealth ranking: Is it valid? World Dev. 1997;25(7):1165–72.
  36. Zeller M, Feulefack J, Neef A. How Accurate is Participatory Wealth Ranking (PWR) in Targeting the Poor? A Case Study from Bangladesh. 2006 Annu Meet August 12-18, 2006, Queensland, Aust. 2006;
  37. Temu A, Due J. Participatory appraisal approaches versus sample survey data collection: a case of smallholder farmers well-being ranking in Njombe District, Tanzania. J African Econ. 2000 Mar 1;9(1):44–62.
  38. Building Stable Livelihoods for the Ultra-poor. 2015.
  39. Banerjee A, Duflo E, Chattopadhyay R, Shapiro J. Targeting Efficiency: How well can we identify the poorest of the poor? 2009.
  40. Blattman C, Green EP, Jamison J, Lehmann MC, Annan J. The Returns to Microenterprise Support among the Ultrapoor: A Field Experiment in Postwar Uganda. Am Econ J Appl Econ. 2016 Apr;8(2):35–64.
  41. Bandiera O, Burgess R, Gulesci S, Rasul I, Sulaiman M, Das N. Can basic entrepreneurship transform the economic lives of the poor? Q J Econ [Internet]. Available from: http://www.oxfordjournals.org/our_journals/qje/forthcoming/
  42. Blattman C, Fiala N, Martinez S. Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda. Q J Econ. 2014 May;129(2):697–752.
  43. Banerjee A, Duflo E, Goldberg N, Karlan D, Osei R, Parienté W, et al. A multifaceted program causes lasting progress for the very poor: Evidence from six countries. Science (80- ). 2015;348(6236).
  44. World Bank Country and Lending Groups – World Bank Data Help Desk [Internet]. [cited 2016 Nov 21]. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519
  45. Bandiera O, Burgess R, Das N, Gulesci S, Rasul I, Sulaiman M. Can Basic Entrepreneurship Transform the Economic Lives of the Poor? 2013;
  46. Haushofer J, Shapiro J. Household Response to Income Changes: Evidence from an Unconditional Cash Transfer Program in Kenya. Mimeo. 2013;
  47. Financials - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/about-us/financials/
  48. Meet our Entrepreneurs - Village Enterprise [Internet]. [cited 2016 Nov 21]. Available from: http://villageenterprise.org/our-impact/meet-our-entrepreneurs/
  49. Introducing our Most Significant Change Series: Ebwalu Joseph - Village Enterprise [Internet]. 2015 [cited 2016 Nov 21]. Available from: http://villageenterprise.org/blog/introducing-our-most-significant-change-series-ebwalu-joseph/
  50. Deming E. Out of the Crisis. JThe MIT Press (first published in 1982 by MIT Center for Advanced Educational Services), Cambridge, Massachusetts. 1986.
  51. GuideStar:Demo-Organization Report [Internet]. [cited 2016 Nov 21]. Available from: https://www.guidestar.org/PartnerReport.aspx?ein=22-2852248&Partner=Demo
  52. Form 990 Village Enterprise Fund Inc. 2013.