Building Evaluation Capacity within a Nonprofit – How Funders Can Help (and Not Cause Trouble)

by Isaac Castillo 

The value of metrics in social change is undeniable. As funders and nonprofits, we understand that the best way to showcase the positive difference our work makes, reinforce our credibility, and influence decision makers is to back up our words with data. We must identify the number of people our efforts are reaching, how our efforts lead to positive outcomes for them and how we can improve our approach.

Measurement and evaluation has become ingrained in our work over the years, and we – as a field – have gotten much smarter about what to measure and how to measure it. As funders, we now have basic expectations that the organizations we invest in will track their progress over time – so we have a baseline at the beginning of any investment and clear milestones along the way toward reaching concrete and defined outcomes.

In the funder community, we can improve even more by making sure that the nonprofits we support have the resources and capacity required to conduct the rigorous high-quality evaluation the social change sector demands. When thinking about nonprofit evaluation capacity building, most funders and grantees agree that it is important, but many have no idea where or how to start. Frequently, there is a disconnect between what the funder and nonprofit want to achieve – all because we lack a clear and simple way to talk about what is ‘low’ or ‘high’ evaluation capacity for nonprofits.

To help both funders and nonprofits, VPP (with the assistance of Dan Tsin – the Director of Data and Accountability at Urban Alliance) developed the Nonprofit Evaluation Capacity Rubric. This rubric is designed to provide a rough estimate of a nonprofit’s existing evaluation capacity as well as way to determine if capacity is improved over time (and by how much).

This rubric can help funders and nonprofits in several ways. First, it can help estimate a nonprofit’s level of capacity at the beginning of an investment or initiative – essentially serving as a baseline for the evaluation capacity of the organization before capacity building services are provided. Second, it can provide a target or goal for where the capacity building services can move the nonprofit over a period of time.

Identifying a Nonprofit’s Evaluation Capacity

In most cases, smaller or younger nonprofits will tend to fall on the low end of the rubric: they will collect minimal information on the population they serve and the responsibility of this task will fall upon a staff person that has many other responsibilities. Nonprofits on the low end of the rubric will have a sense of the number of people they serve and might have basic demographic information (gender, age, race/ethnicity, etc.) on their service population. But these nonprofits will not know how much programming each participant receives, nor will they have data on whether participants improve outcomes over time. Nonprofits at the ‘low’ end of the rubric are still extremely common and funders should not assume that just because an organization is large or well-resourced that it has a robust evaluation and data collection infrastructure.

Nonprofits in the ‘moderate’ part of the rubric regularly track output/dosage information (how much of a service each client receives) and outcome information (changes in knowledge, attitudes, behaviors or conditions) for clients. This information will help the organization determine if participants are improving outcomes, but they will not have a sense of exactly what is leading to improved outcomes, nor will they be able to rule out other potential causes for improved client outcomes. These organizations also tend to have part of a senior manager’s time devoted to evaluation and data collection work or have a full time staff person devoted solely to these issues.

The ‘advanced’ level of evaluation capacity requires a great deal more resource commitment to achieve. These nonprofits (frequently using professionally trained evaluators) have collected enough data to determine that their programming has likely led to client improvements in outcomes and are able to make the case that other initiatives are not the primary causes of client outcomes. An ‘advanced’ nonprofit needs to commit to hiring multiple full time staff for evaluation and data collection work, as well as committing resources to complete high rigor evaluations (which frequently results in the hiring of external evaluators).

How Funders can Provide Quality Assurance for Evaluation

Funders and nonprofits should be realistic in terms of what it takes to move from one level to the next. Moving from the ‘low’ to ‘moderate’ levels can seem relatively easy, but there are many organizational and programmatic decisions and processes that need to be changed – and these often take time and money.

A funder who wants to help build the evaluation capacity of a nonprofit to move them from ‘low’ to ‘advanced’ should think about the necessary resources to identify and collect dosage and outcome data, and the time and effort of funder staff to help manage the process, including:

• Financing the hiring of the appropriate staff in the nonprofit.
• Paying for the necessary infrastructure within the nonprofit (data systems, consulting around data collection and measurement, etc.).
• Providing critical coaching and advice to a nonprofit regarding the challenges related to improving nonprofit capacity.
• Creating reporting infrastructure that builds upon improved and improving evaluation capacity rather than forcing nonprofits to spend time on duplicate data entry or reporting metrics that serve very little purpose.

And it will take time.

Helping to move a nonprofit from ‘low’ to ‘moderate’ frequently takes several years depending on organizational commitment and existing infrastructure. Moving an organization from ‘moderate’ to ‘advanced’ could take five years or more, so funders should consider this from the start.

Nonprofit organizations learn at different paces – and that is especially true in terms of their evaluation capacity. Some organizations can make great strides quickly, while others are slow to make the organizational and cultural changes needed to improve their evaluation capacity. As funders, we need to consider how long we are willing to work with and support nonprofits in their journey to improve their evaluation capacity.

Ensuring that organizations have the ability and capacity to track and evaluate their work successfully is one way to help nonprofits reach the measurable outcomes we all aim to achieve. As a learning organization, VPP strives to continuously evolve how we deploy our resources so they do the most good – with the aim of supporting our investment partners so they are in the best position possible to reach our shared goals. Through our efforts over the years, we have found that investing the time and money necessary to improve evaluation capacity yields outcomes that would otherwise be unattainable. As funders, we must apply the same rigor and dedication to supporting the work of our nonprofit partners that we expect to see in their efforts to improve the lives of young people in this region.