Blog

Evaluation

Designing Better Programs with Randomized Controlled Trials

February 22, 2017

Day after day, nonprofits are always on the search for additional ways to further prove the effectiveness of their programming and the outcomes that result. One method of proving effectiveness through evaluation is by using randomized controlled trials. We sat down with Isaac Castillo, our Director of Outcomes, Assessment, and Learning to discuss randomized controlled trials and the benefits of using this method.

What is a randomized controlled trial (RCT)? And how is it used in the nonprofit evaluation world?

A randomized controlled trial (RCT) is a highly rigorous evaluation methodology that randomly assigns participants or locations to either a treatment group, which receives programming or services, or a control group, which does not receive services. This randomization process is designed to minimize the differences between the treatment and control groups and therefore test whether the intervention/programming is what is leading to different outcomes (as opposed to the differences between the groups).

Due to the high level of rigor, data collection responsibilities and frequently the costs associated with RCTs, only a very small fraction of nonprofit organizations even attempt RCTs. When they do, these nonprofits usually have completed other types of evaluations that have demonstrated the effectiveness of their programming. In most cases, when a nonprofit organization undertakes an RCT, they are seeking to further prove, and not initially prove, the effectiveness of their program/intervention.

What is the greatest benefit to using an RCT? 

The greatest benefit to using an RCT is the randomization process itself, which results in the random creation of a treatment group that receives services and a control group which does not receive services. If done correctly, this creates two groups of individuals that are very similar. Their ages, race/ethnicity, income levels, motivations, etc.,  are all relatively equal which allows for an accurate test of whether the interventions, as opposed to something else, leads to different outcomes for the treatment group.

What are some of the challenges that can be solved by using RCT?

The randomization process of an RCT helps to examine the attribution problem, which is basically testing whether or not the intervention is what leads to differences in outcomes between the two groups. In evaluation methodologies that do not use randomization, it is possible to create two groups that are not identical. One group may be older, or the groups may live in different areas, or one group may be more motivated to succeed. These differences between groups may be what leads to differences in outcomes as opposed to the intervention.  An RCT helps to eliminate as many ‘alternative’ explanations as possible that may lead to positive or negative outcomes over and above the intervention.

What are some of the ethical questions that can arise when conducting RCT as a method of evaluation reporting?

One potential ethical challenge in using an RCT is that can be perceived as ‘denying’ services to individuals that ‘lose’ the randomization and end up in the control group. However, there are several steps that can be taken to address this situation.

First and foremost, nonprofits should usually only pursue an RCT when they have waitlists for programming or they have fewer program slots than needed to serve the population that needs services. In these instances, most nonprofits use a first come first served approach to providing services. The randomization process just modifies the selection process for services. Instead of first come first served, it becomes a random process where every person who needs services has an equal chance of receiving services.

Another potential approach would be to place those individuals that are randomized to the control group on a waiting list where they become the first individuals to receive service once the evaluation is completed. Sometimes referred to as delayed service, this allows for these participant to eventually receive services, albeit at a later time.

However, the potential ethical and real world complications in administering an RCT in a direct service environment are tricky and each nonprofit must be comfortable with, and be ready to justify publicly, the randomization process and its potential, real and perceived, effects on the service population.

How have some of our investment partners been using RCTs? How does VPP support these trials and use them in our engagement model?

Our existing investment partners have used RCTs to further prove the effectiveness of select program models with the intent of replicating or scaling these programs in additional locations. VPP supports the use of RCTs in the appropriate situations and works with our partners to identify potential challenges associated with RCTs and how they can be overcome. VPP also recognizes that not every situation is well suited to the use of an RCT and can also offer guidance on alternative evaluation approaches that are better aligned with the investment partner’s situation.

Are there programs or intervention designs that simply make RCTs impractical?

RCTs work very well with straightforward program designs. As programs become more complex an RCT becomes more difficult to implement to the point that it becomes impractical.  For example, if a program consists of several distinct parts such as tutoring, exercise, nutrition, poetry and substance abuse prevention, it becomes difficult to test the effectiveness of this type of interventions since it will be difficult to determine which parts of the programming are leading to outcomes.

In addition, place-based or community-based programming is also difficult to evaluate using an RCT. In these cases, there are often factors that can’t be controlled when using a randomization process or it is ethically difficult to randomize entire communities.  For example, if an intervention is designed to reduce poverty in a community, it will be difficult to randomly select some communities to receive the intervention while others do not.

What are the major cost drivers of an RCT? Why are they so expensive, and where can there be cost savings?

The primary cost driver of RCTs is data collection for those individuals assigned to the control group. Individuals assigned to the control group frequently lose contact with the nonprofit, which makes it extremely difficult to find these individuals and collect follow-up data. A great deal of time and effort needs to be spent to find these control group individuals and maintain contact for enough time to collect data throughout the entire evaluation.

Some cost savings can be achieved if the cost and burden of collecting data can be absorbed by someone else. For example, if a nonprofit can get access to data collected by government entities such as school systems, health departments, employment departments, etc., the cost for data collection decreases dramatically.