Chairman’s Corner: January 2010

‘Social Outcomes’: Missing the Forest for the Trees?

by Mario Morino, Co-founder of VPP

For the past month, I have worked through draft after draft of this column as I’ve struggled to properly express my concern about the growing movement to advance “social outcomes”—as well as “impact,” “measurement,” “metrics,” “evaluation,” “accountability,” and a half-dozen other related concepts—for nonprofit organizations.

Here is my concern, as best as I can manage to articulate it. I am increasingly worried that the vast majority of funders and nonprofits are achieving, at best, marginal benefit from their efforts to implement outcomes thinking. Granted, there has been some truly meaningful progress. Select hospitals like the Cleveland Clinic and Mayo Clinic have made great strides in assessing their outcomes and being transparent about their performance. And the Edna McConnell Clark Foundation and a few others have keenly focused on the challenge of social outcomes and have dealt with them well. Yet many other efforts may end up misdirecting, even wasting, precious time and financial resources. In some extreme situations, well-intentioned efforts may actually risk producing adverse effects on nonprofits and those they serve.

Keep in mind this comes from a guy who has been strident in stressing the importance of outcomes and assessment for nonprofits for close to 15 years!

To What End?

The three words that have served me best throughout my business and nonprofit careers are “To what end?” I try to return to this simple question constantly during the life of any project or initiative, especially when I fear I’m drifting away from my original purpose or I’m starting to confuse ends and means.

The main reason the dialogue on social outcomes is off track is because we have failed to keep our eyes fixed on the ends we are trying to advance. Every ounce of our effort on social outcomes should be with one end in mind: helping nonprofits create greater benefits for the people and causes they serve. Period.

Most outcomes efforts today have drifted far from that end. Too often, measurement has become an end in and of itself.

• If greater benefits were the end, we would be laser focused on helping nonprofits find and use the right information to manage to their mission in a disciplined way.
• If greater benefits were the end, the sector’s dialogue on outcomes would be 95% about mission and 5% about metrics. Today, we have the ratio reversed.
• If greater benefits were the end, we would be working to help nonprofits clarify the end results they are trying to achieve. Achieving clarity of purpose produces increased benefits even if you never put a single metric in place!
• If greater benefits were the end, funders and nonprofits would not settle for simplistic or surrogate metrics. As one small example, the assessment of a student’s years of learning and development cannot be reduced down to the test scores that end up sampling but one or two facets of what she has learned (to say nothing of what she is capable of in the future).
• If greater benefits were the end, we would properly differentiate between operational performance and organizational effectiveness. What good is it to focus on an organization’s overhead costs or fund development levels if we don’t have a clue as to how effective the organization is at creating benefits for those it serves?
• If greater benefits were the end, we would own up to how much encouragement and support nonprofits need in order to define and assess what they do and how well they do it. We’ve approached this challenge as if it’s about numbers when it’s really about changing cultures. Changing culture requires large and persistent investments of time, talent, and money.

Below, I will try to add meat to these bones. Most important, I want to sound a clarion call that we have a fundamental problem that threatens to undermine years’ worth of well-intentioned efforts to help nonprofits achieve greater impact.

Common Sense Left Behind

A vivid illustration of measurement run amok comes to us courtesy of No Child Left Behind.

I’ve had the opportunity to be engaged in K-12 education through VPP’s work with schools in the National Capital Region, through my participation on a number of national educational initiatives, as an advisor to leaders in education, through my deep engagement with a school in Cleveland for bright students who learn differently, and in my role as a parent of three children. Based on these varied experiences, I believe that the good intentions of the No Child Left Behind Act have led schools and students astray.

Like most people, I believe we need ways to judge our schools and how well our students are doing. But No Child Left Behind does these things poorly. It’s the classic example of metrics over mission.

The current regime of “memorization and testing” and the growing battery of standardized tests risk rewarding targeted test preparation while not informing us or the students themselves whether they are developing the relevant skills and competencies they and our society and economy so sorely need. Yes, it’s very important to achieve—and measure—core competencies like reading and math. But where are the incentives for schools to educate young people to be curious, engaged citizens capable of critical thinking and problem solving? Where are the incentives to encourage collaborative development and learning? Where are the incentives for teaching young people to question, even challenge, “the system?” Where are the incentives to give students practical experience in the ways of life outside of school?

A good friend and mentor who is a nationally recognized education leader shed yet further light on this dilemma. He pointed to the work of Yale Professor Seymour Sarason, who wrote as early as the 1960s about his fear that outcomes theory and products represent a reductionist exercise that looks at only one or two parts of what the organization does, then draws conclusions based on whatever is sampled. To quote my friend, “Sampling may work fine for determining what’s going on in someone’s blood. But at school these days [the only things we’re sampling] are reading and math test scores, because they are easy to acquire and report.”

Too Hard on ‘Soft’ Outcomes

“To what end?” are three powerful words. But as I learned in my Catholic upbringing, two words that carry just as much power are mea culpa.

Here’s an example of how I looked too narrowly at outcomes—and, as a result, risked knocking nonprofits off mission.

In the early years of Venture Philanthropy Partners, we got a lot of resistance to my push for “clearly defined outcomes” from leaders whose organizations placed a premium on being holistic with their services and functioning as “community builders.” Although I agreed with them in concept, I felt that a focus on “community building” was too soft to be a legitimate outcome. Outcomes related to “community building” are, after all, radically ambiguous compared to outcomes like reduction in teenage pregnancy and substance abuse.

I now see better that serving the entire family (holistic services) and building community are some of the very things that create the environment—a web of support and community—that helps youth avoid high-risk behavior, get an education, and prepare for college or a job. But talking about “community building” was too intangible, and not readily measureable to us at the time—and, candidly, difficult to sell to our own stakeholders and the emerging field of nonprofit performance at large.

I regret not being more open in my thinking back then. Instead of pushing back on what we were hearing, we should have done more to understand “soft” achievements that may in fact be every bit as real and important as “harder” outcomes. I aspire to do a better job of making them part and parcel of future efforts to assess outcomes and performance—even if that means using qualitative and/or anecdotal indicators.

The point is this: When public or private funders establish performance metrics and then tie significant rewards or consequences to their achievement, organizations and people will migrate to the behaviors that will allow them to meet their defined targets. If the metrics are appropriate and closely tied to mission, this is a good thing. But if the metrics are overly simplistic and unmoored from mission, then organizations will go racing in the wrong direction. To paraphrase Yogi Berra, they’ll get lost, but they’ll be making good time.

Backseat Driving

Ultimately, the benefits of an outcome orientation must accrue to the nonprofit. Sadly, today most of the discussions around outcomes are being driven by funders, with too little regard for what nonprofit leaders need.

I strongly urge nonprofits and funders alike to be more open to seeing that assessment is most valuable and worthwhile if it is driven by the nonprofit itself. Attempts to define outcomes seldom produce positive results when they are imposed on organizations from the outside.

This is not to say that funders should avoid encouraging nonprofit leaders’ efforts to assess their work. Far from it. My point is that funders should support and encourage nonprofits to self-assess with information that they can and will use to manage the organization. Nonprofit leaders need to own the process and be the primary beneficiary of it.

If we were to take that nonprofit-centric—rather than funder-centric—approach, we would all have a much higher likelihood of achieving what we’re really setting out to accomplish. Nonprofit leaders would not be navigating with intuition alone. They would gain powerful tools to determine where they’re headed, chart a logical course, and course-correct when they’re off. And then, as a big ancillary benefit, they would be able to build broader support by sharing their valuable new information with funders and other stakeholders.

Shining Lights

Some nonprofits have made significant strides in adopting a culture focused on defining and achieving outcomes for the people they serve. One example I mentioned earlier is the Cleveland Clinic, which I serve as a trustee. The Cleveland Clinic, along with the Mayo Clinic and a few others, lead the field in their use of outcomes to assess their own effectiveness. The Cleveland Clinic now openly presents this information via its website.

Thanks to my good friends at the Edna McConnell Clark Foundation, in the field of human services I often hold up the work of Youth Villages. Youth Villages, which helps emotionally troubled children through a wide range of residential- and community-based treatment programs in 11 states, rigorously tracks all of the children it serves, during their treatment and often for two years after they discharge. In the words of CEO Pat Lawler, “The state…shouldn’t be buying beds; they should buy outcomes, successful outcomes.”

Youth Villages continues looking to improve the measures they use. Thanks to a forward-thinking grant from the Memphis-based Hyde Family Foundations, Youth Villages is working with Dr. Richard Barth, the Dean of the University of Maryland School of Social Work, to improve the outcome-evaluation process as a whole, better focus treatment plans so they can target the most effective services to individual children, and share its system with other providers around the country.

First Principles

We can help other nonprofit leaders achieve similar success if we refocus on the first-order question, “To what end?” To do that, we need to remember why we’re engaging in a discussion of outcomes in the first place: to help nonprofit leaders to be more effective—that is, to deliver greater benefits to those they serve. Doing so will provide the basis for the accountability we all seek to have in place.

In my next column, I will offer practical suggestions to help nonprofit leaders—and those who want them to succeed—gain the methods and tools they need to determine where they’re headed, chart a logical course, and course-correct when they’re off.