This essay first appeared on the Pay for Success Learning Hub Blog, published by the Nonprofit Finance Fund.

Pay for Success financing promises to transform social policy: Nonprofit service providers get much-needed capital both to innovate and to scale evidence-based programs, government only pays if programs achieve agreed-upon performance measures, and investors receive both an investment return and a social return.  While the win/win/win inherent in this “promise” has propelled the considerable momentum behind Social Impact Bond deals spreading across the country, the long-term viability of the Pay for Success movement hinges on actual operating experience. Are the resulting programs actually well-implemented and effective in reducing recidivism, promoting self-sufficiency, and improving school performance? Does government get the accountability and performance it expects? How reliable are the measures used? Do investors get the returns they seek?  

Fortunately, as the field pivots from the idealized case that proponents have made to build interest in the model to the reality of making deals and operating programs, the groundwork is now being laid to answer these questions. Late last year, MDRC released Financing Promising Evidence-Based Programs: Early Lessons from the New York City Social Impact Bond, a detailed inside account of the Adolescent Behavioral Learning Experience (ABLE) program at Rikers Island and its Social Impact Bond financing structure.

Disturbingly, one of every two 16- to 18-year-olds incarcerated at Rikers Island jail will return within a year of discharge. By providing cognitive behavioral services at Rikers, ABLE aims to equip these teenagers with the social and emotional skills to help make better life choices when they leave jail, hopefully leading, in turn, to improved life outcomes, a reduction in the recidivism rate, financial savings to government, and eventually returns to private investors.

As our report describes, getting from attractive term sheets to improved rap sheets is not a linear process. Early lessons from the ABLE experience center around three broad challenges: estimating costs and reliably measuring savings, balancing the need for program flexibility against the necessity of contract inflexibility, and finding the right spot on the “risk continuum” to place the bet — from unproven innovations on one end to programs with strong evidence of effectiveness on the other — that will satisfy a given set of investors.

Estimating Costs and Measuring Savings

Knowledgeable partners with a good understanding of social problems, social programs, and government systems are necessary for successfully negotiating any Social Impact Bond deal, but an essential prerequisite is a term sheet built upon reliable cost data. To estimate recidivism savings, for instance, governments must first know how much a jail bed actually costs — both in terms of annual operating costs and quasi-fixed costs — yet many corrections agencies do not have this data. New York City did: the marginal cost of a year at Rikers Island was $4,600. But that figure shot up to $28,000 per year once 100 beds were closed, a number representing the shutting down of an entire housing unit. In the Rikers Island Social Impact Bond, government savings and investor profits are driven by reductions in recidivism that exceed this 100-bed threshold. Achieving savings in multiples of 100 requires serving large numbers of teenage inmates each year over several years.

While recidivism rates are relatively simple to measure, many factors can influence whether recidivism rates rise or fall: an unusually strong or weak economy, a change in policing tactics (for instance, a modification of “stop and frisk” policies), or a change in court practices, since most of these young people are awaiting trial and unable to make bail. To reliably know whether any subsequent change in recidivism was due to ABLE’s cognitive behavioral therapy sessions or to one of these external factors, measuring the program’s effects required a sophisticated quasi-experimental research design that could compare the recidivism rates of ABLE participants with those of a like group not eligible for ABLE — a challenging task assumed by the Vera Institute for Justice. Without a strong design, investors might be repaid when the City received no savings at all or, conversely, investors might lose even though the City gained, since errors in understanding the program’s effects could work in either direction.

Inflexible Contracts and Changing Program Needs

Term sheets, loan and security agreements, liability carve-outs, and guarantee fund agreements are binding documents often requiring months of negotiation among the parties. As our report describes, that was certainly the case with the Rikers Island Social Impact Bond. But the problems faced by incarcerated 16- to 18-year-olds and by the providers that serve them (The Osborne Association, Friends of the Island Academy, and the Rikers school system) and the day-to-day rules and rhythms of the jail are anything but fixed. The list of obstacles is long: Inmates with opposing gang affiliations who can’t be housed together; unexpected but frequent lock-downs cutting into class time; inmates under administrative segregation not able to attend school and having to be served in another way; entry rates that change over time, making it difficult to serve the number of inmates the deal requires; school teachers not working out as program facilitators; and more.

While a pre-deal pilot period was essential for discovering and addressing many of these problems, full-scale operation inevitably revealed still other obstacles requiring additional adaptation. And adaptation often means changes in time frames, staffing, and other issues that have been painstakingly negotiated in binding investment agreements. How to reconcile the contradiction between inflexible contracts and changing program needs is likely the biggest “inconvenient truth” facing each of the operating Pay for Success projects.

Innovation and Evidence

Another tension facing any Pay for Success deal is where on the “risk continuum” to place the bet: from unproven but high-potential innovations to evidence-based programs that have been shown to work and seem capable of producing returns at scale. As our report describes, the ABLE deal was built upon a sturdy evidence base, including a high-quality meta-analysis conducted by the strongest researchers in the field synthesizing more than a dozen rigorous studies.[1] On average, across a range of studies and settings, cognitive behavioral therapy programs, like the one developed for Rikers, reduced recidivism rates by 25 percent over a 12-month time period. But most had not operated at the scale required in New York City, none had operated on Rikers Island (and context does matter), and many had not worked with incarcerated teens. This mix of strong evidence but a new and challenging context probably placed the Rikers program at the midpoint on the risk continuum, requiring some innovation while still being in a position to implement at scale. Balancing evidence with innovation will remain a challenge so long as deals are required to demonstrate government budget savings, a very difficult hurdle to overcome (and one that, by the way, largely ignores benefits to participants and to society at large).

In our new report, we share what we’ve learned so far about these and other compelling issues related both to the process of packaging a Social Impact Bond deal and to the early implementation challenges of the ABLE program. The future of the Pay for Success movement rests on building on the lessons learned from the first efforts to implement these new and potentially transformative financing structures.