Introduction

Many social programs are intended to generate long-term benefits for their participants, but evaluations of those programs have historically not had access to the necessary resources to measure such outcomes over the long run—for 5, 10, or 15 years or longer. Administrative data—data that are created and stored to enable government administration, or as a by-product of it—present a potentially low-cost opportunity for tracking the long-term effects of new policy or program interventions. However, the procedures for gaining access to these data are often idiosyncratic, time-intensive, or undocumented.

The case of the Moving to Opportunity demonstration may be particularly instructive here. Early research focused on the adults of households that were supported in moving from subsidized, public housing to neighborhoods with low levels of poverty, finding little to no economic impact after families completed such moves. However, later findings indicated that living in neighborhoods with low poverty levels had substantial, positive economic impacts on some children of those families after they reached adulthood. As government agencies and their research partners consider opportunities to leverage these data to extend evidence about their programs—and as data privacy and security take on ever-increasing importance—the Administration for Children and Families’ Office of Planning, Research and Evaluation (ACF/OPRE) in the U.S. Department of Health and Human Services is developing resources to support these interests and explore the benefits and limitations of linking study and administrative data for long-term research. The “From Theory to Practice” project represents one of ACF/OPRE’s efforts to support the research community in conducting such explorations.

Purpose

This Guide for Using Administrative Data to Examine Long-Term Outcomes in Program Evaluation is being produced to complement federal efforts to expand the use of administrative data for building evidence—in this case, evidence about the long-term effectiveness of federally funded programs and interventions. This guide is a resource to assist program evaluation teams—including funders, sponsors, and evaluation research partners—in assessing the feasibility and potential value of examining long-term outcomes using administrative data. It describes common steps that are involved in linking evaluation data and administrative data. It will help teams tackle topics such as:

• how to identify worthwhile, policy-relevant opportunities for extending evaluation follow-up
• what study data and infrastructure are required to enable extended follow-up
• factors to consider in selecting suitable administrative data sources
• navigating the legal and ethical requirements that are commonly associated with pursuing extended follow-up
• special considerations for matching study and administrative data
• how to assess the quality of linked study and administrative data

The guide is directed primarily toward research teams considering whether to examine long-term outcomes for evaluations, particularly those whose work has been completed. This guide also includes valuable information that may enable research on long-term outcomes for new or ongoing evaluations.

Key Findings and Highlights

This guide proposes to think about the preparation and work that are required to extend study follow-up using administrative data in three main phases of effort:

• Phase 1 entails considering the value and practicality of long-term follow-up. It is focused on ensuring that there is a solid policy and research justification for extending follow-up and that there are suitable and accessible administrative data that will answer specified research questions.
• Phase 2 involves preparing for long-term follow-up by laying the necessary legal and ethical groundwork. Notably, study teams are advised to ensure that data-related agreements between evaluation teams and other entities governing the research describe and enable the planned research activities. Teams are also advised to assess what, if any, human subjects ethical standards apply to the proposed long-term follow-up research by consulting an Institutional Review Board (IRB).
• Phase 3 centers around assessing administrative data to determine whether they are suitable for answering the proposed research questions and linking to study data. Researchers are advised to assess administrative data providers’ requirements for matching to study data, to consider the extent to which administrative data will cover study participants and their activities, to determine the identifiers that will be used to match and the method for matching, and to establish how the quality of linked data will be assessed.

The full guide describes in more detail the various considerations that study teams might take into account to begin to realize the potential uses of administrative data in researching long-term outcomes. Examples and case studies throughout the guide generally highlight efforts to research long-term economic outcomes, such as participant employment and earnings, but the concepts presented should be applicable to a variety of social policy research contexts.