This post is one in a series highlighting MDRC’s methodological work. Contributors discuss the refinement and practical use of research methods being employed across our organization.

Researchers across disciplines have long taken advantage of natural experiments to study the effects of policies at full scale (one of the first and best-known examples of this being researchers’ use of the Vietnam draft lottery to identify the effect of serving in the military on long-term earnings). In the past decade rapid growth in the number of charter schools and school district choice systems has provided education researchers with exciting opportunities to do the same: to use naturally occurring pockets of randomization to rigorously study the effects of policy-relevant education reforms that are already in place, often on a large scale. What’s more, this work can often be done quickly and relatively cheaply, relying solely on preexisting administrative records. Finally, when done retrospectively, the design allows for long-term follow-up of student outcomes within a short time frame.

What is a naturally occurring school lottery?

When a lottery is used to decide which students receive the opportunity to attend a school, it can create the conditions for a natural experiment. Because students’ school assignment is random, there should be no measurable or unmeasurable differences between the students who win and those who lose the opportunity to attend the school; therefore, any differences in the students’ future outcomes can be attributed to the opportunity to attend the school. In other words, researchers can identify the causal effect of the school model on students. Naturally occurring lotteries most often take place within relatively simple school-managed processes or within centrally managed admissions processes.

School-managed admissions

The simplest version of a school lottery occurs when a school independently manages its own assignment process and students (or their parents) submit applications to the school. Based on its mission and any state or federal funding rules, the school may set priorities for groups (such as siblings, or students from a certain geographic area). If more students apply than there are seats available, within its priority groups, the school uses a lottery to randomly choose which students will be offered a seat. MDRC’s study of the SEED charter school in Washington, DC, followed this approach.[1]

To use a school-managed lottery, researchers must assess whether the process was truly random and the data are sufficient to be used for an evaluation. In our SEED study we attended the public lottery and observed the process; we also walked through the lottery documentation with school leaders and traced students through the assignment process into enrollment the following fall (and beyond). Ultimately, we tracked the results student by student and verified that at every point the students’ lottery results determined whether they were admitted.

It is critical to know how the waiting list is handled by school personnel to determine whether wait-listed students later admitted should be classified as treatment or control group members. If students are placed on the waiting list in random order as a result of the lottery, their admissions offers may be seen as an extension of the lottery process. But if those who lose the lottery must take additional steps to be placed on the waiting list, then wait-list admissions offers may be affected by other student or family characteristics and are no longer randomly determined. In that case, the initial lottery results should be used to determine students’ treatment classification, and any post-lottery movement (even via the waiting list) needs to be accounted for in the analysis. Finally, in areas saturated with school options, students may participate in multiple independently run lotteries and may be selected for admission in more than one school. If all data are available, this situation can be addressed analytically. (These analytic issues will be addressed in a future post.)

Centrally managed admissions

Alternatively, a school district may centrally manage all school admissions, and students apply to one or more schools through a district-run process. The district then uses an algorithm that takes student and school preferences into account to make admission decisions across multiple schools. In the case of a nonselective school, the district would not exercise preferences over which students to admit and would rely on a random process. In effect, the algorithm assigns students a random number and uses this to break ties when more students want to go to a school than the school can serve, creating the equivalent of a lottery.

MDRC has built a portfolio of work taking advantage of such randomization to study programs on a large scale. In 2010 Howard Bloom, Saskia Levy Thompson, and I wrote our first report on the Small Schools of Choice project, which used lotteries within the New York City school assignment process to study the effects of a large-scale education reform initiative: a set of about 100 new small high schools created to serve educationally and economically disadvantaged students (Bloom and Unterman, 2014). For Small Schools of Choice, a lottery occurs when the number of eligible students living within the geographic area who have attended a high school open house and want to attend the school is more than the school can serve; eligible students with lower priorities will not be admitted. (MDRC teams have identified similar lotteries for Boston Public Schools prekindergarten programs and New York City career and technical education programs.)

For work involving centrally managed admissions, a fundamental step is to engage with the district and listen to the staff to gain a clear understanding of the assignment rules. Afterward, researchers must carefully check that understanding with patterns in the assignment data. Once they are certain they understand the process, researchers can deduce where pockets of randomization have occurred. Thus, the practice of identifying naturally occurring lotteries involves both a theoretical understanding of the assignment process and an empirical review of the data. A final check of the lottery’s internal validity is done by comparing the baseline characteristics of students who won and who lost the opportunity to attend the school, to ensure they do not differ systematically.

Once such lotteries are identified, the analysis proceeds as it would in any other multisite randomized controlled trial. However, the lottery identification process (and resulting sample) bring to the foreground a few methodological issues: Statistical power may be weaker than anticipated because of many small lotteries or because of skewed treatment-control group ratios (for example, if a school is very popular, a large number of students may compete for a small number of seats). Compliance may be imperfect if many students who win admission do not attend a school and students who do not win admission attend anyway (via subsequent steps in the assignment process). And if the oversubscribed priority group does not represent the majority of students attending the school or the sample selection process after school assignment is opaque, the generalizability of the findings may be constrained. These issues will be the subject of a future post.