MDRC’s Michael Weiss and Brit Henderson have paired with Harvard professor Luke Miratrix on an important methodological paper, “An Applied Researcher’s Guide to Estimating Effects from Multisite Individually Randomized Trials: Estimands, Estimators, and Estimates.” In this paper, recently published in the Journal of Research on Educational Effectiveness, they explore important choices that applied researchers must make when conducting large-scale multisite individually randomized controlled trials—choices that must be made even when the goal is “simply” to estimate the overall average effect of the program.

The authors explain how researchers make critical choices about how to estimate effects—for instance, they can target the average effect across individuals or sites. The researcher can target the effect for the people and sites in the experimental sample or people and sites in a larger population. These targeted quantities, called estimands, can differ. Once an estimand is selected, there are many ways to estimate the targeted effect. The paper describes 15 such estimators, considers which estimands they are appropriate for, and discusses their properties. Using data from 12 large multisite randomized controlled trials, the authors estimate the overall average effect (and standard error) using each estimator and compare the results. They assess the extent that these decisions matter in practice and provide guidance for applied researchers.

Michael Weiss

Brit Henderson