This post is one in a series highlighting MDRC’s methodological work. Contributors discuss the refinement and practical use of research methods being employed across our organization.

An earlier post in this series discussed considerations for reporting and interpreting cross-site impact variation and for designing studies to investigate such cross-site variation. This post discusses how those ideas were applied to address two broad questions in the Mother and Infant Home Visiting Program Evaluation (MIHOPE): (1) whether variation in the way program services were implemented was related to variation in impacts across local programs, and (2) whether variation in the amount of services participants received was related to variation in impacts. Results from those analyses can be found in the MIHOPE 15-month impact report.[1]

MIHOPE is the national evaluation of the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program. MIECHV was authorized by Congress in 2010 and started a major expansion of evidence-based home visiting programs for families living in at-risk communities.[2] The study is being conducted for the U.S. Department of Health and Human Services by MDRC with several other research organizations. MIHOPE includes the four evidence-based models of home visiting that 10 or more states chose in their initial MIECHV plans in fiscal year 2010-2011. To provide rigorous evidence on the MIECHV-funded programs’ effects, the study randomly assigned about 4,200 families at 86 sites in 12 states to either a program group who received MIECHV-funded home visiting or to a control group who received information on community services.

MIHOPE included a large number of sites so that it could try to answer the two questions related to variation in impacts mentioned above. Three methods were used in the published report:

• Meta-regression. The first method examined whether the features of the 86 local home visiting programs were related to their impacts. Examples of features are the national model used by the local program, whether home visitors were observed by their supervisors, and the average educational level of a local program’s home visitors. To conduct this analysis, the study used the method of Bloom, Hill, and Riccio (2003), conducting a multilevel regression where the explanatory variables included program and individual characteristics interacted with the treatment group indicator. The main concern in conducting the analysis was that local programs were not randomized to have different features, so results would not necessarily indicate causal relationships. For this reason, the results were interpreted as showing associations between features and impacts that might not be causal.
• Causal mediation analysis. In causal mediation analysis, the total effect of an intervention is divided into the direct effect of being assigned to receive the intervention and the indirect effect that operates through mediators: outcomes that might be affected by random assignment and in turn affect other outcomes — the receipt of program services, for example.[3] In essence, the indirect effect is estimated through a regression where the mediators are used as explanatory variables and the outcome is the independent variable. The method relies on an assumption called “sequential ignorability.” Sequential ignorability has a few elements, but an essential one is that there are no unmeasured participant characteristics that confound the relationship between the mediator and the outcome once research group assignment (program or control) and participant characteristics are taken into account. In MIHOPE, this assumption would be true only if the analysis included all participant characteristics and other characteristics that are correlated with both the mediator and the outcome. Even with the wealth of data collected for MIHOPE, it was not possible to include all such characteristics. To reduce the consequences of violations of sequential ignorability, the main analysis in MIHOPE used a “control function” approach in which the residual from a regression of the mediators on research group status was included as an explanatory variable in the regression in which the mediators were included as exploratory variables.[4]
• Instrumental variables. Instrumental variable analysis was developed for situations in which an explanatory variable and an outcome are correlated with a third, unobserved variable, confounding estimates of the relationship between the explanatory variable and the outcome. Instrumental variable analysis relies on finding another variable — the instrument — that is correlated with the outcome and the explanatory variable but is not correlated with the third variable. In MIHOPE, instrumental variable analysis was used to examine the relationship between amounts of program services (such as the average number of home visits a program group member received) and impacts. Randomization provided the instrument, since whether someone was assigned to the program group or control group was associated with the home visiting services an individual received but not correlated with any potential confounding factors that existed before random assignment (for example, motivation to receive home visiting). In MIHOPE, this analysis boiled down to looking at the relationship by site between estimated impacts and the average level of home visiting services received by program group families. For a multisite study with multiple mediators such as MIHOPE, Reardon and Raudenbush (2013) note that the method provides unbiased estimates only if cross-site variation in the impact on the mediators is not related to cross-site variation in the effect of the mediators on outcomes. This assumption would rule out (for example) having some sites with mothers who are both more motivated to receive home visiting and for whom home visiting is more or less effective. If their greater motivation is part of the reason their outcomes improve, results for those mothers could not be generalized to mothers with less motivation. Although this assumption is unlikely to be true for home visiting, the study team thought it was still informative to know whether local programs where families received more home visits had larger effects than those where families received fewer home visits.[5] 

The research team saw value in each of these approaches but also had concerns about the validity of each approach for the reasons outlined above. Meta-regression, instrumental variables, and causal mediation analysis were chosen for several reasons. Each could simultaneously look at several continuous, explanatory factors (such as aspects of program implementation for the meta-regression and types of home visiting services for instrumental variables and causal mediation analysis). The team also considered using two methods that rely on predicting who would be in a subgroup defined by a mediator, such as who would have received program services. The team decided not to use these methods in the published report because they were designed to look at a discrete number of subgroups.[6] In addition, instrumental variable and causal mediation analysis provided alternative ways of examining the relationship between program services and impacts, and the consistency of the results of the two analyses was one criterion used in deciding how much to emphasize specific findings.