Impact evaluations of high school reforms may take years to reach final conclusions, mostly because the outcomes of interest, like graduation, take time to occur. However, within these longer-term evaluations, shorter-term findings from implementation research can reveal insights that are valuable to program operators as well as to researchers.

MDRC’s timely implementation analyses within a larger impact evaluation helped the XQ Institute, a nonprofit organization focused on transforming high school education, adapt their framework for measuring student success. The MDRC research team also used the information to develop a new data collection method to learn about important aspects of the intervention.

Background

The XQ Institute was founded in 2015 with the goal of “reimagining the high school experience” so that all students get the preparation they need for college, careers, and all that their futures may hold. That fall, XQ announced an open-call, open-eligibility competition to apply for a $10-million grant to design an innovative high school. In the first competition phase, nearly 1,500 teams submitted proposed visions for their schools. In the final competition phase, 348 teams were invited to submit full applications. The XQ Institute eventually awarded grants to 16 teams.

Since that time, the XQ Institute has supported these schools as they’ve implemented their school design plans. The XQ Institute has also begun working with school districts and state departments of education to support widescale high school transformation. The XQ Institute does not require that schools have particular structures or follow a manualized set of practices. Instead, it specifies that schools work to build a culture around a unifying set of research-backed “Design Principles” (for instance, having a strong mission and culture and fostering caring and trusting relationships between and among students and adults), and aims for schools to support their students on five overlapping “Learner Outcomes”—that is, to become:

• Masters of fundamental literacies
• Holders of foundational knowledge,
• Original thinkers for an uncertain world,
• Generous collaborators for tough problems, and
• Learners for life.

To date, XQ has funded 57 schools with grants ranging from $20,000 to $10 million.

MDRC’s evaluation of XQ Schools is a multi-methods study involving (1) an impact evaluation of XQ’s initial set of high schools across the country; (2) an implementation study of XQ School start-up and redesign; and (3) administration and analysis of a student survey focused on students’ socioemotional well-being. Findings from the survey support schools in their measurement of socioemotional wellbeing and contribute to the MDRC evaluation.

The implementation research seeks to understand how XQ’s initial set of 16 high schools were designed and operated, the kinds of support those schools received, the unique ways in which each school approached XQ’s guiding frameworks (XQ Design Principles and the XQ Learner Outcomes), and factors that might support or impede schools’ abilities to implement these frameworks. For initial insights into these questions, we interviewed roughly 50 administrators and teachers at 12 XQ Schools across the country in spring 2022.

The Challenge: Timely Analysis and Feedback

As our study team was wrapping up the interviews, XQ’s leadership told us that they were especially eager to learn about how teachers incorporated the XQ Learner Outcomes in their practices. The national organization was revising its guidance to schools about these practices for the school year starting in fall 2022, and findings from the interviews could help inform these revisions.

The interviews yielded information about the ways in which teachers understood and incorporated the XQ Learner Outcomes in their practices, whether the XQ Learner Outcomes mapped to other frameworks the schools were using, how their approaches to the XQ Learner Outcomes shifted over time, and how and whether they measured attainment of the Learner Outcomes.

In response to XQ leadership’s learning needs and timeline, we focused our initial analysis on practices related to XQ Learner Outcomes, with plans to expand analyses to other themes or topics later. We first used a deductive approach to code for topics related to practices around XQ Learner Outcomes that we specifically asked about during interviews. Then we used an inductive approach to identify any themes relevant to XQ Learner Outcomes that were not captured in the earlier round of deductive coding. We assessed the relative prevalence of various themes and identified idiosyncrasies.

The Result: New Approaches

After analyzing the interview data, we met with leadership at XQ to discuss the emerging findings. Particularly salient to XQ leaders were findings about how teachers took integrated approaches to addressing multiple XQ Learner Outcomes at once, how their strategies were often affected by local context (for example, how teachers built local climate change issues affecting their community into their curricula in service of addressing specific XQ Learner Outcomes and Design Principles related to problem solving and community building), and how they adapted their approaches as a result of the pandemic.

Our study findings informed the way XQ leaders provide teachers with an initial orientation to XQ Learner Outcomes. The findings also informed XQ leaders’ ongoing development of a framework that would help teachers operationalize XQ Learner Outcomes in their classrooms. Finally, the findings prompted XQ leaders to consider whether and how to set implementation thresholds that they expect schools to meet, and how these thresholds should be defined.

While the interview data revealed important information about how teachers were conceptualizing XQ Learner Outcomes and the related supports they received around implementation, it lacked numerous examples of how teachers were approaching individual XQ Learner Outcomes within their classrooms. Looking to generate specific examples from a broader sample of schools and teachers, XQ leaders and MDRC decided to field teacher logs (short, weekly surveys about classroom practice) as part of implementation research in school year 2022-23. The logs generated detailed information about teachers’ practices introducing and working with XQ Learner Outcomes. This detailed information will inform the XQ Institute’s continued development of practices and guidance about the XQ Learner Outcomes, and also will inform the study’s eventual findings about the effectiveness of XQ’s approach to high school redesign.