City Year Evaluation
Overview
Helping children and young adults develop their talents and strengths also helps them contribute to their communities in the future. Yet systemic inequities that disproportionately affect students of color and students from low-income households mean too many young people do not have access to such supportive learning environments, and these inequities have been exacerbated by the COVID-19 pandemic.
City Year is an educational and human development organization that partners with schools nationwide to support student success and address the root causes of inequitable educational outcomes. Every year, City Year recruits a diverse group of Student Success Coaches, ages 18 through 25, to deliver its Whole School Whole Child (WSWC) model. City Year Student Success Coaches commit to serving in schools full time for one school year. During that time, they collaborate with teachers, administrators, and other Student Success Coaches to plan and deliver personalized support services to students who need it most and to create a safe and supportive school climate where students can feel committed to and capable of achieving their academic and life goals.
The WSWC model consists of holistic and personalized academic, social, and emotional services that are designed to help students develop the skills they need to navigate and complete high school. City Year Student Success Coaches offer universal services to all students (Tier 1) and targeted intervention services to a subset of students (Tier 2).
- Universal (Tier 1) services are provided to all students in the school. They include after-school programs and school-wide events that recognize positive behavior and student success. They also include in-classroom support for students in English language arts (ELA) and math.
- Targeted (Tier 2) services are provided to students who appear to be at increased risk of not graduating based on early warning indicators related to attendance, engagement manifested as behavior, and course performance in ELA or math. Tier 2 services blend tutoring in ELA and math with social and emotional development and behavior support. Student Success Coaches also provide attendance coaching. These services are provided one-on-one or in small groups in class, during class time but outside of class, or between classes.
In 2017, the American Institutes for Research (AIR) and MDRC began a five-year evaluation of WSWC services in 22 middle schools in five large, urban school districts. The evaluation focused on middle school because it is a pivotal time when students form their identities and learn the foundational academic, social, and emotional skills that they need to succeed in high school and beyond. Building evidence on what works for this age group is an important pathway for improving high school success.
The evaluation included two impact studies. The first study explored the effects of the WSWC model (Tier 1 and Tier 2 services) on the outcomes of students enrolled in the study schools, using a quasi-experimental study design. The second study attempted to isolate the effect of Tier 2 services for individual students who were identified as being at heightened risk of dropping out of school, using a student-level, randomized experiment. The evaluation also included an implementation study.
The evaluation was affected by the COVID-19 pandemic. School closures in March 2020 interrupted WSWC services as well as the availability of data for the evaluation. The evaluation therefore focuses on the implementation and effects of WSWC services before schools closed in March 2020. Despite this change in the study timeline, the evaluation provides valuable insights about the benefits of school and community-partner collaborations that provide holistic support services designed to meet particular needs. It also draws attention to the opportunity to rethink the measures evaluators use to measure student experience and growth.
The research described here was supported by the Institute of Education Sciences in the U.S. Department of Education, through grant R305A170227 to AIR. The opinions expressed are those of the study team and do not represent views of the Institute or the U.S. Department of Education.
Additional Project Details
Agenda, Scope, and Goals
The purpose of the evaluation was to provide rigorous evidence of the effects of the two-tiered WSWC model on the academic, social, and emotional outcomes of middle school students, and to provide information on the model’s implementation. Specifically, there were three research goals:
- To assess whether students enrolled in middle schools that provide WSWC services had better outcomes, on average, than students enrolled in similar schools that were not providing WSWC services.
- To assess the value of Tier 2 services by measuring whether eligible students who received these services had better outcomes than similar students in the same schools who did not receive them.
- To determine how the different components of the WSWC model were implemented in schools, and to identify facilitators of and challenges to implementation.
Design, Sites, and Data Sources
The evaluation was conducted in 22 middle schools in five large, urban school districts. The evaluation included two impact studies:
- A study of the effects of the WSWC model for all students based on a quasi-experimental comparative interrupted time series design. The analysis included two groups of schools: 22 middle schools that were implementing the WSWC model (program schools) and a group of similar, matched schools in the same districts that did not implement the model (comparison schools). The analysis looks at the outcomes of consecutive cohorts (entering classes) of sixth-grade students who were enrolled in the study schools in the years before (pre-intervention) and after (post-intervention) the WSWC model was implemented. Trends over time in the program and comparison schools were compared to determine (1) whether on average, students who were enrolled in the program schools while the WSWC model was being implemented had better outcomes than expected relative to the outcomes of pre-intervention cohorts of students, and (2) whether changes over time in student outcomes were larger and more positive in the program schools than in the comparison schools. The quasi-experimental study focuses on the impact of the WSWC model before the COVID-19 pandemic using student records data collected through the 2018-2019 school year. (The study was supposed to include data from the 2019-2020 school year as well, but student records were not available for this school year because of the COVID-19 pandemic.)
- A study of the effects of Tier 2 services for eligible students based on a student-level randomized controlled trial. In each of the 22 program schools, sixth-grade students who were considered to be at increased risk of not graduating from high school were identified by City Year or the school. A lottery-like process was used to select randomly which of these students were offered Tier 2 services (the program group), and which students were not (the control group). This random assignment process was intended to create two groups of students (the program group and the control group) who were equivalent with respect to all their characteristics except for the offer of Tier 2 services. Random assignment was conducted in the fall of 2018 separately by school and by students’ area of need (ELA, math, behavior, attendance, or a combination). The effect of Tier 2 services for eligible students was evaluated by comparing the outcomes of students in the program group and the control group. The experimental study focused on the impact of Tier 2 services before the COVID-19 pandemic on social-emotional skills, academic outcomes, and suspensions during the 2018-2019 and 2019-2020 school years. Due to the COVID-19 pandemic, data from the 2019-2020 school year were limited to the first half of the year before schools closed, and did not include test scores. In addition, a second cohort of sixth-grade students who were randomly assigned in the fall of 2019 was subsequently dropped from the study because those students received less than a full year of services. All students in the Tier 2 study continued to receive the Tier 1 services, so the findings represent the effect of Tier 2 services over and above the effect of Tier 1 services.
When considered together, the findings from these two studies can provide a better understanding of the effects of the WSWC model than the studies could if conducted in isolation. Both studies examined the effects of the WSWC model and Tier 2 services in three domains: academic achievement (in ELA and math), school engagement (specifically, with respect to attendance and disciplinary outcomes), and social and emotional skills. These outcomes were measured using student records data acquired from school districts participating in the study. For the Tier 2 study, a student survey was also used to measure students’ social and emotional skills. Finally, information from school and district websites was used to illuminate the types of services provided to students in schools not working with City Year. (A principal survey was also administered in comparison schools in the spring of 2020 to collect data about the student support services they offered, but response rates were very low due to the pandemic.)
To put the impact findings in context, the evaluation also examined the implementation of the WSWC model in the 2018-2019 school year and the first half the 2019-2020 school year, before schools closed. The implementation study examined how the components and activities of the WSWC model were implemented in the 22 program schools in these school years, and the amounts and types of services that students received, drawing on City Year staff and teacher surveys and program-monitoring data. MDRC also conducted visits to summer training sessions for City Year Student Success Coaches in the summer of 2019 to observe the training activities and to conduct focus groups, and visited selected schools in the winter of 2020 to speak with staff members and to observe Student Success Coaches as they provided classroom and whole-school services.
AIR led the study of Tier 2 services. MDRC led the quasi-experimental study of the WSWC model and the implementation study.
