Evaluation of Academic Instruction in After-School Programs

Locations

Topics

K–12 Education
Young People

Overview

Many low-income children in the early grades need after-school care. And many of these children score well below their more advantaged peers on standardized tests of reading and math. The confluence of these circumstances suggests that it may be possible to strengthen the academic component of after-school programs (now often confined to providing help with homework) to better enable children participating in these programs to develop solid literacy and numeracy skills.

Whether or not imparting a stronger instructional focus to after-school programs will bring about the desired goals is uncertain. The programs operate at the end of a school day, when children (and teachers) may be tired, less able to concentrate, and expecting a change of pace, so that activities and instructional materials must be engaging. Participation in after-school programs is voluntary, and program attendance is likely to be less consistent than during the school day, making continuity of instruction a challenge. Staff members in after-school programs typically include not only certified teachers but also paraprofessionals and sometimes volunteers, and time for instructor training and daily preparation is generally limited. Finally, while in-school curricula have the advantages of being tied to state standards and (in some cases) being scientifically-based, it is not clear that they can be adapted to an after-school setting. For all these reasons, the U.S. Department of Education selected MDRC to mount a rigorous test of whether children in grades 2-5 who receive enhanced academic instruction in after-school programs have better academic outcomes than their peers who get more typical after-school academic support.

Agenda, Scope, and Goals

Before launching a full-scale test of the effectiveness of enhanced academic instruction, it was important to be sure that the math and reading curricula that were evaluated were strong, that they could be adapted for use in after-school programs, and that they could be implemented successfully. MDRC’s evaluation therefore included both a pilot phase and an implementation phase. The pilot phase involved selecting curriculum developers to modify existing reading and math materials for the after-school setting; developing additional instructional materials as needed; implementing the new materials on a limited scale during the 2004-2005 academic year to assess their feasibility and promise; and refining the curricula in light of this experience.

Upon successful completion of the pilot phase, a full-scale random assignment study of the impacts produced by the two curricula was mounted in a larger number of programs during the 2005-2006 school year, and continued through the 2006-2007 school year. The programs received technical assistance and support to help ensure solid implementation. The questions this test was designed to address include:

  • Does enhanced instruction in after-school programs that use reading and math curricula adapted from the regular school day significantly boost reading and math proficiency, as measured by test scores?
     
  • What are the effects of the after-school reading and math interventions on other in-school academic behavior outcomes, such as homework quality and completion and classroom behavior (for example, engagement)?
     
  • Do the interventions’ impacts differ for students of different grade levels and academic proficiency? Which students benefit the most from participation?
     
  • What is the impact of the enhanced after-school program once the program has been in operation for a year, when after-school teachers may have more experience?
     
  • What is the impact on student academic performance after two years of program implementation?

Design, Sites, and Data Sources

During the pilot phase, four after-school centers tested the reading materials, and another four centers tested the math materials. These centers were located in Atlanta, Georgia; Anne Arundel County, Maryland; Bridgeport, Connecticut; and Norristown, Pennsylvania.

The design for the random assignment study called for a sample of approximately 50 after-school centers in the first implementation year. Twenty-five of these centers used the reading curriculum, and the remaining 25 employed the math curriculum. Within each center, students were randomly assigned to receive the enhanced program or to receive the kind of academic support usually provided by the program. During the second implementation year, 27 of the 50 after-school centers continued to participate in the study.

During the full study phase, data were drawn from a variety of sources. Intake forms and achievement tests administered at intake are used to describe the sample; to compare students in the sample with others in the district, in after-school programs, and in urban schools more generally; and to define subgroups of students on the basis of their pre-random assignment characteristics. Attendance data allow us to examine the intensity of participation in the services under study, while field research, document review, and monitoring reports enable us to understand the nature of the services provided and the fidelity with which the enhanced instruction is delivered. A follow-up survey is being used to describe participation by sample members in other after-school activities that are intended to improve their academic and youth development outcomes. Finally, follow-up achievement tests and surveys are used to estimate differences in student outcomes between the program and control groups.

Supporters
Partners

Staff

Featured Work

All Publications and Resources