Since 2019, high school seniors from low-income households in the state of Washington have received guidance from OtterBot, an interactive chatbot texting tool with artificial intelligence (AI) capabilities. The chatbot, designed by the Washington Student Achievement Council (WSAC), provides information to simplify the complex processes involved in applying for financial aid and enrolling in postsecondary education or training. It is specifically intended to support students in the state’s College Bound program, which offers early financial aid commitments to students in low-income households, helping make higher education more accessible.

In 2022, the Optimizing Texting Technology through Engagement Research with Students (OTTERS) project—funded by the Capital One Foundation and led by MDRC and WSAC in partnership with Mainstay—was launched to study how the chatbot could be improved to better support students and families. This report provides information about and findings from the first two phases of this ongoing study.

In the study’s first phase, the research team identified students’ needs and opportunities for OtterBot to better meet those needs by creating process maps, conducting focus groups, ad­ministering surveys, and analyzing administrative data. This phase revealed several significant challenges faced by students from low-income backgrounds—particularly those who are the first in their families to pursue higher education. Many students experienced confusion over the financial aid process, uncertainty about college costs, and a lack of awareness and knowl­edge about financial aid options such as those available through the College Bound program. To compound these difficulties, OtterBot’s reach remained limited; only 43 percent of College Bound students received its messages, with just 12 percent responding and 15 percent opting out. This low level of engagement can likely be partially attributed to students’ distrust of auto­mated tools such as OtterBot. The research also uncovered differences in engagement, financial aid application completion, and postsecondary enrollment, with gaps evident across GPA, race/ethnicity, and gender.

In the study’s second phase, the research team applied these findings to enhance OtterBot’s messaging approach to better support student engagement and understanding of financial aid application processes. The alternative messaging approach aimed to use communications best practices to provide clearer information and guidance and to better address behavioral barri­ers and student concerns about automated tools. Using A/B testing to compare the alternative and standard messaging approaches, the research team found that the alternative messages increased student engagement with OtterBot. There was also a statistically significant increase in financial aid application completion rates for male students. (A statistically significant effect is one that can be attributed with a high degree of confidence to the intervention being studied.)

As a potential next step, Phase 3 of the OTTERS project could pilot and evaluate an intervention that combines the alternative approach to messaging from Phase 2 with a new, more intensive strategy for providing enrollment and financial aid guidance. This strategy could involve using OtterBot to connect students and parents to more in-depth or localized services, or using gen­erative AI to provide more customized messaging.

The OTTERS project has demonstrated the value of using a phased approach, consisting of user research followed by optimization and testing, to improve educational technology platforms. This approach involves studying the intended user, the platform, and possible modifications to the platform to improve the tools and services before devoting time and resources to evaluating the effects of the platform in an impact study. This strategy could prove valuable in analyzing future educational technology initiatives.