Agenda, Scope, and Goals

The first year of this project (Phase 1) included in-depth market research on the effectiveness of OtterBot messages to encourage financial aid completion. Phase 1 used human-centered design and behavioral science to identify participant engagement problems and generate solutions. The research team conducted process and journey mapping to identify barriers and areas where there is opportunity to use OtterBot to help with the complicated process of obtaining financial aid and enrolling in college. We analyzed administrative data to better understand student usage of and engagement with OtterBot. We conducted focus groups and fielded surveys with students and parents to better understand their needs, questions, reactions to OtterBot messaging, and what could be most useful for them.

In the second year of the project (Phase 2), the team is using insights from Phase 1 to refine and develop messaging campaigns for the school year for seniors and their parents, as well as for juniors. The team will also begin developing messaging campaigns for ninth- and tenth-graders.

The impact of Phase 1-informed messages for seniors will be assessed through A/B testing. A randomly assigned group of seniors will receive the usual OtterBot messages (with very light alterations), while another group will receive OtterBot messages designed using insights from Phase 1 and a behavioral science lens. The Phase 1-informed messages aim to improve OtterBot engagement (for instance, decrease the percentage of students who opt out of receiving messages) as well as to nudge students to act (including participating in financial aid workshops or applying for FSA IDs).   

In addition to these learning objectives, Phase 2 is also focused on dissemination and capacity-building. The team will publish two briefs with findings from the project. The team will also create a toolkit with data science and behavioral science resources, all of which can be used in the future by WSAC and other states that use similar chatbots. Following up on a successful Phase 1 convening, the project will also share findings in a Phase 2 convening for groups who are already using chatbots or are interested in using chatbots to increase college access. A key goal of both phases of this project is to provide training and technical assistance to WSAC so that they will be able to continue this work on their own. Through publications, the toolkit, and convenings, the team also hopes to provide tools and resources for other states and groups using chatbots to increase college access.

After Phase 2, we will seek funding for a rigorous evaluation of a strengthened OtterBot chatbot. 

Design, Sites, and Data Sources

The OTTERS project is focused on learning about the OtterBot interactive chatbot, a tool that aims to serve all students from families with low incomes in Washington state.

The quantitative data analyses focus primarily on student characteristics, OtterBot engagement data, and College Bound Scholarship data. The team will also use data on whether students completed a financial aid application, National Student Clearinghouse Data, and/or data from the Washington State Education Research and Data Center to better understand the outcome levels in this population and to gain initial insights into the extent to which OtterBot engagement may be associated with outcomes, such as financial aid application completion and postsecondary matriculation. The team is using a variety of exploratory analytic methods to better understand patterns in the data, including funnel analysis, text analysis, clustering, and variable importance analysis. The team is also conducting A/B tests to understand the extent to which the Phase 1-informed messaging campaigns may lead to different levels of engagement (and possibly different rates of financial aid completion and postsecondary enrollment) compared to the original messaging that WSAC was using.

Phase 1 collected qualitative data collection to explore insights from the quantitative data. For example, focus groups and surveys explored questions such as:

• Why do people choose to engage or not engage with OtterBot?
• Are messages culturally relevant?
• What environmental and behavioral conditions might explain differences in rates of financial aid and postsecondary education application completion between different groups at key steps?

What additional supports might be required to equitably serve Washington’s College Bound scholarship students?