For decades, MDRC researchers have travelled to schools, job centers, public housing sites, and social services providers locally and around the world. We hopped on trains and subways, rented cars, and flew in planes because we needed to be there to really understand program implementation. Occasionally, we used virtual platforms to speak with program and school staff or with program participants.

In March 2020, as COVID-19 cases swelled in the U.S., it became clear that our site visits would not go on as planned. We halted all work-related travel and transitioned our research to virtual modes: phone calls, Zoom conversations, online focus groups, and virtual site observations. Some projects lent themselves to being moved online more than others.

How did our work change throughout this unforeseen and unprecedented shift? What have we learned that we’d like to carry forward? What adaptations didn’t work well? Do we need to be there to understand program implementation? We’ve considered these questions throughout the 18 months in our Implementation Research Group meetings, Zoom coffee breaks, and Teams chats. This post describes themes we’ve discussed—in the words of implementation researchers from across MDRC.

What We Hope to Carry Forward

We want to continue using approaches like the ones below when we conduct both virtual and in-person site visits in the future.

Developing innovations and engaging study participants. We hope to keep exercising the muscles that the pandemic forced us to grow: the infrastructure to offer virtual interviews, the creativity and skill to engage study participants in virtual formats, and the ability to explore how programs and individuals adapt to urgent needs. For example, some teams share visuals on screen during virtual focus groups and ask participants to annotate on screen and discuss their perceived barriers to implementation.

Study participants and our own staff are now more comfortable with Zoom and other video call technology. “Everyone used to have technical issues with video calls and so we didn’t use them,” commented Dominique Dukes. “Now, a big benefit is that people are really comfortable doing the Zoom thing.”

Using short virtual sessions to prep for and follow up on site visits. During the pandemic, we increasingly used brief pre-“visit” check-ins with site staff. A 20-minute call enables us to gather or verify basic information about the program. We can reflect on that information, adapt subsequent protocols if possible, and conduct a more tailored virtual site visit. After the visit, we can follow up with targeted questions to clarify and confirm. In these ways, the core site visits and interviews can be more efficient and effective. We’ll continue to use these virtual pre- and post-visit calls in both our virtual and our in-person protocols.

Reaching new locations and populations. Before the pandemic, we sometimes had to exclude locations from our studies or technical assistance efforts because of the prohibitive costs or travel time to reach them. But pandemic restrictions forced us to be there virtually, even for locations that we could easily reach prior to the pandemic. With increasing comfort using virtual formats, we can vary the types of locations where we conduct our work, such as more rural or remote areas. Virtual options also allow individuals who found participating in in-person interviews or focus groups costly or inconvenient—often due to competing child care or work responsibilities—to more easily participate in our studies.

What We’re Eager to Return to In Person

Implementation researchers have long drawn on a range of information, including interviews, surveys, focus groups, administrative data, and observations. When pandemic travel restrictions shifted info-gathering from mostly in person to solely virtual, some important things were lost. “Being in person,” said Barbara Condliffe, “provides more opportunities for researchers to understand the context of their data collection.”

Building rapport with study participants. Human connection adds richness to our understanding of programs and the individuals who participate in them. It was more difficult for many researchers to build relationships with staff and participants virtually, especially if they hadn’t previously met in person. “A big part of our work is connecting with people on a personal level,” said Rachael Metz. “That’s a lot harder to do virtually. ‘How’s the weather? How are you doing?’ is not enough.”

Observing context and details. When we visit physical spaces, the information we gather through observation protocols adds context and detail to the data that we capture during interviews or focus groups. “Observations help us more fully understand service delivery, receipt, and relationships,” noted Kyla Wasserman. “They also allow us to see things that a staff person may not even be conscious of—and therefore wouldn’t note in an interview—such as power dynamics.” Our virtual experiences made us more fully appreciate these contextual details, which we aim to more systematically collect and report as we return to in-person interviews and visits.

Understanding group dynamics via in-person focus groups. An important limitation of virtual focus groups is missing the nuances that happen before, during, and after in-person gatherings such as off-hand remarks, body language, and interactions among study participants. As Betsy Tessler put it, “You miss seeing the dynamic—what happens in between your questions.” It is difficult to recreate that dynamic over Zoom, when some interviewees call in from their cars, homes, or shared living spaces and don’t feel comfortable being on video.

“I feel there’s a different cultural connection, a different energy, that goes into being present with people in a space they’re familiar with, as opposed to us inviting them into our Zoom room,” said Jalen Alexander. “It allows for a certain level of comfort for them, coming into a space that’s already communal for them. When we’re in person, we’re the guests. They have a certain level of control and autonomy that doesn’t exist virtually.”

Gaining understanding through immersion. Because in-person site visits involve physically leaving our usual workspaces for concentrated amounts of time, we typically turn on out-of-office autoreplies, update our voicemail greetings, and pause work on other projects. In-person site visits let us immerse ourselves in everything about the site, program, organization, context, and area. “There’s a momentum that comes from going from one interview to another, to an observation of service offerings, from being in the environment,” said Betsy Tessler. “Going from one part of the visit to the next allows you to see similarities across interviews.” During pandemic closures, we tried to consolidate interview scheduling as much as possible to approximate the on-site immersive experience, but it wasn’t the same.

“With interviews stretched out over a longer time period and interspersed with work on other projects, it was difficult to get completely immersed in a particular site or program,” said Marissa Strassberger. “Though we set aside large blocks of time to conduct back-to-back interviews, it was too easy to check email or join meetings for other projects in between interviews, which—though convenient—ultimately detracted from our focus on the site or program. I’m looking forward to returning to in-person site visits.”

Looking Ahead

As the programs we study return to classrooms, community centers, and other locations, and with vaccines and public health protocols in place, we hope that in-person site visits are on the horizon. Like many other aspects of our daily lives—visiting family and friends, going grocery shopping, riding public transportation—we anticipate approaching familiar activities with a different perspective.

In some cases, we will carry forward new habits and innovations from the past year to sharpen our implementation research and reduce the resources we use. Where possible, we’ll continue incorporating new engagement methods and conducting virtual interviews both to make sure that time on physical site visits is used efficiently and to expand whom we include in our studies.

In other cases, we have new appreciation for something we had taken for granted. We’ve had opportunities to step back and consider how aspects of our work, like observations, tours, and immersion, improve our understanding of the implementation of programs. Where possible, we will adapt our data collection protocols and analyses to harness this information more explicitly.

We’re also more attuned to the tradeoffs we face when designing and conducting implementation research. For example, while virtual options can expand the reach of our studies to rural or remote areas and to individuals whose schedules require flexibility, sole reliance on virtual communications risks leaving out those who have limited access to Wi-Fi or cellphone minutes. And while we value the immersive experience of in-person site visits, they are intense and exhausting (for both program staff and our study teams). Virtual site visits that are less concentrated may be less onerous for program staff and participants and, for some researchers, be more conducive to debriefing, absorbing, and reflecting. We will continue reflecting on these and other tradeoffs as we emerge from the devastating COVID-19 pandemic.

Suggested citation: Lewy, Erika B. 2021. “Looking to the Future of Implementation Research: What COVID-19 Adaptations We Will Take with Us, and What We Will Gladly Leave Behind.” Implementation Research Incubator (blog), September.