Overcoming Operational Challenges in a Random Assignment Design

Lessons from the Grameen America Evaluation

By Kelsey Schaberg

This post is one in a series highlighting MDRC’s methodological work. Contributors discuss the refinement and practical use of research methods being employed across our organization.

A previous Reflections on Methodology post discussed the process used to select a research design in the evaluation of the Grameen America program, which uses a group-based model to provide loans to low-income women living in the United States who are seeking to start or expand a small business. We decided to use a random assignment design for the study, which is funded by the Robin Hood Foundation. Our next step was to implement it.

As in any study, we knew there would be tension between research and program needs. And the Grameen America program’s group-based model presented additional challenges in planning for and implementing random assignment. We had to bend the research design at various points to ensure that we were able to evaluate the Grameen America model as it is designed, not a version of the model that better fit the research.

Through good communication with Grameen America’s staff, we were able to quickly detect issues, and most of the solutions came directly from the program’s field staff. Our close partnership was key to the success of the evaluation. (Hear more about MDRC and Grameen America’s partnership in an episode of MDRC’s Evidence First podcast.)

Challenge 1

We needed to ensure that we had a fair test of the Grameen America program’s effectiveness. The study had to be set up to have a treatment contrast — that is, a difference in microloan take-up rates between the program group and the control group. This required carefully planning each design factor.

Factor Considerations Decision

When to do random assignment

Grameen America requires women to form a group of five before applying to the program. The women then participate in group training (learning about the program and loan conditions), have a home visit, and attend a group interview.

  • The earlier that random assignment occurred in the intake process, the greater the risk that program group members would drop out before receiving the Grameen America loan.
  • The later that random assignment occurred, the greater the burden placed on individuals who ended up in the control group, and the greater the chance that they would be exposed to program services.

We tried to balance these risks. Both scenarios would have resulted in a smaller treatment contrast, and therefore smaller estimates of the program’s effects.

We conducted random assignment after women had formed groups of five, but before groups went through training.

Unit of random assignment
  • It was not possible to randomly assign individuals before they formed loan groups because Grameen America required that all group members know one another.
  • If we randomly assigned individuals after they had formed loan groups, it would break up groups and disrupt the intake process.

We randomly assigned loan groups of five women.

Location
  • Adding random assignment at a current Grameen America branch would disrupt regular operations. It was logistically easier and potentially fairer to participants to do the evaluation at a new branch.

The evaluation took place at a new Grameen America branch in Union City, New Jersey.

Challenge 2

The research — and in particular random assignment — affected the program’s marketing and recruitment. Nearly all recruitment for the Grameen America program comes from recommendations from existing members. Some individuals who ended up in the control group were upset about their assignment and unlikely to refer their friends or family members. We made several adjustments to the research design over time to overcome recruitment issues and help the program staff reach the target sample size of 1,500 individuals (in 300 loan groups)

Recruitment issue Design adaptation

We initially used a 50/50 random assignment ratio: Half the groups were placed in the program group and half in the control group.

With this ratio, however, only half the groups who entered the study were potential referral sources for additional groups. This slowed recruitment to an unsustainable pace for the program.

We used a cascading random assignment ratio to allow for as many program groups as possible, while ensuring that we had a large enough control group sample to maximize statistical power. We performed power calculations each time we adjusted the ratio.

Initially we used a 50/50 ratio, then 80/20, and finally 65/35, for a 70/30 ratio overall.

Grameen America organizes loan groups within larger “centers” made up of five to six loan groups. Due to random assignment, many centers were ending up with fewer than five loan groups, affecting the program model.

We considered allowing some groups to bypass random assignment and go into a non-research group that was eligible to receive the loans (these are known as “wildcards”).

We increased the incentive offered to individuals who were randomly assigned to the control group.

At the end of a loan cycle, any Grameen America member can decline a subsequent loan. The remaining group members must find a replacement for her before they can receive another loan.

Initially, these “replacements” were not being randomly assigned, but this seemed unfair to the individuals who had gone through random assignment.

We also learned that some control group members were entering the Grameen America program and getting loans through this route.

We decided to randomly assign replacements, meaning that a group might have to recruit more than once to fill a vacancy. We used an 80/20 random assignment ratio to limit that burden.

This also established a formal process for checking that replacements had not already been randomly assigned to the control group.
 

 

 

 

Learn more