Randomistas: how radical researchers changed our world

(QESP Editor’s Note: The following is a reprint of a 01/03/2018 article in The Mandarin. The original, with related posts, is available at https://www.themandarin.com.au/89090-randomistas-how-radical-researchers-changed-our-world/)

Randomised trials are in your life, whether you like it or not. In most advanced countries, governments won’t pay for pharmaceuticals unless they’ve undergone a randomised evaluation. Increasingly, the world’s smartest aid agencies are looking for the same level of evidence before they allocate funds to a project.

After four decades of conducting randomised trials in social policy, Judith Gueron has dozens of maxims for researchers: ‘Never say that something about the research is too complex to get into.’ ‘If someone is unreservedly enthusiastic about the study, he or she doesn’t understand it.’ It’s hard-won wisdom, emerging from more than thirty large-scale social policy trials, involving a total of 300,000 participants.

In 1974 the Ford Foundation and six federal government agencies created the Manpower Demonstration Research Corporation, now known simply as MDRC. Its mission was to improve understanding about what worked in social policy by conducting random assignment studies. Judith Gueron, then aged thirty-three and having received her Harvard economics PhD just a few years earlier, became MDRC’s first research director.

Raised in Manhattan, Gueron attributes her ambition and confidence to having a father who told her ‘for as long as I can remember, that girls, and I in particular, could do anything’. In her work at MDRC, she would need this self-belief. Not only were the fields of economics and policy evaluation heavily male-dominated, but experiments were a radical idea. At that time, academics didn’t get tenure by doing randomised trials, but with complex mathematical models. MDRC was ‘a lonely band of zealots’.

Gueron’s first major experiment tested whether long-term welfare recipients and people considered ‘unemployable’ could be supported into jobs. Random assignment had never before been attempted on a large multisite employment program of this scale. Gueron’s team were warned it would be impossible, that ‘asking program operators to turn people away would be like asking doctors to deny a patient a known cure’.

To address the criticism that they were being cold-hearted towards a deserving group of people, MDRC came up with a clever solution: they would expand the size of the treatment group so that it used every last dollar of available funding. That meant their detractors could not credibly claim that having a control group denied worthy recipients from getting a supported job. Even if you scrapped the control group, the number of people who received the treatment would be unchanged.

Gueron recalls how the staff felt as the program was rolled out. They all hoped it would succeed, but had to keep reminding themselves that it probably would not. ‘Fortunately for MDRC in these formative years, we had random assignment – with its inexorable comparison of experimental versus control outcomes – to keep us honest and help us avoid the pitfalls of advocacy research.’

When the results from the first evaluation rolled in, they showed that the supported work program helped women, but not men. And even for the women the impacts were small. When participants found employment, the government reduced their welfare benefits. Since the jobs didn’t pay much, the net effect was only a slight fall in poverty rates. The program was effective, but no panacea. Yet to Gueron, what mattered most wasn’t the results, but how they had judged the program. A naive evaluator might simply have looked at the raw outcomes, which showed that more men found jobs than women. Yet a randomised evaluation showed that this had nothing to do with the program: men in the control group were just as likely to get jobs as men in the treatment group. It took randomisation to reveal the truth. Gueron was ‘hooked . . . on the beauty and power of an experiment’.

Throughout the 1980s and ’90s, Gueron worked with state and local agencies across the United States. Getting started with a new randomised evaluation could be tough. In San Jose, Gueron wanted to evaluate a job-training program for young migrants from Mexico. The managers of the program told her that turning away people at random was inconsistent with their mission – the staff would never agree to it. So she met with the staff and explained why random assignment was uniquely reliable. A positive finding, Gueron told the team, might convince the federal government to fund programs like theirs. ‘They agonized about the pain of turning away needy young people, and they talked about whether this would be justified if, as a result, other youth gained new opportunities. Then they asked us to leave the room, talked more, and voted. Shortly thereafter, we were ushered back in and told that random assignment had won.’ The evaluation showed positive results, and prompted the federal government to fund an expansion across fifteen more sites.

Taking over as MDRC president in 1986, Gueron became better at explaining to people who ran social programs why random assignment was fair. In one instance, a county commissioner continued random assignment after the research ended, seeing it as a just and unbiased way of choosing who would be assisted. Others weren’t so warm. In 1990 Florida legislator Ben Graber tried to shut down MDRC’s evaluation of ‘Project Independence’, an employment program. Random assignment, Graber said, was treating welfare recipients like ‘guinea pigs’. He foreshadowed legislation that would ban the use of control groups. The media was quick to pick up the story. Testifying in Florida, Gueron focused on the fact that the program had never been properly evaluated. ‘If we had a readily available wonder drug to help people be self-sufficient and off of welfare, I’m sure there isn’t a person in this room who would not use it. If Project Independence is that, I assume you would put more money into it. Since we don’t know that and the funds are not there, you would be wise to get the answers first.’ The legislature approved the trial. A few years later, the evaluation results were in: the employment program saved the taxpayer about as much as it cost to run. It wasn’t a wonder drug, but it was worth continuing.

Before the use of randomised evaluation, debates over social programs often featured conflicting studies, using a tangle of approaches. When the experts diverged, policymakers and the public were torn. As economist Henry Aaron remarked, ‘What is an ordinary member of the tribe to do when the witch doctors disagree?’ In the Florida case, the official in charge of Project Independence came to Gueron after commissioning three non-randomised studies, all of which used different methodologies and arrived at different conclusions. ‘The only way to get out of the pickle of these duelling unprovable things,’ he decided, ‘was to get an evaluation of unquestioned quality.’

To Gueron, randomised evaluation was a powerful communications tool because of its simplicity. In the late 1960s, Gueron recalls, ‘researchers knew in theory the power of random assignment. They just didn’t believe it would be useful to evaluate real-world social programs and address important policy questions.’ But over her career Gueron successfully deployed policy experiments in courtrooms, schools, community colleges, job training centres and community organisations. Over time, she ceased warning her colleagues that ‘no one wants to be in a random assignment study’. MDRC was turned down more often than accepted, but the popularity of social experiments steadily grew. They were straightforward, and made it harder to hide failure. ‘The key thing about random assignment,’ Gueron concludes, ‘is that it is the epitome of transparency. You flip a coin. You create two groups or more. You calculate average outcomes. And you subtract . . . That power is to be treasured.’ The chief lesson she draws from her career: ‘Fighting for random assignment is worth it.’

Randomistas: How radical researchers changed our world by Andrew Leigh is published by La Trobe University Press. $29.99

Available online at www.blackincbooks.com or wherever good books are sold.