One of the greatest concerns the modern economist faces has nothing to do with convoluted real estate securities or global financial policy. Instead, it has to do with the growing reliance on lab-based experiments used to test economic theories. The question is, do those experiments actually have anything to do with the real world? Well, the answer appears to be yes, according to a new study, at least in one prominent case.
At issue in the new research, published in Science, is what’s known as external validity, the idea that what works in the lab also works in real life. Fields like physics take external validity largely for granted, since, by all accounts, the laws of physics are the same throughout time and space, in or out of a research laboratory; electrons don’t act differently based on where they’re being scrutinized.
There’s a funny irony in the debate over external validity: That debate is almost entirely theoretical.
On the other hand, as Steven Levitt (of Freakonomics fame) and John List argued in a 2012 paper, external validity could be a big deal for researchers using experiments to understand the social world. For example, there are so-called demand characteristics, subtle cues in a researcher’s body language or procedures that might reveal what experimenters hope the results will be. Subjects might therefore either consciously or unconsciously try to help researchers get those results. If so, then people’s actions during a lab experiment could be entirely different in a more natural setting—and, Levitt and List argue, that’s just one of many problems experimental social scientists face.
That’s all well and good, economists Daniel Herbst and Alexandre Mas explain, but there’s a funny irony in the debate over external validity: That debate is almost entirely theoretical.
“In spite of the importance of laboratory studies in the social sciences, and of the many articles engaging in this debate, to our knowledge there has not been a systematic comparison of the same economic parameter estimated in laboratory experiments and field studies using more than a small number of studies,” Herbst and Mas write.
So, Herbst and Mas conducted their own. Thye chose to examine a relatively well-studied effect, the observation that increasing one worker’s productivity can increase others workers’ productivity as well. Over the last 15 years, 11 experimental and 23 observational—that is, real-world—studies have been published that estimate the size of the spillover effect.
Experiments and real-world studies, Herbst and Mas found, produced essentially identical results. The average estimate in experimental studies was 15 percent spillover, with most estimates falling between four and 24 percent (roughly speaking, a 0.15 percent increase in productivity for every one percent increase in co-workers’ productivity). Most observational estimates fell between six and 15 percent, with an average of 11 percent—somewhat smaller on average, but still markedly similar overall to experimental results.
“This is a surprising finding because even proponents of laboratory experiments have argued that laboratory experiments may only generalize qualitatively,” Herbst and Mas write, “and it suggests that laboratory experiments have more external validity than previously recognized.”
Quick Studies is an award-winning series that sheds light on new research and discoveries that change the way we look at the world.
