A new study highlights a subtle problem with the growing number of experiments being conducted online.
By Nathan Collins
(Photo: Sebastien Wiertz/Flickr)
In a newonline experiment, researchers discovered an unbelievable way to lose weight: Thinking about eyeliner makes you weigh around 23 pounds less than thinking about aftershave.Well, at least that’s what you’d conclude if you didn’t look too carefully at the data. In fact, the results highlight a subtle problem with online experiments: Because it’s so easy to quit experiments midway through the exercise, researchers are sometimes looking at incorrect results.
There are a lot of good reasons to conduct psychological research online, write University of Chicago Booth School of Business researchers Haotian Zhou and Ayelet Fishbach. Online experiments cost a lot less and gather a more representative sample of the population than other methods (which mostly just recruit undergraduates from introductory-level psych classes).
Those benefits in part stem from the fact that it’s very easy to participate in an online experiment, and people will do it for next to nothing—often as little as 10 cents per minute. But the ease of participating in an online experiment through services such as Amazon’s Mechanical Turk also means it’s very easy to quit the experiment before it’s done—and different people quitting based on which experimental condition they get assigned to can have very strange consequences.
That’s what happened in the eyeliner example, Zhou and Fishbach argue. Basically, a lot of participants dropped out after they saw what the task was—writing down their feelings on eyeliner/aftershave. After that, the proportion of women was higher in the eyeliner condition; that is, 43 percent of the people who answered the eyeliner question were women, while 30 percent of those in the aftershave were women.
The results had nothing to do with changing minds, and everything to do with who quit the experiment midway through.
If you didn’t know about participants bowing out of the experiment early, you’d think that asking the eyeliner question made people lose weight, when, in fact, it was just that more women were asked that question, and women generally weigh less.
Of course, a survey question can’t make you immediately lose 20 pounds, so far as we know. Could variable attrition rates have a less noticeable, more pernicious effect?
It’s obvious that surveys can’t make you suddenly lose weight, so the researchers tested to see whether variable attrition rates could make a more pernicious, less noticeable difference. To do so, they got 198 people to state a position on gun control, asked half of them to write an explanation of their position, and then asked again for their position on gun laws.
The raw results would indicate that asking people to defend their views makes them more supportive of gun restrictions, and that’s plausible—perhaps thinking through gun control could increase a person’s support for some gun-control measures.
That is not, however, what happened. In actuality, no one dropped out of the control condition (where participants did not have to write anything), while 36 percent dropped out when asked to justify their position—and most of those dropouts, it turns out, opposed gun control. The results had nothing to do with changing minds, and everything to do with who quit the experiment midway through.
All is not lost, however. Zhou and Fishbach found, in a separate experiment, that they could cut attrition rates in half with two simple instructions: telling potential participants how important it is that they not quit experiments midway through, and asking them to write out the sentence, “I will answer open-ended questions,” such as the gun-control justification.
||
