The key is to poll a crowd, then do what they least expect.
By Nathan Collins
(Photo: stressedtechy/Flickr)
There’s a popular idea known as the “wisdom of the crowd,” which holds that the best way to find an answer to a question is by asking a whole bunch of people and combining their answers to find the best one. For instance, say you want to find out whether it’s better to take Sunset Boulevard or the 10 freeway to get across Los Angeles at rush hour. To get a good answer, just put a bunch of people to a vote.
The trouble is, crowdsourcing commuting routes and such is tricky business. If you poll a random selection of Americans about the L.A. traffic question, many of them would assume the 10 freeway is always faster, just because it’s a freeway, but it is, in fact (as Southland residents know), often better to take surface streets.
Still, it may be possible to use crowdsourcing methods to produce reliable answers, Dražen Prelec,Sebastian Seung, and John McCoyargue this week in Nature. Their idea: Don’t go with what the crowd thinks, but rather with what they least expect.
Here’s how it works. Imagine you ask 50 people whether Philadelphia is the capital of Pennsylvania (it’s not), and then you ask each person to predict the outcome of the crowdsourced vote. Suppose the average prediction comes out to 40 “yes” votes, but when you tally the actual vote, there are only 30 “yes” votes. Then, according to the researchers’ algorithm, the answer we should infer is the one most people would find surprising, namely, “no.”
That’s how the algorithm, dubbed “Surprisingly Popular,” works. But why exactly does the algorithm work? It relies on the plausible assumption that, even when the crowd is wrong, it will be more accurate than what people expect of it—that is, that the crowd will move in the direction of the truth, even if it doesn’t get all the way there.
To see if that plays out in practice, the team presented 50 statements about state capitals to 51 students at the Massachusetts Institute of Technology, where two of the researchers are based, each time asking whether the statement was true or false and what percentage of people would answer “true.” The Surprisingly Popular method reduced the number of incorrect decisions—crowds answering “yes” to the Philadelphia question, for example—by 48 percent compared with a simple majority voting scheme. Additional experiments in the domains of art and medicine produced similar results.
The method isn’t perfect. For example, if one person’s beliefs influence others—as is often the case—Surprisingly Popular can break down. But, the authors write, it will give the right answer in a number of cases where more basic methods fail.
