A study in the latest issue of the Journal of Peace Research suggests that the particular “cluster sampling” method of estimating fatalities in the Iraq conflict is biased toward generating unrealistically high numbers.
Physicist Neil F. Johnson of the University of Miami and four co-authors aren’t alleging any political motive in this reputed overcount — which generated shock waves when its topline figure of 655,000 dead in the 2003 invasion was publicized — just a bias in taking samples near the corners of major arteries. Johnson and co-authors Sean Gourley of Oxford and Michael Spagat of the University of London have called it “main street bias.”
“Cluster sampling itself is not problematic,” they write, “but the micro-level details on how households are selected at the final stage of sampling are crucial and widely overlooked. … These studies often initiate the sampling process from some easily accessible geographical feature, such as the centre of a village, in order to economize resources and ensure staff safety.”
In the well-publicized study of Iraqi civilian deaths led by Gilbert Burnham that appeared two years ago in The Lancet, those geographic features were random intersections. “A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.”
As Burnham told The Washington Post when the study was published: “We’re very confident with the results,” both the number and the trends, which matched the direction (but by no means the lower numbers) of other reputable mortality studies.
Nonetheless, the study generated its own conflict, with President Bush saying he didn’t consider the report credible and others citing the numbers – then up to a dozen times the best figures from independent observers – as further evidence of the war’s evil. Some critics picked at the study’s funding or its on-the-ground coordinator, a former Hussein-era bureaucrat; the resulting controversy even spawned its own lengthy Wikipedia listing, “Lancet surveys of Iraq War casualties.”
The Journal of Peace Research authors didn’t pick up the political gauntlet. A nontechnical summary of the technical criticism of the Lancet study is that street corners are busy and violent places and that numbers drawn from there applied uniformly across less-violent urban areas provide too high a body count.
“For conflicts like the one in Iraq,” Johnson and the others write, “violent events tend to be focused around crossstreets, since they are a natural habitat for patrols, convoys, police stations, parked cars, road-blocks, cafes, and street-markets. Major highways would not offer such a wide range of potential targets – nor would secluded neighborhoods.”
The journal study suggests that the resulting bias “may be quite large” and then notes that the Iraq Family Health Survey suggests the Burnham numbers may be off by a factor of four.
