Last year, Miller-McCune.com reported on a paper in the Journal of Peace Research that suggested some high-profile casualty claims — specifically from Iraq but by extension anywhere a certain methodology known as “cluster sampling” is used — produce death tolls that are unrealistically high.
The poster child for “cluster sampling” has been Gilbert Burnham, a professor at the Johns Hopkins Bloomberg School of Public Health and co-director of the university’s Center for Refugee and Disaster Response. He and three co-authors examined deaths following the U.S.-led invasion of Iraq in 2003, and, in a paper in the British journal The Lancet, reported the then-eye-popping figure of 655,000 “excess deaths” since the invasion.
“Our estimate of the post-invasion crude mortality rate represents a doubling of the baseline mortality rate, which, by the Sphere standards, constitutes a humanitarian emergency,” Burnham wrote. It also constituted a political emergency for the U.S. and its allies, since, as the authors acknowledged, “Our estimate of excess deaths is far higher than those reported in Iraq through passive surveillance measures.” Far higher — in some cases, by a factor of 12.
Burnham and company were both hailed and castigated — President George W. Bush said he found the report not credible —but the figures retained public currency. (Although, as one letter writer to The Lancet noted, “Gilbert Burnham and colleagues’ Iraq mortality survey was summarily dismissed by politicians as not credible, yet studies with identical methods were applauded by the same policymakers.”)
Today, the American Association for Public Opinion Research announced that it conducted an eight-month investigation of Burnham and determined he violated its code of ethics for not fully disclosing his survey methods. As the organization’s code of ethics states, “Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted.”
While some detail was provided in The Lancet and this paper, the AAPOR sought to drill down deeper, especially since cluster sampling proved a contentious tool.
Quoted in a release from the AAPOR, Mary Losch, the chair of its standards committee, explained the genesis of the probe:
AAPOR formally requested on more than one occasion from Dr. Burnham some basic information about his survey including, for example, the wording of the questions he used, instructions and explanations that were provided to respondents, and a summary of the outcomes for all households selected as potential participants in the survey. Dr. Burnham provided only partial information and explicitly refused to provide complete information about the basic elements of his research.
The group — to which neither Burnham nor the School of Public Health are members — did not question (nor endorse) his results, only his refusal to be transparent. Burnham was not immediately available for a response, several media report.
