Will Old Media Learn a Lesson from Lancet? Doubtful

NationalJournal.com has news (HT Instapundit) about the reality of the October 2006 Lancet report on civilian deaths in Iraq -- a report that was breathlessly and gullibly cited at the time by Old Media outlets and reporters (including David Brown here at the Washington Post).

Here is background for those unfamiliar with the original story:

Published by The Lancet, a venerable British medical journal, the study [PDF] used previously accepted methods for calculating death rates to estimate the number of "excess" Iraqi deaths after the 2003 invasion at 426,369 to 793,663; the study said the most likely figure was near the middle of that range: 654,965. Almost 92 percent of the dead, the study asserted, were killed by bullets, bombs, or U.S. air strikes. This stunning toll was more than 10 times the number of deaths estimated by the Iraqi or U.S. governments, or by any human-rights group.

In December 2005, Bush had used a figure of 30,000 civilian deaths in Iraq. Iraq's health ministry calculated that, based on death certificates, 50,000 Iraqis had died in the war through June 2006. A cautiously compiled database of media reports by a London-based anti-war group called Iraq Body Count confirmed at least 45,000 war dead during the same time period. These were all horrific numbers -- but the death count in The Lancet's study differed by an order of magnitude.

Editorials in many major newspapers cited the Lancet article as further evidence that the invasion of Iraq was a bad idea, and the liberal blogosphere ridiculed Bush for his response. Prominent mainstream media outlets quoted various academics who vouched for the study's methodology, including some who said they had reviewed the data before publication.

Here is what the National Journal has found:

..... Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros's Open Society Institute.

..... (concerning Item 1) The Lancet study was based on techniques developed by public health experts to determine rates of illness and death from epidemics and famines in large populations. This "cluster" sampling is a relatively new methodology that attempts to replicate the logic of public opinion polling in Third World locales that lack a telecommunications infrastructure.

Following this method, questioners undertake a house-to-house survey in certain areas and then extrapolate the results from that statistical sample to the entire national population. According to this study's design, teams of Iraqi questioners would visit approximately 47 randomly chosen clusters of homes.

..... (concerning Item 2) "The authors refuse to provide anyone with the underlying data," said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at Harvard University. Some critics have wondered whether the Iraqi researchers engaged in a practice known as "curb-stoning," sitting on a curb and filling out the forms to reach a desired result. Another possibility is that the teams went primarily into neighborhoods controlled by anti-American militias and were steered to homes that would provide information about the "crimes" committed by the Americans.

..... (concerning Item 3) Virtually everyone connected with the study has been an outspoken opponent of U.S. actions in Iraq. (So are several of the study's biggest critics, such as Iraq Body Count.) Whether this affected the authors' scientific judgments and led them to turn a blind eye to flaws is up for debate.

None of these issues kept Old Media from jumping on the story three weeks before the 2006 mid-term elections. The discrepancy between Iraq Body Count (IBC) at the time (IBC's total now is 80,320 - 87,731) and Lancet alone should have been cause for serious concern (that, and "where are all the dead bodies being hidden?"). Brown's Washington Post story noted the degree of the difference between IBC and Lancet, but "somehow" wasn't able to find anyone skeptical about the report's results for a quote.

This should be a lesson to Old Media that a little digging is in order when something so out of line with previous reports shows up. But it's one that probably won't be learned -- at least when outlier studies like Lancet's fit their advocacy template.

Cross-posted at BizzyBlog.com.