Prominent Statistician And Bush Opponent Calls Lancet Study “Unreliable”
And he puts the low range at as little as 100,000, or about 15% of what the study authors claimed:
Donald Berry is Chairman of the Department of Biostatistics and Applied Mathematics at the University of Texas MD Anderson Cancer Center.
In an e-mail Berry, apparently no supporter of President Bush’s Iraq War policy, described the study published today in The Lancet, the British medical journal, as “unreliable.”
“…The last thing I want to do is agree with Bush, especially on something dealing with Iraq. But I think ‘unreliable’ is apt. (I just heard Bush say ‘not credible.’ ‘Unreliable’ is better. There is a certain amount of credibility in the study, but they exaggerate the reliability of their estimate.)
“Selecting clusters and households that are representative and random is enormously difficult. Moreover, any bias on the part of the interviewers in the selection process would occur in every cluster and would therefore be magnified. The authors point out the possibility of bias, but they do not account for it in their report.
“It is true that the range reported (392,979–942,636) is huge. Its width represents only one source of variability, the statistical error present under the assumption that their sample is representative and random. I believe their analysis to be correct under these assumptions. However, it does not incorporate the possibility of biases such as the one I mentioned above. Incorporating the possibility of such biases would lead to a substantially wider range, the potential for bias being huge. Although there is no formal way to address bias short of having an ‘independent body assess the excess mortality,’ which the authors recommend, the lower end of this range could easily drop to the 100,000 level.”
Does this prove anything? Well, it proves it pays to be skeptical of huge outliers…
UPDATE 3:41 p.m.: More from the BBC:
Those who had faith in an earlier report from 2004 – also published in the medical journal The Lancet – are now able to say that this larger survey proves their point that Iraqi deaths have been far greater than publicly reported, and have now reached what the report calls “a humanitarian emergency”.
Those who thought that the 2004 survey was exaggerated – it estimated 98,000 additional deaths up until September 2004 – think this one is even more wide of the mark.
Les Roberts, one of the report’s authors said: “It may not be extremely precise, but it gets us into the ball park.”
On the other hand, Michael O’Hanlon of the Brookings Institution, which tracks statistics in its Iraq Index, said: “I do not believe the new numbers. I think they’re way off.”
The Brooking[sic] Index, relying on the UN (which gets figures from the Iraqi health ministry) and the Iraq Body Count (IBC), estimates the civilian death toll at about 62,000.
The IBC, which counts the number of reported civilian deaths, puts them between 43,850 and 48,693, though it adds that this is a baseline and that the true figure could be much higher.
The IBC reaction to the Lancet report is awaited.
If I understand the methodology used here correctly, the authors of the study used an actual death figure of 547, then extrapolated that number into…654,965. So their estimated deaths account for 99.9% from the original figure that they’re basing their study on. Also – they give no validity for who’s been actually issuing all of these “death certificates” for the massive body count that they’re quoting here – how on earth can they claim to be accessing any credible data on this, when the original data is already highly questionable?
A standard criticism of all research is that the sample chosen is not representative and random. This is how the sample for the Lancet study was chosen, undermining an assertion of investigator bias.
And I quote: As a first stage of sampling, 50 clusters were selected systematically by Governorate with a population proportional to size approach, on the basis of the 2004 UNDP/Iraqi Ministry of Planning population estimates (table 1). At the second stage of sampling, the Governorate’s constituent administrative units were
listed by population or estimated population, and location(s) were selected randomly proportionate to
population size. The third stage consisted of random selection of a main street within the administrative unit
from a list of all main streets. A residential street was then randomly selected from a list of residential streets
crossing the main street. On the residential street, houses were numbered and a start household was randomly
selected. From this start household, the team proceeded to the adjacent residence until 40 households were
surveyed. For this study, a household was defined as a unit that ate together, and had a separate entrance from the street or a separate apartment entrance. (page 2-Burnham etal. Mortality after the 2003 invasion of Iraq: a cross-sectional clustere sample survey, Lancet).
Remember the investigators have no idea who lives in any given house or the number of deaths that have occurred in the household. The death certificates only serve as corroboration of the number of deaths elicited verbally during the interview. I guess people could go out and procure death certificates for all sorts of reasons other than someone died.
Hey, don’t tell it to me, tell it to the Chairman of the Department of Biostatistics and Applied Mathematics at the University of Texas MD Anderson Cancer Center…
I just bit thru my tongue.
Since the Lancet report authors had access to death certificates, why didn’t they just find out how many death certificates were issued during the period since the war began, and compare that to the number of death certificates issued during an equal period of time prior to the war?
It appears that every blog on the web is struggling to deal with this study without anyone pointing out exactly where it went wrong. Let me give it a shot.
The authors made two decisions, neither of which is catastrophic but taken together they destroy randomness of the sample and thereby invalidate the study.
First, they relied on word of mouth to pre-announce the study, and its purpose to households before they arrived at the door:
As I said, this is not necessarily a problem as long as this in no way affects the way a household responds to the questions, invites outside influence, or affects the samples.
Second, they allow households to decline participation. Page 2:
OK, now we have a problem. Households can decline participation since they know what is being asked. This may not even be an intentional attempt to skew the results, it may just be that people that did not have any deaths in the last five years felt no reason to talk. It may also be that a local authority was encouraging them not to participate without the knowledge of the survey team. I give the survey team the benefit of the doubt and assume they tried their best to be impartial. (Why do all this work if you suspect the results are invalid!)
This method can be very precise, but it is of vital importance that the sample truly be random. This is difficult enough without a glaring error like this, and given this error the survey is worthless. I do not give the authors the same benefit of the doubt. Only a fool could produce this paper without seeing the flaw.
One piece of data that could help explain how serious this problem was is number of households that refused to participate. With this data we could re-run the analysis assuming there were no deaths in that additional sample. That data is not in the paper.
-jcp-
jcp makes some good points about some possible issues with the study, but the biggest issue, in my opinion, is a lack of common sense. I am a statistician and my research is demogrphic modeling, I use mathematical modeling to predict population change based on a varity of factors. Some of my results turn out to be just nonsense, population levels FAR too high to be viewed as realistic. I do not then decide that “this is what is going to happen in the future”; rather, I look at my model to try and decipher why it failed. The worst thing any scientist can do is remove common sense from their research, plug some numbers into a statistical model, and then just accept the values that are spit out. These numbers are an order of magnitude higher than some estimates and the investigators should have looked long and hard at their data to determine why. Instead in the discussion section they mostly talk about how their projected numbers are too low. This needs to be addressed as a reply in the journal, it is not good science, not nessicarily bad science, but not good either.
I think understand the statictical probabilities in projecting the spread of contagious diseases, or casualities from radioactive fallout. But I fail to see how modeling is relevant to estimating a body count from war, let alone ‘excess’ deaths.
If it is accepted that Saddamn ‘violently’ killed 500K over time in his regime, what of the increased mortality rates from poor hygiene, inadequate health care along with environmental pollution leaching into the average Iraqi dietary & living space? Are those considered natural or excess deaths. Or will we blame the US for imposing sanctions that led to coarsened living conditions. I’d like to see the Lancet address that, but I won’t hold my breath.
The problem with modeling war deaths is tenous on many levels.
1) Planning for war, we were solemnly caustioned that given SoDamn Insane’s WMD capabilities, we could experience up to 200K Coalition fatalities, hence the excess number of body bags. I’ll give you 3 guesses on who led the charge on this front in demanding that we prepare for planeloads upon planeloads of returning dead.
2) It was also projected that there would be massive outflux of refugees pouting out of Baghdad and other hotspots into primarily Jordan and to a lesser extent into Syria among other neighboring countries. Along with the flood of humanity, diseases would run rampant and the corresponding death toll. Where are we on that scientific metric?
3) Anbar region has been a hotspot for clashes. So if the Lancet polled 1000 people there and gets high counts of ‘excess’ deaths which more or less correspond to official body counts of clashes. How does that correspond to milder regions? Apparently the Lancet has no qualms about applying the same ratio to those regions. Somehow, I detect a dissonance in applying that ratio to the Kurdish region that is experience high economic growth rates and peace relative or better than most US metropolitian areas. That the Kurds make up what % of the Iraqi population?
4) The anti-American portion of Iraqis are well aware of the media value in ‘exposing’ American collateral damages/deaths. Why aren’t we baraged daily with 40+ deaths at the hands of Americans? Haditha, the cold-blooded rape & murder of a girl and family by soldiers, the missile attack on the ‘wedding’ party and others come to mind. Occam would say what we get reported are pretty much it, hyperbole (the wedding party was all family & friends, not a group of terrorists having a get together) notwithstanding.
5) That the media can’t get out and report all the deaths is pure BS. After all every so often we are treated to live documentations of violent deaths by Reuters, AP and other news outlets where the media just happened to be filming in the immediate vicinity. What, the media can witness live executions by the insurgents, but going out and reporting on actual body counts is too dangerous. Yeah, right and I’ve got a bridge for sale.
6) So if the contention is that bodycount.org’s tally is too low, even a 25% under-reporting is still less than 100K, which by the way is the bottom-range of the Lancet error margin. This is an interesting numbers game. I wonder if I can recreate the formula:
1) Take a hot region, get an average death rate, apply it to the 28 million Iraqis. Call it the high end.
2) Take the bodycount.org number, juice it up by a ’scientific’ error margin. Call it the low end.
3) Find the median value. Wait for the ’scientifically’ desginated date of maximum impact before election day to ’soberly & solemnly’ issue a press release.
Yep, that’s the ticket
here’s the other side of the story:
http://www.cjrdaily.org/politics/debating_the_body_count_in_ira.php