Associated Press lead reporter Liz Sidoti, other contributors (AP Director of Surveys Trevor Tompson, AP News Survey Specialist Dennis Junius and AP writer Alan Fram), and the wire service's supposedly vaunted editors apparently don't understand what a polling margin of error is.
In a Wednesday story I found in four different places (CBS News, AP-Google, Breitbart, Yahoo! News), Sidoti et al let a paragraph stand claiming that a 3.5% margin of error in the poll results they were reporting meant that the real results could vary by as many as 14 points.
Here are the key paragraphs found in each story (bold is mine):
The presidential race tightened after the final debate, with John McCain gaining among whites and people earning less than $50,000, according to an Associated Press-GfK poll that shows McCain and Barack Obama essentially running even among likely voters in the election homestretch.
The poll, which found Obama at 44 percent and McCain at 43 percent, supports what some Republicans and Democrats privately have said in recent days: that the race narrowed after the third debate as GOP-leaning voters drifted home to their party and McCain's "Joe the plumber" analogy struck a chord.
..... Polls are snapshots of highly fluid campaigns. In this case, there is a margin of error of plus or minus 3.5 percentage points; that means Obama could be ahead by as many as 8 points or down by as many as 6. There are many reasons why polls differ, including methods of estimating likely voters and the wording of questions.
Uh, no. A 3.5% margin of error means Obama could be ahead by as many as 4.5 points or down by as many as 2.5. That's a seven-point spread.
Zheesh. If I didn't know better, I would be thinking, with their apparent failure to grasp basic math, that Sidoti et al are auditioning to join Martin Crutsinger and Jeannine Aversa as AP business reporters.
Cross-posted at BizzyBlog.com.
UPDATE: Commenters are saying AP is correct in its interpretation.
I don't think so, because if the commenters are right, newspapers would routinely write up, say, a 6-point lead by McCain or Obama as being "within the 3.5-point margin of error." That is, the one trailing could be 3.5 points higher, and the one leading could be 3.5 points lower, meaning that the person reported as leading could conceivably be trailing. But I don't think I've ever seen it reported that way, because, as I understand it, the margin of error is 3.5% for the entire poll, not 3.5% for each poll component.
—Tom Blumer is president of a training and development company in Mason, Ohio, and is a contributing editor to NewsBusters




















Editor at Large
Comments Policy
Now, Tom, normally I'd
October 23, 2008 - 14:19 ET by HockeyKidNow, Tom, normally I'd agree with your assessment. But we have to remember that ACORN is very active this cycle. Therefore, the observed sample size itself is also subject to MOE, causing the potential results reported by Sidoti and friends.
Either that, or journalists can't count.
I'm not sure you quite have it right either.
October 23, 2008 - 14:49 ET by mcgIt’s not hard to see how they arrived at those numbers. A 3.5-point error means that Obama could range anywhere between 40.5-47.5, while McCain could range from 39.5-46.5. Taking the extremes on both ends puts Obama between 6 points down and 8 points up.Now that’s not right; the margin of error on the difference doesn’t double. But is it the same as the MOE? According to this guide, for instance, it’s not: the MOE of the difference is larger than the MOE of the individual numbers. They recommend a multipler of 1.7, which means the difference is about 6 points. Thus Obama could be up by 7 or down by 5.
Standard deviation?
October 23, 2008 - 14:57 ET by Prester JohnIf the +/- 3.5% MOE is the same as one standard deviation (which would cover about 68% of possibilities), then two standard deviations would equal +/- 7% and would cover about 95% of possiblities which is a typical range statisticians use to define the likelihood something might/might not happen.
+/- 2 standard deviations equals a 14 point range.
Yes, but that's +/2 SD on an individual number.
October 23, 2008 - 15:12 ET by mcgWhat I'm saying is this: the standard deviation on the difference is not the same as the standard deviation on the individual numbers themselves; nor is it twice as much. Again, as the item I linked to suggests (though it could be wrong), they claim the standard deviation of the difference is about 1.7 times larger.
Actually, I think their description is accurate
October 23, 2008 - 14:50 ET by buzzbearWith any survey that's just a sample of the total population (and I've worked on quite a few in my profession), you get a result, and that result has a +/- factor attached to it. Sample size plays a role in this (though it's not the only factor). But bottom line, if we accept the stated margin of error, it suggests that the "true" percent, in the total population, supporting Obama lies between 40.5% - 47.5%, while the corresponding "true" total pop figure for McCain would be between 39.5% - 46.5% .
Because those two ranges overlap, it's much more accepted to just declare the two "statistically even". Stat guys often refer to "confidence intervals", and in this case, one would NOT be confident that there's a true difference between the candidates. In other words, MOE is more of a nuisance variable for scientists, not a launching pad for speculation about what the "true" numbers could be.
So, the news orgs are being tacky, but not outright wrong in their math.
More important for this election in particular is how the different pollsters define "likely voters". I won't bore readers here - look it up yourself and you'll see why it's so significant this time around.
Well actually
October 23, 2008 - 14:57 ET by JackRBThe reporter is correct in the assignment of the MOE. A 3.5% margin of error means that both Obama's 44% and McCain's 43% can move either way by 3.5% at the same time.
That means it could be McCain 47.5%, Obama 41.5% (+6 McCain) or Obama 48.5%, McCain 40.5% (+8 Obama). That is a 14 point swing.
Massage Them Numbers!
October 23, 2008 - 15:15 ET by stratman14 point spread?
Based on your and AP's conclusions, this poll must have a really crappy confidence level.
It seems like the sample size for a MOE of 3.5% was around 800 instead of the 1100 they toss out. Extraneous data. Was someone counting the number of Obama votes and stopped when the magic 800 number was reached?
Interesting article on polling and elections that some (Tom Blumer and ??) might enjoy: http://terrytao.wordpress.com/2008/10/10/small-samples-and-the-margin-of-error/.
Statistics make my gasket blow.
Tom, Seems AP is using
October 23, 2008 - 15:01 ET by Clear thinkerTom,
Seems AP is using that new Math.
Peek Into Liberal Minds - If You Dare
Making Fun of AGW http://giovanniworld.wordpress.com/
Asinine Palaver
October 23, 2008 - 15:03 ET by mattmAsinine Palaver
These folks
October 23, 2008 - 16:35 ET by NorthCoasterare challenged by basic statistics. It's no wonder that polls get mis-interpreted. They do the same thing when reporting on Climate Change, but that's another story.
Hey Al Gore, have you heard about the growing glaciers in Alaska and the recent 100year plus low record temps.? I guess Gorebull Warming is postponed for a few more years. You can sell the future beachfront property that you bought in the Appalachians because it will continue to be the foothills into the forseeable future. (sarc off)
I must be ignorant then. I
October 23, 2008 - 16:58 ET by guefyI must be ignorant then. I would have thought it was a 14 point spread too. Guess I learn sumpin' new every day.