The Democrats hit General Petraeus and Ambassador Crocker with the results of an ABC/BBC poll of Iraqi citizens during the two days of testimony. Barbara Boxer was so immersed in the poll results that she couldn't even muster up a question for General Petraeus. Since the poll results were not released until Monday September 10, 2007, it left little time for an indepth look at the poll, the sampling size, the surveyors and the results from all the questions.
First of all - the sample size. The number of Iraqis questioned for the poll was approximately 2100 people. 2100 people in a country with an estimated population of 27,499,638 according to the CIA Factbook. That means the poll results were from 1/1000 of the population. How can a sample size that small even be considered partially representative of the population?
Next - the polling companies. The polls were managed by D3 Systems of Vienna, VA and KA Research Ltd of Istanbul. Both polling companies work with an all-Iraqi staff from in-country. I have no way of knowing the ulterior motives of the Iraqi staff members but recalling the number of "in-country" media stringers who have been involved with insurgents makes one wonder. Both companies have performed similar surveys for ABC and BBC prior to this one.
Last but not least the results that didn't make it into the media. For argument's sake lets ignore the small sample size and just look at the results. Several questions had the choices of "much better", "somewhat better", "about the same", "somewhat worse" and "much worse" but "stayed the same" was ignored completely. Keep in mind that this poll was done in August 2007. The surge did not officially start until June 2007 when all the troops were in place. Those responses in between make quite a bit of difference when looking at the results.
Take Question 2 of the poll: "What is your expectation for how things overall in your life will be in a year from now?" 61% answered either "much better", "somewhat better" or "about the same". 39% answered "somewhat worse" or "much worse". The results don't sound as gasp-inspiring as "Only 29% think things will get better in the next year" as reported by the BBC.
Question 10 asked "In the past six months has the security situation in this neighbourhood/village become better, become worse or stayed the same?" 69% of those responding said security had become better or stayed about the same. 31% said it was worse. Since we were only 2 months into the surge, those results are not as dire as reported by the Democrats in Congress.
One question that got a lot of media exposure was Question 21 "How long do you think US and other Coalition forces should remain in Iraq?" 47% answered "leave now". BUT 53% answered that US and Coalition forces should remain until security is restored or the Iraqi government is stronger.
And about withdrawal before civil order is fully restored? 46% believe that withdrawal will make it more likely for Iraq to become a base of operations for international terrorists. 46% also believe that Iran will take control of parts of Iraq if there is an early withdrawal.
Results that painted Iraqis as on the road to reconciliation were completely ignored by the media. Question 13 asked "Which of the following structures do you believe Iraq should have in the future?" 62% of Iraqis responded "one unified Iraq with a central government in Baghdad.
"Confidence in the Iraqi Army - 66% answered "great deal of confidence" or "quite a lot of confidence"
Confidence in the Iraqi Police - 69% answered "great deal of confidence" or "quite a lot of confidence."
Willingness of members of the National Assembly of Iraq to make necessary compromises to bring peace and security to the country? - 50%
Separation of people on sectarian lines - 98% responded it was a "bad thing"
The Iraqis' responses to questions about Al Qaeda in Iraq and Iran were conveniently left out of the media reporting. 21% of Iraqis surveyed blame Al Qaeda and foreign jihads for most of the violence (up from 18% in 2/07). While 19% blamed the US or Coalition forces for most of the violence, that number is down from 31% in Feb 2007.
Other questions about Al Qaeda included:
attacks on US and coalition forces - 51% unacceptable
attacks on Iraqi civilians - 100% unacceptable
attempts to gain control in local areas - 98% unacceptable
Countries actively engaged in encouraging sectarian violence within Iraq:
Syria - 66%
Iran - 79%
Saudi Arabia - 65%
So while the small sample size did reveal some negative results for the US and Coalition Forces, there were many more positive results that never made it into the media. These positive results actually back up the maligned testimony of Iraq's reconciliation progress.
Once again the media reports the poll results that make America look bad and ignore the results that point to progress. Typical.
Editor at Large
Comments Policy
Sample Size
September 12, 2007 - 22:55 ET by RevolvrYour criticism of the sample size suggests MRC needs some additional statistical resources. Wrong tree to bark up. Typically polls use a sample size of about 1100 which leads to a +/-3% margin of error - ASSUMING the sample is random and representative of the population. This survey has a margin of error of close to 2%.
It is the assumption of being a representative sample that should be questioned, not the size.
_____________________________________________
Igitur qui desiderat pacem, praeparet bellum
Thank you for clarifying
September 12, 2007 - 23:01 ET by Robin BoydThank you for clarifying that for me. I'll edit the blog to correct it.
Revolvr
September 12, 2007 - 23:16 ET by botgASSUMING the sample is random and representative of the population i am sure you would agree that the smaller the sample the more likely it is to not be representative or random. Thus sample size is a legitimate concern. Were any Kurds sampled? Was this survey taken in Falluja? Bagdad? Bosra? Tekrit?
I don't believe there is enough baseline information to predict this to a 2% accuracy without multiple qualifiers (such as this is accurate for inner city dwellers)
Support our Troops
Oh, those sample statistics
September 12, 2007 - 23:57 ET by hungarianfalconI'm no expert on polling statistics but I do have a Ph.D. in analytical chemistry where we are expected to have a fundamental understanding of sampling criteria, standard deviations, signal noise, etc.
Based on all that:
1. From every poll I've ever seen, their "margins of error" appear to be derived explicitly from poisson statistics where the relative standard deviation (margin of error or RSD) is defined as 1 over root N where N is the sample size. For example, a sample size of 400 would yield an RSD of 1/20 or 5%. One other note- this is the absolute fundamental best result that one can exact if polls are truly Poisson in nature. If there are other sources of variation (and there almost always are), the RSD values will be worse. Any deviations that are better than our buddy Poisson are anomolies when it comes to random counts.
2. These MoEs or RSDs have ABSOLUTELY NOTHING TO DO WITH ACCURACY and everything to do with reproducibility or precision. All polls will have their biases or inaccuracies. What the MoEs / RSDs capture is how tight (precise or reproducible) the distribution of results would be if the same poll was conducted several times in a row under THE EXACT SAME SITUATION (sample population, wording of the question(s), outside factors like pertinent events, etc.). Probably 90+% of the general population does not understand this point and the pollsters knowingly let them read things into these polls that are false. That's why you can have two independent groups get completely different results and neither of them are really wrong. It isn't automatically a case of INTENTIONAL bias (we all have our subconscious biases), although I think some polls are dubious at best at times.
3. For the lay people- precision is your ability to hit the same spot on the target, whether it be the bullseye or a section on the perimeter of the target, time and time again. Accuracy refers to the average location of your shots on that overall target. One can have great precision with lousy accuracy, vice versa, and everything in between.
There will be a test on this at 8AM tomorrow morning.....
hungarian
September 13, 2007 - 00:19 ET by botgas i understand you
precision=reliability (is it reproducable)
accuracy=validity (does it measure what it is supossed to)
it's amazing that they use the 1 over root(n) to fool people. They would not know a "T" test or chi square analysis if it bit them.
Support our Troops
"reliability" troubles me a
September 13, 2007 - 01:14 ET by hungarianfalcon"reliability" troubles me a bit. I wouldn't use it. In a nut shell-
precision w/no accuray = adding 2+3 10 times and getting 8 each of the 10 times. That's precise w/o accuracy.
accuracy w/no precision = adding 2+3 10 times and getting the following- 4, 6, 3, 7, 2, 8, 1, 9, 5, and 5 for an average of 5. That's perfect in terms of accuracy (2+3 = 5 after all) but the precision is in the toilet.
I don't look at the use of 1/root N as trying to fool people. There probably is some legitimate usage for it (if polling is considered Poisson in nature then it is definitely legit, although I can't imagine how the Poisson equations are valid for 3-way, 4-way, etc. polling like some of the polls we see before the party candidates are chosen or when Perot was around. A count is a yes/no concept not a yes/no/maybe concept). I can't say for sure that the 1/root N equation is the equation in use w/polls but in every poll I've EVER paid attention to, the stated margin of error ALWAYS fits the expected 1/root N relationship so there probably is some established standard among those learned in the polling sciences.
P.S. I did stay at a Holiday Inn Express last night...
What qualifies a
September 13, 2007 - 00:11 ET by Del DolemonteWhat defines a representative "sample", though?
As for the poll sample sizes, I think that you're right, which raises an interesting question-why did they use twice as many people in their polling sample than we use here in the US?
Could be that some readers would see the story on the poll and see how many people said what they said, and concluded that the poll was an accurate barometer of current public opinion.
Yes, my master ...
September 12, 2007 - 23:45 ET by KC MulvilleUsing a media poll reflects one crucial distinction between liberals and conservatives. Liberals want the media to control our national conversation. They want the media to be authorities. Note that Democrats tried to rebut Petraeus with media reports. Democrats admitted openly that they trust the media, not in the military. (Does it dawn on them that the country is exactly the opposite?) The senators threw the media reports around as if they were the Oxford dictionary. The Democrats’ obedience to the media was never more obvious than during this testimony.
Liberals want the media to declare the winners and losers. Is it any wonder that the networks claim to call the winners and losers of the election? Of course, in reality they have no official status. Each state election commission calls it. That’s why Florida was such a mess in 2000, and Ohio in 2004. The Democrats remain convinced that those results must be wrong, because they conflict with what the media expected and wanted.
The founding fathers had a different idea. Our national conversation was supposed to take place in Congress and in each state legislature. Say what you will about the clowns and buffoons that fill our legislatures, at least they’re elected. Journalists aren't elected. They're following their own sense of truth. No one (anymore) falls for the myth that journalists are impartial. Journalists don't go into the media simply to “inform” fellow citizens; that reduces journalists to mere observers. They went in to become players. They don't want to report reality; they want to control it. They want to control what fellow citizens think about. They highlight whatever advances that; they block and distort whatever doesn't.
One scary thought: doesn’t it scare you that the Democrats would have paid much more attention to Jon Stewart’s testimony than Petraeus’?
Search your feelings. You know it to be true.
Barbara bozer whined like a stuck pig
September 12, 2007 - 23:53 ET by SportPoliticsnote she whined specifically about 80% of the shia and 90% of the sunni think their children will have a worse life than they did.
I hate to point it out, but 100% of the democrat party tells us that is true for their and all of the USA children. So, the shai and sunni actually have a brighter outlook than the democrats.
Iraq poll
September 13, 2007 - 14:17 ET by buckhiloJudging by his comments on the sample size, it's apparent to me Boyd doesn't understand how random sampling works.
The margin of error for the Iraq poll -- if based on a truly random sample -- is 2 percent. It takes slightly fewer than 400 recipients for a 5 percent margin of error. The mathematical formula for the margin of error is 1.96 times the square root of .25 divided by the random sample size.
Rather than the size of the sample, I would be interested to see how the sample was chosen.
buckhilo
September 13, 2007 - 16:01 ET by Robin BoydFirst of all it's not "his comments", I am female. I admit that I am far from a statistical analyst - I'm just a concerned citizen. Sometimes we get so bogged down in minutia that we fail to see the big picture. Sometimes we get so caught up in being "experts" that we forget to look at things like a regular "Joe" who listens to the news with one ear while eating dinner.
When someone tells you that "Iraqis want the US out and the surge has failed" based on a poll of 2100 people in a country of 26 million I say that size is significant. Most people don't give two hoots about how a sample is chosen or the margin of error. They hear "the Iraqis" and "poll" and base their opinions on that.
Believe me I will never post anything here about polls again. I'll let the experts handle it.