I have never much liked the trite phrase 'there's lies, damn lies and statistics'. Having taken two stats courses, and taught elementary stats for residents twice, I have great respect for statistics. The problem comes when the wrong model is chosen, and more importantly, when data are deliberately skewed.
Statistics is nothing more than mathematical modeling. While the number crunching can be tedious, conceptually it is nothing more than looking at data using well defined shapes and seeing if the data behave that way. Of course, the right shapes must be used, and the more concepts and shapes the statistician knows, the better the data can be described. The data itself must have been obtained rightly, must be a right image of the population being studied.
One of the cardinal sins of statistical analysis is to generalize data outside the subset from which it was obtained. This is obvious in some cases- if life expectancy data are obtained from dogs, we can not apply them to humans. Sometimes it is less obvious, such as applying heart attack survival data from men to women.
Unfortunately, sometimes data are generalized outside a subset for deliberate manipulative purposes. It is this kind of behavior that makes for disresepct for statistics. I got interested in looking at how political polling is done when I kept seeing wildly different poll data for the same election, in this case the 2004 presidential election. I went to the polling organizations' web sites, and reviewed how they did their poll, and what groups they looked at. The results were very revealing, and have influenced my view of polls ever since.
First, how questions are asked makes a big difference. Anyone who has received donation requests from the political parties knows this. Also, the order in which questions are asked makes a surprising difference. Finally, who is asked, where they are found, and how they are interviewed all make a difference.
Interviews may be done by mail, email, phone or in person. While it would seem at first to be best to do an interview in person, there are man problems with this. The interviewer may influence the results by choosing the subject. The interviewer's body language, closeness of body, gestures and facial expression may all influence the result. Mail and email have problems with response bias, and in the case of email and internet use, income, education and sex bias. Phone is generally the best.
Where people are found and how they are chosen have a profound impact on results. People may recall the quite embarrassing Quinnipiac exit poll of 2004 that showed Kerry beating Bush by 5-6%. Right. Dewey beats Truman except in reverse. In doing political polls there are a couple of important parameters.
The poll mix must reflect the demographics of the community by race, age, ethnicity and political party. This is not nearly as easy as it sounds when using a sample of less than about 10,000. To reproduce the diversity of teh US in a sample of 1000 is quite a feat, and requires great care.
Also, the sample is quite different if adults, registered voters or likely voters are sampled. Moving across this spectrum the results trend increasingly republican. Likely voters are more often white, older educated and employed, all trtaits which favor republicans, as opposed to the general adult population. If you are interested in the opinion spread at a football stadium, it is reasonable to poll adults, but if you want to know how people will vote, it is best to poll likely voters.
There are two firms whose results are broadly accepted and trusted- Rasmussen Reports and Gallup. Everyone else is using biased samples or simply is calling adults. It is easy to look at the data and even the questions at the various polls' web sites.
Here is where today's entertainment came from. Every Rasmussen poll for months gives the generic congressional vote to the republicans, and by margins at least as high as in 1994. This, and other issues, portends a republican gain of 25-40 seats, a number you may have read or seen.
Today, CNN released the results of their poll, indicating a generic congressional vote advantage to the democrats of 4% (Everyone else has it 5-9% republican favorable). I thought this was an oddity, so I looked at why. The answer is they polled adults. Part of their poll was of registered voters, but none was of likely voters.
Not very helpful. It is important to know who is being sampled and how before accepting poll data. This is more true for the political parties and candidates than it is for individuals. One of the marks of folly is selectively accepting only that data which proves your own case. This is an error to which we are all prone, but which seems to affect preferentially politcal parties.
Every bit of data from likely and registered voters for months points to a republican win in November. The democrats woudl do well not to ignore this, but to seriously address why it might, be.
Ted
Statistics is nothing more than mathematical modeling. While the number crunching can be tedious, conceptually it is nothing more than looking at data using well defined shapes and seeing if the data behave that way. Of course, the right shapes must be used, and the more concepts and shapes the statistician knows, the better the data can be described. The data itself must have been obtained rightly, must be a right image of the population being studied.
One of the cardinal sins of statistical analysis is to generalize data outside the subset from which it was obtained. This is obvious in some cases- if life expectancy data are obtained from dogs, we can not apply them to humans. Sometimes it is less obvious, such as applying heart attack survival data from men to women.
Unfortunately, sometimes data are generalized outside a subset for deliberate manipulative purposes. It is this kind of behavior that makes for disresepct for statistics. I got interested in looking at how political polling is done when I kept seeing wildly different poll data for the same election, in this case the 2004 presidential election. I went to the polling organizations' web sites, and reviewed how they did their poll, and what groups they looked at. The results were very revealing, and have influenced my view of polls ever since.
First, how questions are asked makes a big difference. Anyone who has received donation requests from the political parties knows this. Also, the order in which questions are asked makes a surprising difference. Finally, who is asked, where they are found, and how they are interviewed all make a difference.
Interviews may be done by mail, email, phone or in person. While it would seem at first to be best to do an interview in person, there are man problems with this. The interviewer may influence the results by choosing the subject. The interviewer's body language, closeness of body, gestures and facial expression may all influence the result. Mail and email have problems with response bias, and in the case of email and internet use, income, education and sex bias. Phone is generally the best.
Where people are found and how they are chosen have a profound impact on results. People may recall the quite embarrassing Quinnipiac exit poll of 2004 that showed Kerry beating Bush by 5-6%. Right. Dewey beats Truman except in reverse. In doing political polls there are a couple of important parameters.
The poll mix must reflect the demographics of the community by race, age, ethnicity and political party. This is not nearly as easy as it sounds when using a sample of less than about 10,000. To reproduce the diversity of teh US in a sample of 1000 is quite a feat, and requires great care.
Also, the sample is quite different if adults, registered voters or likely voters are sampled. Moving across this spectrum the results trend increasingly republican. Likely voters are more often white, older educated and employed, all trtaits which favor republicans, as opposed to the general adult population. If you are interested in the opinion spread at a football stadium, it is reasonable to poll adults, but if you want to know how people will vote, it is best to poll likely voters.
There are two firms whose results are broadly accepted and trusted- Rasmussen Reports and Gallup. Everyone else is using biased samples or simply is calling adults. It is easy to look at the data and even the questions at the various polls' web sites.
Here is where today's entertainment came from. Every Rasmussen poll for months gives the generic congressional vote to the republicans, and by margins at least as high as in 1994. This, and other issues, portends a republican gain of 25-40 seats, a number you may have read or seen.
Today, CNN released the results of their poll, indicating a generic congressional vote advantage to the democrats of 4% (Everyone else has it 5-9% republican favorable). I thought this was an oddity, so I looked at why. The answer is they polled adults. Part of their poll was of registered voters, but none was of likely voters.
Not very helpful. It is important to know who is being sampled and how before accepting poll data. This is more true for the political parties and candidates than it is for individuals. One of the marks of folly is selectively accepting only that data which proves your own case. This is an error to which we are all prone, but which seems to affect preferentially politcal parties.
Every bit of data from likely and registered voters for months points to a republican win in November. The democrats woudl do well not to ignore this, but to seriously address why it might, be.
Ted
Comments
Post a Comment