The criticism against KLR VU research’s poll on Morgentaler’s reception of the Order of Canada has been fierce.
But uninformed.
(For background about the poll, see this article Massive New Poll: 56% of Canadians Oppose Morgentaler Order of Canada).
I would like to address the several issues raised by critics of this poll. However, I do not wish to write overly long blog posts. Therefore, I will deal with one issue per post.
As I am nursing a newborn baby and chronically sleep-deprived—and the kids like using the computer now too—it can take a while to respond.
The issue I would like to address blog post is the survey technique: the use of machine to conduct the survey. In the research business it’s known as Interactive Voice Reponse (IVR).
IVR can refer to a variety of methods that may or may not include a human interviewer (and indeed, is used for non-survey purposes such as voicemail), but for the purposes of this blogpost, IVR will refer exclusively to a completely automated phone survey. In the KLR VU Research poll, the sample was weighted by population density, and such geography-based weighting is typical with this method.
IVR is a relatively new method. It first came to the fore in US political polling around the turn of the millennium.
At first, IVR wasn’t taken very seriously. This article in Slate magazine by David Kenner and William Saletan said of those who used automated polling during the 2004 presidential elections:
Clearly, the automated pollsters are onto something, and the human pollsters who have fallen behind will have to figure out how to beat it—or join it.
Another IVR booster is Democratic pollster Mark Blumenthal, who blogged about this topic at Pollster.com in 2006.
As the time, this form of polling was dismissed by “insiders”. However, their criticisms are not based on evidence:
Skepticism about IVR polling based on theoretical concerns is certainly widespread in the survey research establishment, but one can look long and hard for hard evidence of the lack of reliability of IVR, or even Internet polling, without success.
He says that over 55% of the polls archived at pollster.com were conducted by IVR.
The American Association of Public Opinion Research (AAPOR) takes this method seriously enough to have let Blumenthal give a conference on this subject in 2007 at their annual conference (scroll way down).
These survey methods have made opinion polling accessible to many small election campaigns, and has become popular in states where polling large numbers of people would be cost-prohibitive.
In the US there are two major companies that use IVR—Rasmussen and SurveyUSA.
SurveyUSA specializes in quick polls with a huge number of respondents:
Minutes after the first debate in the 2004 Presidential Election …
ABC News completed one poll of 531 debate watchers.
CNN completed one poll of 615 debate watchers.
CBS News completed one poll of 655 debate watchers.
NBC News did nothing.
SurveyUSA completed 35 separate polls in 35 separate geographies, of 14,872 debate watchers.
Here’s a word about SurveyUSA’s methodology.
SurveyUSA’s methodology is so reliable that in 2005 50 American television stations regularly relied on them to track voter intentions on various ballot issues.
I can hear the objections: But we don’t know who picks up the phone!.
That’s true. However, the bottom line is: it works. The way it works is still something of a mystery as research into this method is still in its infancy. But, it’s worked well enough to produce results that could predict elections. If it wasn’t reliable enough to predict the results of elections, campaigns would not spend money on this method.
As this academic paper says:
SurveyUSA and Rasmussen Reports are two of the most established polling firms using IVR. They have faced much criticism from a theoretical perspective but both have yielded strong results that have been at least comparable to established live telephone interviewing firms like Mason-Dixon.
(…)
According to Bloom and Pearson, SurveyUSA performed at “roughly the same level as other nonpartisan polling organizations in 2002” despite its high criticism (2003). By other measures, SurveyUSA performed slightly better than the other polling organizations (2003). IVR’s success was supported in subsequent papers. In Bloom’s second paper, “consistent with the previous paper, is that despite criticism over the use of automated interviewers, SurveyUSA and Rasmussen Reports provide accurate and reliable data, consistent with other polling organizations” (Bloom, 2005). In the two most prominent firms using IVR, IVR has statistical proof of its reliability in the 2002 Senate and 2004 Presidential elections.
But how can it work? We don’t have a lot of research (or at least I don’t have easy access to it) but we can speculate. One reason it works is that conventional telephone surveys aren’t perfect, and IVR surveys can compensate for those imperfections. For instance, the method of selecting the adult with the most recent birthday—to make sure that men and women are chosen—is not perfectly random. The human interviewer can lead respondents, and respondents may feel the need to give socially desirable answers.
In fact, one of the greatest strengths of the IVR method is that it can provide the anonymity necessary to let people truly speak their minds on sensitive and controversial issues. This may in fact explain the discrepancy between the recent polls from Ipsos-Reid and Angus Reid, and the KLR VU one. It is perfectly plausible that on a sensitive issue like abortion, respondents gave what they considered to be a more politically correct answer to a human interviewer, and a less politically correct one to an anonymous machine.
Is IVR as good as traditional telephone surveys in gauging public opinion?
The tendency is still to rely on the traditional human interviewer method as the gold standard. That being said, there are benefits and drawbacks to any research method. IVR has shown its ability to predict support in elections and ballot votes in the states. It is therefore a reliable method, especially for simple surveys. The more simple the survey, the more reliable the information. The KLR VU Morgentaler survey was extremely simple.
Okay, if it works so well, you might ask, why aren’t more companies using it?
As a matter of fact, this interview of Ken Peterson, Vice-President of Ipsos-Loyalty, shows that the big players are adopting this method:
Ken: In recent years, we have seen an increased use of IVR systems for survey research. IVR has always met timeline and budget requirements, but it has often been questioned as a reliable data source. Recent developments in sampling processes and inbound participation, such as point-of-sale systems that generate receipts that refer customers to a toll-free number, have caused us to reconsider IVR.There is greater appreciation for outbound calling too, whereby the system automatically calls targeted customers. Advances in IVR technology make it so three basic needs are met: timing, price – which has always been a benefit using IVR – and now data quality.
The beauty of IVR is that you do not need a lot of employees to do the survey. So say, a self-employed individual, could conceivably conduct a valid poll, with some hired help.
Which is what KLR VU Research has done.
Follow-ups to this post will be published as necessary and as my time and energy allows.
I'm exhausted now. I'm going to feed my baby. Again.
For more social conservative news check out BigBlueWave.ca
