With the outcome of the recent BC election surprising the largest polling firms in the country, a debate has emerged over which method—web panel surveys or telephone surveys—most accurately reflects public opinion. Notwithstanding the fact that several other factors also led to the discrepancy between polling results and actual election results (e.g., low voter turnout, ability of the BC Liberals to get their vote, last minutes shifts in voter preferences, etc.), the question of whether online polls accurately reflect public opinion is a good question. The BC Liberal party, using a telephone methodology, reported that throughout the campaign their numbers were showing a much closer race than the media pollsters; and on the eve of the election, the BC Liberal polling was showing a tied race while the media pollsters were showing 8-9 percentage point gaps between the BC Liberals and NDP. Understandably, companies whose business models rely primarily on web panel polling are fighting back in an attempt to preserve their credibility. Unfortunately, a few myths are being spun in the process.
One of the main myths is that web surveys can access a broader segment of the population than telephone surveys. In fact, web samples are drawn from opt-in or pre-recruited panels, a database covering a minor population (5% at most) instead of from a database covering the majority of the population. So essentially, this 5% is reflecting the opinions of the other 95% of the population.
As telephone surveys can include cell phone numbers in the sample, almost 100% of the population is included in the sample frame. Note that market research firms are not prohibited from contacting respondents on cell phones. And since many people have unlimited or at least high limits on their plans, if they are interested in providing their opinions, they do so. Including cell lines also is an effective way to reach younger respondents, who are a challenge to reach even for online surveys.
Another factor to consider in the phone vs web argument is ethnic reach. It is well known in the market research community that ethnic communities, particularly those of Asian origin, tend to be under represented on web panels (another reason why the media pollsters using online methodologies underestimated the Liberal vote). Participation levels are simply better on telephone surveys, particularly if language translation is provided.
Both web and telephone surveys are useful tools in opinion research and each has its advantages and disadvantages. But telephone surveys are not an ‘old fashioned’ tool as some claim and are still used by even the most prestigious research providers such as Pew Research Centre in Washington, DC.
Mustel Group offers both methodologies with approximately half of our studies conducted by telephone and half on-line (using our randomly recruited “Giving Opinions” panel, rather than an opt-in panel). We recommend the methodology which is most appropriate for the study objectives. Online is just another tool in our toolbox. Other methodologies will surely emerge with evolving technologies.
Evi Mustel
Principal, Mustel Group