The Key Factorof Opinion Poll Quality

Shaw Tao, Ph.D.Director, Data Services

Future of Public Opinion Poll• “I mean clearly something's wrong. I am

concerned about the impact on thecredibility of the profession.”--Bruce Cameron, 2012

14/05/2015The Key Factor of Opinion Poll Quality2

Jeff McArthur, Radio AM 640

Pollster Bruce Cameron, on CBC

• “Alongside Tim Hudak who announced he’dbe stepping down as leader…should be a lotof pollsters stepping aside as well…”--Jeff McArthur, 2014

• “A Dark Election Day for U.K.’s Pollsters”--Charles Forelle, May 8, 2015 Charles Forelle,Editor, The WSJ

• “If you really want to know a likely electionresult, ignore the polls and look at the bettingmarkets.”--John Fund, May 9, 2015

John Fund, columnist, National Review

Recent Polling Results 1

14/05/2015The Key Factor of Opinion Poll Quality3

British Columbia Election 2013

GRN LIB NDP Firm MethodError of

Top 3ElectionResult 8.1 44.1 39.7

5/12/2013 9.0 37.0 45.0 Ipsos Reid Online 13.3

5/12/2013 13.0 34.5 40.5 EKOS IVR 15.3

5/10/2013 9.0 36.0 45.0Angus Reid PublicOpinion Online 14.3

5/09/2013 14.0 31.0 45.0Justason MarketIntelligence Online 24.3

5/08/2013 8.0 41.0 43.0 Forum Research IVR 6.56

14/05/2015The Key Factor of Opinion Poll Quality4

Quebec Election 2014

CAQ LIB PQ Firm MethodError of

Top 3

Election Result 23.1 41.5 25.4

4/04/2014 25.0 39.0 27.0 Angus Reid Online 6.0

4/03/2014 23.0 44.0 24.0 Forum Research IVR 4.0

4/03/2014 23.0 38.0 29.0 Léger Marketing Online 7.2

4/03/2014 21.1 39.8 27.0 EKOS IVR 5.3

4/02/2014 19.0 37.0 28.0 Ipsos Reid Online 11.2

Recent Polling Results 2

14/05/2015The Key Factor of Opinion Poll Quality5

Ontario Election 2014

LIB NDP PC Firm MethodError of

Top 3

Election Result 38.7 23.7 31.2

6/11/2014 35.0 26.0 32.0 Abacus Data Online 6.8

6/11/2014 33.0 30.0 31.0 Ipsos Reid Online 12.2

6/11/2014 41.0 20.0 35.0 Forum Research IVR 9.8

6/10/2014 36.6 21.5 30.2 EKOS IVR 5.3

6/10/2014 36.0 26.0 32.0 Angus Reid Online 5.8

6/08/2014 35.0 24.0 36.0 Oraclepoll ResearchTelephone 8.8

Recent Polling Results 3

14/05/2015The Key Factor of Opinion Poll Quality6

Alberta Election 2015Date NDP PC WR Firm Top 3 Error

5/6/2015 40.6 27.8 24.2Election Results5/4/2015 45 23 23Forum Research 10.45/4/2015 42 23 27Insights West 9.05/3/2015 44.3 22.5 24EKOS 9.25/2/2015 42 21 24Forum Research 8.4

4/29/2015 44 21 26Mainstreet Tech 12.04/29/2015 37 24 26Ipsos Reid 9.24/29/2015 42.2 23.1 21.3EKOS 9.24/28/2015 39 20 27ThinkHQ 12.24/28/2015 38 30 24Leger Marketing 5.04/28/2015 38 24 21Return On Insight 9.64/27/2015 44 20 191ABVote (Google Surveys) 16.4

Recent Polling Results 4

Historical Poll Results

14/05/2015The Key Factor of Opinion Poll Quality7

Election # of Polls Top3 ErrorOntario 2014 10 9.5Ontario 2011 13 6.1Ontario 2007 6 4.0British Columbia 2013 11 15.0British Columbia 2009 3 8.3British Columbia 2005 4 7.4Quebec 2014 6 7.0Quebe 2012 7 7.6Quebe 2008 3 7.3Quebe 2007 4 11.5

Current vs. Past: Errors have DOUBLED

published studies

14/05/2015The Key Factor of Opinion Poll Quality8

• Models used for polling results (such as the“likely” model) ;

• Effectiveness of different polling methods(IVR/telephone vs. online) ;

• Low response rates from polls;• Low turnout of voters;• Voters changed their minds at the last minute.

What Went Wrong?

• Declining trend of listed telephone directories

14/05/2015The Key Factor of Opinion Poll Quality9

Year Households Records Coverage

1996 10,899,427 9,650,412 88.5%

2001 11,546,320 10,963,435 95.0%

2006 12,386,625 11,162,433 90.1%

2011 13,320,615 12,015,463 90.2%

2015 14,206,364 9,626,078 67.8%

Coverage Rate in Canada

Data Source Is The Key

14/05/2015The Key Factor of Opinion Poll Quality10

• Random Digit Dialing (RDD, landline or cellphones)• Address Based Sampling (ABS)• Online panels• Social Media

Remedies

14/05/2015The Key Factor of Opinion Poll Quality11

• Fast turnaround time• Low cost• Coverage of people missed in listed

telephone directories (young householdsand apartment renters)

• Low intrusions to people

Advantages of Online Panels

14/05/2015The Key Factor of Opinion Poll Quality12

• Extremely low coverage rate (below 5%): How toget the responses from other 95% population?

• Low response rate (4% or lower)• Acceptance rate: true response rate =• Distribution Bias

– Internet accessibility (seniors, rural areas)– “Professional respondents”

• Respondent Fatigue

Limitations of Online Panels

4% * x% = y%

14/05/2015The Key Factor of Opinion Poll Quality13

• Canadian Internet Use Survey, 2012 (conducted byStatistics Canada)– In 2012, 83% of Canadian households had access to the

Internet at home, compared with 79% in 2010– 98% households with incomes of $94,000+ had home

Internet access, compared with 58% of households inthe low incomes of $30,000 or less

– While 83% of Canadians aged 16+ used the Internet forpersonal use, the rate of those who are 65+ was only48%

Internet Accessibility

14/05/2015The Key Factor of Opinion Poll Quality14

• Disinclined to continue participating;• Lead to invalid responses (‘‘test wise’’ );• A panel contains 200,000 members:

– 38.4% in Ontario (76,800);– Based on 5% response rate, 20,000 samples required

for 1000 respondents;– The company conducted six polls between May 9 and

June 11 for Ontario Election 2014;– Many panel members could have received 2+ polling

requests in one month’s time.

Respondent FatigueAn overlooked issue

14/05/2015The Key Factor of Opinion Poll Quality15

Alf Landon, Republican Franklin Roosevelt, Democrat

Who Won 1936 PresidentElection?

14/05/2015The Key Factor of Opinion Poll Quality16

• Sample Size: 10 million prospective voters, 2.4million responded.

• Prediction: Landon would get 57% of the voteversus Roosevelt's 43%.

• Actual result: Roosevelt received 62% versus 38%for Landon, and won 46 out of 48 states.

1936 Liberty Digest Poll

14/05/2015The Key Factor of Opinion Poll Quality17

• Sample distribution bias– From telephone directories, club member lists, magazine

subscribers;– They are wealthier than average Americans;– They tended to vote Republican;– “Garbage in, Garbage out.”

• Non-response bias– 2.4 million out of 10 million responded (24%);– People who respond to surveys are different from people

who don't.

History repeats, lessons not learned

What Went Wrong?

14/05/2015The Key Factor of Opinion Poll Quality18

Error of Top 3Online IVR/Tel

Mean 11.2 7.9Median 11.2 6.6Standard Deviation 5.9 3.9Sample Variance 34.8 14.9Kurtosis 2.4 1.7Skewness 1.4 1.3Range 18.5 11.3Minimum 5.8 4.0Maximum 24.3 15.3Count 9 7

Online vs. IVR/Tel•Mean: 41.8% higher•STD: 51.5% higher•Range: 63.7% higher

Recent Poll Result Analysis

14/05/2015The Key Factor of Opinion Poll Quality19

• Selection Bias: If each member of the relevantpopulation does not have an equal chance ofending up in the sample.

• Any self-selected sample is basically worthless as asource of information about the population beyonditself.

• Low response-rate samples invite bias.

Statistics Principles

14/05/2015The Key Factor of Opinion Poll Quality20

• Mainly due to extreme low coverage, onlinepanels can only be considered asconvenience, instead of probability, datasources.

• They can only be used for pilot studies, notfor studies representing whole population.

• Need to educate clients: while timing andcosts are important, reliability is VITAL!

Conclusions

14/05/2015The Key Factor of Opinion Poll Quality21

Wrong!Election Polls

Non-Election PollsMarket Researches• • • • • • • • •

How to regain our credibility?

14/05/2015The Key Factor of Opinion Poll Quality22

For further discussion: Questions?

Thank you!Detailed analyses and references are available at:http://www.marketingresearch.org/article/key-cause-polling-quality-deterioration

14/05/2015The Key Factor of Opinion Poll Quality23

• Pressures from clients is the main driving factor– Timing (“Clients want results yesterday.”)– Costs

Why Online Method in Favor?