the death of polling?
TRANSCRIPT
The Death of Polling Version 1 Public 1
Death of
Polling?
The
#ipsosmorilive
The Death of Polling Version 1 Public 2
Ben Page, Ipsos MORI
Agenda
Julia Clark, Ipsos USA
David Ahlin, Ipsos Sweden
Nando Pagnoncelli, Ipsos Italy
Darrell Bricker, Ipsos Canada
Q&A
4 The Death of Polling? Version 1 Public
Sadly not everywhere looks like this … Average candidate error in US Presidential Elections (INDIV CAND SHARE)
1936
6%
1. 5%
2012
5 The Death of Polling? Version 1 Public
There’s a global
Ipsos has carried out election
conversation going
on about polling
polling in c30 countries since 2007
6 The Death of Polling? Version 1 Public
And pollsters all around the world are
How do we achieve a
representative sample?
How do we predict
turnout accurately?
having to face up to hard questions
How do we make best
use of the increasing range
of methodologies open to us?
How do we ensure our polls
are reported well and
understood among the
media, politicians
and the public? How do we avoid other
biases, such as social
desirability?
7 The Death of Polling? Version 1 Public
... although our answers
Size/diversity of population
Turnout levels/compulsory voting
Stable party system or new insurgents?
Generational declines in party loyalty
Societal/cultural context Media expectations vs budgets
and levels of
polling transparency?
Survey researchers or
moving towards big
data modelling?
National polls vs
local/state-level polling?
Traditional face-to-face
methods vs new
or mixed modes?
And in any one country
likely to be good and
bad examples of each!
may be very different!
The Death of Polling Version 1 Public 8
UK
experience
The
9 The Death of Polling? Version 1 Public
British Polling Council
Selection of samples
(their main explanation)
Representativeness of sample
(key among explanations)
Correction for likelihood
of voting (less important)
Late swing (some signs,
but not that important)
Herding
(unproven)
Inquiry Interim findings
10 The Death of Polling? Version 1 Public
Our final poll – all parties less than 2% points away
How would you vote if there were a General Election tomorrow?
from actual – except Labour, overestimated
36%
35%
11%
5%
8% 5%
Ipsos MORI final poll GB final result
Conservative lead = +1 Conservative lead = +6.5
CONSERVATIVE
LABOUR
UKIP
GREEN
LIB DEM
OTHER
37.7%
31.2%
12.9%
3.8%
8.1% 6.4%
11 The Death of Polling? Version 1 Public
‘Shy Tories’ not our problem – instead too
How would you vote if there were a General Election tomorrow?
many Labour voters and not enough non-voters
Conservatives
Labour
Non voters
11.3m
12.5m
9.3m
12.2m
20.5m
15.4m
Actual
Implied from final poll
The Death of Polling Version 1 Public 12
Ipsos MORI’s view:
We need to take a two-pronged approach
– tackle the problem of more politically
engaged taking part and also making
sure we can detect differential shift
in turnout over-claim between parties
The Death of Polling Version 1 Public 13
Ipsos MORI’s view:
Healthy scepticism
not the death of polling
14 The Death of Polling? Version 1 Public
We need to improve representation of
No easy answers to this
(especially bearing in
mind budget and time
Constraints)
politically disengaged/non-voters in our samples
Already introduced newspaper weighting (for example to reduce
proportion of broadsheet readers). In four months from
September to December 2015 this:
• Reduced the proportion of claimed likely voters by an average of 3
percentage points a month
• Primarily at the expense of the Labour share (down on average by 3 points,
Conservatives up by 1.75 points)
But will continue other experiments (for example changes in quotas, and so on)
15 The Death of Polling? Version 1 Public
Random samples after election though do not appear
to be hugely more representative than our quota ones
Election
result
Final telephone
polls (average)
Final Ipsos MORI
poll
British Social
Attitudes
British Election
Study1
Ipsos MORI post-
GE past vote
(June-July)
Voting
intentions2
Difference
from result
Voting
intentions
Difference
from result
Report of
vote
Difference
from result
Report of
vote
Difference
from result
Report of
vote
Difference
from result
% % % % % %
Con. 37.7 34.5 -3.2 36 -1.7 39.7 +2.0 40.6 +2.9 37.9 +0.2
Labour 31.2 34.3 +3.1 35 +3.8 33.6 +2.4 32.7 +1.5 32.5 +1.3
Other
parties 31.2 31.2 0.0 29 -2.2 26.7 -4.5 26.7 -4.5 29.6 -1.6
The Death of Polling Version 1 Public 16
The Death of Polling?
So much more than
a horse race
17 The Death of Polling? Version 1 Public
So much more
richness in the
polls to help
us understand
public opinion
18 The Death of Polling? Version 1 Public
Not to mention giving voters a
chance to express their views
19 The Death of Polling? Version 1 Public
Or using twitter analytics to get live, real-time reactions
to the big events: 239,000 tweets in the 2nd debate
Twitter ‘worm’ – real time
analysis of reaction to
second leader debate
(2,500+ per minute)
20 The Death of Polling? Version 1 Public
And using new digital techniques to get closer to voters
7426
posts across
340 forum
topics
c.2000 members
from across
the UK
Over
7500 survey
responses
THE THEME IS:
More Data Than Ever
0
20
40
60
80
100
120
Jan
-60
De
c-6
0
No
v-6
1
Oct
-62
Sep
-63
Au
g-6
4
Jul-
65
Jun
-66
May
-67
Ap
r-6
8
Mar
-69
Feb
-70
Jan
-71
De
c-7
1
No
v-7
2
Oct
-73
Sep
-74
Au
g-7
5
Jul-
76
Jun
-77
May
-78
Ap
r-7
9
Mar
-80
Feb
-81
Jan
-82
De
c-8
2
No
v-8
3
Oct
-84
Sep
-85
Au
g-8
6
Jul-
87
Jun
-88
May
-89
Ap
r-9
0
Mar
-91
Feb
-92
Jan
-93
De
c-9
3
No
v-9
4
Oct
-95
Sep
-96
Au
g-9
7
Jul-
98
Jun
-99
May
-00
Ap
r-0
1
Mar
-02
Feb
-03
Jan
-04
De
c-0
4
No
v-0
5
Oct
-06
Sep
-07
Au
g-0
8
Jul-
09
Jun
-10
May
-11
Ap
r-1
2
Number of Polls per month - 1960 - 2013Number of Polls per Month – 1960 - 2013
17,058 polls in 2012
Current US Political Polling Methodologies
PHONE
RDD
Lists
IVR (Robo)
TRADITIONAL ONLINE
Probability
Panel Only
Lists
Blended
NONTRADITIONAL ONLINE
SurveyMonkey
Great Variation Among Pollsters
25
Constantly
Changing
Questionnaire
(daily)
Election
Day ‘Exit
Poll’ of
40,000
Voters
Continual Survey: 11,000/month (24/7/365)
Daily Assessing Events Same-Day (Parsing) 26
State-level
polls (2k)
with rolling
reporting
What Does Ipsos Do?
A VERY QUICK WORD ON THE US ELECTION…
To replace the image: Delete existing image below, click picture icon, select new image & “send to back”
ANTI-ESTABLISHMENT THEME:
But Differential Framing of Problems & Causes
The rich are to blame Middle Class Economics
Americans First Restarting American
Exceptionalism
VS.
Trump & Sanders are the Response
17%
5% 5% 6%
13%
3% 3%
29%
19%
Tru
mp
Cru
z
Car
son
Oth
er o
uts
ider
s (P
aul,
Fio
rin
a, e
tc)
Esta
blis
hm
ent
(Bu
sh,
Ru
bio
, etc
)
Wo
uld
n't
vo
te
O'M
alle
y
Clin
ton
San
der
s
Republicans (46%) Democrats (51%)
46% of Americans are
supporting “nontraditional”
candidates
Political Fundamentals Speak to a Republican Year
65% 60% 55% 50% 45% 40% 35%
Government Approval Rating
Od
ds
of
Win
nin
g Incumbent Party’s Odds of Winning White House
SUCCESSORS
INCUMBENTS
(2012)
… AND LOW TURNOUT BENEFITS THE REPUBLICANS TOO
Source: Reuters / Ipsos Poll; Sept-Oct 2014
30% 35% 40% 45% 50% 55% 60% 65%
51 50
49 49 48
47 47 47
44 44 45
46 46
48
Republican Vote
Democratic Vote
2012 Turnout Actual 2014 Turnout
Generic C
ongre
ssio
nal V
ote
Share
Turnout Level (by Likely Voter Model)
2014 Generic Congressional Ballot by Turnout Levels
The Death of Polling Version 1 Public 33
Nando Pagnoncelli
Ipsos Italy
#ipsosmorilive
34 © 2015 Ipsos.
THE PREDOMINANCE OF COMPLEXITY THE ELECTORAL SCENARIO IN ITALY - 2013
169 parties and movements
2
35 © 2015 Ipsos.
A GAME CHANGER EFFECT THE EVOLUTION OF THE ELECTOR
ELECTORATES become fluid, reactive to the political offer, they lose their sense of belonging
2013 is the end of a 20-year period of almost perfect turnover between centre-left and centre-right, during which
• neither coalition was in office for two turns in a row
• voters switches between the two coalitions were residual
3
36 © 2015 Ipsos.
A GAME CHANGER EFFECT MOVIMENTO 5 STELLE
in 2013 we experienced a political earthquake or, a tsunami
4
37 © 2015 Ipsos.
A GAME CHANGER EFFECT MOVIMENTO 5 STELLE
… with victims, politicians …
… and pollster alike
they got a shock
we got it wrong
5
38 © 2015 Ipsos.
EVERYBODY BECAME A POLLING EXPERT, except the experts THE POLLSTERS UNDER ACCUSATION
Discussions went on for weeks on the limits of polls
Methodologies
Sample size
Coverage
Bias
…
were discussed and commented and criticised by anybody and all on TV, in
newspapers, online forum, social networks, workplaces and cafés
6
39 © 2015 Ipsos.
Tecnè Demos Ipsos
diff vs. actual results
RIVOLUZIONE CIVILE 0,9% 0,9% 1,0%
SEL 0,0% 0,4% 0,4%
PD 4,9% 3,7% 4,9%
ALTRI CENTRO SINISTRA -0,4% -0,5% -0,4%
TOTALE CENTROSINISTRA 4,5% 3,6% 4,9%
CON MONTI PER L'ITALIA -0,9% 2,0% 0,4%
UDC 0,8% 1,0% 0,7%
FLI 0,0% 0,2% 0,2%
TOTALE CENTRO -0,1% 3,2% 1,3%
LEGA NORD 1,0% 1,2% -0,1%
PDL -0,3% -1,5% -1,4%
ALTRI CENTRO DESTRA -0,7% -0,7% 0,6%
TOTALE CENTRODESTRA 0,0% -1,0% -0,9%
MOVIMENTO 5 STELLE
BEPPEGRILLO.IT -5,8% -6,6% -5,6%
ALTRE LISTE 0,5% -0,1% -0,7%
POLLSTERS WERE INACCURATE THE POLLSTERS UNDER ACCUSATION
WHAT WENT WRONG WHY IT WENT WRONG
For all the major agencies in Italy
• overestimated the Democratic Party’s
• underestimated the M5S result
• No past voters behaviour on M5S, which is a key component of the weighting process
• The difficulties of intercepting the potential M5S voters
• reticence in centre-left supporters to declare their intention
• last minute swing (25% estimated to have decided 2 days before)
• A high refusal rate, affecting differently the various groups of respondents (10 contacts to get a valid interview)
• Respondents lie
7
40 © 2015 Ipsos.
BUT SOME WERE MORE ACCURATE THAN OTHERS THE POLLSTERS UNDER ACCUSATION
WHAT WENT RIGHT WHY IT WENT RIGHT
IPSOS correctly described the scenario
• The high level of abstention (26,5%)
• The dramatic growth of M5S
• The victory of the centre-left at the Chamber of Deputies
• The hung Senate
• All the other parties’ results
Because
• A mixed sampling method was used
• Sample sizes were large enough
• We had been polling continuously – weekly, even daily – for a long time
• We had a wealth of data
• We had lots of analyses on targets, geographical areas, voters switch dynamics
8
41 © 2015 Ipsos.
THE PREDOMINANCE OF MADNESS THE POLLSTERS CHALLENGE
Our stakeholders show a schizophrenic attitude • one minute, they attack us, because we couldn’t predict the future as
exactly as we are expected to • the minute after, they ask for data, comments, explanations, because
they need our expertise in interpretation
We would like to see expectations decrease, towards the predictive nature of our work but we know it’s inconceivable
9
42 © 2015 Ipsos.
THE FlashForward Effect THE POLLSTERS CHALLENGE
… and everyone acted consequently. Polls act as an activator of two powerful, contrasting forces in electors
• Self-fulfilling prophecies vs. • Self-defeating prophecies
“On October 6, the planet blacked out for two minutes and seventeen seconds. The whole world saw the future...”
10
2,9 (2006)
5,7 (2010)
12,9
(2014)
0
10
20
30
Electoral support for the Sweden Democrats 2006-2016
2006 2010 2014
0
30
60
Left-Right Parties Election 2006 Left-Right Parties February 2016
From two political blocks to three – in ten years
Immigration most pressing issue and Voter support for Sweden Democrats
5 8
13
20 20
40
6 7
9 12
14
17
0
20
40
June-2010 June-2014 Aug-2014 Jan-2015 Jun-2015 Jan-2016
Immigration/integration most pressing issue Voter support for the Sweden Democrats
Immigration most pressing issue and google searches for “refugee” in Swedish
8 13
20 20
40
0
20
40
60
80
100
June-2014 Aug-2014 Jan-2015 June 2015 Jan-2016
Number of people say immigration/integration issues are most pressing issue
The number of searches on the word ”Refugee” on Google in Swedish during the same time period
17.2
23.9
0
20
Average voter support Telephone based methods Average voter support Web based methods
Voter support Sweden Democrats based on method of measurement Feb 2016
49 © 2015 Ipsos.
The state of polling 2016 SWEDEN
Social effects / Interviewer effects
Higher share of non-response in strong Sweden Democrat regions
From 45 % – 25 % in the last six years
SOCIAL EFFECTS AND NON-RESPONSE
DROPPING RESPONSE RATES
Less than 50 per cent with younger Swedes
Not a problem with seniors
SAMPLE COVERAGE
Telephone calls only for close friends and family? (Pew 2015)
Web based and telephone based give different answers
Transparency!
Mobile first strategy?
NEW BEHAVIORS
NEW METHODS
The Death of Polling Version 1 Public 50
Darrell Bricker
Ipsos Canada
#ipsosmorilive
51 © 2015 Ipsos.
Ipsos Final Poll vs. Actual Election Results—National ELECTION DEBRIEF
Ipsos Final Poll Election Results
38% 40%
31% 32%
22% 20%
4% 5%
4% 4%
Base: Final Call Poll (Decided Voters, Leaners Included n=2,226; Online n=1,328; CATI n=898) Weighting: 50/50 telephone/online, education, region, thumb
52 © 2015 Ipsos.
Online/Telephone Research (Call Poll) ELECTION DEBRIEF
2 waves consisting of minimum sample sizes of:
1/3
1/3
1/3 Ampario
Allocated
Reallocated
1,000 Telephone Completes
1,000 Online OMNI
roughly: 40% cellphone
3-day field window
Sample blended together to create 4 quadrants: 1) telephone landline; 2) telephone cell; 3) online panel, and 4) online non-panel
- Result somewhere in the middle…
and
KEY BENEFITS: Telephone poll to act as a gut check on our online greater confidence in our figures Triangulation among various sample sources mitigates bias of any one methodology Allows us to formulate a weighting scheme to mitigate panel bias
53 © 2015 Ipsos.
Call Poll Sample Breakdown ELECTION DEBRIEF
Base: Final Call Poll (Total n=2,503; Online n=1,502; CATI n=1,001)
16%
24%
13%
47%
CATI Mobile
CATI Landline
Online Ampario
Online Panel 60% Online
40% CATI
54 © 2015 Ipsos.
What We Learned
54 © 2015 Ipsos.
55 © 2015 Ipsos.
Lessons Learned and Next Steps ELECTION DEBRIEF
Mixed mode methods show promise.
Ipsos will continue to explore and share what we learn.
Offset biases associated with specific modes of data collection with minimum weighting.
56 © 2015 Ipsos.
Lessons Learned and Next Steps ELECTION DEBRIEF
Online and offline methods consistently under and over-estimate specific demographic groups. Also, over and under-estimate specific voter mindsets. Evidence building that:
…online overestimates progressive voters
…telephone overestimates conservative voters.
57 © 2015 Ipsos.
Lessons Learned and Next Steps ELECTION DEBRIEF
LIKELY VOTER ESTIMATES, while logically attractive, work better in theory than in practice. What’s missing? How do we make them better?
58 © 2015 Ipsos.
Likely Voter Model A REGRESSION MODEL BASED ON…
PAST BEHAVIOUR
INTENT TO VOTE
INTEREST IN ELECTION
59 © 2015 Ipsos.
Final Weighted Call Poll by Likely Voter ELECTION DEBRIEF
38%
37%
37%
37%
37%
31%
32%
32%
33%
32%
22%
22%
22%
22%
22%
4%
2%
2%
2%
3%
4%
5%
5%
5%
4%
Final
55% Turnout
60% Turnout
65% Turnout
70% Turnout
Liberals Conservative NDP Green Bloc
Base: Final Call Poll (Online Total Decided Voters n=1,328; I-Say Allocated n=427; I-Say Re-Allocated n=624; Ampario n=277) Weighting: 50/50 telephone/online, education, region, thumb
The Death of Polling Version 1 Public 60
Thanks for listening
Q&A #ipsosmorilive