The dark night of the polls: How did the polls get the 2015 election so wrong?

Professor Charles Pattie and Dr Todd Hartman from the Sheffield Methods Institute at the University of Sheffield explore some explanations for what went wrong with the pre-election polls.

This piece also features on The Sir Bernard Crick Centre blog

The UK’s 2015 General Election was widely touted as the most unpredictable in recent memory. The polls pointed to a dead heat between the Conservatives and Labour, as well as a deeply hung Parliament. The unpredictability, most thought, was going to be around what sort of multi-party deal would emerge from the chaos. But later that night the BBC/ITN exit poll pointed to another – quite sensational – result: a very good night for the Conservatives with 316 seats, edging them close to an overall majority. So shocked were pundits by this revelation that the former Liberal Democrat Paddy Ashdown famously quipped: ‘If this exit poll is right…I will publicly eat my hat’.

Much to Ashdown’s chagrin, the Conservatives did even better than the exit poll predicted. With 331 total seats, the Tories won their much-prized and little-expected majority, the first time their party had done so since John Major’s 1992 victory. For pundits, pollsters, and the media, this was a shock result. For the polling industry, it was a disaster – not just a long dark night of the soul, but a long dark night for the polls!

The long road to failure

The 2015 contest was one of the most polled in modern British history. Yet, despite the sheer volume of survey data, the polls had been remarkable in their uniformity. For months, they suggested that little had changed in the battle between the parties. Labour and the Conservatives remained deadlocked at around 34% each, while UKIP were stuck on around 14%, the Lib Dems languished on 8%, and the Greens on about 5%. This was a recipe for a hung Parliament if ever there were one. Even on the eve of the election, 11 separate polls all produced virtually identical results – identical to each other and to the general trend in the polls since the start of the year.

graph1

As we now know, the polls got it wrong. Far from a neck and neck race between the top two parties, the Conservatives’ national vote share, at 36.9%, was a comfortable 6.5% ahead of Labour’s share. And as if to drive home the point, there were no secret negotiations and scrabbling to build some deal in a hung Parliament – the Conservatives gained a majority of MPs and could govern alone.

graph2

In hindsight, the polls underestimated Conservative support by almost 3 percentage points and over-estimated Labour’s share by a similar amount. That pollsters had done better on predicting the other parties’ shares was little compensation.

graph3

So what went wrong with the pre-election polls? There are number of possible explanations. Deciding which is correct will be difficult and require more information than we currently have available. Even so, some explanations are more plausible than others. Below, we look at the runners and riders.

Was it all just sampling error?

Of course, it could be argued that the deviations between the polls and the actual results were within conventional sampling error (as a rule of thumb, most poll estimates are accurate to within ±3% for samples of about 1,000 people). But this misses the point. To get one party’s vote share wrong by this much would be unfortunate. To get both of the main parties’ shares so wrong looked careless. To do it repeatedly in survey after survey was disastrous. The huge number of polls – all saying virtually the same thing – suggests something deeper was wrong than just simple sampling error.

We have looked at 212 different national opinion polls over the period between January and May 2015. They tell a remarkably consistent story. The histograms below count up the number of polls over the period which gave the Conservatives or Labour 28% of the vote, 29% and so on. As is clear, very, very few polls indeed put either party more than just a percentage point or so above or below 33%. The Conservatives average 32.9% over all the polls we looked at, while Labour averaged 33.5%. The standard deviations (a measure of the spread of data around this average) are low – just 1.8% for the Conservatives and 1.6% for Labour.

These figures reveal a troubling pattern. Conventional statistical theory tells us that even a completely random, unbiased sample may occasionally give us incorrect estimates of the thing we are trying to measure. But it also suggests that if we take many samples, the average of the estimates from each sample should converge on the true value in the population. So the average Conservative and Labour support from our samples should begin to converge on the actual election result.

Clearly, that did not remotely happen in 2015.

graph4

graph5

Of course, one objection is that we shouldn’t expect too much of polls taken several months before the election. After all, much could have changed between January and May. But even if we focus on the 8 polls in our data set taken in the last week of the campaign, the same problem occurs: Conservatives and Labour have average shares of 33% with standard deviations of 1.1% and 1.4%, respectively. What is more, the maximum Conservative share, 34%, is well below the party’s actual vote share (36.9%), and Labour’s minimum poll rating over the period of 31% is above its actual vote (30.4%).

Adding to the confusion, the polls generally did a good job of estimating the Lib Dem, UKIP and Green vote shares. This is particularly ironic in the case of the latter two parties. In the run-up to the election, many pollsters were extremely nervous about their ability to estimate the votes for UKIP and the Green Party correctly. Neither (especially UKIP) had ever before polled at these levels as a General Election approached. As a result, the pollsters had no past precedent to go on to guide them on whether they were on the right lines. As it turned out, the pollsters were right about the minor parties: they should have been worrying more about their estimates for the two parties for which they had most past precedent!

Whatever was going on, therefore, we would be foolish to assume it was just a result of random sampling error. The discrepancies between the polls and the actual election results for Labour and the Conservatives are too large, and too consistent in direction, for that.

Was it the pollsters’ methodologies?

Pollsters have different means of contacting potential respondents to ensure that they reach representative samples of voters. Some polls are based on face-to-face interviews, some utilise telephone interviews, and still others are conducted over the Internet. Each method faces both general and specific problems.

All polls, for instance, will tend to over-emphasise the views of the politically interested and under-play the views of those who are less engaged with politics (for the very simple reason that people who are not interested in politics are less inclined to talk to the pollsters, and so are under-represented in all samples).

This specific problem is exacerbated by the general trend over time of a drop in overall survey response rates. We are increasingly over-sampled as a population, and people understandably get fed up with being asked their views again and again. Fewer and fewer of us are willing to give time to pollsters. Those who do agree to be surveyed become less typical of the population as a whole.

What is more, there are well-known social and demographic biases of those who take part in opinion polls – participants are often older, more middle class and better educated than the population at large, and that will have consequences for how representative they are of public opinion. The pollsters are well aware of these challenges, and all have various weighting schemes to rebalance their data.

There might also be mode effects. Telephone interviewing, for instance, is becoming harder as more of us go ex-directory, adopt call blocking services, and move to mobile rather than landline phones. Internet polls, meanwhile, are biased towards the more Internet-savvy sections of the public. Again, the pollsters are well aware of this: their weighting schemes and interview methodologies also do a lot to correct for these biases.

But ultimately, explanations based on differences in polling methodology run into an obvious problem in 2015. Virtually all the major polling companies, whatever their methods, were producing very similar estimates of the parties’ standings. The very unanimity of the polls militates against this explanation. Whatever went wrong, it affected all of the pollsters and all major methods.

Maybe the polls were right: Was this a very late swing?

Pollsters will point out – and they are right – that no opinion poll is really a prediction of the election result. All a poll can do is provide a snapshot of public opinion at that moment in time. Even on the eve of the actual election, there is still scope for people to change their minds. In this (apparently) most unpredictable of elections, perhaps a significant body of voters, faced with what they saw as a risky or uncertain outcome, stepped back at the last minute.

This explanation fits with some elements of the last weeks of the campaign. For instance, while the polls remained deadlocked, it was also clear that in Scotland the SNP was heading for a major upset. The post-election arithmetic of the House of Commons suggested by the polls seemed to imply that one quite likely outcome would be a Labour-SNP deal of some kind. Speculation about this coalition came to dominate the final week of the campaign. The Conservatives in particular made much of the claim that such an arrangement would signal chaos and would hold English voters hostage to a cohesive group of Scottish MPs who were not only on the left politically but were also committed to the break-up of the UK – a fear luridly captured in Boris Johnson’s vision of a post-election ‘Ajockalypse Now’.

What is more, a large group of voters were still undecided on the eve of the election, and in close contests the undecided can swing the result. Faced with uncertainty, a natural reaction for the undecided might be to vote for stability: better the devil you know… And that would tend to benefit the Conservatives, as the main party in the incumbent government.

So there might have been a last minute swing to the Conservatives. The problem, though, is that (so far at least) there is not much evidence of it. On the final day of the campaign, YouGov re-contacted a group of voters they had already interviewed earlier in the same week. This was a tactic they had employed successfully during the Scottish Independence Referendum in September 2014, and on that occasion they had picked up evidence of a late swing away from independence towards staying in the UK: the result of that vote was a larger vote against independence than the polls in the previous weeks had predicted.

So YouGov’s strategy for picking up late swing was tried and tested. However, they found no evidence of late swing in 2015. In the last hours of the campaign, some voters were changing their minds. But movements in one direction were countered by similarly sized shifts in the other. The net effect was negligible. Late swing does not look a likely contender for the failings of the polls in 2015.

Shy Tories: Do people tell the pollsters the truth?

Another popular explanation is the so-called ‘Shy Tory’ effect. This is a variant of Elisabeth Noelle-Neumann’s well-known Spiral of Silence, in which individuals are reluctant to express a particular viewpoint that might be unpopular for fear of being ostracised. Supporters of a party that generates strong antagonistic views might, therefore, be reluctant to declare their loyalties publicly, even to pollsters. Rather than show their hand, they might say they are undecided, or do not intend to vote – or they might refuse to take part in the survey altogether.

But in the privacy of the polling booth, when no one can see and no one can comment, they cast their vote for their unpopular preference. This, of course, will tend to depress opinion polls’ estimates of the affected party’s support, which suddenly jumps in ‘the only poll that matters’, the election result itself. This seems to have been the explanation for a previous dramatic failure of the opinion polls. At the 1992 General Election, the polls also underestimated the Conservatives’ vote share and over-estimated Labour’s.

In the aftermath of that polling disaster, the pollsters conducted lengthy post-mortems, and most adopted methods to counter the ‘shy tory’ effect (for instance, by weighting by vote in the preceding General Election). Could it also be that the problem has resurfaced in 2015? Perhaps so. For instance, there is evidence that, despite the post-1992 methodological adjustments, the polls have persistently (though not always as dramatically) over-estimated Labour voting and under-estimated the Conservatives. There does seem to be something ingrained among some voters about not wanting to admit publicly to being a Conservative.

However, the Conservatives were not the only party that generated strongly antagonistic views in 2015. The Liberal Democrats, for instance, had been through a torrid five years since joining the coalition in 2010, widely attacked from the centre and left for propping up a Conservative government (and for, inter alia, abandoning its commitment on student fees), and from the right for thwarting some Conservative policies while in coalition with them.

Similarly, Labour in Scotland had been in the sharp end of widespread and vocal opprobrium since the 2014 Independence vote, not least for campaigning jointly with the Conservatives to oppose independence. Among the gentler insults thrown their way by pro-independence campaigners in the months after the referendum was ‘Red Tories’. And, let us not forget UKIP, a party which has endured constant ridicule and belittlement, so much so that voters agreed that they would be most embarrassed to tell friends and family that they supported the party.

But the Liberal Democrats, Labour in Scotland, and UKIP showed no sign of a spiral of silence. The polls accurately predicted these parties’ vote share in 2015. (Labour’s 2015 vote share in Scotland was 24%: eve of election polls put them on 25% there.) The challenge for a spiral of silence explanation, then, is ‘why just the Conservatives’?

Another argument against the ‘Shy Tory’ theory is that many of the pre-election polls were conducted over the Internet in the privacy of voters’ homes. Research suggests that people are more likely to reveal their true preferences to sensitive questions when they are not interacting with a live interviewer (either in person or over the phone). The reason is that people are motivated by social situations to avoid embarrassment and will hide information that could be potentially damaging. Yet, we know that the results from the online and telephone surveys were similarly inaccurate, both of which underestimated support for the Conservatives. In fact, the online surveys should have been more likely to pick up on shy Tory voters because the anonymity of this survey mode should have mitigated embarrassment about revealing support for a potentially unpopular party.

Question ordering and wording effects?

A final possible explanation was advanced in the days after the election by James Morris of Greenberg Quinlan Rosner Research. Morris had worked closely with the Labour leadership during the 2010-15 Parliament, conducting much of their internal market research, so had an insider’s view. He claimed that Labour’s internal polling throughout the parliament had always been more pessimistic than the polls publicised by the media.

The reason: question ordering. Morris argued that while commercial polls often led with the party choice question, Labour’s private polls began by asking a series of questions tapping voters’ evaluations of the state of the economy, their evaluations of the party leaders and so on. Only after they had been primed to really think about the issues were respondents asked their vote intention. The consequence was that support for the Conservatives was greater than in other polls because voters saw them as better managers of the economy and had a more popular leader than Labour, and as having a more Prime Ministerial leader.

What is more, he claimed, Labour’s internal focus group work over the last week or so of the campaign did pick up growing concerns among erstwhile Labour voters about the risk of strong SNP influence over a minority Labour government (exactly the effect hoped for by Conservative strategists, who pushed the ‘vote Labour, get the SNP’ line remorselessly).

We are not yet in a position to evaluate these claims independently. We have not carried out a systematic analysis of question ordering in all the commercial polls for which we have data. And we do not, by definition, have access to Labour’s private polling. But Morris’ arguments are at least plausible.

Except… If Labour’s internal polling was consistently more negative than the public polls, why were senior figures in the Labour campaign so visibly shocked when the election produced such a good result for the Conservatives?

What about question wording effects? For the most part, the pollsters seem to ask the same type of vote intention question: ‘If there were a general election held tomorrow, which party would you vote for’? This consistency is important, as it allows us to compare the polls by different companies over time. Yet, some people have pointed out that there appear to be differences in responses – and by extension, accuracy – when asking respondents about their general vote intention versus constituency-level vote.

These differences suggest that there could be another problem with question wording: all of the polls asked the vote intention question about the party; no one really asked about the specific candidates who were actually running for office (and whose names would ultimately be on the ballot). Obviously, asking about the specific candidates within each constituency drastically complicates polling. But we might expect some people to hold different evaluations toward the national party and their local representative (either positive or negative). And, these differences could have been responsible for certain MPs over- or under-performing relative to the pre-election polls.

No easy conclusion

We are left, then, with a series of puzzles. Some explanations (shy Tories, question ordering/wording effects) look more plausible than others (late swing, methodological differences between pollsters, and sampling error). As after their 1992 election fiasco, the polling companies have to conduct a full and frank post-mortem. That will give us a better idea of what went wrong for the pollsters this time. Watch this space.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s