Advertisement

We need your help now

Support from readers like you keeps The Journal open.

You are visiting us because we have something you value. Independent, unbiased news that tells the truth. Advertising revenue goes some way to support our mission, but this year it has not been enough.

If you've seen value in our reporting, please contribute what you can, so we can continue to produce accurate and meaningful journalism. For everyone who needs it.

Alamy Stock Photo

Elections 2024 General election polls are poor predictors of local elections – here’s why

Stefan Müller examines the effectiveness of election polling and says we should change how they work and how we report them.

OPINION POLLS ARE one of the main talking points of the recent local and European Parliament elections. Tánaiste Micheál Martin and Jack Chambers, Fianna Fáil Director of Elections, publicly criticised pollsters using online panels. 

Fianna Fáil has good reason to be frustrated. General election polls have significantly underestimated support for Fianna Fáil and vastly exaggerated support for Sinn Féin. For them, this creates a mistaken media narrative about popular support for Sinn Féin.

But does this underestimation of support for FF in general election polls mean that we should completely ignore opinion polls? Do they still have value for politicians, journalists and voters? How can the media, politicians and voters engage more constructively with public opinion polls? The starting point is to recognise that general election polls do not measure support for parties in local elections. Irish general election polls have always been a poor predictor of local elections, but are good predictors of general election results.

Polls for general elections are accurate

One criticism raised by politicians and commentators after these elections relates to the huge difference between support in current opinion polls for the upcoming general election and the local election results.

Let’s untangle local and general election polls. Are opinion polls for general elections accurate? This is the first question I will address.

The Irish Polling Indicator project, which Tom Louwerse (Associate Professor, Leiden University) and I started 10 years ago, aggregates all opinion polls into a single estimate of public support.

We have collected all opinion polls published since the mid-1980s, resulting in a sample of 717 surveys.

This tedious exercise involved going through newspaper archives, checking various websites and contacting academics. The resulting dataset not only allows us to present aggregated estimates but also helps us understand how well polls predict elections.

To assess the accuracy of polls, I extracted the Polling Indicator estimate on the day of the election and compared the figures to the actual first preference vote shares. Next, I calculate the absolute mean error, which is the difference between the first-preference vote share at the election and the aggregated estimate. If a party obtained 25 per cent of first-preference votes, but the poll of polls estimated support at 21 per cent, the absolute error for this party amounts to 4 percentage points. This analysis is performed for all parties to calculate the average absolute difference.

The graph below reveals that the absolute mean error results to be just 1.32 percentage points across the eight general elections between 1989 and 2020. In other words, on average, the poll of polls is just 1.3 points off the election result. This is very accurate.

polls_vs_election_general Stefan Müller Stefan Müller

From a political science perspective, it is also very encouraging: most estimates fall within the margin of error. There are, of course, exceptions to the rule. In 2002, the polls overestimated support for Fianna Fáil by 4.6 percentage points but underestimated the party’s support by 4.4 points in 2007 and 3.4 points in 2016.

In 2020, the aggregated polls overestimated support for Fianna Fáil by 2.4 points. In 2016, the polls overestimated support for Sinn Féin and Fine Gael by around 3 points each. Yet, overall, polls have done a pretty good job in predicting party support in Irish general elections over the past 25 years.

Not so for Local Elections – this is no surprise

What about Local Elections? Pollsters received a lot of criticism for underestimating Fianna Fáil support and vastly overestimating the Sinn Féin vote last week. Yet, opinion polls do not intend to predict local election results.

Looking at the second figure here supports this argument. The absolute mean differences range from 2.46 (2004) to 3.77 (1991) points. Last weekend, the mean difference was 3.18 points. At almost every election, the estimates for at least one party are completely different to the outcome.

polls_vs_election_locals Stefan Müller Stefan Müller

Why are the polls so far off? Irish local elections are unique. Incumbency plays a huge role in Local Elections. Our research shows that even incumbents who got elected on the last count face a huge advantage in rerunning and getting re-elected compared to the candidate who lost on the last count. Canvassing, name recognition and community ties are highly relevant. Door-to-door campaigning is a unique feature of Irish politics and it is a crucial condition for getting elected.

An Ireland Thinks poll fielded on the day of the local and European Parliament elections mirrors these claims. 75 per cent of Fianna Fáil voters cast their first preference for a candidate rather than the party the candidate represented. 66 per cent of Fine Gael voters ranked the party over the candidate.

The focus on candidates is lower for smaller parties – possibly because they ran fewer incumbents and their candidates were simply not as well known.

Fianna Fáil and Fine Gael also canvassed very successfully. 35 per cent of respondents said they met a Fianna Fáil candidate at their doorstep. 34 per cent of respondents had direct contact with a Fine Gael candidate. Only 18 per cent met a Sinn Féin representative. This might be part of the explanation why Sinn Féin performed even more poorly than anticipated: they are not only losing support in opinion polls, but they had few incumbents and their candidates were simply not as active on the ground as Fianna Fáil and Fine Gael. To sum up, incumbency and local ties are crucial, and Fianna Fáil and Fine Gael outperformed Sinn Féin in this hugely important aspect of campaigning.

We need to keep in mind that the electorate is becoming more volatile. According to Exit Polls from the past 25 years, around half of the voters make up their minds about the first preference vote in the last month before an election. Since the financial crash, around four in 10 voters have switched between parties from one general election to the next. Campaigns and canvassing are crucial in general elections and can reshuffle the Irish political landscape.

Online vs. face-to-face polls

The debate between the efficacy of online and face-to-face polls continues to be contentious. While some, like Fianna Fáil’s Micheál Martin and Jack Chambers, argue that online polls tend to underestimate their support, face-to-face polls also present challenges, such as low response rates and social desirability biases. Red C Research, despite being criticised at the moment, accurately predicted Fianna Fáil’s performance in the 2020 general election.

Ideally, we rely on a mix of methodological approaches to better understand Irish party politics. Telephone polls, face-to-face surveys, and online surveys all have advantages and shortcomings. Michael A. Bailey, Professor at Georgetown University, describes these issues in an excellent recent book “Polling at a Crossroads”. Having an open mind towards public opinion will ultimately improve our understanding of public opinion.

Moving forward: suggestions for journalists, politicians and voters

What can we learn from this debate on opinion polls and how can we move forward the debate? I have four recommendations:

1. Media outlets should adjust their reporting. We must consider structural differences between polling companies, focus on trends rather than individual polls and not overplay small “changes” lying within the margin of error as “bounces”, “rebounds”, and “recoveries”. Sampling variation drives most of these “changes”, which have usually been within the margin of error of ±3 percentage points. Political scientists Erik Gahner Larsen and Zoltán Fazekas show that Danish news reports frequently interpret deviations within the margin of error as substantive changes. The authors conclude that “a horserace coverage of polls about change can rest on a foundation of stability.” Irish newspapers often apply the same rhetoric. A move away from horserace coverage is necessary.

2. In addition to changing the language, polling companies and newspapers could present party support as within windows (i.e., the 95 confidence interval). Reporting single numbers projects a false sense of precision. Other countries are slowly adopting this approach. For example, Italian media reported the results from the exit poll for the European Parliament elections in intervals. Rather than focusing on a single number – and suggesting a wrong sense of certainty – their headline finding was that “Meloni’s FdI to get 26-30% in EU elections – exit poll Centre-left Democratic Party to come second with 21-25%”. Irish media should follow suit.

3. We should focus more on trends. For example, the polling trends clearly showed the loss of support for Sinn Féin in general election polls – across all companies. While pollsters deviate systematically in their survey mode and baseline results, these developments are unambiguous. Voters should approach poll results critically, considering the broader context and acknowledging uncertainty.

4. We need to keep in mind that polls provide a single snapshot and that using polls to predict elections is extremely difficult. Paul Perry, back then Vice Chairman of Gallup Polling, focused on these problems in a Public Opinion Quarterly paper published 45 years ago:

All one has to do is use a properly drawn sample of the electorate large enough to minimise random sampling error, get honest answers from everyone, do the questioning close enough to the time of voting to minimise changes in voting intentions, anticipate how the undecided will vote, and, finally, distinguish between voters and non-voters in the electorate – (Perry, 1979: page 312).

Pollsters should need to be even more transparent in their reporting methods. While most outlets now list the poll’s margin of error and time when the survey was in the field, the exact approach of selecting households for face-to-face polls, selecting respondents from the online panel, non-response rates, and the weighting of respondents is still a black box. This is understandable because polling is highly complex and relies on cutting-edge statistical methods. Media outlets could add a few sentences about the sample selection and weighting.

Finally, it is important to note that polls are extremely useful for understanding voter preferences and behaviour. Journalists and politicians need polls to understand the electorate. I believe the current debate on the usefulness of polls is constructive and ought to lead to important reflections among all actors involved. I hope that mentioning methods and uncertainty will increase trust in public opinion. Ultimately, these changes would help us to focus on policy issues and the solutions that parties offer.

Stefan Müller is Assistant Professor in the School of Politics and International Relations at University College Dublin. He maintains the Irish Polling Indicator and the Irish Politics Data project.

Readers like you are keeping these stories free for everyone...
A mix of advertising and supporting contributions helps keep paywalls away from valuable information like this article. Over 5,000 readers like you have already stepped up and support us with a monthly payment or a once-off donation.

Close
52 Comments
This is YOUR comments community. Stay civil, stay constructive, stay on topic. Please familiarise yourself with our comments policy here before taking part.
Leave a Comment
    Install the app to use these features.
    Mute Andrew Kenny
    Favourite Andrew Kenny
    Report
    Sep 11th 2024, 1:11 PM

    I wouldn’t exactly be excited to be endorsed by Taylor Swift. Pass the bucket..

    275
    Install the app to use these features.
    Mute Tony Skillington
    Favourite Tony Skillington
    Report
    Sep 11th 2024, 1:20 PM

    @Andrew Kenny: I’d be on the same page as you but there’s a 300 million younger vote in the US so her word will definitely impact on Trumps campaign so it definitely carries weight.

    227
    Install the app to use these features.
    Mute Paul O'Mahoney
    Favourite Paul O'Mahoney
    Report
    Sep 11th 2024, 1:24 PM

    @Tony Skillington: 300million ?

    116
    See 8 more replies ▾
    Install the app to use these features.
    Mute Brendan O'Brien
    Favourite Brendan O'Brien
    Report
    Sep 11th 2024, 1:26 PM

    @Tony Skillington: 300 million? Did everyone get Botox?

    44
    Install the app to use these features.
    Mute Jack Hayes
    Favourite Jack Hayes
    Report
    Sep 11th 2024, 1:27 PM

    @Andrew Kenny: I would if I welcomed support from her huge fan base.

    43
    Install the app to use these features.
    Mute Molly Mal
    Favourite Molly Mal
    Report
    Sep 11th 2024, 1:31 PM

    @Tony Skillington: average age of her concert goers is 40+. Taylor has outlined her reasons and research behind the decision to endorse Harris, not that any of you would care to read them.

    84
    Install the app to use these features.
    Mute another one? what's going on is the semi state sec
    Favourite another one? what's going on is the semi state sec
    Report
    Sep 11th 2024, 1:34 PM

    @Tony Skillington: In 2020 there was 168.31 million people registered to vote in the US. Sure the total population is around 345 million…… Did you get mixed up with the amount of worldwide followers she has or something?…..
    The general jist of what you’re saying is right

    47
    Install the app to use these features.
    Mute Brendan O'Brien
    Favourite Brendan O'Brien
    Report
    Sep 11th 2024, 1:46 PM

    @another one? what’s going on is the semi state sec: No. It’s completely wrong.

    10
    Install the app to use these features.
    Mute Con Cussed
    Favourite Con Cussed
    Report
    Sep 11th 2024, 1:57 PM

    @Brendan O’Brien: Are you a numpty Trump supporter? In which case everything that anyone says against him is wrong, not just anyone on here!

    33
    Install the app to use these features.
    Mute another one? what's going on is the semi state sec
    Favourite another one? what's going on is the semi state sec
    Report
    Sep 11th 2024, 2:22 PM

    @Brendan O’Brien: Good man Brendan. Get out of the wrong side of the bed this morning?!?!

    10
    Install the app to use these features.
    Mute SerotoninWars
    Favourite SerotoninWars
    Report
    Sep 11th 2024, 6:42 PM

    @Andrew Kenny: It’s great to see even as a non-Swiftie! Not my bag but hugely influential. The Trump’s of the world have had the run of the place for too long. Male, stale and beyond the pale, in his case. Stick together sisters and bring the old guard down! I don’t agree with everything Harris stands for but she’s not so much an upgrade as hi-tech cutting-edge futurism beside the cat-crazed mob.

    21
    Install the app to use these features.
    Mute John Kenny
    Favourite John Kenny
    Report
    Sep 11th 2024, 1:16 PM

    See comments closed on fine gael always open for the shinners how come

    156
    Install the app to use these features.
    Mute Shimo F
    Favourite Shimo F
    Report
    Sep 11th 2024, 1:27 PM

    @John Kenny: the Churno is a mouth piece for FFG, no mystery really

    152
    Install the app to use these features.
    Mute Brendan O'Brien
    Favourite Brendan O'Brien
    Report
    Sep 11th 2024, 1:38 PM

    @John Kenny: Because I said so.

    11
    See 3 more replies ▾
    Install the app to use these features.
    Mute 9QRixo8H
    Favourite 9QRixo8H
    Report
    Sep 11th 2024, 2:57 PM

    @John Kenny: they closed the article where FG want to bring in public-funded childcare!!! Socialist childcare!!! From a party that’s supposed to be for privatisation/capitalism!!!

    14
    Install the app to use these features.
    Mute John Kenny
    Favourite John Kenny
    Report
    Sep 11th 2024, 6:46 PM

    @Brendan O’Brien: no just your mammy

    1
    Install the app to use these features.
    Mute Sun Rise
    Favourite Sun Rise
    Report
    Sep 11th 2024, 7:48 PM

    @John Kenny: why type words can make no sense question mark.

    2
    Install the app to use these features.
    Mute cup of tea
    Favourite cup of tea
    Report
    Sep 11th 2024, 1:18 PM

    Means little. Americans with a brain like Elon Musk are voting for Trump in November.

    133
    Install the app to use these features.
    Mute Dere
    Favourite Dere
    Report
    Sep 11th 2024, 2:41 PM

    @cup of tea: the only winner of last nights debate was Vovager 1, the furthest thing from Earth.

    44
    Install the app to use these features.
    Mute Con Cussed
    Favourite Con Cussed
    Report
    Sep 11th 2024, 3:19 PM

    @cup of tea: You mean most Americans with half a brain will vote for Trump. Elon Musk may be intelligent but he has no compassion or empathy, which may be part of his autism. Just watch the way he fired people at Twitter – there are even videos of him doing it. If he is the type of leader which encompasses Trump thinking then Attila the Hun would be considered a nice chap.

    54
    See 3 more replies ▾
    Install the app to use these features.
    Mute Shane Kinsella (Kinsey)
    Favourite Shane Kinsella (Kinsey)
    Report
    Sep 11th 2024, 4:08 PM

    @Con Cussed: musk has Asperger’s not autism. And they way he fired those twitter employees was proven right. Twitter still ran without a hitch.

    18
    Install the app to use these features.
    Mute ItWasLikeThatWhenIGotHere
    Favourite ItWasLikeThatWhenIGotHere
    Report
    Sep 11th 2024, 5:43 PM

    @Shane Kinsella (Kinsey): Aspergers was classed as a type of autism.

    Twitter has lost half its value since Muskys takeover.

    And Twitter even DDoSed itself.

    31
    Install the app to use these features.
    Mute Patrick Hamill
    Favourite Patrick Hamill
    Report
    Sep 11th 2024, 9:33 PM

    @cup of tea: What’s wrong with his brain

    1
    Install the app to use these features.
    Mute Kush OMeara
    Favourite Kush OMeara
    Report
    Sep 11th 2024, 1:13 PM

    I knew she was trouble when she walked in

    57
    Install the app to use these features.
    Mute Ned
    Favourite Ned
    Report
    Sep 11th 2024, 4:26 PM

    Not supporter of trump or Harris but if a person young or old need to follow some pop star to help them to decide who to vote for well that is a sad state of affairs for that person who can’t think for themselves.

    43
    Install the app to use these features.
    Mute Patrick Hamill
    Favourite Patrick Hamill
    Report
    Sep 11th 2024, 9:40 PM

    @Ned: If you actually read her statement she did not tell people to vote for Harris. She told people to look at what both candidates and decide for themselves who to vote for. What she recommended her followers was for them to make sure they register to vote.

    27
    Install the app to use these features.
    Mute Edward O'T.
    Favourite Edward O'T.
    Report
    Sep 11th 2024, 9:47 PM

    @Patrick Hamill: her intent was to drag her fans who can’t think for themselves to vote for Harris that is so obvious even you should see that

    12
    Install the app to use these features.
    Mute AnthonyK
    Favourite AnthonyK
    Report
    Sep 11th 2024, 1:26 PM

    Genuine question: if a man gets a woman pregnant should the man have a say in what happens to the foetus, unborn child, sprog, spawn, or whatever people want to label the result.

    41
    Install the app to use these features.
    Mute Brendan O'Brien
    Favourite Brendan O'Brien
    Report
    Sep 11th 2024, 1:29 PM

    @AnthonyK: No: it’s the woman’s body, she she ultimately decides.

    110
    Install the app to use these features.
    Mute Brendan O'Brien
    Favourite Brendan O'Brien
    Report
    Sep 11th 2024, 1:30 PM

    @AnthonyK: ‘so she’, I meant.

    22
    See 7 more replies ▾
    Install the app to use these features.
    Mute Jack Hayes
    Favourite Jack Hayes
    Report
    Sep 11th 2024, 1:31 PM

    @AnthonyK: “or whatever people want to label the result’.” Wow.

    26
    Install the app to use these features.
    Mute Brendan O'Brien
    Favourite Brendan O'Brien
    Report
    Sep 11th 2024, 1:33 PM

    @Brendan O’Brien: Stay on topic.

    13
    Install the app to use these features.
    Mute Jack Dermody
    Favourite Jack Dermody
    Report
    Sep 11th 2024, 1:35 PM

    @AnthonyK: Yep, as soon as men get pregnant…

    40
    Install the app to use these features.
    Mute Tim Brennan
    Favourite Tim Brennan
    Report
    Sep 11th 2024, 1:45 PM

    @AnthonyK: no

    9
    Install the app to use these features.
    Mute Louis Jacob
    Favourite Louis Jacob
    Report
    Sep 11th 2024, 1:51 PM

    @AnthonyK: Genuine answer: Ideally the man should be allowed state his case. But be shouldn’t ever be allowed to dictate what a woman does with her body.

    64
    Install the app to use these features.
    Mute Éamonn O'Kane
    Favourite Éamonn O'Kane
    Report
    Sep 11th 2024, 4:45 PM

    @AnthonyK: No, it’s the womans body.

    16
    Install the app to use these features.
    Mute Paddy C
    Favourite Paddy C
    Report
    Sep 11th 2024, 11:53 PM

    @AnthonyK: what if the man feels like he’s a woman now surely that’ll give him rights he can argue he has very same rights as a woman then with equality crap.

    4
    Install the app to use these features.
    Mute Frank
    Favourite Frank
    Report
    Sep 11th 2024, 2:08 PM

    She was criticised because she’s friends with someone who “liked” a Trump comment on social media. So there are sad people who actually check on what comments people like. The inquisition has returned.
    Meanwhile being endorsed by a singer is a major thing! Granted Harris is a better choice than Trump (so is my cat). But the dumbing down of politics is rather worrying.

    44
    Install the app to use these features.
    Mute ItWasLikeThatWhenIGotHere
    Favourite ItWasLikeThatWhenIGotHere
    Report
    Sep 11th 2024, 5:56 PM

    @Frank: It was probably forced into peoples “news” feed by the algorithms used on these social media platforms.

    4
    Install the app to use these features.
    Mute Michael o Dwyer
    Favourite Michael o Dwyer
    Report
    Sep 11th 2024, 1:15 PM

    Haven’t seen any posts from buster this morning

    28
    Install the app to use these features.
    Mute Dermot Blaine
    Favourite Dermot Blaine
    Report
    Sep 11th 2024, 1:28 PM

    @Michael o Dwyer: that’s probably him pretending to be Brendan O Brien

    45
    Install the app to use these features.
    Mute Jack Hayes
    Favourite Jack Hayes
    Report
    Sep 11th 2024, 1:30 PM

    @Michael o Dwyer: Trolls gonna troll. He’ll be the first to whine like his Daddy when trolling closes down more comment sections.

    28
    See 6 more replies ▾
    Install the app to use these features.
    Mute Mike
    Favourite Mike
    Report
    Sep 11th 2024, 1:32 PM

    @Jack Hayes: You’re the only one whining here, lady.

    14
    Install the app to use these features.
    Mute James Leahy
    Favourite James Leahy
    Report
    Sep 11th 2024, 1:36 PM

    @Michael o Dwyer: probably gone to bed to cover his head. Can’t believe what happened last night.

    23
    Install the app to use these features.
    Mute Jack Hayes
    Favourite Jack Hayes
    Report
    Sep 11th 2024, 2:02 PM

    @Mike: So you don’t understand the meaning of ‘whining’ then?

    14
    Install the app to use these features.
    Mute Mike
    Favourite Mike
    Report
    Sep 11th 2024, 2:14 PM

    @Jack Hayes: Keep on moaning, girl. It’s entertaining.

    11
    Install the app to use these features.
    Mute Jack Hayes
    Favourite Jack Hayes
    Report
    Sep 11th 2024, 3:46 PM

    @Mike: Add moaning to the list of things you don’t understand. Calling me ‘girl’ is great. You’re another low IQ, low info Dim Bulb and you’re not afraid to hide it.

    18
    Install the app to use these features.
    Mute Mike
    Favourite Mike
    Report
    Sep 11th 2024, 5:39 PM

    @Jack Hayes: Fix me a sammich, lady! And hurry t fk up.

    4
    Install the app to use these features.
    Mute Louis Jacob
    Favourite Louis Jacob
    Report
    Sep 11th 2024, 1:33 PM

    I wish trump would clear up one thing? Are the Dems talentless, incompetent bunglers or are they complex, genius, masterminds?
    I bet the Trump supporters onhere believe they are both.

    27
    Install the app to use these features.
    Mute Tricia G28
    Favourite Tricia G28
    Report
    Sep 11th 2024, 1:58 PM

    @Louis Jacob: You just reminded me of this paragraph from an article I read recently about Trump supporters – Boebert took the opportunity to boast on Trump’s behalf about one of his favorite topics: his supreme intelligence—as evidenced by the fact that, as Trump loves to mention, he allegedly once “aced” some cognitive test.

    But here’s what Boebert actually said: “President Trump, when he was in office, underwent testing for his cognitive dissonance.”

    You couldn’t make up those types of slips…….

    42
    Install the app to use these features.
    Mute Louis Jacob
    Favourite Louis Jacob
    Report
    Sep 11th 2024, 2:22 PM

    @Tricia G28: Lol. I have to say I’m addicted to Boebert videos. She’s straight out of Monty Python.

    17
    See 2 more replies ▾
    Install the app to use these features.
    Mute Con Cussed
    Favourite Con Cussed
    Report
    Sep 11th 2024, 3:19 PM

    @Louis Jacob: Sounds like she’s a lot smarter than yourself!

    8
    Install the app to use these features.
    Mute Louis Jacob
    Favourite Louis Jacob
    Report
    Sep 11th 2024, 5:10 PM

    @Con Cussed: Good one.

    3
    Install the app to use these features.
    Mute Padraig O'Brien
    Favourite Padraig O'Brien
    Report
    Sep 11th 2024, 2:13 PM

    Taylor Swift, well, maybe. Let’s hope she doesn’t get endorsed by Bouncey, Pinky, Dolly and the other pretentious “feminists”. That would sink her with a lot of voters!

    18
    Install the app to use these features.
    Mute Jack Hayes
    Favourite Jack Hayes
    Report
    Sep 11th 2024, 3:47 PM

    @Padraig O’Brien: Ok. You’re one of those types of people ….

    31
    Install the app to use these features.
    Mute Robert Halvey
    Favourite Robert Halvey
    Report
    Sep 15th 2024, 6:43 PM

    It never stops amazing how so called alfa males worship every hateful lie the crying orange thing spouts out ,

    1
Submit a report
Please help us understand how this comment violates our community guidelines.
Thank you for the feedback
Your feedback has been sent to our team for review.

Leave a commentcancel

 
JournalTv
News in 60 seconds