How accurate were election surveys?

Scientific statistical polling applied to the political landscape has its origins in the early 20th century. A famous episode is narrated from 75 years ago in which for the US presidential election in 1936, a survey was conducted by Literary Digest (a major publication of that time) with close to three million readers participating in the postal ballot.

The contest was between Landon and Roosevelt for president of the United States. The digest suggested that Landon would win. However, Princeton trained Dr Gallup used a far smaller (but statistically chosen random sample), predicted that Landon would lose, and Roosevelt would win and went on to also predict by how many percent the Literary Digest would be wrong in its prediction.

When the election results came, the non-random sample of the Literary Digest was wrong and the few thousand strong sample by George Gallup was not only right about the horse race but also about how wrong the media survey was. This was the start of political polling in the US and in the coming decades political polls became part and parcel of any political campaign and discussion. Pakistan got its first statistical sample political poll in 1979 when Dr Ijaz Gilani, trained at MIT USA set up what was then called the Pakistan Institute of Public Opinion. Since 1979, political polling has gained currency in Pakistan slowly but surely.

Together with political polling came criticism -- some substantive, others purely hypothetical and speculative. The key question asked is: are pre-election surveys able to predict election outcomes? If yes, how accurately? I as a pollster get asked this question a lot and my response is: Yes they are able to predict the elections significantly better than any other tool or instrument available for political analysis.

For any pollster, Election Day is like the Day of Judgement -- you pass or you fail. How has General Election 2024 been for political polling in Pakistan? As I explain below, it has been a success and a triumph of not just Gallup Pakistan as a pollster but also a triumph of the scientific tool and method we and many others across the industry apply. Let me explain.

In January 2024, Gallup Pakistan released on its website a report titled ‘Political Weather Report One Month Before the General Election 2024’ (available for public scrutiny on the Gallup Pakistan website).

The report made the following key assertions which were proven correct on February 8, 2024.

Punjab: The PTI and PML-N were in a tough competition in Punjab, 34 per cent and 32 per cent of the sample survey respondents saying they would vote for the two parties respectively. The General Election 2024 results show that the predicted outcome was razor-thin close to the actual outcome. PTI-backed independents got more votes than the score was 35 per cent for the PTI and 34 per cent for the PML-N.

The same survey suggested that, despite significant efforts, the PPP was unable to raise its vote bank and the study suggested a 6.0 per cent vote bank for the PPP which has also been proven correct. Lastly, the report said: “Survey results show that Punjab vote bank reflects return to 2018 General Election political landscape”. This has proven to be correct as well as the share of votes that the top three parties has received is almost the same as what they got in Punjab in the 2018 general election. In 2018, the PTI was 1.0 per cent ahead of the PML-N in Punjab.

Khyber Pakhtunkhwa: The report suggested that KP has an overwhelming support for the PTI (45 per cent of the survey respondents said they would vote for it), and trailing much behind would be JUI-F at 15 per cent, PML-N (9.0 per cent), PPP (7.0 per cent) and ANP (7.0 per cent). The election results match almost completely within this prediction. The PTI has received 44 per cent votes, JUI-F 15 per cent votes, PPP received 6.0 per cent votes and ANP got 7.0 per cent. The prediction and the election result is within a 1-2 per cent margin.

Sindh: The report suggested the PPP would be the single largest party in Sindh with 44 per cent votes. The current election data shows 44 per cent of the voters in Sindh have voted for the PPP.

In other words, the electoral outcome predicted by Gallup Pakistan matched the actual outcome.

This however does not mean that all public polls are able to predict accurately. Polls like any other research can be professional or unprofessional. The level of predictability depends on adherence to the correct method, observance of professional integrity. We therefore encourage readers to be ‘sophisticated poll watchers’ who might decipher between good and bad polls. The judgement must be done scientifically and some of the key things to look at in a good poll are: working of question, sample composition (is it representative of the various social and demographic groups within the society it is claiming to be representative of), the sample selection method, and lastly if the firm doing political surveys has any track record of doing surveys.

Survey research is a science and it can only be done and supervised by those who are trained in this field. All scientific surveys are prone to errors but their error is measurable. Pollsters should have the courage to face reality when predictions are in error and celebrate when they are on the dot. Fortunately this prediction falls in the latter category. The overall electoral outcome is remarkably similar to the predicted ratios at the macro level. Any irregularity at the macro level can thus be addressed on a case-to-case basis.

An important implication of pre-election surveys and election data broadly matching is also a vindication of the election’s integrity and transparency. For the 2024 general elections in Pakistan, this source of evidence should also be used and this author’s personal assessment is that Election 2024 did reflect the broad will of the nation and the trust it posed in the parties.

There might be irregularities at the constituency level and those must be investigated under law thoroughly and to the satisfaction of all principles of justice and law. However, claims of widespread stealing of the people’s mandate are not proven through pre-election surveys.

The writer is executive director of Gallup Pakistan and holds a Masters degree from the School of Oriental and African Studies (SOAS), London.