What Went Wrong With Polling? Some Early Theories

The poll results among seniors are another symptom of a deeper failure in this year’s polling. Unlike in 2016, surveys consistently showed Mr. Biden winning by comfortable margins among voters 65 and over. The final NBC/WSJ poll showed Mr. Biden up 23 points among the group; the final Times/Siena poll showed him up by 10. In the final account, there will be no reason to believe any of it was real.

This is a deeper kind of error than ones from 2016. It suggests a fundamental mismeasurement of the attitudes of a large demographic group, not just an underestimate of its share of the electorate. Put differently, the underlying raw survey data got worse over the last four years, canceling out the changes that pollsters made to address what went wrong in 2016.

It helps explain why the national surveys were worse than in 2016; they did weight by education four years ago and have made few to no changes since. It also helps explain why the error is so tightly correlated with what happened in 2016: It focuses on the same demographic group, even if the underlying source of the error among the group is quite different.

Polling clearly has some serious challenges. The industry has always relied on statistical adjustments to ensure that each group, like white voters without a degree, represents its proper share of the sample. But this helps only if the respondents you reach are representative of those you don’t. In 2016, they seemed to be representative enough for many purposes. In 2020, they were not.

So how did the polls get worse over the last four years? This is mainly speculation, but consider just a few possibilities:

The president (and the polls) hurt the polls. There was no real indication of a “hidden Trump” vote in 2016. But maybe there was one in 2020. For years, the president attacked the news media and polling, among other institutions. The polls themselves lost quite a bit of credibility in 2016.

It’s hard not to wonder whether the president’s supporters became less likely to respond to surveys as their skepticism of institutions mounted, leaving the polls in a worse spot than they were four years ago.

“We now have to take seriously some version of the Shy Trump hypothesis,” said Patrick Ruffini, a Republican pollster for Echelon Insights. It would be a “problem of the polls simply not reaching large elements of the Trump coalition, which is causing them to underestimate Republicans across the board when he’s on the ballot.”