The Q&A: "It's getting harder, not easier, to do this kind of work"

Pollster David Coletto on AI, fear of losing everything, and how hard it is to measure the Conservative vote

David Coletto of Abacus Data

David Coletto is the founder and chair of Abacus Data, a full-service public-opinion research firm that is most familiar to readers of this newsletter as a tireless producer of political polls. For the longest time, his regular horse-race poll reports would end with a paragraph or two of analysis from Coletto. But lately he’s been writing a lot more. His Substack newsletter, which he launched way back in 2021, has been in higher gear since mid-2023. He’s recently broken out two separate websites, abacus-weighting.com to discuss why the firm weights its samples according to past voting behaviour, and precaritymindset.com, about what happens when voters worry they could lose the things they hold dear.

Lately he’s been arguing that most pollsters have been underestimating the Conservative vote in election after election. But in Monday’s by-elections the Conservative vote was way down from where it’s been in recent years. This caused great hilarity among Coletto’s detractors online. I thought it was a good chance to ask him about the challenges of polling in general, and his responses to those challenges in particular. The transcript of this interview has been edited for length and clarity.

---

Paul Wells: I wanted to talk to you because you’re doing more writing than I remember you doing in all the years that you’ve been a political pollster. How come?

David Coletto: There’s just so much going on in the world. There’s no shortage of topics to write about, and I find sometimes that I’m constrained by just simply describing data and not being able to dig a little deeper to help explain — at least from my vantage point — why I think things are happening and what they mean. I’ve freed myself up in my day-to-day work to enable me to do it, partly because I’ve come to really enjoy it. It’s now become an important part of working through the data that I’m seeing — the responses to surveys and the interviews and the like that we’re doing, and the behavioral things that we see happening, whether it be in politics or in how people are feeling about what they’re buying. It’s really become an outlet to add more value to the research that I’m doing, and I’ve gotten a pretty good reception from it, so that means there’s a demand for this kind of stuff out there.

PW: You launched a Substack a couple of years ago. I happen to be very interested in Substacks. How has that experience worked out for you?

Coletto: I started it as a place that I thought would be my lab. On the Abacus website, we would release much more clean, descriptive types of polling releases, and on the Substack, it would be a place where I could play around with some ideas. That’s still very much what it is, but it’s also become a place where I could take a survey that I might have done on some subject, maybe go a little bit deeper, because I think the audience on the Substack appreciates that. They are looking for a longer form, more consistent kind of take on what’s happening in the world.

PW: You’ve even broken out two standalone URLs, which are: The Precarity Mindset, and Abacus Weighting. I think the weighting discussion has gotten a lot more attention this week. Some people online are making great sport of the fact that you’ve been arguing that the Conservative vote has been undercounted — and then the Conservatives face-planted in those by-elections the other night. Tell me about your argument. And tell me how it stands up to the week’s events.

Coletto: For the last three elections and beyond, every time I look at our final poll, and if I hadn’t weighted by past vote, I would have severely underrepresented the Conservatives and slightly overrepresented the Liberals. Our calls — which are important for pollsters, it’s the one time you can actually hold us to account and say, “Did you actually measure what you think you were measuring?” — would have been very wrong. For 15 of the 16 years that I’ve been doing this at Abacus, we have done some weighting based on the “past recall vote”, as pollsters would say. We ask people: How did you vote in the last election?

The thing I sit around worrying about is the question: Are we talking to the right people? Are the people who answer our surveys like the people who don’t? Are the people who participate in politics from time to time, fully captured in the way that we do our research? At Abacus, it’s mostly done online using a variety of different panel sources that we pull to build what we believe is representative of the population. Particularly for the last few weeks, I’ve been getting lots of commentary asking: Why is it that one pollster can say the Liberals are ahead by 15, and another says they’re ahead by 6? Those two things can’t possibly be true because they are substantially different.

I’m not calling out the other pollsters, more to say, this is why ours is what it is. It’s an effort to try to show a little bit behind the cloak. To show a little bit about the choices I’m making, and always thinking about, and obsessing about. I’m constantly asking: Are we doing this right? I’m not adjusting all the time, but I’m testing all the time. And what I think I’ve come to conclude is that this approach isn’t perfect. I do think we’re probably overweighting Conservatives. I think there are some people who likely voted Conservative who, today, are now Liberals, and are not telling us they voted Conservative in the last election. So, it’s not a perfect test, and I’m still refining it, but I’m not afraid to show my work a little bit, and I think this was an attempt to do it in a more detailed way than if I just wrote a 600-word post on our website.

PW: Give me the short version of what “weighting by past vote” means.

Coletto: When pollsters do a survey, oftentimes, the sample we get — say, a thousand people we’ve interviewed — isn’t fully representative of what we know the Canadian population is, and usually the things we know are from census data. So, what’s the distribution of age, and gender, and education, and where people live in the country? We can adjust, oftentimes very slightly, to match what our sample says of that population.

Right now, approximately half of the people we survey who say they voted in the last election say they voted Liberal. And about a third, or a little less sometimes, 32%, say they voted Conservative. Now, we know from the last election results that it should be around 44% [Liberal] to 41% [Conservative], with the NDP at 6%, and the Bloc at 6 or 7%. What I worry about is that because a sample source is structurally biased towards people not inclined to vote Conservative — when I use the word “bias” I don’t mean the way we talk about media bias, or how someone’s politically biased one way or the other — but that the sample source is biased.

Think of some of the people you might think of who vote Conservative. They’re probably not going to answer the phone and talk to a pollster for 20 minutes. They’re probably not signing up for panels and spending time sharing their perspectives. This phenomenon has been seen consistently in my data in Canada. We’ve always underestimated Conservatives. They’ve always been harder to reach. It’s true in the United States, and in the UK, and elsewhere. So, there’s something about conservative-oriented people that they’re just less likely to participate, and so “weighting by past vote” means we’re adjusting it closer to what that vote should have been.

There are different ways we can do that. We don’t use a full “heavyweight”, where we try to get that 44% – 41% because we do know, as time goes on, that people either misremember how they voted, or they don’t want to admit it. What I’ve seen… and maybe this is getting too deep in the weeds here… is that there’s a statistical relationship between how popular a government gets and how unlikely someone is to admit they didn’t vote for that government. We are now slightly adjusting for that, and that’s something I’m still testing out. But it’s showing up in the data that I have which tells me there’s no perfect way of doing this. But if I’m not adjusting for past weight, I think that I’m leaving out some voters who would otherwise vote Conservative in our estimates.

PW: Let me get a sense of the methodological world that you’re operating in. Out of 100 calls that you would make, how many people respond?

Coletto: If it’s a live caller, which we don’t do very much of anymore because it’s incredibly expensive, and depending on if you do callbacks and you’re using the proper methodology, you could get anywhere between 8 to 12 of those 100 [legitimate] phone numbers actually answering your survey. You have an 8-12% response rate. Less than half of people are actually even picking up the phone. Way less than half. Which is the biggest problem. We’ve now become a culture where just answering the phone is something we don’t do, even sometimes if we know who’s calling, let alone a random phone number that maybe Rogers or Bell tells you is spam, or likely to be spam. So, very low response rates now on telephone.

When you do an online survey, we don’t have response rates per se, because we are accessing what we call “panels”, basically databases of people that have been recruited in different ways. People who have agreed to answer surveys through email. We have access to a marketplace of panels that add up to about 400,000 to 500,000 people. We are blending those panels to create what we think is a more representative sample. What’s different between telephone surveys, typically, and an online survey is that the person completing the online survey also gets some form of incentive — some points or some money. And it’s being done, obviously, on a screen in a visual, written form, rather than audio through a phone.

PW: The hills that you and your colleagues and competitors are climbing up… I’m actually impressed that the polls are as accurate as they are, because it sounds like you’re living in a methodological hellscape. It sounds like you’re all like the makeup artists who are prepping Stalin’s corpse for the memorial service. Does it ever feel like it’s impossible work?

Coletto: I started Abacus 16 years ago, and it wasn’t quite the golden age of polling that people who are slightly older than me describe, when you just literally needed to call a random sample of phone numbers, and you got very accurate results because probability theory says that that should work. That no longer really applies, unless you’re Statistics Canada, and you have an incredible budget to do this research methodologically really, really well, which they can do. I don’t have a client who has $200,000 or $300,000 to spend to collect data. So we try to make decisions that don’t decrease the quality of the research but make sure that the return on that investment is worth it. In a census, you need to be accurate within percentage points, because a one-percentage-point difference between the population X or Y means big things for government funding, for the rise of certain kinds of behavior. That’s not the case when it comes to most of the research we do. If I tell a client, your positive reputation is 85, but let’s imagine in reality it’s 82, what’s the material difference in the decision that the client’s going to make? It’s still positive, and most people have a good view of them.

But to your question, yes, it’s constantly becoming more difficult. Because, first, technology is changing rapidly. Before, we were up against people not answering the phone. Then we’re up against people who become more fragmented in the ecosystem of information that are becoming harder to reach. Now we’ve got AI, and so-called, synthetic polling, that is going to be competing with real people’s opinions. I think there was a story in Axios, in which they reported on a poll that was completely generated from AI agents acting like real people responding to questions. Two smart profs wrote in the New York Times that if we go down this path, humans will never need to be asked questions again. We will just let the machines tell us what we think. I wrote a piece saying that I do think people need to be part of polls.

There’s all this change happening, and the thing that keeps me up at night is the question: Are the people answering our surveys the same as the people who aren’t? Because if they aren’t, then we’ve got a problem because the people we’re talking to are not representative of the rest of the population. I know that everyone in my industry, whether here in Canada or around the world, is seized with this question. We are always trying to adapt and learn, and there’s different methods that we can adjust. But yes, it’s getting harder, not easier, to do this kind of work.

PW: All the time you’re doing this, you’re getting a constant barrage of assumption and criticism from people who say that if the effect of your work is to nudge the Conservative vote higher, it’s because you’re a Conservative fangirl, or you’re getting paid by the Conservatives, or you’re trying to build a conservative world. And vice versa for the larger number of your competitors who are showing fantastic numbers for the Liberals. The armies of online Statler and Waldorfs must be just a constant.

Coletto: Yes, that’s true. After I released this piece on weighting, a number of commentators on the right were saying: “Aha! The conspiracy has been unveiled!” And that’s not why I did that. Nor do I think there’s a conspiracy. I don’t believe that my colleagues in the industry are choosing to do what they do because they are Liberals, or they want to cheer on the Liberals. I think we all have our methodological frameworks in which we come at these things. And we’ve done things for a long time, and so that criticism happens every day. There’s not a week that goes by in which someone [doesn’t say], “Oh, David, you’re a Liberal” this week, and then “a Conservative” the next week. I’m mindful of this.

The reason I care about it is that I don’t want us to be surprised or shocked when things happen. I remember the night of the 2016 presidential election. There’s this image of a woman at Hillary Clinton’s “victory” party. Her hands are on her face, and she’s in shock when she sees the results. I think most of the world felt that way because the polls were systemically and structurally missing a group of voters that voted for Donald Trump. Most American pollsters — up until recently — never weighted by education. Which we know is a strong correlate to how people vote. We’ve been doing that for many years.

The reason I do this work is that I want us to have as accurate a sense of the world as possible. Now, I could be wrong, right? I think the by-elections definitely prove that the Conservatives are having a difficult time either convincing people to vote for them or motivating their base. And those are two different problems. The question, I think, is this: Is there still latent support for the kind of politics Pierre Poilievre represents? I believe there’s this internal debate that I watch as an objective observer, that says, what should Poilievre do? Should he be more like Carney, or should he be more like Poilievre? I don’t know if there’s an obvious answer, given the way I see the polling and some of the results in those by-elections. But that’s a separate conversation.

PW: How much of your business is polling on partisan political preferences?

Coletto: Oh, I make no money off that. I won’t do any partisan political polling at the federal or provincial level. I’ve done some municipal work. This is a personal interest, and it’s marketing for our firm. It’s how we get our names in the news, and how we stay top-of-mind to those people we actually want to buy our research, which is primarily associations and unions and corporations who are trying to understand how their audiences are thinking. Politics is just a way that we look at the world. Now, I do use a lot of the research I do about politics and public policy to help inform my clients, but none of them are political parties or political leaders directly.

PW: Let’s talk about this “Precarity Mindset,” which I think is your way of reminding the elites that there’s another Canada. Is that a good way to put it?

Coletto: Yes, but it’s also my way of trying to create a framework to understand and explain why we are where we are today. Think of the Precarity Mindset as a conceptual framework to understand where most people in the country’s heads are at. It really became apparent to us — and my colleague Eddie Sheppard, who coined the term in our team — because we had been describing a scarcity mindset, post-pandemic up until basically the end of 2024. There was an almost universal feeling people had that the things they needed in their lives were harder to get, more expensive, and if they had them, they were worried they were going to lose them. This is where inflation bites. This is where rapid population growth had its impact on housing prices, on availability of access to healthcare, and so people became very zero-sum thinking in their way. There was this perception that they were fighting, even if most Canadians actually had nothing to worry about, because they were pretty well off.

And then Donald Trump comes along, again, and that shifts rapidly. We went from seeing responses to a question we ask on every survey — What’s keeping you up at night? — being things like cost of living, housing, affordability, which are still very much front and center to people today. But the answers to that question shifted to Trump, to tariffs, to Canada’s sovereignty, and then in 2026, it basically became Venezuela, Greenland, Iran, and Trump, Trump, Trump. The Precarity Mindset is basically moving from a world where most Canadians were asking the question, “Will there be enough?” to, “Will we be okay?” And it changed the calculus.

Now, this is not everybody. And this is why I think the last election so perfectly showed us the divide in the country. I think largely those people who voted for Mark Carney and the Liberals were voting because they were worried they were going to lose something. They were going to lose Canada’s sovereignty. They had something to lose. They were the Baby Boomers who — particularly the men — voted for Erin O’Toole and Andrew Scheer, who despised Justin Trudeau, and then en masse shifted over to the Liberals, because Mark Carney was going to protect the equity in their home, or the value of their investments against the uncertainty of Donald Trump.

But there was another group that doesn’t have a precarity mindset, but are living in precarity, which is very different. They are people who have nothing to lose. They want a disruptor. “Bring it on. It can’t get any worse than this.” And I think that is one of the divides in Canada right now that has been muted because of the Trump threat. It is still very much underlying the mindset of how people see politics and their choices right now.

PW: And then you bring in the writings of Kahneman and Tversky, the Israeli economists that Michael Lewis wrote about, whose argument, I believe, is that people are more worried about what they might lose than excited about what they might gain.

Coletto: Yes. And the loss aversion, the feeling of risk, became so profound. And different than I think many people had experienced in a long time. It wasn’t that people actually thought that Canada would become a 51st state, or that Donald Trump would invade us. It was simply the complete… upheaval of what people took as normal and that became very, very threatening to them. The simplest way to see it, and research confirms that there always has to be a villain in the story, and the villain is the thing that threatens us.

We released some data earlier this week that showed almost half of Canadians say either the first or second most threatening thing to their quality of life right now is Donald Trump. And 70… or 65% of them would vote Liberal, and love Mark Carney, because they see him as really the antidote to that chaos. There’s this sense of… not quite impending doom, but the fear of losing all the gains people think they’ve made, and they don’t want to lose, and they want somebody to protect them.