Can You Trust a Poll No Human Ever Took?

Can you trust a poll when no human ever picked up the phone? Eric sits down with Matthew Hanauer, co-founder of Civly, to unpack synthetic polling — building AI simulations of real, individually-modeled voters (using public voter file, demographic, and donation data) to predict how they'd respond to a survey. Matt walks through how Civly back-tests its models against real election results, why asking AI yes/no questions produces unrealistically extreme answers, why piling on more data (like social media) can actually make predictions worse, and how turnout modeling benefits from more than just "did they vote last time." Eric and Matt land on where AI polling actually fits into a campaign's toolkit in 2026: filling the gap for downballot races that could never afford a traditional poll, and pressure-testing messages before committing budget to the real thing. 

Visit our website: CampaignTrend.com

Transcript

Copy Transcript Copied

Eric Wilson 0:03

Welcome to the Campaign Trends Podcast, where you're joining in on a conversation with the entrepreneurs, operatives, and experts who make professional politics happen. I'm your host, Eric Wilson. Joining me today is Matthew Hanauer, co-founder of Stivly, a campaign intelligence platform built around AI research. Matt's background is in research methods, uh, his PhD, and a lot of his work lately is on synthetic polling, stimulating an electorate of individual voters, and then seeing what those uh voters have to say and comparing them to actual election results and polling. He recently ran a synthetic poll on the closely watched 2026 Maine Senate race, uh, and we'll get into some of the results from that and and see where AI polling can fit into our campaign toolbox. So the question with we're really wrestling with today is when can you trust a poll when no human uh ever picked up the phone? And we've been talking about this a lot on the podcast, and I'm I'm really fascinated uh about it. So uh excited to talk to Matt about that today. Matt, before we get into the main numbers, let's kind of give people an orientation who may not be familiar yet with the synthetic poll or AI polling. And you didn't talk to a single voter. You built a simulated panel of, let's see, 4,597 real main registrations, and then you asked AI how each one would lean on the the survey instrument. So walk us through what's actually happening there. Um what what is what's driving that that that vote, if you will.

Matthew Hanauer 1:50

Yeah. We have a real state voter file. So one of the things I want to differentiate, because you know, with sometimes I think Nate, we've been thinking about the name and whether that's a misleading name. We aren't making up stuff about people. We aren't just asking AI, what do you think? And we are we are giving it real information about, like you said, real registered voters. So we're giving it their demographic information, you know, any uh education, um their back if they had make any donations, and in some cases, things like social media. Um we we partner with L2 on this, and then we have our own big list of different data sources, and we put together basically a dossier of a person, a bunch of information that is publicly available. That's another important differentiation I want to bring out. Uh, we don't try to hack into anybody's private accounts, and we don't try to hack into your text messages and stuff. And I'll talk about why that is could be a weakness of ours as well, but it's an important weakness that we will continue to have. Um so this lot relies on people's public information. And so we put all that in there, and then what we do is we ask AI, hey, you know, given your information that you know about this person, how do you think they would vote in whatever question we're asking, or how do you think the they would respond to this message? And in some cases, we can uh be more accurate in others. For example, when there is a clear um answer, like when there's a 2024 election or when there's a previous election when we're doing a horse race, we can do back testing and we can say, hey, um we give them we give them the information. We only give information that happened from you know that that year or before. We don't want to uh uh dilute it or or uh poison it with information that happened in the future before that back test. And that's something that that folks will do. And when you get really, really accurate results, like you're 100% right, um, that means that you did something wrong. Every data scientist, AI person who who has been doing this for a while knows when you have a hundred percent accuracy that you screwed something up. Um no one's at no one's ever that accurate. Um so we take that information, uh, plug it in there, and if we're doing like a back testing, which is basically just seeing can we predict the past? Because that is how these models, all of your data science traditional ML models, that's that's the way they do it. There are some weaknesses to that because we know things, you know, 2024 is not the same as 2026, but that is, you know, that's the best way we have to be able to say, hey, you know, when I did this in 2024 results, I was, you know, for the main one, we had a we were plus or minus four points off from what happened before. Um and so we're able to get a level of accuracy, which is different. Um, one of the other things that I've noticed uh gets a little confused is the difference, what what a margin of error is, and then kind of what our accuracy is. So the margin of error is really just a function of how big your sample size is, and it says how precise you think this estimate is. It doesn't necessarily mean you're gonna be accurate. So it kind of the way I explain this is I play tennis. Um, I can be really, really precise at hitting the ball in the net. That that is not good. If you haven't played tennis before, you want to hit the ball over the net. Um and I could I could be very precise at that, but I would be wrong and lose all the time. What we do is we try with these back testing, we try to see how accurate are we, how accurate were we to the actual result. And so for the example for that, if I'm serving, we are pretty good, very we're very good at hitting it in the box. Sometimes I hit it to the left side or the right side of the box, but we are we are very good at getting it in that box at some point. And sometimes we're a little bit off, but generally, like for example, we're in the right direction. Uh, we did um some analysis for um where we posted some of our results for New York and Maryland. And in that example, we were nine out of ten. We uh we got we got the one wrong that I think most people did. Uh it was uh one of the ones that was a surprise in New York. Um, but we've uh yeah, go ahead. Yeah.

Eric Wilson 5:51

So uh here here's something that I learned from Ben Leff at Verisight, who who wrote a paper on this, as you're probably familiar with, and we had him on the the podcast that AI polling really struggles when it's current events. And I can't think of a race that has more turmoil in it than the main Senate race um and and and the different uh different candidates. But what you were doing is you were sort of taking these, I guess like nine candidates that could potentially you know run against Susan Collins, then looking at the the the previous elections um using the same model. Uh you you know one I guess how how do you know that this method is capturing reality um and not just getting lucky a few times? It's like the the the 2020 um race, you know, I think your model had it at plus one, Collins actually won it by nine. So I I I think that that that's something we gotta address.

Matthew Hanauer 6:53

One other thing, when we did our kind of big back testing, we did the like the 2024 election and some ballot measures, we did that for I think 293 different counties. So basically we did 293 different tests to be able to see how accurate we were. And so for these, I try to do you know as many tests at as many counties and try to predict as many counties as we can to see how accurate we can get. So we try and do not just, hey, we got one thing right, we try to see can we get all these different counties right? But one of the big things um we found is um the way we we the way we ask the question. When we go through, we ask it, you would ask several questions in a row. And when we ask the question yes or no, one of the things that these models do is they get really, really extreme. They get, you know, 100% yes and 100% no, especially on really, really hot topics.

Eric Wilson 7:42

Yeah, which is not how actual people think about things. Yeah.

Matthew Hanauer 7:46

Exactly. Exactly. So one of the ways we've been able to help and get around that, and that's why we've been able to get some more accurate results, is we ask AI on a scale of one to 10. And then we start to get some variation for folks, and then that ends up averaging out and getting us a little bit closer to reality. But yeah, when we get into extreme situations, uh, another one I did last night, we were, um, and I'll talk about this in a little bit, we were looking at the North Carolina race. And one of the new things I'm wanting to do is um not just ask it one model, but I want to ask it from different models because every model has its own worldview and its bias. Like I like Claude. I think Claude tends to go, you know, to my worldview and my biases, and it's great for what I do, but I, you know, not everyone uses Claude. And in fact, most people don't use Claude, they use Chat GPT. Um, and so I think it's also really important and can help us get more accurate and get different types of worldviews and get to different types of current events in there by asking different models. So when we asked it, you know, who was gonna win the governor's race, um, I think what whatever race Mark Robinson was in, it was like Mark Robinson was gonna lose by a landslide. And he did lose by 14 points, but the model was predicting like 68, he was gonna lose by 68 points or something. And then other models, which was interesting, other models that we ran with like Deep Seek and some other ones, had him losing like by five or six. And and so one of the things I found, you know, doing this recently is is it can help when we get some of these current events or these volatile events, if you run it across different models, and then it can give you an early indicator of maybe the case that it's just too volatile. Like in that case, we I probably wouldn't have put the poll out for North Carolina because I had models all over the place. They were ranging from minus 68 to plus five. And so in that case, I would either need to do a lot more work or just say, hey guys, this isn't this isn't this isn't going to be for us for this model, it's too volatile. Yeah.

Eric Wilson 9:41

And I don't want to be too hard on the the AI polling uh um thought because I I think it is telling us something. The challenge is for those of us in the field working on campaigns is we just don't know what it's telling us yet. Um I think sort of the same way about um prediction markets, right? There's useful data in there. We just aren't sure um how to use it. And I think it's particularly useful in this case where you're you're back testing it on election results. Of course, it gets really difficult when we're trying to back test it on messages and opinions rather than than behavior. This comes up in every conversation I have with with AI, where on one hand we say, look, you know, we can't afford to do a poll. And on the other hand, hey, talking to real people over the telephone is the gold standard of polling, and you know, it's gonna cost you a lot of money to do that. So the the decision in reality is not uh AI poll or a a human poll, it's a human poll or no pole, and that's where kind of AI comes in. And so um where do you think it should fit in? I guess right now. Yeah.

Matthew Hanauer 10:56

Yeah, so where we're really excited about it is for down ballot races where it just it wouldn't be feasible, you couldn't get enough people on the phone. Um, and so for folks like you said, the no poll or poll, we really think this can be really helpful because there are some limitations with real polls. You have to get people to pick up the phone. People don't want to pick up the phone. And the people who pick up the phone are specific types of people. There are response biases, and this this has been a problem in polling since the beginning of time. Uh the people who pick up the phone are systematically different than the people who don't pick up the phone. With the polling we're doing, everybody has to pick up the phone. They don't have a choice. They're they're getting polled randomly. The neat thing about this too is we can pull any sort of when we can get really, really deep into the crosstabs. Like we can go four or five levels deep. Whereas with the traditional poll, that's really, really difficult and really, really expensive. So I think there's two things. One is the simple one, down ballot races would never be able to afford the poll. They would never be able to do the message testing. And so we can do things like some of this back testing to be able to say, you know, whether or not we feel confident or this is going to be accurate enough. We can always choose not to do the poll. Like if we do the back testing and the analysis and we see that we cannot get accurate results, we can always choose not to do it. Um, so that's one area where we can we can help down ballot races. The other area is to supplement and complement. It's really tough to get good random samples of all kinds of different people. You have to weight them differently, and you do a bunch of statistical analyses that would you really don't want to have to do, but you have to to be able to get those samples equal. And so instead, we can just sample how we want, and then we can ask as many questions as we want. One of the neat things about these polls is uh we our respondents don't get tired. They don't, they don't, they don't have any fatigue, they don't have anything like that. And so we can ask them as many questions as we want. And so someone who wants to test 30, 40 different messages, which we've done before, because the first 20 didn't there was wasn't anything there, and then we thought, oh, let's try something else. And then we were able to find like one of the things is for a candidate, we went back to their um pre uh uh pre-politician history, and they they were a nurse and had some healthcare background and did a lot of stuff, and that stuff really pulled well. We didn't do it in the first round because we didn't think about it, but because it was relatively inexpensive and quick to be able to do that, we did that, and that polled really well for that individual, and so that's what that's what we kind of put forward.

Eric Wilson 13:30

Yeah. I agree with the the the critiques of traditional survey research. I'm just very skeptical about how we measure all the various inputs that people get from media and and their information diets, which is is is totally unique to them, which was not the case when the methodology was established. Um so I I'm with you on that. Uh this benchmark that you're using then back checking on the previous election. So my question for you is if the the AI model, the synthetic sample nails the the sort of election result previously, what kind of confidence should we have that it's getting the new forward-looking messaging stuff accurate, or those totally different vectors that we we shouldn't read into?

Matthew Hanauer 14:23

Predict when I used to work in healthcare, we would try to predict to be able to predict what someone's um costs were gonna be the next year, we would use the previous year. And so there are issues in that too. I always gave the example I hurt my wrist one year and my healthcare costs went up. Um but you can you can include those sorts of things in your model where, oh, I know Matt is young, he plays a lot of tennis, younger, I guess I'm not as young as I used to be, uh plays a lot of tennis. Um, you know, he does it, his history of injuries and in chronic issues hasn't been a case. So we know from his entire profile, even though there was this one blip, that we know that we're not gonna over-index on that one blip, and we'll probably assume his cost may go up a little bit, but not a lot for the next year. And that's kind of the same concept we can do here. We can build in if we have lots and lots of years of history of data to be able to see how these trends ebb and flow, we can help build some security in what might happen in the future. But at the end of the day, yeah, it's the best we have. And if there are huge shifts and we haven't, we don't have a variable in there that is accounting for that, then then we could be then we could be off by more than we would like.

Eric Wilson 15:31

Yeah. You're listening to the campaign trend podcast. I'm speaking with Matthew Hanauer, co-founder of Cively, about real real-world applications of AI research. You did something really interesting here, which I uh you call I think called the honesty adjustment. Because uh and and and so you you do anchor things to a real poll. Can you just talk us through that and and you found something interesting, which is adding more voter detail made you less accurate. We have consistently seen this now, where it's not necessarily the the straight line of adding more data gets you better results. So walk us through both of those things.

Matthew Hanauer 16:14

That that was surprising. Um because most of the time I've been doing this for a while, and I've been doing it in other fields like healthcare and stuff. Um it's rare to see that more data made it made it less accurate, but the I've seen that a couple of times since I've been now in politics. I haven't been in politics super long. I've been in politics maybe, you know, since the start of this company. I've been predicting uh healthcare outcomes and things like that. And that was pretty rarely the case that more data didn't help. Usually what happens is more data just doesn't really add any value. Um, and so you know it doesn't make sense to really add it in, especially if it's missing all the time. Um so I I actually bringing that up, that is one of the cases where we did see um data, uh uh more data make it less accurate. If you have, and this is kind of what happened with social media data. One of the one of the cases we added a bunch of social media data and it made it worse. And so for two reasons that could have made it worse. Most people don't have social media data. Um, I know we we talk about it a lot, we think about it. Most people aren't posting all the time on Twitter, Facebook, things like that. And so if you're missing a bunch of data, what it'll do is really overweight um those people who do have it and really um amplify what what they're what they're saying. And and also um what I'm finding is with social media data, and this isn't a surprise, people aren't always honest um on social media and they tend to uh have a type of persona. So even though we think, yeah, all of this scraping and all of this data can really help. What I found in in politics, and frankly, what I found a lot in when I was doing healthcare is it generally comes down to seven or eight different factors that account for about 90% of the variation comes from. And really, for politics, it's your basic demographics, education, um race, things like that that tend of your voter registration, who you registered for, um, where you live, things like that. That tends to predict about all of it. Um donation history can help a lot um if we have it, because that can kind of tell if you're registered one way but you're donating another way. Maybe you just haven't updated your registration. Um but those are kind of things that can help and you can think about um with with uh when you're when you're doing this type of analysis.

Eric Wilson 18:31

Yeah. I I think that's uh such a fascinating point because you're um you you would think that knowing more about a person will help you help you understand them, but in in this case it just does not it does not add up. And I I really do think it is because the the you know the political decision making is is really kind of core to who you are and and who you are doesn't doesn't change. The the big challenge though has to be figuring out who's gonna turn out. I mean that that's a challenge for traditional modeling, traditional polling. I saw there was a study recently about you know the 2026 New York primaries compared to the turnout in 2025. And you know, a lot of people who turned out for Zoran Mamdani um did show up, but a part of it, and you couldn't really predict who was gonna show up. It's just I think that's the big wrinkle in all of this is the the the randomness.

Matthew Hanauer 19:29

Yeah, I'm looking at so we actually did some analyses on turnout, so we took um a lot of the way I've seen in politics and what I've heard consultants say, and and this is not a bad way to do it, but you brought up why it can be problematic. Usually that seems like the rule of thumb is um someone's gonna turn out if they voted in the last two elections. And I think that is a decent, solid, like rule of thumb, a mnemonic kind of to go by. But we actually looked at that, and there was um the there was a metric in some of our data that that L2 has that is kind of very similar to that. They take the percentage of how many people, uh how many elections they were eligible for, and then how many they voted in, and that was their metric. And it misses out on people like you were just talking about who come out every once in a while for a big race or something like that. And it also misses newly registered voters who are really excited to vote. So what we did was take um the same model we took, except for we didn't, we didn't use AI in the sense of I didn't throw it into a large language model. I used a traditional uh uh gradient, they call it gradient boosty model, it's just a traditional statistical model. And so we threw that in with all the with all the factors that I was just talking about, and it was the same kind of factors, those five, 10 different factors of demographics, education, location, and we along with did you turn out in the last elections, like that percentage? And then we put that in and we were able to improve that accuracy. I'm looking at my results over here about six percent in general election, and then in low turnout models, we were able to really improve by about uh 13, 14 percent. And so I think um we can get more accurate with turnout. We just need to, we just can't rely on did they come out the last election? That is definitely an important factor, but there are also a lot of other factors in there that that can help us predict that better.

Eric Wilson 21:18

Yeah, that's that's one of the reasons I've stopped talking about low propensity voters and instead call them infrequent voters because it it's not that they are um disengaged, it's that just voting might not be convenient for them. And so whether they do their their their civic duty or not depends on you know what's the weather, like what's their schedule, what have they got going on. Yeah. One one interesting thing about the social media data that I think is worth pointing out social listening has been something that people have been selling me for decades. And and we know about participation inequality, right? Which is the idea that for any given social network, you've got about 10% of people who are creating the content and interacting with the content and Creating those signals. But 90% of people are just consumers, the sort of lurkers. And so if you are adding in social media data back to your model or maybe relying on that more for a model, what you've done is you've just recreated the response bias of the people who pick up the phone. It's the hardest thing we have in polling right now is finding those people who don't want to share their opinions yet still vote. And so this is one of the ways to kind of imagine what they look like.

Matthew Hanauer 22:27

Yeah, I think this is probably our best way. It's really tough to get at those people. And like you're exactly right. I hadn't thought about it that way, but yeah, the social media uh adding that in is kind of the same exact thing, like you said, of just of just feeding that response of these are the people who engage. Um, and you know, so I I think social listening is really important. It's something that we do at our company, but we do it more for um thing was said today that could help your candidate. Maybe somebody said something on a particular topic that is important for your candidate and you can use it in whatever race you're doing. So I find it more useful for that as opposed to I haven't used it really much in any of my turnout models just because it's only it's so infrequent and it's I don't know how honest it's being. Yeah.

Eric Wilson 23:10

So here Matt, here here's kind of my current model of how to understand AI polling in campaigns, if I'm a campaign manager. And I want to get your honest assessment of this, not just as someone who sells this, but as someone who understands m research methods. But it seems like I I start from the position that we we don't yet have enough pattern recognition to know, hey, this this model is telling us this, and it's typically off by about this. And in the way that we know this about surveys, or hey, the responses, you can look at a crosstab and say, Oh, that that Republican lean is not quite right, and you know, in a subgroup. So but we so we have that pattern recognition. Uh I I think that that AI polling can tell us something. Again, we just don't know what exactly. So the two use cases I really see this for are bridging the gap between budget, right? So you're still gonna want to try and do more traditional survey research, but in the times that you can't do that, here's here's a solution. And then two, for sort of testing out that survey and getting a read on it before you push it into the field. Because I I think a lot of people burn up budget on questions that come out, well, it's 50-50. And it's like, well, why don't we spend all that time? And so there's an efficiency gain there. Do you think those are probably the two best ways that campaigns in 2026 should be using uh AI polling?

Matthew Hanauer 24:48

I think there's another one, but yeah, the the second one you just described, the pre-poll. We uh we did that for a client. Yeah, that was exactly what we were trying to do is help them because there's a limited set of things you can test. And so we were doing like that pre-poll cheaper. So we we limited it down to okay, here the example I was giving you earlier where we went through like 50 of them so that they could ask 10 to 15 messages that we had some confidence and some data on, that those would be ones that would be important for them to ask. So that's definitely one, yeah, the the down ballot folks who can't afford it. I think um the follow-ups, like if you do 10, 15 messages, um then, you know, and you want to do more and you want to do that follow-up, but you just like you've spent all that money, it took all that time, and you didn't quite get what you wanted, but you want to keep following up those messages. I think it can be helpful for that if if you're if you're doing that in a time crunch. The thing that one of the things that's always gonna be a weakness for this is what we talked about at the beginning. A lot of people are in group chats. They're on their group chats on their phone, they're on their text, and that's like so that's where you know a lot of our conversations happen. I've mentioned I played tennis, I'm in a group of, and that's how I got started with this. Um, I'm in a group of maybe 200 guys of uh about the same age, same level, and we have tons of group chats and all this stuff, and I'm on the court with guys, and we're at wing night and things like that. And now those types of conversations where you know real honest conversations are happening about how people feel. I'm not gonna get that with AI because it's not capturing that. That stuff is happening in person, and that's where polls, real polls, that's why they're not gonna go away. Real polls are gonna be able to ask people how they're feeling, and those types of feelings can come out when you're asking them about that. And we will, if people are capturing that, then they're doing something fundamentally illegal, and they should never be doing that because that that needs to stay private. So that's why polls are always gonna be around.

Eric Wilson 26:39

And and I think this lands us in a really interesting spot, which is where we come back to with a lot of AI, which is we're never going to take the humanity out of understanding humans or creativity or you know, the voice uh uh of say a candidate. But what we can do is make the the time being spent by humans more efficient. So in this case, we're only gonna hopefully be going out and surveying the human beings for the things that only they can tell us, like what's your opinion about this thing, and we can rely on the AI for for some other uh elements. So, with that, I want to say thank you to Matt for a great conversation. You can learn more about his work and Sively in the show notes, including some links to some of their research, and sign up for a demo, learn more about them. If this episode made you a little bit smarter or gave you something to think about, you know that all we ask is that you share it with a friend or colleague. You look smarter in the process, they get smarter, they find out about the show, it's a win-win all around. Remember to subscribe to the Campaign Trend Podcast wherever you listen to podcasts, so you never miss an episode. And you can visit our website at campaigntrend.com for even more. With that, I'll say thanks for listening. We'll see you next time. The Campaign Trend Podcast is produced by Advocacy Content Kitchen, a media production company.