Mystery AI Hype Theater 3000

Crunching the Numbers (with Decca Muldowney), 2025.10.20

• Emily M. Bender and Alex Hanna • Episode 65

Share episode

So-called AI tools are increasingly infiltrating newsrooms, particularly when it comes to data analysis. DAIR writer-in-residence Decca Muldowney joins us to discuss the need for journalists to distinguish between "AI" and reliable, verifiable research methods.

Decca Muldowney is a journalist and writer who was our fact checker for The AI Con. She's also a writer-in-residence and web editor at the Distributed AI Research Institute.

References:

Fresh AI Hell:

Check out future streams on Twitch. Meanwhile, send us any AI Hell you see.

Our book, 'The AI Con,' is out now! Get your copy now.

Subscribe to our newsletter via Buttondown.

Follow us!

Emily

Alex

Music by Toby Menon.
Artwork by Naomi Pleasure-Park.
Production by Ozzy Llinas Goodman.

Alex Hanna: Welcome everyone, to Mystery AI Hype Theater 3000, where we seek catharsis in this age of AI hype. We find the worst of it and pop it with the sharpest needles we can find. 

Emily M. Bender: Along the way we learn to always read the footnotes, and each time we think we've reached peak AI hype, the summit of Bullshit Mountain, we discover there's worse to come. I'm Emily M. Bender, a professor of linguistics at the University of Washington. 

Alex Hanna: And I'm Alex Hanna, director of research for the Distributed AI Research Institute. This is episode 65, which we're recording on October 20th of 2025. Today we're joined by a very special guest, Decca Muldowney. Decca is a journalist and writer who was our fact checker for The AI Con. She's also writer in residence at DAIR, where she's been working on our newsletters and other communications. So glad we're finally getting to have you on the show. Welcome! 

Decca Muldowney: Thank you. I'm so happy to be here! I've watched and listened to so many episodes of the podcast. It's like having a walk on role in like, Buffy, or something I've watched a billion times.

Emily M. Bender: Wow. 

Alex Hanna: Wow. Who- 

Decca Muldowney: Big words. 

Alex Hanna: Yeah, I guess, who's Spike and who's Angel? 

Emily M. Bender: I'm outta my depth there, but thank you for joining us. Decca has done investigative journalism and data analysis throughout their career, so we thought they'd be the perfect guest to talk about today's topic, which is data journalism versus so-called AI. Should we get into our first artifact here? 

Alex Hanna: Yeah. 

Emily M. Bender: Okay, here it is. This is a recent piece. The sticker is data journalism. The headline is "An Early AI Pioneer Shares How The Vibe Coding Revolution Could Reshape Data Journalism." The journalist for this piece is, the Center for Health Journalism Fellow Clara Harter. And this was published on October 9th of this year, so not too far back. The headline already is like, do we have to go like, you know, colonialist with pioneer, and also vibe coding? 

Alex Hanna: Yeah, it's a little weird, and it's also interesting just calling this person an early AI pioneer. Anyways, there's other reasons for that, which we'll get into below. So it starts off, "The word AI-" which I guess it's two words- "can invoke a sweeping range of emotions in the reporter. Paralyzing fear of more job losses, sheer awe at the new ease of massive data analyses, gnawing resentment at intellectual property theft, or simply the ick. And the link "the ick," links to something from neuroscience. I don't know. I don't think it's germane. Anyways. 

Decca Muldowney: Massive intellectual property theft does give me the ick. I've gotta be honest. 

Emily M. Bender: Yeah. 

Alex Hanna: "These emotions are all valid, explained Chase Davis, an early pioneer in journalistic uses of artificial intelligence and the keynote speaker at the USC Center for Health Journalism's 2025 Data Fellowship. But it's possible, and in fact, encouraged to hold both a healthy dose of skepticism towards AI and optimism about how we can advance our mission as journalists, said Davis, speaking to fellows gathered in Los Angeles for a week of data training and talks." Okay. There's already a lot to unpack. 

Emily M. Bender: Yeah. But I think it gets worse. So unless- 

Alex Hanna: It gets a lot worse.

Emily M. Bender: Yeah. Unless, Decca, you have any initial reactions, we can keep going.

Decca Muldowney: Well, I mean, the first thing I'm gonna say is that this is written by a fellow, which is usually a role that you have really early on in your career as a journalist. Like, one of my very first roles was as a research fellow at ProPublica. And you know, I messed up that early in my career, so I don't want Clara to think we're coming down too hard on her personally. But there's a lot of, I guess, presumptions being made in here that I think we could get into.

Emily M. Bender: Yeah. So let's maybe keep the focus on Chase Davis here, rather than Clara, or Clara. So, okay, so I'll keep reading. "A helpful framework for doing this is to realize that most conversations about AI can be split into one of two buckets. In one bucket, you have the big picture, theoretical, think-piece questions: How can we prevent AI-generated misinformation from further eroding our perception of the truth? What will be the environmental toll of our increased reliance on AI? Could authoritarian regimes exploit AI to suppress dissenting journalism?" And I have to say, none of those felt big picture or theoretical to me. Like these... 

Decca Muldowney: Those are very practical questions at this point. 

Emily M. Bender: But, so, okay. "In the other, you have the narrow and practical problem-solving questions: How can I use ChatGPT to isolate dataset entries that represent hotel addresses? What's the best way to prompt Google Pinpoint to pull instances of kidnappings from thousands of police reports? Am I able to use AI-generated code to verify my analysis of overdose data? While it's important to dwell on the high-level questions, Davis believes reporters have the most to gain by engaging with AI on this granular level." Thoughts? 

Decca Muldowney: Well, it's interesting, there's always this dichotomy that's presented between like, well, should we be worrying about the big picture, you know, misinformation, data privacy, the destruction of the environment, these big picture problems with AI? Or should we just be thinking about like, how can I best prompt ChatGPT to give me what I want? As though those are like, two separate things that we can really consider in isolation from each other. But then the other thing is these like, imaginary kind of news articles that are being written, you know, like hotel addresses, pulling kidnappings out of police reports, or verifying overdose data. Like those are all tasks that journalists do as part of their reporting and have been doing for a long time without these tools. I just wanna sort of flag that at this moment, that like, none of these questions for potential news stories are new. They're things journalists will have been doing, and especially local reporters or like crime reporters. But here it's all being framed in like, how do I make ChatGPT do this for me?

Emily M. Bender: Right. As opposed to doing it in a more reliable and verifiable way. 

Alex Hanna: Yeah. 

Decca Muldowney: A hundred percent. 

Emily M. Bender: Yeah. So, quote from Davis: "'The way I look at applications of AI today are very much focused on very small, very practical problems,' said Davis. 'I tend to think that the best uses of generative AI are extremely, extremely boring.'" And then we get his bona fides. He was leader on the New York Times' Interactive News Desk and former head of the Minnesota Star Tribune's AI Lab. And then, "Davis has more than three decades of coding experience and 15 years of working with artificial intelligence tools." 

Alex Hanna: Yeah. That last sentence really sent me because it gives, like, "I have 200 years of ChatGPT experience," like, okay, you're ret conning. What is artificial intelligence? You know, what I'm anticipating is some basic, like, natural language processing or other types of tools, maybe just automatic speech recognition. And it's getting lumped into putting everything into this. So, I mean, it's really, and I guess like the coding experience thing also kind of sent me, 'cause I'm like, okay, so? Like, I'm more kind of interested in what you're doing maybe as a journalist, and like what kinds of ways you've been doing things methodologically. But it's been, yeah, that really was an annoying sentence. 

Decca Muldowney: Yeah, it's funny, 'cause mostly when we introduce ourselves as journalists like this in a, you know, things that we are proud of, usually you talk about like stories that have won awards or have had really big impact, or a beat that you've covered. It's sort of sending me that it's like, "Minnesota Star Tribune's AI Lab." I mean, I wish I knew more about what that was, but it is also giving me the ick. 

Emily M. Bender: Yeah. And at the same time, I feel like somebody who's been using natural language processing to assist data journalism for 15 years ought to be more skeptical about ChatGPT.

Alex Hanna: Yeah. 

Emily M. Bender: Like that, mm, no. 

Alex Hanna: Right. And it's also interesting too, 'cause so like, in the next piece they say, "One early example of a newsroom successfully doing so is a 2015 investigation published by the Los Angeles Times. They used natural language processing to analyze police reports or keywords, identifying crimes with serious or minor assaults. And they talk about, you know, the kind of findings and misclassification. But that's 2015, right? I mean, that's not the use of LLMs. I'm going to the article, I'm just kind of skimming this article, 'cause I haven't clicked through to this. Trying to identify what they're analyzing. And it's not great, 'cause it just says in, even in an example, "To analyze the data, the Times used a computer algorithm." Like, okay. Like, I pressed control find on my Google Sheets. That is a computer algorithm. "The program used crime data from the Times' earlier investigation to learn keywords that identified a crime as a serious or minor assault. The algorithm then analyzed nearly eight years of incident data and the findings were sampled and manually checked by reporters for accuracy." So this is a bit infuriating, too, just from the Times's, their example, like an algorithm. And they use like the term "learn," which I'm like, was it a machine learning system? Was it probabilistic? Was it deterministic? Like, what's actually happening here? 'Cause that has a lot more downstream, you know, like, implications for how you're actually analyzing those data.

Emily M. Bender: Yeah, it sounds like they may have done something like TF-IDF, term frequency inverse document frequency, which is basically a way to find words that are characteristic of one set of documents as opposed to the rest. Like, that would be a way to extract a set of keywords. I'm projecting here, I don't know what they did.

Alex Hanna: Yeah, it's possible. The word that confuses me is like, "learned." You know, like, and I'm like, okay, did you train a model? Like, and that's, if you did that, then like fine, say so. 'Cause TF-IDF is not machine learning. I mean, it's distinguishing what are kind of the most significant words in the document relative to the document corpus, right? I mean...

Decca Muldowney: I think this is coming out of an era, and it's kind of the one that I was trained in and came up in, where you're not very good really in the article at explaining the data analysis that you did on the back end, or you explain it in the simplest possible terms, 'cause your assumption is the audience isn't that interested in how you got the information and they're more interested in what you found and that, you know, you're a journalist, you're telling a story. I think now when journalists employ these tools, it's actually, they're morally obligated to talk about the process in much more detailed terms. Because of the context that we're in, because of the environment that we're in and because of the owners of these tools, right? They're not just neutral tools sort of circulating in a vacuum that you as a journalist can pluck out and use. And maybe they never were, but maybe, you know, if you just used language processing to identify key terms, that didn't come with the baggage that using of something like ChatGPT comes with now. So I think this poses basically, these new tools pose new problems to data journalists. And I think it's a responsibility now to foreground your method in a much clearer way. People are not only interested in what you found, but also how you found it, what tool you used to find it, and how you verified it. 

Emily M. Bender: Yeah. And I, when you said morally obligated, I was sitting here nodding vigorously, at least inside my head, because there are appropriate use cases of automation, but every time you call what you did "AI," you are basically helping to build up the idea that it's good to use ChatGPT this way.

Decca Muldowney: Totally. And I think Alex said something before about, a lot of these tools that journalists have been using for a long time are now getting retroactively lumped in to some nebulous category called AI. We never once, when I was in the data journalism specialization at journalism school, no one ever said the word AI. But these are a lot of the same tools and likely some of the same tools that the LA Times used in this. So, you know, there's this retroactive understanding of what we've been doing all these years as AI, when it's not. 

Emily M. Bender: Yeah. And you certainly like, I've got the ick from these coming paragraphs, and I would hope that serious data journalists would like to differentiate what they're doing from this. So, the article says, "While AI can be applied to any form of journalism, it's especially useful for data journalists as it can decrease the burden of manual computation and code writing. Davis recommends using large language models, LLMs, such as ChatGPT, as thought partners to help brainstorm solutions to roadblocks in data journalism projects. It may not have all the answers, but a back and forth conversation can yield new ideas on how to retrieve data, clean it, and then craft code to both analyze and verify the end result. 'What AI allows us to do is to transform the problem we're dealing with into a problem that's easier to solve,' he said." 

Decca Muldowney: Where to start? 

Alex Hanna: Yeah. It's pretty- but I mean, the thought partner thing is terrible. And then the kind of like, "how to retrieve, clean, and craft code." So it's sort of intended to be used in all points of the process and you're like, will any of that actually work? And how do you assess any of that code? And they, you know, they talk about that a little bit below about generating new code, even if you don't know how to use that. Yeah.

Decca Muldowney: The sentence that's upsetting me is that "it can decrease the burden of manual computation and code writing," because I think there's a parallel when people talk about like generative AI helping you as a writer. Or do you use generative AI in some way as a writer? The idea that there's part of the process of writing that's boring, that's a burden. That's, you know, that's a waste, it's, that it's wasting your creative energy, like, no. That you, we think as we do, we think as we write, and it's the same, I believe, with manual computation and code writing. It's not necessarily a burdensome part of the journalism. I mean, there are things to be discovered in the process of looking at data, you know, running different analysis on it, manipulating it. And you know, the very basic thing that we were taught when I was coming up is that data's not neutral. It's not, you know, it's something that you must interact with, that you must understand the biases of, that you have to understand the source of it as much as you understand the source of anything else that you use in your reporting. And I just, I'm not convinced at this idea that this is a burdensome part of the process. I think it is the process, it is the journalism. 

Alex Hanna: Yes. Yeah. 

Emily M. Bender: Yes. There's some great stuff in the chat. abstract_tesseract says, "Thought partners? Yuck! Friend, I would literally buy you a rubber duck if you need one." And mjkranz replies, "Rubber ducks are pretty excellent thought partners, for what it's worth." abstract_tesseract: "New Mystery AI Hype Theater 3000 swag idea just dropped." PartyStreets: "Take my money." 

Alex Hanna: Yeah, so just a rubber duck that says "thought partner" on it. 

Decca Muldowney: Great. 

Alex Hanna: Which would be hilarious. But yeah, I completely agree with you, Decca. I made like a LinkedIn post about this that went weirdly kind of viral, like ages ago. And it was effectively, like, I got an email that was like, "Don't you hate doing research?" And it was like, "Now we have a code thing that-" I was like, the research is the point. You know, like that's the process, that's how you think about it. And I think, yeah, I mean, there's no, the writing is the thinking. It's the only way to, the only way to get there is through, you know, I mean that's, and I think that's the whole sauce.

Emily M. Bender: Yeah. So, back to the article: "For example, he was recently working with a large dataset containing a list of many different places and wanted to isolate entries that had addresses so he could later use this information to geocode them for a map. However, there is no coding formula that can make a computer realize that Dogwood Coffee Shop will have an address, but Lake Superior will not. So he asked ChatGPT to distinguish between entries with and without addresses, and then generate a list of those with addresses. So, first of all... 

Alex Hanna: But hold on, read the quote next to it because it's, this sent me as well. "'It's highly accurate at being able to answer that very, very specific, tiny question that otherwise would be very hard to do with conventional technology,' Davis said." And I'm like, what? 

Decca Muldowney: I mean, that's funny because that's the opposite of what it's good at doing. 

Emily M. Bender: Right. And on top of that, there's a contradiction in here, right? There's no coding formula that can make a computer realize this, but ChatGPT knows. ChatGPT, still software.

Alex Hanna: Is that not a formula? Yeah. 

Emily M. Bender: Yeah. And it's like, okay, so go back in NLP, to rule-based things. We had this notion of a gazetteer, which is like a list of named entities. And those probably are still floating around, right? 

Alex Hanna: Yeah. I used a gazetteer for my dissertation, like it's literally like, there was, in like 2013, you know, there was something called the Data Science Toolkit, which was like a little VM that you could spin up, and it would effectively just index things based off of like, an open places gazetteer. I'm like, literally, this is like, a pretty old technology. If you even wanted to get, like, stupider than that, you could have a fun regular expression, like X to addresses. You know, it's not like this is, this is not a, you know, revolutionary type thing. So now we have plenty of things like that. We have the Google Maps API, like there's stuff that you can resolve addresses for. So it's so interesting that this is like, posed as like, the thing that's very hard to do. And as the specific example. 

Decca Muldowney: It seems like this is a case of like, data cleaning. Really that is just like, a very, very normal part of dealing with a large dataset as a journalist.

Emily M. Bender: Yeah, okay. So, "As a result, he was able to take a task that would've taken him days to do manually and complete it within an hour." Which means he didn't verify the output. Right? Okay. "In addition to making journalism-" sorry, I just got off a very long flight yesterday. Not super coherent. Lemme try again. "In addition to making data journalism faster-" parentheses, and more inaccurate, that was me- "AI is also making it more accessible. Whereas in the past, a reporter would need to study R or Python before crunching complex datasets, that same reporter can now prompt AI to write the code for them. Instead of inputting a series of numerical commands and functions, they can simply type out what they're trying to do in English, or 'natural language,' and tweak the output accordingly, a change that Davis finds empowering." Thoughts, Decca? 

Decca Muldowney: I mean, yeah, I have thoughts. That's incredible, like, use of the word "accessible" here, which you know, is so often how a lot of this stuff is justified, in these like terms of accessibility. To me, it's not saying that not knowing R or Python, you know, makes doing journalism more accessible. It means that it's devaluing the skills of experienced data journalists who know how to use R and Python. And expecting younger and less experienced journalists to be able to produce like, complex, highly verified, highly fact checked work with massively inferior tools. And this is like basically my red flag that I'm waving about our industry, about this problem. Is it seems natural to me that when this gets introduced into actual newsrooms where just the labor standards have been so degraded already, this is just gonna end up falling on the youngest, you know, and newest journalists who will get ideas thrown at them, like, you should be able to do this with ChatGPT, we should be able to reproduce a data investigation that was previously done by people who could use R and Python alongside reporters or whatever. Now you are gonna be expected to do it, you know, in one hour with ChatGPT. It's a recipe for just, increasingly challenging labor circumstances in our industry that is already just so hollowed out. 

Alex Hanna: Exactly. And you can't really, you can't verify this. thx_it_has_pockets says, "Congrats, you made slop quickly." And so I think that's, yeah. And any examples that they go into are just, ugh, like, the next sentences were really brain melting. Pun intended because of what is next. 

Emily M. Bender: It's coming. 

Alex Hanna: Yeah. So Davis says- 

Decca Muldowney: Oh god, I see it. 

Alex Hanna: Yeah. Davis says, "'It allows us to spend our brain calories-'" so first off, that's the most cursed, like, phrase I've seen this week so far, and I don't think it's gonna get beat. 

Decca Muldowney: Cursed. 

Alex Hanna: Cursed, yes. "'Thinking more about the how and the why and the methodological questions of data analysis,' he said, 'and less about, wait a minute, what's the second argument in the VLOOKUP function again?'" And I'm like, is it that hard to look up an argument in a documented function? Like, is that really the thing that is interrupting your thought processes?

Decca Muldowney: Yeah. Again, I just don't see why these things are opposed. Like, this is all part of the process of doing data journalism, but it's being artificially opposed here. That, you know, remembering a lookup function is so arduous, but thinking about the methodological questions is, you know, what we should be spending our time doing. Guess what? Data journalists that are good do both the whole time and have been doing it for as long as there have been computers. In the old days, they used to call it computer assisted reporting, before we even had the term data journalism, because this has been going on. You know, journalists love new technology and they will try and use it and they will try and use it to the best of their abilities. It's bosses I think, and I'm gonna put this guy in the category boss, who have these sort of like bonkers interpretations of how this work happens. They're not the people, I believe, doing the work every day in the newsroom. 

Alex Hanna: Yeah. It says he was a former leader of the New York Times' Interactive News Desk and was the former head of the... 

Emily M. Bender: So, boss.

Alex Hanna: Yeah. So this man is squarely within the boss category, yeah. 

Emily M. Bender: Yeah. And I just have to say, Decca, to your point about this is the data journalism. Like if you are someone who cares about the veracity of what you're turning up, which I take to be sort of fundamental to being a journalist, then you are gonna care about every step of the process between the data you've gathered and the way you were gathering it and the words you put into the article.

Decca Muldowney: Absolutely. I mean, I wouldn't, say if I'm reporting out a story, especially a complex one, especially one that involved data, I wouldn't go to another reporter and say, "You know what? I find this bit of the reporting process kind of boring. Could you do it for me and then I'll just like slot it in?" You know, that is what is being suggested here. Is that you go to ChatGPT or some other chatbot or LLM, and you give them part of your reporting process, and then you take it back and, you know, basically accept the results or you do some minimal verification on it. Now why on earth would one do that? I wouldn't do it if it was another person's brain, and I'm certainly not gonna do it when it's OpenAI's product. 

Alex Hanna: Yeah, absolutely. There's some great things in the chat. So a first time chatter, petite_argen in the chat says, "Don't you get a little suggestion popup saying what each argument it is? Seems like the issue is just reading." And then PartyStreets replies, "No time. Gotta save brain calories." 

Emily M. Bender: Yeah. Okay. I think we should be thinking to get to our examples soon. Is there anything else in this that you wanted to dog on, Alex? 

Alex Hanna: We gotta talk about citing Karpathy, like this is, like, it's, yeah. So, "AI researcher Andrej Karpathy coined a term for this process in early 2025. It's called 'vibe coding,' and it shifts the developer's role from writing code line by line to guiding an AI assistant, enabling the developer to focus on the bigger picture. Or, as Karpathy put it in a viral tweet, 'The hottest new programming language is English.'" And then it's- 

Emily M. Bender: It has everything. It has conjunctions, it has conditional- sorry. 

Alex Hanna: Yeah, it's got loops. "Vibe coding is not just a method for reporters with no knowledge of coding. It's being embraced by the top echelon of AI experts." 

Emily M. Bender: But what about the actual data journalists? 

Alex Hanna: And it's just, yeah. And then, "A prominent data journalist recently sent Davis a note that said, 'Oh my god, this vibe coding thing is insane.'" And I'm just putting in like, brackets, "bro." 'Cause this feels like a bro statement. "'If I had this during our early interactive news days, it would've been a godsend. Once you get the hang of it, it's like magic.'" 

Decca Muldowney: Interesting. Interesting, willing to be quoted in the article as a prominent data journalist, not willing to put their name on the record.

Alex Hanna: Interesting. So fascinating. 

Decca Muldowney: Who knows why. 

Alex Hanna: Yeah. And then I did want to read this beat because this is, you know, the, "of course, a healthy dose of skepticism is needed" section. So, "At the same time, he urges reporters to hold on to their healthy sense of skepticism when it comes to AI. Remember that LLMs can hallucinate information, and the code and data analysis they produce will often have flaws." Like, as a journalist, first off, like, why the hell would you want a tool that has flaws like that? I mean, hopefully you want that kind of thing with any kind of veracity in any job you are. But as a journalist it seems quite terrible to do so. And then I do wanna like, this part was very alarming and I thought it was hilarious that this was even here because the whole rest of the thing seems so Pollyannaish. So they wrote, "It's also good to remember that data entered into LLMs like ChatGPT may be stored on external servers of technology companies, and could in some circumstances be subject to subpoena. Therefore, if a reporter is handling extremely sensitive information, it is best to keep it away from AI, and from the internet in general, he said." Thoughts on that, Decca? 

Decca Muldowney: Yeah. Yeah, totally. It could, and you shouldn't have sensitive information anywhere near anything like- I mean, this is just basic stuff- but it's horrible to imagine, you know, sensitive information getting anywhere near any of these tools. 

Emily M. Bender: Yeah. And the subpoena angle is interesting, 'cause that was one that I think is maybe more specific to journalism. We're used to worrying about like, privacy in general, but not specifically subpoena power, which is an interesting point. 

Decca Muldowney: Yeah. And I don't know the legal ins and outs of this, but you know, this is a thing in journalism about protecting one's sources. And we've seen very prominent examples of when, you know, outlets like The Intercept or whatever, were not able to always protect their sources. But not because they were subpoenaed, but because it was, because the federal government figured it out. But in the political situation that we're in, journalists should actually be incredibly worried about their information being subpoenaed as well. So, yeah, this is just such a nightmare. This is nightmare fuel for me. 

Alex Hanna: Yeah. Well, and it was the case too, wasn't there that, the thing about like, even if you say ChatGPT shouldn't retain your conversations, they still have held onto everything because of the ongoing litigation, right? So it's, you know, there's, those checkboxes are in effect meaningless. 

Emily M. Bender: Right. PartyStreets asks, "Regurgitation is a bigger risk than subpoena, no?" I think it depends. I think in many cases we don't know for sure that the companies are using these queries as additional training data. And if they're not being used as additional training data, then you don't have the regurgitation problem.

Decca Muldowney: One problem is we have no idea how they're training these models. I mean, and that's the other thing as a data journalist, right, is that I don't really think that you should use a tool that you do not understand. That's a very, very basic thing. But you know, I basically understand the functionality of an Excel spreadsheet, and that's honestly the tool that I would use most often. And much is, I mean, earlier I was thinking, an Excel spreadsheet can probably figure out whether something has an address or not. But anyway, not the point.

Alex Hanna: I think one fun art project, if anyone wants to do it, is you should make an Excel spreadsheet clone, but it just introduces like random fuckups in formulas, and just call it like SheetsGPT.

Decca Muldowney: But yeah, I think it's very dangerous to use, I mean, okay, you might be able to code in R and Python and not know the, you know, absolute capabilities of absolutely everything that could do. But you sort of, you do, you have a greater understanding of it. With ChatGPT in these models, we actually just don't know what the backend looks like at all. We don't know how it's trained. So in that sense, regurgitation is as much a risk as subpoenas, as much a risk as data privacy, as much a risk as anything else, when we just don't understand the tool. 

Emily M. Bender: And even if you've got somebody who is like, supposedly doing due diligence and like testing their prompt with ChatGPT with a bunch of known things so they can see that it's working. First of all, there's no guarantee that this probabilistic system's gonna keep doing that, but also OpenAI could update the model in the background or do anything else. And you know, it's not a model that you have control over. 

Alex Hanna: Yeah. And, apparently, wisewomanforreal told a wild story in the chat. She said, "I've been trying to track down a plagiarist and in frustration asked ChatGPT. It happily returned everything I had already found, including her address at the time she submitted her doctorate, as that information was in the published version of the thesis. So it does store private information and blithely reports it." Yeah, I mean, I guess the question is whether that was trained on something that was either already in an existing corpus, you know, that was in, say, Common Crawl or, I'm assuming if it was a thesis, something from ProQuest or one of these data stores, or if that was something that had been uploaded via prompt. And I think that's the un-clarity there. 

Emily M. Bender: Yeah. All right, so let's get to a couple of examples where journalists did this thing. 

Decca Muldowney: Hooray! 

Emily M. Bender: The first one comes from the Financial Times. This was published on September 23rd of this year, by Melissa Heikkilä, Chris Cook, and Clara Murray. And the headline is, "America's Top Companies Keep Talking About AI, But Can't Explain The Upsides." Subhead, "FT analysis of hundreds of filings suggests the S&P 500 businesses are clearer about the risks than the benefits." And I have to say, seeing this headline, I was super excited, and then when we got to their methodology, I was super disappointed.

Alex Hanna: Yeah. And so what they said that they did in this is basically, we can scroll down. I mean, they effectively said, "The FT used AI tools-" and they don't say what it is in here- "to identify these mentions of technology in the US Securities Exchange 10-K filings and earnings transcripts, then to categorize each mention. Results were then checked and analyzed to help draw a nuanced picture about what the companies were saying to different audiences about the technology." But it's so weird that they used, they didn't, they just said that they used, did they say here that they used ChatGPT? Or that they-

Decca Muldowney: No, Alex, this is because you and I became obsessed with this story and ended up- 

Alex Hanna: We became obsessed with this story. And I messaged Melissa, who to be clear, like, is a great journalist. Like, I respect her work and have seen a lot of her work, especially in tech reporting. But then one of the other authors was the one who did the analysis and then was, like, encountering a lot of, like, David Gerard from Pivot to AI was asking him what he did in this analysis, was basically like saying, this analysis is bullshit. They used ChatGPT. And his defense was that he used ChatGPT. Like it, he was just confirming what David was saying. I'm like, okay, and that's different as... what? 

Decca Muldowney: Yeah, and in terms of the journalism, the question that this piece is asking and the idea of looking at these documents in order to answer that question. That, those are good. They're good. 

Alex Hanna: They're great questions. Yeah. 

Decca Muldowney: And, you know, in journalism you're always encouraged to develop a document mindset, which is like, where may there be documents that exist that I can get access to that will help me tell this story? And in this, you know, their document mindset has been to identify these earning documents, these SEC 10-K filings and earning transcripts. And then the idea to analyze them by locating certain phrases is also fine, but this is one of those examples of, you know, the FT has used AI tools where, okay, once upon a time, we might have been able, as data journalists, to fudge and say, we wrote an algorithm to identify this. And, you know, the reading public doesn't really care about how we did it on the back end. We're just not in that universe anymore, you know? People did want to know, as you pointed out, what they used, and then it turns out they used ChatGPT, and people were not very happy to hear that. 

Alex Hanna: Yeah, exactly. abstract_tesseract says, "Oh yeah. I was surprised by the defensiveness. Quote, 'I don't understand why people are so upset. We didn't use AI for the data, we just used it for the analysis!'" Yeah. As if that's better, you know, and I thought it was really wild on the journalist's part, I think it was Chris that was responding, Chris Cook, and it was, it's like, yeah. And you're absolutely right, Decca, that because people are really keyed in to the way that LLMs get things wrong, the way they operate now. I mean, I think it's made people really attentive to methodology and, you know, as we say, always reading the footnotes, and really trying to try to do things. And I think that's, I mean, that's a great kind of discipline that it's fomenting in people, but it's bad that it's happening because people are using these tools so much.

Decca Muldowney: Yeah, and it's bad that the findings of the story, which are themselves interesting, are then undermined by the method, you know, used to find them. And also, I do think that this is not a story that couldn't have been done prior to ChatGPT that's the other thing. I mean, in the simplest possible version of the story, it's a control F, you know, that takes a hundred years, right? But there are ways of doing this that did not involve ChatGPT. 

Emily M. Bender: You could write a Python script. 

Decca Muldowney: You absolutely could. Yeah. 

Emily M. Bender: And just to say, if there's a story that couldn't be written without the aid of ChatGPT, the story can't be written. Because ChatGPT is not actually gonna be a solution to anything. 

Decca Muldowney: No. No. No, it's a thought partner, Emily. 

Emily M. Bender: We gotta make those rubber ducks. 

Alex Hanna: But even some of the statistics on this are not, like, not beyond the pale to do. You know, there's a slew of analysis tools that are off the shelf. So, for instance, "374 of the S&P 500 mentioned AI in earnings calls in the past 12 months-" 

Emily M. Bender: That's grep. 

Alex Hanna: That's, yeah, that's a grep. That's a control F. That's a regular expression. If you just wanna, you know, include different types of, you know, capitalizations or punctuations. "With 87% of calls logged as wholly positive about the technology with no concerns expressed." That's Vader, that's Luke. Like, that's- 

Emily M. Bender: That's sentiment analysis. Yeah. 

Alex Hanna: That's basic sentiment. You know, there's like, I could name maybe two more packages if you gave me five more minutes that could do that. Like it's, and I don't think it's unreasonable for your job as a data journalist in doing things with document analysis to understand that and to explore that and to use things that are not LLMs.

Emily M. Bender: Yeah. 

Decca Muldowney: But I think, the, sorry, it's a bit like the cruelty is the point. I think the LLM is the point in some of these articles, you know, it's being used on purpose, either for the sort of clout of like using a new tool, or because the push that's coming down from the top of newsrooms is that they wanna see these AI tools used, you know? Business Insider sent out this email that was leaked to all of their staff saying like, we actually want you to be using ChatGPT in your research. These newsroom bosses truly have this, you know, have swallowed the pill of "this is gonna increase efficiency and it's gonna, you know, make stories better and blah, blah, blah, blah, blah." This story's a good example of how the reception of it was actually really damaged by the fact that it was using the tool. And so I hope there's a lesson there. But I mean, it, unfortunately, I think the blame and everything falls on the journalists and obviously, I think the problem's higher up.

Emily M. Bender: Yeah. And it's hard to be in that position where these journalists are, right? Except maybe they could actually go back to regular expressions and existing sentiment analysis tools and tell their bosses, this is AI. But then actually have a solid methodology, but then they have to be careful about how they write about it, because as we're saying, the audience is now sensitized to this and, you know, wants to know, did you use a real methodology or did you ask the bullshit machine? And like, as you were saying before, Alex, I love the questions they're asking here. And now I'm not even bothering to look at what the results are, because they're fake. 

Decca Muldowney: Yeah, exactly. 

Alex Hanna: Yeah, and this is very funny too, because it's, you're writing an article about AI FOMO? Maybe all the AI critics are gonna pay attention to this article. And then use ChatGPT? Okay, you've lost us, you know. 

Emily M. Bender: You've been experiencing the FOMO! Yeah. So, ndrwtylr, who I actually now think is pronounced Andrew Tyler, writes, "I saw a piece years ago about how the big tech, big data projects could mostly be done perfectly well on a normal laptop with built-in Linux tools. And I think this is the same. People who don't know what they're doing are pretty amazed by pretty easy stuff." 

Decca Muldowney: Yeah. And I mean, there was this admonishment towards journalists, you know, in the past kind of 20 years of horrible layoffs and the hollowing out of the industry, right? "Learn to code." And yet the journalists who learned to code, and there were many of them, and they've done incredible work with it, are sort of now being told, yeah, learn to use ChatGPT. 

Alex Hanna: There was actually a line in the other piece that we didn't read, and I'm gonna read it because it's terrible. Where the fellow says, "While other technological innovations have been, quote, 'inflicted' upon the newsroom- Google Search, Craigslist, Facebook, and other social media platforms- AI is still in its infancy. So this is a perfect time for reporters to learn how to leverage it." And I'm like, as if this is not something that's being inflicted on the newsrooms as well.

Decca Muldowney: Of course. 

Alex Hanna: And we've already seen the reporting and the kind of, the implication that traffic to sites is going to basically go to zero via Google referrals, and that's just gonna collapse their revenues even further. So it's as if, you know, adopting these tools is any kind of way to shift power away from the kind of political economy of the industry.

Decca Muldowney: Yeah. There was also the graph that came out, I think earlier this week, that said the number of, like, chatbot authored or LLM authored articles on the internet is now higher than that of human authored text on the internet. You know, so how you could think these tools aren't going to decimate us as human journalists and human writers, I don't know. It's naive beyond, I think it's malicious naivety. 

Emily M. Bender: Yeah. So we have one more example, which is not from as well known of a news source, sort of surprising name of a news outfit called Sentient Media. I don't know what to make of that.

Decca Muldowney: I've actually never heard of this outlet. 

Emily M. Bender: Yeah. Okay. So this is another example, it's not just the well-respected things like the Financial Times. So this is, the date is September 26th, 2025. The tags are analysis, climate, and research. And then the author is listed with the tag "Words by Cherry Salazar," which I guess means that someone else did the numbers. I don't know. Headline, "Analysis: 96.2% of climate news stories don't cover animal agriculture as a pollution source. An opportunity for climate journalists to expand their coverage by including the largest source of food-related emissions." So they're basically, they took a sample of articles from news outlets, Washington Post, Reuters, Boston Globe, et cetera. And then, "Methods: Sentient Media ran two computational methods. One categorized articles by using the same search parameters used in a 2023 study published by Sentient and data researchers Faunalytics, while the other analyzed them using Anthropic Claude API, with each category defined by detailed classification criteria." And then, they seem to be giving the Claude results. So, "Findings: Using the Anthropic method, the analysis showed that 96.2% of stories did not mention animal-based farming as a cause of greenhouse gas emissions." 

Decca Muldowney: It's like, "and here are the fake findings!" 

Emily M. Bender: Yeah! 

Alex Hanna: Yeah. And then, using, ugh, okay. "Using the the Anthropic method, the analysis showed that 96.2 are-" is that supposed to be percent?

Emily M. Bender: I think so. According to the headline, yeah. 

Alex Hanna: Okay. I was just like, there's a missing percentage sign. 

Emily M. Bender: Yeah. 

Alex Hanna: So, proofreading. "Did not mention animal based farming as a cause of greenhouse gas emissions. This means only 36 out of 94...." So they had, they're basically doing a kind of a mention classification situation again, which again- 

Decca Muldowney: Yeah, it's the same kind of analysis. 

Alex Hanna: Yeah. Again, control F exists. 

Emily M. Bender: Right. And then you can find, you know, basically a few keywords, you can find the ones that mention, and then you can go read and check about contextualization. Like, these are not enormous numbers. 940 articles is not an enormous dataset. 

Decca Muldowney: No. It's also, I guess in the process of this, you are potentially feeding all of these articles to Anthropic. 

Emily M. Bender: Yeah. 

Alex Hanna: Yeah. 

Decca Muldowney: Which is deeply depressing as well. I didn't care so much when it was earning reports, but now that it's like, you know, now that it's reporting, I'm upset.

Emily M. Bender: Yeah. All right. And we don't know anything about this except they have a really strange name, so- 

Alex Hanna: Well, there's like a, I looked up the, they report on factory farms. Which, again, you know, good thing to investigate, but also like, why? Why are we doing this? 

Emily M. Bender: Yeah. So abstract_tesseract in the chat, "Really gives you the Anthrop-ICK, am I right?" 

Alex Hanna: Hey! 

Emily M. Bender: Ah, okay. So with that, I think we're going to transition to Fresh AI Hell. Alex, this time you said musical ahead of time. I've got you as one among a chorus of rubber ducks, all right? So musical genre that's appropriate to that. Singing about AI slop journalism.

Alex Hanna: Yes, yes. Quack quack quack quack quack, quack quack, quack quack. I'm sick of the slop. Quack quack, quack quack. It appears on my page. Quack quack, quack quack. From Financial Times to Sentient Media. Quack quack, quack quack. Buy our merch. Quack quack, quack quack. Ask me. Quack quack, quack quack. I am very pretty, and I can give you misinformation, and serve as your thought partner. Quack, quack. 

Decca Muldowney: Wow, straight to number one. 

Alex Hanna: And it's very funny that you mentioned ducks and I immediately went to, what is that, Blue Danube, Swan Lake? 

Emily M. Bender: Oh, yeah. 

Alex Hanna: Yeah, sorry. 

Emily M. Bender: Very aquatic. 

Alex Hanna: Yeah. 

Emily M. Bender: All right. aczhou gives you five flames, 10 out of 10. 

Alex Hanna: Thanks Amy. And then Ozzy, our producer in the chat says, "That was beautiful. We're so back." 

Emily M. Bender: Okay. So, Fresh AI Hell. 

Alex Hanna: Yeah. This is from the Wall Street Journal. "Teen Sues Maker Of Fake Nude Software. There's a growing concern over AI sites that generate fabricated naked photos of minors and non-consenting adults," by Julie Jargon, incredible surname, October 16, 2025. So yeah, we mentioned this in the book, we mentioned the kind of Nudify apps. And so, what is the company here that's named in this article? 

Emily M. Bender: I'm not logged in. 

Alex Hanna: Oh, it is, okay. So yeah, well there is a, suing a developer. Okay.

Emily M. Bender: Yeah. So that's some pushback, which, you know, unfortunate that it's needed, but I'm glad that it's happening. 

Decca Muldowney: Yeah. Sad the teens have to do the lawsuits over this. 

Alex Hanna: Right. 

Emily M. Bender: Yeah. Okay. And then just back to the chat, petite_argen adds, "Did you mean five ducks, 10 out of 10?" 

Alex Hanna: Yes. 

Emily M. Bender: Okay. 

Alex Hanna: So Ozzy dropped the archive link in the chat. Thanks, Ozzy. And it says, "The plaintiff is suing AI/Robotics Venture Strategy 3 Ltd.," which is, what a scammy name for a company, "the company behind the web tool ClothOff." Gross. 

Emily M. Bender: Ugh. Great. Okay. So then we have some reporting from 404 Media by Samantha Cole, October 14th, 2025. Sticker, "ChatGPT." Headline, "ChatGPT's Hail Mary: Chatbots You Can Fuck." And basically, because OpenAI is not finding a revenue stream, they are pivoting to sexbots. 

Decca Muldowney: Shout out Samantha Cole, a reporter who would never use ChatGPT.

Alex Hanna: Yeah, she was on the show a while ago. And we were chatting about the kind of various collapses in journalism. But go to the lede, 'cause it's- go up a little bit, 'cause there's a, on the, yeah. "As recent reports show OpenAI bleeding cash, and so on the heels of accusations ChatGPT caused teens and adults alike to harm themselves and others, CEO Sam Altman announced that you can soon fuck the bot."

Emily M. Bender: And that's what's in this tweet here from Altman saying, "We've solved mental health issues! So we can do this now." 

Decca Muldowney: Well, okay. This is like, I got into conspiracy theory brain when this story, when Sam Altman made these comments. I shouldn't really indulge in conspiracy theory brain as a journalist, but I'm gonna pretend that we're off the record right now, but, okay-

Alex Hanna: We are very much on the record. 

Decca Muldowney: I know, we're recording. Okay. But I was like, is this in fact what ChatGPT was always supposed to be? 

Emily M. Bender: Oh, interesting. 

Alex Hanna: Yeah, yeah. 

Decca Muldowney: Like, this is the financial plan. 

Alex Hanna: Well, it's sort of like, I've been threatening to write a paper, and I've been chatting with a few people, and it's sort of like, the paper will be subtitled something like, "Mommy, Nanny, Mary, Mammy." But it's about how like, chatbots either serve as like, moms via Hinton, they are nannies, they clean up for other people, they are Marys, they're like our secretaries that people want to fuck, or they're like mammies in which they are racialized types of people that they get to boss around. So yes, that is, I've recorded this. I have to chase my Donna Haraway dreams and, but I think that is very much there, Decca. 

Emily M. Bender: Absolutely. petite_argen says in the chat, "Spicy autocorrect has a whole other meaning now." Okay, I think you get this one, Alex. No, I want this one. We're gonna switch. You can have Salesforce. So this is Kathy Tewson on Bluesky, quote tweeting someone called The Questionable Authority. And it's pointing to this new company or existing company called GitLaw that has just announced vibe lawyering, GetLaw Alpha: Vibe Lawyering. And The Questionable Authority says, "Lord have mercy." And Kathryn Tewson says, "The bros are at it again! And make no mistake, they are literally calling it 'vibe lawyering.'" 

Decca Muldowney: Journalists, when you get subpoenaed, do you want a lawyer or do you want a vibe lawyer? 

Alex Hanna: Do you want a vibe? Yeah. 

Emily M. Bender: Okay, Alex, you get this one. 

Alex Hanna: So this is very fresh hell, this is from the New York Times. "Salesforce Offers Its Services To Boost Trump's Immigration Force. The San Francisco based firm has told ICE that it could use AI to help the agency nearly triple its staff. The company's CEO-" this is Marc Benioff- "once a progressive tech titan, has embraced President Trump." And there's a picture of Benioff somewhere. This is by Heather Knight. Yeah, so this is what the tin says. I mean, Benioff, you know, the week before said that he would welcome the National Guard to come to San Francisco. That Trump says now he is gonna do, so, thanks, Marc. 

Decca Muldowney: He's really auditioning for like biggest San Francisco villain right now.

Alex Hanna: Yeah, I mean, that's a pretty dense field. And yet, you know, he could have shut his mouth and kept on, you know, encouraging things like homelessness measures that tried to house people. Instead he was like, no, we're going full villain. 

Emily M. Bender: Oof. Okay. This is mixed news. October 12th, 2025 in the Washington Post, by Daniel Wu. "Towns are saying no to AI data centers. One got sued over it." Subhead, "A developer sued a Michigan township after it voted against a data center proposal. Cities in Ohio and Missouri have explored data center bans." And it's interesting, like, on what grounds are they bringing these lawsuits? And I'm guessing it's probably in here, but just, I'm glad to see this increasing, you know, pattern of towns rejecting these things. And I guess it's gotten to the point that the other side is lawyering up. 

Decca Muldowney: Hopefully vibe lawyering. 

Emily M. Bender: Yeah, right. 

Alex Hanna: Yeah. Hopefully. Good lord.

Emily M. Bender: Okay. Alex, you didn't read this one. Did you read this one or should I do it? 

Alex Hanna: I didn't read this one. I didn't see this one. 

Emily M. Bender: Yeah, so I'll do this and you can take the lead on the next. So this is in TechCrunch, sticker "Startups," by Amanda Silberling on October 16th. Headline, "Rent A Cyber Friend will pay you to talk to strangers online and will show off its platform at TechCrunch Disrupt 2025." So this is basically a platform where people pay to talk to strangers who are actual people as opposed to chatbots. And it's not, so it's not really an AI story, but it's also like really, really sad that somehow like access to real people, A, has to be mediated by a platform, and B, is becoming commodified. 

Decca Muldowney: I don't even understand the words in this headline, in the order in which they are presented. 

Alex Hanna: Yeah. It's really jumbled words. 

Emily M. Bender: Yeah. All right. 

Alex Hanna: This is our palate cleanser. So this is from an interview. So this, Juno Rylee Schultz is doing a clip from an interview that Ari Shapiro did with David Simon, the creator of The Wire. So we're gonna act this out. So Emily, you're gonna be Ari Shapiro. 

Emily M. Bender: Yep. And this is, by the way, October 8th is the Bluesky post. Okay. "So you spent your career creating television without AI, and I could imagine today you thinking, boy, I wish I had had that tool to solve those thorny problems."

Alex Hanna: "What!?" 

Emily M. Bender: "Or saying-" 

Alex Hanna: "You imagine that?" 

Emily M. Bender: "-boy, if that had existed, it would've screwed me over." 

Alex Hanna: "I don't think AI can remotely challenge what writers do at a fundamentally creative level." 

Emily M. Bender: "But if you're trying to transition from scene five to scene six, and you're stuck with that transition, you could imagine plugging that portion of the script into an AI and say, give me 10 ideas for how to transition this."

Alex Hanna: "I'd rather put a gun in my mouth." 

Decca Muldowney: Yes, Simon! 

Alex Hanna: Yeah. So. 

Emily M. Bender: And what's up with Ari Shapiro here? Like, hello?

Alex Hanna: I don't know. Well, as, you know, David Simon would say, or no, not David Simon, as Stringer Bell would say, "Tell everybody we're back up." Sorry. 

Emily M. Bender: Ah, wow. This was fun. As usual, looking, you know, straight down the barrel of a whole bunch of awfulness. But Decca, thank you for helping us take it on. That's it for this week. Decca Muldowney is a journalist and writer in residence at DAIR. Thank you again for coming on the show. 

Decca Muldowney: Oh, thanks for having me. 

Alex Hanna: Our theme song was by Toby Menon, graphic design by Naomi Pleasure-Park, production by Ozzy Llinas Goodman. And thanks as always to the Distributed AI Research Institute. If you like this show, you can support us in so many ways. Order The AI Con at thecon.ai or wherever you get your books, or request it at your local library.

Emily M. Bender: But wait, there's more. Rate and review us on your podcast app. Subscribe to the Mystery AI Hype Theater 3000 newsletter on Buttondown for more anti-hype analysis. Or donate to DAIR at dair-institute.org. That's dair-institute.org. You can find video versions of our podcast episodes on Peertube, and you can watch and comment on the show while it's happening live on our Twitch stream. That's twitch.tv/dair_institute. Again, that's dair_institute. I'm Emily M. Bender. 

Alex Hanna: And I'm Alex Hanna. Stay out of AI hell, y'all. Quack! Quack! 

Emily M. Bender: Quack, quack, quack. 

Head Shot

Alex Hanna

Co-host
Head Shot

Emily M. Bender

Co-host