Mystery AI Hype Theater 3000

Who's Responsible for "Responsible AI"?, 2026.07.06

• Emily M. Bender and Alex Hanna • Episode 82

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 58:53

Lately, a new framework has been creeping into AI boosterism: "harm reduction," or "responsible use." Education technology scholar dr. sava saheli singh joins Alex and Emily to unpack narratives around automation inevitability in the classroom. Spoiler alert: they're not actually "reducing harm" to students at all!

An interdisciplinary scholar and filmmaker, sava is currently working on a research project examining secondary school teachers’ experiences of genAI use, and making a film about the future of AI in education. Her speculative short film series on the harms of surveillance technology is available at screeningsurveillance.com.

References:

Fresh AI Hell:

Check out future streams on Twitch. Meanwhile, send us any AI Hell you see.

Find our book The AI Con here, and MAIHT3k merch here.

Subscribe to our newsletter via Buttondown.

Follow us!

Emily

Alex

Music by Toby Menon.
Artwork by Naomi Pleasure-Park. 
Production by Ozzy Llinas Goodman.

Alex Hanna: Welcome, everyone, to Mystery AI Hype Theater 3000, where we seek catharsis in this age of AI hype. We find the worst of it and pop it with the sharpest needles we can find. 

Emily M. Bender: Along the way, we learn to always read the footnotes, and each time we think we've reached peak AI hype, the summit of Bullshit Mountain, we discover there's worse to come. I'm Emily M. Bender, professor of linguistics at the University of Washington. 

Alex Hanna: And I'm Alex Hanna, director of research for the Distributed AI Research Institute. This is episode 82, which we're recording on July 6th of 2026. Our guest this week is dr. sava saheli singh, who's an assistant professor of digital futures and education at York University in Toronto.

Emily M. Bender: An interdisciplinary scholar and filmmaker, sava works at the intersection of education, technology, surveillance, speculative futures, critical digital literacy, labor abolition, and research creation. She's currently working on a research project examining secondary school teachers' experiences of gen AI use and making a film about the future of AI in education. Welcome to the show, sava! 

sava saheli singh: Hi, thank you so much! I'm excited to be here. And I realize every time someone reads out an introduction, I'm like, "I should write it so that it's readable, read-outable rather than just-" 

Emily M. Bender: What an amazing intersection to be working at, though. This is gonna be really fun.

Alex Hanna: Yeah, it's so good. This week, we're diving into the idea of, quote, "responsible AI use" or "harm reduction." I'm a big fan of harm reduction, but that is not a term that makes sense to use about bullshit generators. 

Emily M. Bender: When people start using these terms, they're often conceding the point that automation is somehow inevitable. So today, we wanted to really dig into the way AI boosters are using these harm reduction arguments, both in general and specifically in the context of education.

Alex Hanna: And we're of course going to talk about why automation is not actually, quote, "responsible" or "reducing harm" at all. So let's get into it. 

Emily M. Bender: So our first artifact is not an education artifact. But we just wanted to have one place where someone was very clearly using and misusing this term. So this comes from something called GARP. Do we know what GARP is? 

Alex Hanna: The World According to GARP. It's Global Association of Risk Professionals.

Emily M. Bender: Okay, there we go. So this is a little bit old, October 27th of 2023, so one year into or 11 months into ChatGPT being out in the world. And the title here is "Harm Reduction: A Strategy to Mitigate the Risks of AI," and it's by Jesús M. Gonzalez and Laura M. Gonzalez. Similar names there, maybe not by accident. So, "AI advancements have brought a lot of benefits to the financial system, but also come with risks. To minimize the impact of negative consequences arising from the use of AI, financial institutions and regulators can mimic a harm reduction approach that's proved successful in the public health sector." So sava, what, do you think of that analogy? 

sava saheli singh: It's interesting, right? I think one of the things about harm reduction is to first look at what is happening and then figure out what needs doing, and here they're just taking "Let's mimic a harm reduction approach that's proved successful in public health sector." And it's interesting that they're connecting the public health sector to like, I guess this is finance. And it's problematic on so many levels. Like, where to even start? The financial institution is a harm. So it's weird, just like, there's so much to be done before you even start thinking about the harms of AI, AI in the financial kind of context. So it's weird to have them bring this language into that, and we see this a lot. We see it in education, we see it in academia, we see it in research. A lot of language that's being used in communities to do things for communities that come from the communities are then taken and used as "This is the cool new buzzword. Let's apply this to this other thing," right? And you see it a lot in academic research, and it really breaks my heart because I know that people are trying to do good there, but it's just like, you're taking something that's come from a community for that community, and now you're applying it to something that has nothing to do with the community or is already harmful to that community, and then you're trying to do this thing, right? So it always feels like you're taking buzzwords from things that sound good because they are good for the people that came up with them, and then applying it to this thing to make yourself look better in some way, and that's always just so upsetting. 

Alex Hanna: Yeah. What I think is actually fascinating about this piece- because it's not just a financial firm, but it's a piece that they're doing risk management, and so they're doing things like fraud protection and this is what they say. The first sentence is very helpful to read here, which it says, "The rise of artificial intelligence tools over the past few years has improved modeling performance and has yielded risk management benefits across areas like fraud protection, anti-money laundering, and credit underwriting." And so those are areas which are really fascinating to highlight and to have some discussion of things coming from public health. Especially because, when we talk about harm reduction, is often used in relation to certain communities that are engaging in different behaviors that we know are likely, may have detrimental consequences for them personally. But it puts blame on those people directly for everything that's happening to them, not in the societal elements and the structural elements that have led people into engaging into some of those kind of behaviors. So it's very weird that this is then, harm reduction is then the paradigm being brought into this by these risk management professionals. It's really ironic. 

Emily M. Bender: Yeah, and we see it, I think, also somewhat in the education context. So we're gonna talk more about "responsible" and "ethical use," like those catchphrases. But the thing that really gets me about it is that when I hear harm reduction, especially in the context of let's say needle exchanges or something along those lines, it really is- it sounds like it anyway- coming from a perspective of someone who's really caring for people who are among the most marginalized in society and saying, "What do we need to do for these people now? And how do we address the most immediate risks so that these people have a chance to experience less adverse consequences of what's going on?" And to use that terminology and analogize, "Okay, we're gonna have safe injection sites or we're gonna have needle exchanges," to "We're going to teach people how to use this ethically," that just doesn't match. It's not the same kind of thing. And it also has this notion of inevitability, right? So if you've got somebody who has substance use disorder, then you can't just say, "Just stop using," right? That's not gonna work. So there is a certain local inevitability there that you have to deal with. But to then just say, "Yeah, gen AI exists and it's great, so it's gonna have to exist and we're gonna have to use it," is not the same thing and it makes me really mad. 

sava saheli singh: I wanna say also, recently I've, for better or worse, I've put myself on committees and working groups to think through some of this at our university to be like, what are guidelines for AI in teaching and learning and stuff? And I put myself on these committees with the knowledge that I will at some point rage quit, because that's a thing, and I haven't so far. I think I'm making some sort of leeway because I, much like you were saying, Emily, I change the language that they use. And just a side note, Emily, if I start using language that's not accurate, please correct me, because I think it's useful for people to know what are better ways to say certain things. And I do a lot of that work in the documents that I'm working with. Like I make sure that we use the word "product" instead of "tool," for instance, because people often use that term, right? So please, even I, who I consider myself somewhat aware, always slip up and I try to correct myself. So I appreciate if you would correct me because your post was so useful. So do- 

Emily M. Bender: I will try to do some. I don't want to interrupt you too much, but I do want to just flag what this post is. So this is on our newsletter, and it's something co-authored with Nanna Inie, where we have a bunch of suggestions for rephrasing words to get away from the anthropomorphizing language that is so deep in the discourse, which is like another kind of rephrasing in addition to this, let's not talk about harm reduction or responsible use. So yeah. So you've put yourself on these committees. 

sava saheli singh: Yeah, and it's interesting to have some of these conversations. Like it's taught me a lot about how to talk to people a little bit, because I think, like I was saying earlier, year one for me, I was rage. In year two, I'm in despair, and in year three, I'm trying to bring people back, right? And when I'm talking to them, I'm sharing what I know about it, or trying to. I think it was wrong of me to assume that we were operating off of the same kind of understanding of some of these things, and I've learned that the hard way. But some of these things, it's in a document we had to put together, whatever, draft report of where our group was at that time, right? And someone puts it together and I'm in there going "Cancel tool, make it product," or whatever. And the language is important. Like one of the places, I had included, written out, what are all the different kinds of harms of these systems, right? And one of my colleagues had put a comment there saying, "These are only highlighting the negatives." And instead of having a purely harm reduction approach to this, we should also talk about how these things can be useful. And I'm just like- I have decided, like I haven't communicated with them, but I've decided to be like, "You know what? That was a great idea. We should have a harm reduction approach to this, and this is what that looks like." But people use that language as though, like, it's meaningless. Like the term to them has become meaningless. Do you understand when you say the words, where we shouldn't use a harm reduction approach- we are educators. We should have a harm reduction approach to everything we do in the academy. And it's wild to me that people cannot see that or do not understand the words that they are using. So it's really words, what do they mean anymore? 

Emily M. Bender: Yeah. And if you want to reduce harms, you gotta know what the harms are. And I love this comment in the chat from magidin. He says, "To paraphrase War Games, maybe the only responsible AI use is to not use AI." And yeah, so when I'm on these committees- and I'm often not allowed on the committees, because I'm too good at making people uncomfortable, I think. But the thing that I'm always advocating for is: keep refusal on the table as an option. That we cannot start from, "Okay, it's here, how do we use it responsibly?" It has to be possible to say no. 

sava saheli singh: Absolutely. I think one of the things I've also done, and this was like, I did this and I read your piece and I'm like, "Oh my god, this is perfect." Because the way that they describe the two sides or the two extremes of AI use is, one is it's restrictive and punitive, two, it's permissive, allowing for exploration and innovation. And I'm like, "No, those are not the two poles." The first pole is critically informed rejection. And the other one is uncritical wholesale adoption. Let's be clear about what these two things are, right? So the way that it is framed is very important, and I'm trying to do some of that work in these committees as well. I'm always that guy in the room, and I'm like, "Okay, I will be that guy now." 

Emily M. Bender: Yeah. Absolutely.

Alex Hanna: I think, the history of what harm reduction is and what traditions that comes out of, it's very weird to then import it here without really considering, if you're making that analogy, what is the implications of making that analogy? What are you treating- AI, you're treating quote-unquote "AI" as like, through an addiction framework. And it's like, no, that doesn't seem like the right- like these are things that you purchase at an institutional level. I'm not sure what's happening here. But yeah, I think really good point in highlighting the language and the abuses of language here. 

Emily M. Bender: Yeah. All right, before we get to the UNESCO thing, this is making me think of one other thing. So my institution has just rolled out our own "AI assistant," in big scare quotes called Purple AI, 'cause purple and gold are our colors. And it's just a wrapper around some other models, but the idea is that any data that we put in from the UW side doesn't go further. And so it is harm reduction in the sense of, the whole point is to limit the liability of the university, 'cause the university is afraid of cases where our data gets improperly shared. But it's not being portrayed that way. It's, "We have our own tool and this is the one you should use," but it's exactly what it is. 

sava saheli singh: So, I think that's a good point, right? What is the question there is like, harm to whom? Where are you reducing this harm? To whom? And we get that too, like, "This is a closed system where our data is not going anywhere." But I was like, it's already been, right? What do, like, the things that we're doing has already harmed, and so how are we, what are we doing to reduce that? It's- ugh. 

Emily M. Bender: Yeah. Ugh. All right. So this is, our next artifact is actually a UNESCO report, under the rubric Education 2030. it was published in 2024, and the title of the whole report is "AI Competency Framework for Students." And there's an obnoxious graphic here, with a white femme-presenting person with dark hair sitting cross-legged holding a book open, and behind them is a monitor sitting on top of two books with an absolutely giant pencil about as big as the person's leg. And then there's a robot that is larger than the screen, and then there's little chat boxes, like speech bubbles, where it seems like the person who's probably a student is having a conversation with the robot. 

sava saheli singh: Is the robot coming out of the screen? 

Emily M. Bender: It looks like it, doesn't it? 

Alex Hanna: Yeah. It looks like, I believe the robot is pulling a "Ring," the girl in The Ring where she's emerging from the television. 

sava saheli singh: Horror. 

Alex Hanna: Yeah, very horror. And I don't think it's, as keydelk says, I don't think it's a synthetic generated image. I think it's just a really terrible image. So this is a piece, this comes out of UNESCO. And it's worth reading the short summary to say what this is, and then we'll get into the pieces of it. And so, "Short summary: Preparing students to be responsible and creative citizens in the era of AI. Artificial intelligence, AI, is increasingly integral to our lives, necessitating proactive education systems to prepare students to be responsible users and co-creators of AI. Integrating AI learning objectives into official school curricula is crucial for students globally to engage safely and meaningfully with AI." I'm gonna just pause there to get y'all's thoughts.

sava saheli singh: Like you were saying earlier, Emily, like there's never the option to not, right? Here they're like, "responsible users," "co-creators of AI." That makes it sound so nice, friendly, useful, like you're co-creating this stuff. And it's just- no, you're not. It's being taken from you- non-consensually, by the way, right? But it's being reframed into this thing where we're working with the system to do a thing. But nothing is being said about the people who had to work before to make the system. Were they working with the system? Were they co-creators? Again, language that is used to obscure so much of what, number one, went into making this stuff, and number two, how we are made to interact with it, right? And so it's like, "AI learning objectives," and forcing it into official school curricula, and engaging safely and meaningfully. It's like, a lot of students that I talk to, once they learn about this, they're like, "We don't want anything to do with this." Like, where is that, right?

Emily M. Bender: Yeah, exactly. I was gonna say, thinking about your work on surveillance in ed tech, sava, what does it mean to be safe, right? If these systems are so much surveillance of the students, whose notion of safety is really being applied here?

sava saheli singh: That's exactly it, right? It's this weird paternalistic thing, which is like, "I'm going to keep you safe, and my definition of safety is the one that is the most important, because somehow I've made myself an expert on this, and I've not consulted what you need to make you feel safe." And students talk about that, right? Like we have on campus, we've had students talk about how unsafe they feel, especially now since, in the ongoing Palestinian genocide, there are a lot of students who are on our campus who do not feel safe in those ways, right? Their safety has never been given any sort of thought, right? And it's very sad, because when students tell you what they need to feel safe, institutions rarely do that stuff, right? And like we were saying earlier, who is the harm being reduced for? Institutions are very much looking out for themselves. And so when they set the standard for what it looks like, they make it okay for them to surveil students. In the classroom, it's weird. We use something called eClass at York, and often in all of my classes, I show my students the view that I have as an instructor of the thing, right? And the thing is, okay, I've created it. I'm trying to get off of it, but I've created it, and you have the lessons you have every week. You have the readings and everything. And then I show them that where their names are, I can see what time they logged on, how long they were there, what they looked at, whether they did stuff. And it's like, they're always shocked. A lot of them do not know that this is happening, right? They don't know that they're being surveilled in this way. So then how, now they've integrated AI into the system and it's just like, what the hell? Like, when I opened it last month, we got a notice saying, "Oh, now there are these AI tools baked into this, and it's really helpful, and students can do the thing." And you read it, and the only button there is "I acknowledge." I think they've changed some of the wording now, but that's it. You don't have a choice. It's like, they have decided what it's gonna look like and how you're gonna access things. So it's it's really, yeah, it's really not great. 

Alex Hanna: It reminds me a bit of the point that I know Britt Paris recently had some research around the network of content management systems. Pick your, whatever one, they are just sort of all tied together. And the kind of data gathering that they're doing in the surveillance mechanism, and then they're then adding these third-party vendors to then ingest that and do whatever. And it's becoming even more and more opaque. So it's really, yeah, really absurd stuff. And I wanna point out just this language of co-creation that this document focuses on. So they mention it even twice on this page. There's another sentence here that just says basically the same thing, except they changed the word from "responsible users" to "responsible citizens," I guess to bring in some kind of element of democracy. But yeah, the idea that co-creation is in any kind of meaningful way happening with regards to these technologies and not as if they are just foisted upon us, is really obscuring so much of what's happening, and really frustrating. 

Emily M. Bender: I want to also get, for this little thing in the circle, it says, "By 2022, only 15 countries had included AI learning objectives in their national curriculum." Ah, '22, before the whole world went mad. On this, in this way. 

Alex Hanna: What a weird circle. I thought it was like a coffee ring. 

Emily M. Bender: Yeah, it's a weird graphic. So where are we going with this, Alex? 

Alex Hanna: So let's go to page 12, 'cause there's a first paragraph here that I want to get into, which is their introduction. So this is chapter one, introduction. "1.1: Why an AI competency framework for students?" So, "The rapid iterations and proliferation of artificial intelligence, AI, across all aspects of life and all sectors are posing new challenges regarding the nature of machine intelligence." What? "The collection and use of personal data, the role of humans and machines in decision-making, and the impact of AI on social and environmental sustainability. It is essential that education systems prepare students not only with the knowledge and skills to use AI, but also with the insight into the potential impact of technology on societies and the environment at large. Given the transformative potential of AI for human societies, it is crucial to equip students with the values, knowledge, and skills needed for the effective use and active co-creation of AI." So there's a lot here to be upset about. It's mixing the kind of, they're like, "Oh, there are these impacts and on society and the environment, but it is, we need to have this co-creative element." And yeah, I'm gonna pause and try to let y'all comment on this too.

sava saheli singh: It's really, it's interesting how I read through some of these. I've read multiple types of these kinds of guidelines, and I'm actually working on one, right? And what's really always interesting is like, even in this one paragraph, there's so many contradictions, right? And there's always this notion of, "Oh, we have to look at both sides. We have to allow for both sides. We have to find a balance." I'm like, how do you find a balance between existential threat and annihilation and, "Oh, look, this cute robot that comes out of your TV like the scary lady from-" So it's weird, again, language, and I know that this is your thing, Emily, in some ways, which is just how language is bent and shaped and made into a pretzel to make a certain point. And I think what I'll also say is, when you look at the authors of these things- and this is something that I'm grappling with, right? When we're writing these things, okay, they have me in the room to make those points, which is, you have to let people opt out. You have to include the environmental issues. You have to do this, right? So someone has said, "We have to include that," and then someone else has said, "But we also have to include this other thing." And it's this weird dance between like, who you are will read through these things, and we're trying to tell you it's bad for the environment, by the way, but also, it'll help you do this thing, but it's also gonna destroy- like, climate change is bad. And it's almost like a comedy act, which is just, what are you actually trying to say here? Because none of these things work together, right? And so it's wild. It's again, it's wild to me how people don't read this and be like, "What are you actually saying here?" 

Emily M. Bender: Yeah. And so frustrating. So yeah, this contradictory, written by committee thing, and the people who are in there saying, "We need to have no as a legitimate answer," the opt-out option are often just like, "Well, and then critics say..." And you're off in this box on the side. And the last sentence here was really getting me. "It is crucial to equip students with the values, knowledge, and skills needed for the effective use and active co-creation of AI." What about the values, knowledge, and skills to say, "Hell no"? That's, again, just not on screen here. And I haven't read this whole document. Alex, did you spot a definition of artificial intelligence anywhere here? 

Alex Hanna: No. There's definitely not, no. No definition. There's not a lot of definitions of many things here. In the beginning, this also talks about ethics, it talks about sustainability, and really doesn't go into what any of those really mean. What sustainability? What environmental cost, motherfucker? What are you talking about? Like, are you actually gonna define that? I'm thinking of the goose chase meme. What are you actually referring to, when you're saying this? And what do you mean by ethical? And I think it's such a, there's a lot of hand-waving here, in terms of this. And the intention of this is to provide a bit of a framework of competency, of navigating this to do this with enough sufficient warning, but what does that even entail? And it's really, it's a frustrating document, as many of these types of UN, and other international NGO documents are. But at the same time, shouldn't you define this to some degree? What are we actually getting into?

Emily M. Bender: Yeah. There's a comment here from ElizabethwithaZ that I love. "The idea that we should be putting these products in the hands of students and saying, 'Make sure not to do racism with the racism machine' is bonkers." 

sava saheli singh: Right? So many of these documents, like when you were talking about only 15 countries have done whatever- when they do that thing, it's almost setting you up for FOMO. Which is, hang on, are you a school that hasn't done this? Have you not prepared your students to be like good co-creators, good little co-creators with the AI, of the AI, right? And so many of these documents do that. And I think it's interesting, because a lot of people are working on these guidelines documents now, and how many years out from ChatGPT are we, right? And we have learned so much more about how it works, and so people are trying to bring this "responsible" kind of use of this stuff. And I was talking to my sister earlier, and she was like, "That's like saying 'Smoke responsibly.'" And so you know- and we have, right? A lot of people or critics have made the connections, or the comparisons to the tobacco industry and to smoking in terms of AI, right? What are you doing here? There is no responsible use for this. And even when cigarettes first came out, they were touted as a medical kind of thing, that was gonna help you for various things. And then we learned, and we stopped, right? And we can stop now. We're learning. 

Alex Hanna: Yeah, absolutely. There's some other elements of this that are pretty bad. So on page 14-

Emily M. Bender: Yeah, shall I read it? 

Alex Hanna: Yeah, go ahead. 

Emily M. Bender: So, "Section 2.1: Fostering a critical approach to AI." Somehow I don't think that's what this is gonna be. All right. "Critical thinking is a fundamental skill that students need to meaningfully engage with AI as learners, users, and creators. Students also have the responsibility to determine what types of AI should be developed and how they should be used to drive human societies towards inclusive, environmentally sound, shared futures. School students need to be supported to become active co-creators of AI, as well as potential leaders who will define further iterations of AI and its interactions with human society for the present and future generations." I can't with this. Like, it's just, there's- "No" is just absolutely not on the table here, right? "What types of AI should be developed? AI that will drive societies towards inclusive, environmentally sound shared futures." And also, it's this responsibilization, like, individual students should be doing this, when this is all happening because of the massive centralization of wealth and power, where it's being imposed on everyone. 

Alex Hanna: I also wanna bring in, I don't know, sava, if you've had a chance to read the new Canadian AI, whatever it's called. I don't- 

sava saheli singh: A little bit. Let's say yes. 

Alex Hanna: Yeah. It's the Canadian AI strategy, and it's, literally the main title is "AI for All." Which, first, very funny that you took Fei-Fei Li's weird nonprofit and used the same name. That also has many problems, but that's for a different episode. But then the kind of element of that national strategy document is like, "We need to get people involved in the process and upskill them to then get them to get involved with the creation of this technology." And yeah, just what you're saying, Emily, the idea that there is meaningful development of any of the people involved in this, rather than something that is just foisted upon them, is such a weird rhetorical move that's so consistent with this document and so many other documents from large institutions. And so, yeah, I'd love your thoughts on that and anything else, sava. 

sava saheli singh: It's weird, right? again, a lot of this is framed, like the sentence, "Students also have the responsibility to determine what types of AI should be developed and how they should be used." It's like, there's nothing saying that, number one, which students? Number two, if the students do this, look at all of this and follow the, quote-unquote, "ethical frameworks" that you're setting out and they decide that this is not a good thing, do you then allow students to have that voice to say no? There's nothing that's actually setting students up to be empowered to make the right choices once they learn what these systems are. And so that's a lie right there, right? Like they're assuming that students are gonna be like, "Oh my god, this is so great. Let's do it!" You know what I mean? And in the ways that it has been hyped to be done, everyone's told them that this is gonna change everything for the better. And once they start learning that it's not really, number one, it's not really that great. Everything is so mid, everything that comes out of things like ChatGPT, just really, right? And there are students who recognize that. I have students in my classes who are like, "Thank you, sava, for saying don't do this, or that this is bad, because we hate it. We don't want our students to use it. We can see our brains deteriorating as we use it." So it's not that young people are unaware, right? Again, these kinds of documents have that really, "We know what's best for you," and then we're gonna use language that pretends to empower you to make your decisions, but then we're gonna ignore your decision anyway, right? And that's like- that's what it feels like here. And the Canadian stuff is weird because the Canadian kind of push is just, "We need to make Canadian AI, and that's somehow going to be better than the American AI." And it's like, dudes, it's the same thing. There's really not that much difference between Canada and the US, I'll be honest. There really isn't. I've lived in both places for long enough. I came, I moved to Canada, I'm like, "Oh, yay, Canada." I'm like, "Oh, no, you're the same, if not worse in different ways." And to do this whole nationalistic thing about "Canadian AI is going to save us, because we're just gonna make more pipelines in the meantime, just ignore that part," right? So it's really wild. And it's really upsetting because- it's upsetting but not unexpected- that the politicians and the people in power are gonna say, "We wanna hear from you." And then when we do tell them, they're not actually hearing from us, right? Who are they hearing from? Who gets to have meetings with the Prime Minister of Canada? Who gets to have meetings with those people? We don't, right? Only people with money do. And so they're just setting all of those things, and they're creating a Canadian version of the American setting. And then one day those two things are gonna be married anyway because Canada and America cannot exist without each other. They share everything. They share data, they share surveillance, all of it, right? So to pretend like this is gonna be some separate, better thing, is just such a lie.

Emily M. Bender: It'll say sorry when it surveils you. 

Alex Hanna: The Canadian element of it, it's like, "Well, it's gonna reflect Canadian values." And so just like other types of things in Canada, they make some gesture towards First Nations, and then they say something about truth and reconciliation, and then they don't do anything about it. But then that's actually integrating people and listening to, making Canadian AI. But I don't, I'm taking us a little far afield from this, yeah. 

Emily M. Bender: Sava, you clocked how in this it says, it's taking this attitude of, "We know best. Here's what you should do." And this isn't written on the basis of actual expertise, right? This is written, it sounds like somebody who got wowed by the tech companies, and that's how they know best, is that these are the people with the money. They're the successful people. We want students to be successful, so we are going to pass through and put the UNESCO imprimatur on these ideas that are actually just coming from the people selling the products. Which, yeah. So I just wanted to ask if you wanted to do the cover going on in the chat, which is hilarious. 

Alex Hanna: Oh, no, I don't know what it refers to, so if you wanted to- 

Emily M. Bender: Oh, so I can't sing well enough, but-

Alex Hanna: What is it referring to? Sorry, I can't-

Emily M. Bender: "I'm not enjoying this cover of 'I Believe the Children Are Our Future,'" and the next line is something like, "Teach them well and let them lead the way." It's an '80s song. And I just can't sing, so I can't produce it for you. But-

sava saheli singh: I know the song well. I remember it, and they all sang it together and everything. It was just like, oh my god. Yeah.

Emily M. Bender: And then sjaylett says, "At least it's not, 'I believe the thousandth generation are our future.'" But it gets even better, because Oedipvs says, "Slop them well and let them vibe the way? Show them all the tokens they possess inside?" So, anyway, for the entertainment of our Gen X listeners. 

Alex Hanna: Sorry, this is one of the cases I'm gonna be like, "I missed this." At first I thought it was an Evanescence riff, which it was like, "Slop me up, slop me up inside." But anyways, and I apologize to all the millennials on that are listening. I wanna read one thing here, and then let's go to that terrible table. So the next page on there is the section on prioritizing human-centered interaction with AI. They say, "Critical AI competencies as proposed in this framework can also guide students to understand the unique value of social interaction and of the creative works produced by humans that should not be replaced by AI outputs." Sure. But then, "By developing competency for human-centered engagement with AI, the framework aims to prevent students from becoming addicted or dependent on AI, and to foster behaviors that maintain human accountability for high-stakes decisions." And I'm like, okay. 

sava saheli singh: Oh my goodness. This is, I feel like this was the influence of someone who was critical about it, right? Being like, "No, you have to have this critical language." But I also wanna point out, because we talk about harm reduction, right? Addiction is being used here. And I think the language around addiction is used around technology and phones and AI psychosis- I don't like that term. I wonder if there's a more updated term for it.

Emily M. Bender: I think it's Dr. Damien P. Williams who suggested "AI ensorcellment." 

sava saheli singh: Okay. 

Alex Hanna: I think it's not quite, it doesn't quite get at it. 'Cause ensorcelment is more about the condition of, being ensorcelled- which is, I know that's tautological. But there should be a bit of a better term.

sava saheli singh: Yeah. But like, all these terms around addiction and mental health and stuff like that. And then, it makes sense that they're trying to use the term "harm reduction" for some of this, because they're taking all of these things and applying them, and this is always what happens when you take concepts and you abstract humans away from them, right? And you abstract these things away from them, they become just concepts that you can just slap onto anything else. What is a critical AI competency? Just curious. I don't, know what that is. 

Alex Hanna: I think it's getting into this thing that's on the third section, which is the structure of the competency. This is my transition. 

Emily M. Bender: Great segue! 

Alex Hanna: So do you wanna talk about the first table here? 

sava saheli singh: Oh my goodness. 

Emily M. Bender: I'm, like, having visceral reactions here. So table one, "AI competency framework for students." And then there's, the rows are the "competency aspects," and the columns are the "progression levels," subdivided into "understand," "apply," and "create." And the first competency aspect is "human-centered mindset," and under understand it's "human agency," apply is "human accountability," and then create is "citizenship in the era of AI." And it's like, those first two, fine. Let's keep agency with the people, and let's keep the people involved- especially those with power- accountable. But the era of AI, I think we can reject that, right? And then the next competency aspect is "ethics of AI," and the progression levels here, understand is "embodied ethics," apply is "safe and responsible use," and create is "ethics by design." It's like, this create column just seems bananas to me. And again, it's because they've lost track of the possibility of saying maybe just no, right? If instead of "create" it was, like, "act," maybe, and then you could have, my action is refusal in that column. 

Alex Hanna: I would say that a little bit of confusion is that they spend the bulk of this report even going through this. And the embodied ethics, first off, are really, it's- so first off, a really weird deployment of the term embodied, because the comments behind it have no reference to bodies and corporeality. I'm like, I don't know exactly what this is getting at. But they have some subprinciples which are "Do no harm..." 

Emily M. Bender: 'Cause all students are doctors now. 

Alex Hanna: Yeah. First off, I'm like, okay, so then you're mangling another oath or another ethic from another field that isn't quite... So like, where's the "do no harm" that should be, don't use this at all for what you're suggesting and what you're doing. But they are narrowly scoping harm to purposes that might be harmful for humans, which they say is, "such as facial recognition for surveillance or assigning social status or predictive algorithms for grading examinations." So then that might get us close to what kind of definitions that they're considering for, quote-unquote, "AI." But then what are the myriad other types of harms that you're just patently leaving out of this "do no harm" principle? 

sava saheli singh: It's so weird. And for whom, right? Nobody's asking for whom. Like even the citizenship stuff, right? Who are these citizens? Of where are they allowed to be? Who are we talking about? There's so many populations, demographics, people with wealth. Citizenship status means something in different contexts, right? And who gets to have it, who doesn't? Who are the people who are being harmed? Who are we doing no harm to? Like, when you start taking apart some of these things, you're like, hang on, who has already been harmed? Are we not gonna talk about them at all? We're just looking at, here's- it's just, it was born yesterday. You downloaded ChatGPT on whatever thing, and that's when it starts existing. And then from that point on, you have to assess how- Like who assesses what harm is, as well, right? Have they described what harm is beyond saying things like environmental harm- that seems to be the main one that they talk about, but they're also talking about grading examinations or assigning social status or predictive algorithms. This is already happening, so it's very weird. Like there is- has someone secretly written this to be like, "Listen, this is happening. Read this and tell them to stop. I'm not allowed to say, 'Don't do it' in this document for whatever reason, but I'm encoding it into this document so that you somehow stop it." I don't know. Is that a secret person who is giving, who's trying to tell us something here? 

Alex Hanna: I think it is 100% reflecting the nature of a type of document that is like this, that is written by committee, and has a lot of people trying to fight for their little place. I'm looking at the author list near the, that's in the peer review. And so it starts with, the framework was drafted by someone named Fengchun Miao, chief of unit for technology and AI and education at UNESCO, Kelly Shiohira, director of Global Science and Learning Education Network, and Natalie Lao, executive director of the App Inventor Foundation. And then there's a lot of reviewers who are people who are AI in education and a lot of engineering- 

sava saheli singh: Co-founders of companies. 

Alex Hanna: Yeah. So it's, I don't have a lot of confidence in the kind of critical aspect to winning out here, that just being a vague hand-wavy element of this.

Emily M. Bender: Yeah, so there's, LozzaHouse in the chat says, "Do no harm! Except to data workers, writers, artists, the planet, yourself..." I think that captures it nicely. 

sava saheli singh: Exactly! Exactly.

Alex Hanna: Yeah, ezysigh is saying, "AI Club. We're all part of the AI Club now." I think AI Club is the, most- 

sava saheli singh: There's a Pro AI Club in there too somewhere, I think. 

Alex Hanna: Oh, yeah. Wait, "Nisha Talagala, co-founder and CEO of AIClub and AIClubPro." AIClubPro! That's the paid version without ads. Oh my gosh. 

Emily M. Bender: Yeah, so, any sort of concluding thoughts here, sava? 

sava saheli singh: I think, again, drawing on my recent experiences around trying to craft a document like this, I don't know where it's gonna go. But one of the things that came up, I remember, was somebody shared a form, which just looked like a standard form, and then said, "Fill this in." And then at the bottom of it said, "Full disclosure, I used-" you know, I forget what thing they used, but they used one of these products to create it. And I replied and said, "This makes me really uncomfortable." And they were taken aback by that, which is interesting to me. But I started this long conversation about consent because I made very clear that I don't want any of my contributions to be touched by this stuff, right? And that really confounded everybody present. They were like, what... a little bit they were just like, "What do you mean?" And I'm like, "How do you not understand what this means?" And it's very interesting to think about, in that same kind of context of like, students have to have agency to do things, right? And what does consent mean? I don't think consent means the word- Again, words, what do they mean, right? If consent is like, you have to click yes to access something, that's not consent. If you're not gonna get, be able to access it without clicking yes. And so people don't understand, when I said, "I do not consent to this," they're like, "We don't know what to do with that," right? And it's just, yeah, right? Now when you actually enact or embody meaningful consent, it breaks your brain because you're just like, "Hang on, someone can say no to this?" And I think that's interesting, too, right? Like you were saying, there's nowhere in here that anybody's allowed to say no.

Emily M. Bender: Thank you. So Alex, all right, for our transition. 

Alex Hanna: Yes. Let's do it. 

Emily M. Bender: You are an elementary school teacher in a far future where our civilization has collapsed, but somehow this document has survived. So there aren't even any computers, but you're trying to teach the AI lesson on the basis of this document.

Alex Hanna: Wait, I'm trying to teach an AI lesson, or the AI- 

Emily M. Bender: You're teaching, it's all people. There's no AI around. 

Alex Hanna: Ah, okay. Yes. 

Emily M. Bender: But you have this surviving document from the previous civilization that tells you how to teach, and so you're going to teach the students about AI from this document.

Alex Hanna: Got it. Okay. So sure. Okay, class, sit down. Okay, there's something called AI. we're supposed to co-create it, everybody get out your sticks and your rocks, and we're gonna just, let's just make A's and I's together, and then that's it. It's just, we're in a burnt down house, that's in a field in the middle of nowhere. That's it. Yeah. I wanna actually riff- 

Emily M. Bender: Let's reflect on how ethical we were. 

Alex Hanna: I know. I also want to riff on the chat 'cause there's lots of AI club mentions, so, Lozzahouse says, "The first rule of AI club is that no one talks about the possibility of not joining AI club." Very good.

sava saheli singh: We're all in AI club.

Alex Hanna: I responded and I said, "The final rule of AI club is if it's your first time at AI club, you have to prompt." And sjaylett said, "Sama: 'I just wanted to destroy everything pretty.'" Great Tyler Durden quote. And then magidin says, "The middle rule of AI club is to hand over all your money to the AI hucksters."

sava saheli singh: I wonder if when you were saying, Emily, about teaching this document to students of the future, I wonder if you could use it as a historical document. I'm reminded of like, when people- someone was sharing online a New Yorker cartoon where there's a guy in a suit sitting around a fire being like, you know, "For a brief moment, we were," whatever. "We destroyed the world, but for the brief moment, we had like, really great market cap" or whatever, right? 

Alex Hanna: Yeah, it's like- and we created so much shareholder value. Yeah. 

sava saheli singh: And so this could be like, you find this thing and you're like, "Children, back in the day we did really stupid shit like this."

Alex Hanna: But boy, did we create such a bubble. I will create, and I'm referencing a meme, I will create a bubble so toxic that... I don't know how to finish that. 

Emily M. Bender: I'll create a bubble so toxic that I can drown Altman and Musk and Amodei and all of them in it. 

Alex Hanna: Oh, wow. Ooh, I got it. 

Emily M. Bender: All right, so first you get the first one here, Alex.

Alex Hanna: Yeah, and I know sava will have thoughts on this one. So this is a screenshot of a paper by, and this is on Bluesky, the person screenshotting it is Peter Whitewood, and says, "Rebranding hallucination as, quote, 'a critical confabulation that can fill the archival gaps' is a big no from me." So basically, I'm not gonna read this whole thing because it's long and I hate it. But I will read the first half, which says, "LLMs hallucinate, yet some confabulations can have social affordances if carefully bounded. We propose-" in bold- "'critical confabulation,' inspired by 'critical fabulation' from literary and social theory, the use of LLM hallucinations to, quote, 'fill in the gap for omissions in archives due to social and political inequality and reconstruct divergent yet evidence-bound narratives for history's, quote, 'hidden figures.'" Reading that aloud just made my skin crawl in a way that I wasn't anticipating. But there's so much that's offensive about this. The idea of critical fabulation is not from literary and social theory. Like this vagueness. It comes from Saidiya Hartman writing about the fucking slave trade and the ways in which stories from enslaved Africans are lost and basically only written into colonial records. And it's really fucking offensive to see it in an abstract like this, not to mention the, quote, "hidden figures," taking from that, the narrative of the three Black women working at NASA who worked on the Apollo program. So yeah, this pissed me right the fuck off. 

sava saheli singh: Really! This is an insult to the great Saidiya Hartman, absolutely. And it's weird, like, where do you even start with, what critical fabulation was meant to do as, I don't know, weirdly enough, as potentially thinking of it as some sort of harm reduction in terms of bringing the stories that we lost, right? What was between the gaps? The concept of, again, you've abstracted it away from what, who it was meant to be for and then created this thing. And this seems, I don't know, is it like, I recently saw, I don't want to name him, but I recently saw a fairly well-known writer who has a fairly popular portmanteau that he's come up with, no, neologism that he's come up with, kind of use Audre Lorde to talk about LLMs and say the master's tools. And I'm like, does nobody read the rest of that, or does nobody read the very short essay that it is? You know, it was a talk. Just read the whole thing and understand how you're using this, right? And so this is the second kind of example of taking something from Black women who thought very deeply. And I can't, I don't even know- the words escape me, how enraged I am by this. Like, how are you, number one, misrepresenting, number two, misunderstanding, and then number three, misapplying what they're trying to do for us, right? And for us! That's, it's true. I have learned, we learned so much about what was done to peoples, right? And how dare you do this to that? It's so upsetting. 

Emily M. Bender: Yeah, and to just sell it as, "We are now doing this special new thing," and it's just, it's so gross on so many levels.

Alex Hanna: Yeah, and as if an LLM making shit up could, that is trained on all this past data can somehow point to some kind of liberatory future or fill in a gap in an archival record or speak to archival silences. On, it's trained on the archives, my man. Or the digitized archives. That is not what filling an archival silence, or imagining that is, that's not going to happen with an LLM. No. 

sava saheli singh: We see this a lot in research things, right? Like they're using this and then they, you're not doing the work then, right? What is the point of going deep into archives and noticing what's missing or noticing the differences? You cannot do that with these products, right? It is, the process of doing it is part of where, how you arrive at the thing you're arriving at. It's wild to me. I have so many colleagues who straightforwardly tell me how they use ChatGPT or Claude or Perplexity or whatever to do their research, that, "I fed all of my stuff into this thing." I'm like, "What do you mean you..." Like, I'm just like, "What did you just do? What are you doing? What have you done?" I'm just like, "What have you done?" And to claim that this is part of a research process is so egregious, right? It's like you're short-cutting yourself. You're taking away from the work. It's tedious work. It's not fun to do a lit review. It's not fun to comb through archives, right? It's tedious. It's a lot of work. It's boring even. But then you arrive at the thing, and that's the process, right? And it's, so, 100%. It's, really sad. Really sad. 

Emily M. Bender: Yeah. All right. I'm gonna make us go faster through the rest of this, 'cause I would like to get to a few more. So this one, I am personally offended. Character AI apparently has a character in their system called Emilie, spelled the French way. And here the headline is from May 13th, 2026, in a publication called National Law Review. The headline is, "Emilie is not a psychiatrist: Pennsylvania Board of Medicine alleges unlawful practice of medicine by an AI chatbot." And basically, Character AI, one of the things that they were able to access in it, had a thing that basically was pretending to be a psychiatrist, so with an MD, with actual practice experience, and behaving in contradiction with the laws of the state of Pennsylvania. And so there's a lawsuit here, which I am excited about that happens. Alex, you wanna take us through this one real fast? 

Alex Hanna: Yeah. Next one, The Verge, by Stevie Bonifield. The title is, "Google Home will now get better at recognizing you. Google's smart home AI will be able to use clothing to identify people and recognize specific sounds caught on camera." Terrible. There's also something here where, it's something like, it's using the Google Photos' recognize people element, where it recognized other people in your home that come over without their consent, which is awful.

Emily M. Bender: Yeah. Ugh, okay. Next one is some reporting in The Seattle Times, June 23rd, 2026. Do I have a journalist? Jessica Fu. Headline is, "Medicare AI programs made them suffer in pain. Now they want answers." And so this is basically, Medicare implemented automated claim denials basically in the name of efficiency, and then all of these patients couldn't get their care covered, and all of these doctors spent all of this extra time fighting with the system to try to get care for people. So really efficient, definitely. Your turn, Alex. 

Alex Hanna: Yeah, this I take personally. This is written deep in Pride Month, so it was June 10th, by Cydney Hayes, in something called The Gazetteer for San Francisco. The title: "Why Do These Castro Gay Bars Have TSA-Style Face Scanners? A handful of neighborhood bars have installed facial recognition devices that collect and share customers' names, addresses, genders, and indiscretions with each other." Really fucking frustrating. 

sava saheli singh: Indiscretions!? 

Alex Hanna: Indiscretions, yeah. I guess, I don't know what that... I don't know what that necessarily, what they actually count. I don't know, the Castro is not gay. That's my hot take. 

Emily M. Bender: Yeah. All right. One more bad one, and then we have two palate cleansers. So this is 404 Media. We love 404 Media. This one is by Matthew Gault, June 30th, 2026, and I think the headline says it all: "County With 37 Data Centers Asks Schools To 'Conserve Electricity.'" And the sort of mechanism here is that the electricity costs went up, and so all public services are trying to cut costs, and things like schools are not gonna have enough money for the electricity that they were previously using. Because why? Because we need all those data centers, apparently. All right. So this is CNBC, Monday, June 1st, 2026, by Ashley Capoot. And the headline is, "Florida AG sues OpenAI, seeks to hold CEO Altman personally liable for alleged harms." And, love to see it, and like, did not expect Florida to be the first up to the bat doing that. And finally. 

Alex Hanna: Yeah, so this is Reuters, just by Reuters. "Norway imposes near ban on AI in elementary school." So, good job! And, let's see details. "'Near ban on use of generative AI tools by elementary school pupils while also restricting their use in the education of older children to prevent a negative impact on learning,' the country's prime minister said on Friday." Wow. Damn, it can be done. 

Emily M. Bender: It can be done. Inevitable, no!

Alex Hanna: Inevitable, no. And I was about to type a response to LozzaHouse where they say, "My indiscretion is wearing the Google logo underwear. Please don't share with anyone." And I want to respond in Devil Wears Prada voice, "Is that the..." "Yeah, the Google logo underwear." Anyhow, that's it for this week! dr. sava saheli singh is an assistant professor of digital futures in education at York University in Toronto. She has a speculative short film series on the harms of surveillance technology available at screeningsurveillance.com. Thanks so much again for joining us!

sava saheli singh: Thank you so much for having me. That was a lot of fun, and I was nervous about it, but I really enjoyed it. And I also wanna say, I am working on a film about AI and education, so hopefully that's coming along. And I'm not sure what form it'll take because that's part of my research practice, and I'm not going to use any, quote-unquote, "AI things" to do any of it. And I'm happy to continue talking about all of these things, having conversations with people. If people are showing my films in your classrooms and you want me to come and talk to your class, I'm happy to do that. I'm always happy to talk about stuff. Thank you so much, Emily and Alex. This was amazing. I really enjoyed myself. 

Emily M. Bender: Yeah. Thank you, sava! This was such a fun conversation, and I want to encourage all listeners to take sava up on that really generous offer because, as you can see, she's a wonderful interlocutor. 

Alex Hanna: Absolutely. Our theme song is by Toby Menon. Graphic design by Naomi Pleasure-Park. Production by Ozzy Llinas Goodman. And thanks as always to the Distributed AI Research Institute. If you like this show, you can support us in so many ways. Order "The AI Con" at thecon.ai or wherever you get your books, or request it at your local library. 

Emily M. Bender: But wait, there's more. Rate and review us on your podcast app, subscribe to the Mystery AI Hype Theater 3000 newsletter on Buttondown for more anti-hype analysis, or donate to DAIR at dair-institute.org. You can find our merch store there, too. That's dair-institute.org. You can find video versions of our podcast episodes on Peertube, and you can watch and comment on the show while it's happening live on our Twitch stream. That's twitch.tv/dair_institute. Again, that's dair_institute. I'm Emily M. Bender. 

Alex Hanna: And I'm Alex Hanna. Stay out of AI Hell, y'all.

Head Shot

Alex Hanna

Co-host
Head Shot

Emily M. Bender

Co-host