It's literally making stuff up and it has no idea what it's making up. Therefore, it is a bullshit engine. It's a bullshit engine in that it makes up stuff that is essentially has no semantic content, grounding causality or any of that in it, and it's bullshit because its only goal is to be plausible.
SPEAKER_01Now, obviously, this is a conversation that is very relevant right now, as Chat GPT and these image generation tools get a lot of media attention, public attention. You know, there's a lot of big promises and expectations for what they might mean in the world, you know, often very positive things, most likely or almost certainly overpromising on what they can deliver, as is so often with the tech industry. So I think it's important that we have some conversations that, you know, throw a bit of cold water on this, throw a bit of reality on what is likely to come of these technologies and whether we should be so quickly embracing them as Sam Altman and these other billionaires expect us to do. One thing that was really fascinating to me in having this conversation was that Dan told me when he set out to write the book, it was actually called AI for good, right? And he was looking for a good version of AI to promote. And as he did the research, as he further looked into artificial intelligence, these technologies, how they work, the actual impact that they're having in the world, he changed that to what is now resisting AI, you know, a very different take on these technologies and how we should approach them. And I think that this conversation is really instructive, not just to understand how these technologies work, but also how they are deployed and the risks for how they can continue to be deployed as they are improved and advanced and made more capable or have more computing power behind them, not just in the private sector as they are rolled out by these companies, but also in the public sector, right? As more of these tools are used to deliver services in governments that are increasingly subject to austerity spending cuts, don't have the money to deliver the services that publics expect from them, and they look for quote unquote efficiencies in order to supposedly deliver these things in a better way. But actually, what they end up doing is creating more exclusion, making it more difficult for the poorest and most marginalized people in society. And so if we think about these technologies and the impact that they're going to have, it's much more likely to be more of that rather than you know being your doctor and your teacher and all these wonderful things that Sam Altman would want us to believe. So I think this is an important conversation. You know, I really enjoyed talking to Dan. I think you're going to benefit from it, learn from it, enjoy it. At least I hope so. And I think that especially in this moment where AI is getting so hyped up, you know, we need these kind of conversations to give us a bit of a reality check. So with that said, if you like this conversation, make sure to leave a five-star review on Apple Podcasts or Spotify. You can also share it on social media or with any friends or colleagues who you think would learn from it. And if you want to support the work that goes into making Tech Won't Save Us, you know, having these conversations with people like Dan to give us the critical perspectives on these technologies that are being hyped up by the tech industry, you can join supporters like Bayorn from Bern in Switzerland, Pete in Pauling, New York, and Randy Seldinger in Seattle by going to patreon.com slash tech won't save us and becoming a supporter. Thanks so much and enjoy this week's conversation. Dan, welcome to Tech Won't Save Us. Thanks very much for having me. Very excited to chat with you. You know, obviously these AI tools are everywhere today. And last year you had a book published called Resisting AI, which is very relevant to uh everything that is going on in this moment. Great timing on the book, but it gives us something great to talk about and to dig into it, right? Because I think it's a really important perspective on AI to have, especially in this moment where there is so much hype and excitement about it. And so I want to start just by getting your kind of initial impressions of what has been going on the past few months. Obviously, last year we had the kind of slow rollout of these kind of image generation tools, things like Dolly, stable diffusion became slowly more popular throughout the year. And then, of course, in the past few months, we've had ChatGPT emerge. We've had the deal between Microsoft and OpenAI to kind of build that into the Bing search engine. Google is moving forward with its own kind of AI search engine integration. Facebook is talking recently about how it also has its and and it might do something with it soon too. You know, because of course now that there's the hype, everyone has to get in on it. So I wonder just generally what you've made of, you know, kind of the responses that there have been to these technologies over the past few months as they have slowly kind of become much more common and the public has been kind of interacting with them.
SPEAKER_00Well, I suppose what's kind of currently on my mind is just because being stuck in academia, you you tend to see what's most immediately around you. And I'm really, really distressed right now about the way a lot of academics are just rolling over and going, oh yeah, well, uh, you know, Chat GPT or the all the language models, they're here, you know, they're inevitable, and you know, we have to learn how to live with them. And they even valorize this, you know, they're saying, uh, ah, you know, we can adapt our ways of writing to, you know, include particular forms of creativity and we can use these tools. No one could say that there's absolutely nothing to be got from these tools, but I think these kind of statements are anyway what I call in the book AI realism for starters, which is this sense of inevitability around AI, no matter how visible it's sort of toxicity. And also the fact that they just seem to be able to sweep this toxicity under the carpet. I mean, the absolutely appalling sort of um varieties of harm that are immediate and already rolled up in something like a language model just don't seem to trouble them. So I guess that just happens to be my little fishbowl at the moment. I'm not searching for something good to say, because I have nothing good to say about large language models, but I suppose there's a kind of collateral benefit in, as you say, it's just pushed it further to the front of everyone's agenda. And so in that sense, I was reading a great thread just now by Emily Bender on OpenAI and their sort of delusionary statements about their careful custodianship of the future of artificial general intelligence and all this kind of nonsense, which is just basically marketing stuff, really. But it also represents that kind of ideology. And you know, that's that's really great that lots of other stuff is coming kind of out from under the stone, some things that were perhaps only of concern to those of us who spent too much time thinking about this kind of thing anyway, and worrying about it and trying to get people alerted to it. Now it's out in the open. It is mostly hype, and people seem to be getting swept along with it. But, you know, those of us who are in the skeptical corner, you know, are trying our best to highlight the urgent downsides, and I think some more of those will come out. I mean, I won't want to go on about it too long, but just to say it's it's a kind of weird time for me because my main concern and my concern in the book wasn't really to focus on these most sort of spectacular forms of AI in a way. I mean, I think they're important, but my gut feeling has always been actually that the most impactful applications of AI or effects of AI are those things that sort of much more invisibly permeate through the institutions and come to touch people's lives in much more ordinary ways, you know, when they're claiming benefits or when they're seeking housing or when they're getting educated or whatever it is. And I don't think this is diverting, I think it is just shining a light on that stuff. But my sort of final resentment for the moment would be that by adopting these things, like Microsoft, for example, still stuck in with them, GPT in Bing, to my mind is also kind of it's just adding another brick of legitimation to these kind of technologies in general and their general application to our lives. And that's what resisting AI is really about, is the broad resistance to these kind of computational assemblages being given any kind of influence over our everyday life at all.
SPEAKER_01No, I I appreciate how you put that. And I guess that is kind of a bright side that you've discussed, right? Is even as the hype kind of accelerates around these technologies, that does also give us an opportunity to have more of these kind of skeptical views about these technologies, get more prominence and attention. And as you say, Emily Bender is someone I definitely need to have on the podcast to discuss this with her as well. I want to go back to what you were saying at the beginning about, you know, kind of the reaction of the academics that you're seeing, because it does seem like, you know, obviously there's this kind of booster-ish response to it, where, oh my God, this is going to change everything. This is great, the kind of Sam Altman perspective on this and the people like him. But then it does seem like there's also kind of a pragmatic response that also kind of plays into this as well, that's really beneficial to them, which is associated with what you describe, right? These are here, we need to figure out how to adjust to them rather than saying, hold on a second, do we actually need these technologies at all? And should we be embracing this and and acting like it's just inevitable, or can we push back on it?
SPEAKER_00Yeah, I mean, just to sort of skip to the punchline in a way, I'm, I guess, a rejectionist or an abolitionist in that sense. And I think we will talk about this. I think it's that's not just about saying no, it's about saying, what are the alternatives and how we go about that? But what you made me think of just then was I do agree that there's first off that there is this more self-styled sort of realistic approach to it. What I think in a way is that that's kind of more dangerous because it's easy to dismiss the hypsters and the and the boosters, and people understand that positioning, you know, there's hypsters and boosters in in every area of life. So people are somewhat inured to that. I think it's the it's the positioning of people as responsible to be able to sort of, well, yeah, absolutely, there are problems with these things, but you know, given that we have them, this is the inevitability, given that we have them, we can what's really frustrating about that is that implication that they are the people who should make that decision. You know, that somehow these responsible, liberal, bourgeois intellectuals should be the ones who can take on this uh essentially monstrous technology and decide for the rest of us how best it should be managed for the common good, which is coincidentally something that they get to decide. So um I think it's I mean, in a way, and um, I'm not, you know, I don't really make any sort of secret of my positionings around these things. One of my bigger concerns about AI, I suppose, and its openings towards facilitating reactionary solutions to things, is not that that is either a sharp break in technology, it's not sci-fi, or that that's a sharp break in politics. You know, that the the harms that we're talking about are an intensification of the harms that are happening right now under our, you know, uh nominally democratic, apparently liberal regimes. You know, so it's not just that we've got lunatic, spacefaring, long-term visionaries, if that's the right word for them. May they all go off in rockets on a one-made ticket. It's that we've got the responsible authorities who are responsibly closing off borders so that people die in the desert or on the Mediterranean or responsibly doing all the other things that are causing harm right now. And those are the same people who are coming along and saying, don't worry, you know, we have the capacity and we have the responsibility to manage these things sensibly for everyone's benefit.
SPEAKER_01Yeah, it's a huge concern, right? And we'll obviously come back to it through the course of this conversation to talk about how, you know, this does make AI technologies much more concerning and worrying, especially when, you know, they're not the very visible Chat GPT style tool, but are much more kind of secretly integrated into the systems that enable all the problems that you're just talking about, right? And so before we get to discussing those things, I think it's better for us to get an understanding of how these technologies actually work, right? Because that I think is going to help to illustrate how these problems are caused and why they're difficult to avoid. So can you talk to us a bit about how you know AI technologies, so to speak, actually work? You know, how they kind of learn, quote unquote. You know, I'm putting scare quotes on that for listeners who can't see me. Very scary. Totally. So, how should we understand these technologies and how they're actually developed and how they function?
SPEAKER_00Okay, well, thanks for that. It gives me a chance to try and ground, you know, at least sort of backfill some of my opinions with some semblance of being grounded in an understanding of what's going on. And maybe if we work backwards from the language models, the large language models, which are the ones that are so front and center of everyone's attention. I mean, they're, you know, essentially text prediction machines. Inside of AI in general, inside of language models per se, you know, there are some sort of clever ways of doing things. I'm not questioning that. You know, they're kind of somewhat deft and skillful ways of manipulating things under their own terms. And what they're largely manipulating are statistical optimizations of one kind or another. So it's machine learning. And so language models are just have used transformer models essentially, which is just a particular way of setting up that kind of optimization, to really just learn how to fill in the blanks. I mean, they've learned how to, you know, the cat sat on the blank. That's how they've learned to fill in that gap because they have quite sophisticated methods for doing that, uh huge volumes of data to do that, quite a large amount of invisible ghost labor, which you know, invisible human input into that, and you know, oodles and oodles of computing power, then they're they do that that very well. And doing that very well gives them a number of capacities. I mean, they can produce grammatical, mostly contextual and sometimes creative seeming texts uh that are doing what they're meant to do, which is be plausible. They don't have any capacity to understand. They don't understand anything. They don't understand anything more than a teletype machine understands anything. They produce sequences of letters, and they've learned how to do that in a way that achieves a mathematical minimization. Now that does gift them with certain affordances, and producing sort of plausible-looking text is one of them, and it's plausible exactly because it's been trained against text that was actually produced by humans to be a good imitation, and that's all it is. It's an imitation, it's you know a somewhat randomized imitation of what humans can produce given the right set of inputs and prompts, and given the vastness of the different examples and subjects and topics and contexts and sources that have been sucked into these things, you know, what they can therefore produce is also quite diverse. But I mean, I guess, and this is where you could really help me by prompting me. I guess having spent a little bit too much time looking at that, I sometimes get a bit flat-footed when I'm faced with people who kind of read it as anything more. You know, I mean, I can understand it. I'm not trying to belittle anybody, but because it's plausible text doesn't mean what it's saying is plausible. It doesn't mean that it has any conception of what it's saying. It's us reading into that pattern that there's some meaning in it. You know, we are bringing the meaning to it. There is no innate meaning in what an AI is doing other than a mathematical operation of statistical minimization and a reproduction of that by deploying that model, that set of learnt parameters basically. There's nothing else in it. If it seems reasonable to us or if it seems interesting and weird to us, that's us. That does seem to place some pretty strong limits on what we'd trust it to do. You know, I think of it as a like it's a really, really interesting party trick. But why would you do anything else with it? I don't know. Why would you allow it to be an in an integral part of our knowledge searching structures and search engines? And why would you let it write your essay? I mean, that sounds like a ri I'm not trying, I'm not getting at my students here if anybody's listening, you know, that's you want to try it, that's up to you, right? But it's inevitably going to be produce what the industry itself calls hallucinations, which I think is still overreading it because it implies that there's something doing the hallucinating. But let's use that word for the moment. Definitely, you know, it it always literally making stuff up and it has no idea what it's making up. Therefore, it is a bullshit engine. It's a bullshit engine in that it makes up stuff that is essentially has no semantic content, grounding causality or any of that in it, and it's bullshit because its only goal is to be plausible, just like somebody who tries to bullshit you at a party that they know all about fusion physics or something like that, and they they don't, you know, they're just bullshitting and they're just trying to sound convincing. Um, so yeah, I I don't know. I mean, that's that's language models, and what they're doing is is a particular application to this whole business of trying to predict blank words and sentences. That's the learning method. That's just a very specific instance of the general what we call AI right now is a very particular slice of what could possibly be called AI. And AI over history has been lots of different things, as I'm sure you've covered before, and you you you very well know yourself, right? So there's lots of different kinds of AI. This is one very particular kind, it's called connectionist AI. So it's not relying on the idea that there's rules in the system, there's no expert stuff in the system. What it's relying on is that if you present this kind of learning with a huge enough data set, you give it a certain amount of desired outputs and you make lots and lots of interconnections in between those two things. And then you sort of turn the handle and you keep on doing a sort of self-correction process called backpropagation so that you try to more closely approach the quote-unquote correct output given your input. I mean, that's all it's it's just a kind of turning the hand like a mang. It sort of mangles this stuff until it learns how to fake it, it fakes it till it makes it, basically. And that's pretty much the whole of sort of, I mean, deep learning, reinforcement learning. They're all variations on a theme, you know, um, they all but they all have these common characteristics. They're all based essentially on mathematical optimization. There's absolutely no reasoning involved in in any sense to understand it. Um they all require vast amounts of computation, vast amounts of data, which have their own political implications, of course, and their own ethical implications and their own social implications. So yeah, I don't know. That's my starter for 10.
SPEAKER_01No, I I think it's a very uh concise explanation of how these things actually work. And I think what you get at there is why some of the kind of reporting around it has been concerning, right? As a lot of these publications have been publishing transcripts with ChatGPT and saying, like, oh my God, look at all the things that it's been saying to me after I fed it these particular prompts to lead it in that particular direction. You know, this is something that we should be very worried about, in like, not so much in the ways that I think we'll be talking about, but more in the way that, oh my God, look at the things it's spitting out. What if it actually means these things or gets people to believe them or what have you, right? It does seem like uh looking at this through the wrong lens of what the problem is here and how we approach it. You mentioned there the degree of computation that is necessary for these AI tools, Chat GPT and things like that in particular, but many of the others. Would you say that part of the reason that the hype around this is happening in this moment is because there has been so much progress in computational power and creating these kind of cloud data centers where there's so much computing power available? Or would you link it more to a development in kind of the politics and how that has evolved in our society in recent decades?
SPEAKER_00I try to earwig a bit on industry conversations. And actually, a really good podcast for that kind of thing for anybody who wants to sort of try and listen along is called Twimmel, right? Which is This Week in Machine Learning. And the host on there, Sam, you know, interviews key practitioners in all of these areas and does a very good job, I think, of uh some some of it would be um just very technical or very industry focused. But if you can sort of listen along, you understand some of the more um internal narratives. And I think it's pretty well understood in the industry that the main thing that's going on at the moment is scale. There have been some actual changes, and there's a little bit of competition between, you know, do we a deep learning or reinforcement learning, and now we've got transformer models, and they have, well, perhaps not surprisingly, transformed a lot of things, but not fundamentally. You know, fundamentally these are just there are variations on a theme, and essentially it's about scale. It's about compute power, as they say. But I think something that I should have said a little bit earlier, in a way, is, and I'd like to sort of emphasize this to people is that I may sound like I'm going on about the computational technology, and I am, because I think it's important to be materialist about these things, you know, across the board. I mean politically materialist, uh or rather, sort of a materialist in the sense of political economy and follow the money and everything else, but also actually look at the technologies themselves as, you know, not being simply neutral or passive technical frameworks in which people um invest their politics. I mean, of course they do. We know that. We can talk about the the weird politics of supremacist politics that gets invested by Silicon Valley in these machinery. But they're not just passive vessels for our imagination. They have particular characters, they make certain things easier and certain things harder. But my reading of the politics of them, it does start from, or at least that's how I try to do it in the book. I do try to start from let's look at what AI actually does, not what you know the newspapers think it does, or certainly not what sociologists think it does, or maybe not even what some computer science think, but what is it really doing as a material technology? Because but that's because it never exists like that by itself. You know, that stuff didn't just appear. We tend to talk about it as if it kind of exists over there somehow and then is being brought into society. That is a kind of dangerous fiction. I mean, this stuff emerged out of the same social political matrix that we currently inhabit and immediately goes back into it. So it's really impossible to say. Separate the operations that I find really interesting from the institutional assemblies that they're always part of, of the kind of you know, ideas and ideologies and and sort of values that are already present throughout their creation process, throughout the process of even conceiving of these things, throughout their operation and throughout their application. These things are can never be separated, they just act as a particular condenser or a particular channel, you know, within that process. So a real understanding of what these mean in a way in the world can never really be separated from the wider social matrix. And so I guess for me, trying to read the significance of AI is trying to say, well, what are they actually doing and how could this affect people's lives? But it's trying to say, but we're not starting from a blank here. You know, we're starting from the societies that we all have, as they say, we're all experts by experience, right? We all inhabit this world and we understand some of its very stark and peculiar topologies. These same landscape is that out of which AI has emerged, from a particular corner of it, yeah, but out of this particular world, and that's where it actually operates. There's no other space. So if we need to understand what AI means, where it comes from, what it's going to do, you know, we can't really do that without trying to see it as part of a jigsaw puzzle in this wider jigsaw puzzle of society as we know it and history as we know it, the ongoing dynamic and unfolding of history. I think AI is part of that history right now, and it's certainly playing an active part in trying to foreclose certain futures and open up certain other futures. That's why I do find it something important to pay attention to.
SPEAKER_01Yeah, you know, I would say that kind of addresses my question on politics, but we can get more specific about it, right? Because in the book you describe how, you know, obviously we had this neoliberal turn in the in the 70s and 80s that led to a greater kind of privatization of the functions of government, a greater reliance on the market to kind of provision services for people, you know, cuts to the welfare state, government seeking efficiencies, quote unquote, to deliver services and things like that. And as things get marketized, there's also a vast production of data that goes along with kind of creating these systems. And that seems to be a type of environment, you know, less investment in the public service, the government needing to seek out efficiencies, the market providing more of the services that we rely on. That seems to be kind of something that would be beneficial to an AI type of approach to, you know, all of these sorts of functions. So maybe you can talk about how AI fits into this broader kind of political picture or economic picture that we've created over these past couple of decades and how it kind of exacerbates these trends rather than seeking to ameliorate them.
SPEAKER_00Yeah, absolutely. I mean, you're what you're describing a bit, I think, is I mean, I originally signed up with Bristol University Press to write a book called AI for good. I'm in a computer science department, I feel like, but I also have experience of, you know, working in the what they call the social sector, you know, that's but that's my twin concerns. And yeah, maybe I can help, you know, to usher in this idea of uh sort of AI in the public interest or something like that. And I just went through this series of of kind of wait, what sort of moments, uh, you know, as I was kind of really trying to engage with the AI and at the same time have a kind of sense of history and sense of political and social understanding. I was trying to understand what was going on with AI and went through a series of sort of uh negative revelations, you know, sort of going, wait, this is exactly like neoliberalism, and you know, wait, this is in its own way, it's it's entirely recapitulating a sort of Hayek idea of what markets are supposed to do, you know, the sort of optimal distillation of all available information. So, yes, basically, I mean, you know, AI in that sense, you could have almost predicted AI based on a reading of neoliberalism. And it does, and I think the aspects of that that are most interesting to me are it plays into this anti-social in the literal sense, you know, anti-society, this anti-society privatizing, centralizing, seamless construction of a space of free flows for capital and all of that. It does play into that. Okay, so one of the useful things I think about the analysis of neoliberalism that's equally important for AI and is somewhat overlooked sometimes, perhaps, is you know, the idea of the simultaneous production of the subjects, in other words, us, you know, how we experience ourselves, how we see ourselves and each others under neoliberalism. I think that's one of the really strong elements of the critique of neoliberalism as opposed to just like all the capitalisms that we've had, you know, is its hyperemphasis on hyper-individualism and you know, the fact that we exist as very isolated subjects and entities, and that what's relevant for me is a sense that AI, in that sort of feedback way, I was saying, comes out of and feeds back into that process as well, that it feeds back into subjectivation, it feeds back into the, you know, AI produces its own subjects in a way. You know, it's not just sort of a mechanism for manipulating us, it's a mechanism for producing or producing certain kinds of us or producing certain understandings in us of who we are and what we should want. And I I think that's always how hegemony works, you know, that establishes a s a set of cultural and sort of experiential framework and field, as much as it does governance, as much as it does, you know, sort of industrial relations or sort of relations of work or labour and social reproduction. It establishes these things across the board, you know, from the psyche to the political economy. And I think AI is active in that way. The other thing about it, from what you said, was one of my first kind of revelatory moments in sort of unpicking what actually goes on inside AI and realizing that it doesn't really just do anything except come up with these ways of dividing things and ways of sort of doing a sort of utilitarian optimal division, you know, a way of sort of ranking in essentially the desirable and the less desirable, that again think about the moment, slightly longer political moment where in the sense of the financial crash, for example. What I'm saying in a very long-winded way is I just looked at this and said, Oh my god, this is a machine for austerity. This is an austerity machine. It's a machine for reproducing and scaling this mechanism of austerity and this mechanism of precaritization. You know, when you're looking at AI, you're looking at delivery. You're looking at Uber. These are inseparable from the core. We want to call them dark patterns. Somebody, somebody called it to me the other day, taking that analogy from web design. You know, these are the dark patterns that perhaps weren't intended to be dark, but they came out of a certain logic and they're embedded in these systems. So yeah, it's um it does absolutely not conflict with neoliberalism, but seems like a kind of to me, a kind of turning up of the volume of the agenda that's been rolling since, you know, let's pick a date 1973 and the coup in Chile or whatever. You know, it's a machine for the reproduction of Chicago boys' values or whatever. But at the same time, I think that's pretty bad. But what really concerns me, maybe, is that this is coming at a time when I think the wheels are coming off that dominant world model. You know, there seems to be quite a lot of unintended friction in trying to sustain this neoliberal world of sort of just-in-time transglobal economic operations, and actually for a number of reasons. One of the financial crashes, one of them, you know, the so-called refugee crisis, or the you know, the sort of massive displacements for war and and climate change and famine and other reasons that happened around the world, plus the climate crisis, you know, sort of overarching everything, is causing shocks and tensions in the system that uh not to mention the pandemic, actually. That was also another kick in the nuts for the for the existing system. And it's really, you know, it's it's staggering. It's staggering. And that's kind of that is worrying because it's not like the relations that have accrued themselves such power over this period of time, even compared to sort of previous versions of capitalism, let's say, you know, the the the massive integrication happened over that time. It's not like they're gonna go, oh yeah, it's not working anymore. We better go for a sort of socialist republic or something. They're gonna dig in, you know, they're gonna dig in, and they're going to seek other means by which they can support and sustain the existing asymmetries, let's say. Historically, that is the kind of recipe that people like Robert Paxton point to as being you know pretty generative of forms of far-right politics, if not fascism, which is you've got a crisis, you've got a sort of ruling class panic, if you like, and you've got a complementary presence of a sort of solution-offering ideologies, you know, and we don't have to look anywhere, you don't have to look outside of you know, down the street at the moment in many, many places in the world to see rising far-right movements. So that is in my mind, I mean, that's something that has concerned me for a long time. You know, politically, I I've always felt myself to be an anti-fascist, but the last few years is incredible, the rise of the far right. So, again, if you're if you're really considering AI as a technology that's enmeshed and entangled and involved in creating circuits and feedback loops in the society that it comes from and the society that is currently emerging, then the fact that this machine for division and ranking and separation and a sort of, you know, this machine that tends to essentialize the attributes that it imposes on people, for that to come about at a time when not only do we have an existing injustice, but those injustices are are veering in a far right direction in in so many ways, whether that's sort of um fascistile far right or sort of uh more religiously justified far right or whatever kind of nationalisms or identitarianism we're talking about, it's breaking out all over. And the possibility of that happening in conjunction with the emergence of AI is partly also why I wrote the book.
SPEAKER_01You know, I I think it is concerning to see how these things develop, right? And I want to come back to the point on fascism and the relationship to the rising far right in just a minute. But, you know, even if we think about how this affects our governments that have not elected far-right parties, which is still the case in some countries, luckily, if we look at that, you know, I was talking to Rosie Collington recently about how, you know, bringing these kind of private sector technologies into the public sector still has an effect on what governments can do and what they do, right? Because it does help to shape the responses. And we can already see, you know, people like Sam Altman and these other kind of boosters of AI technologies, whether it's Chat GPT stuff or more broadly the kind of suite of AI tools, that these tools are going to be your doctor in the future. They are going to be your teacher in the future. So we're going to need less human teachers. And, you know, it won't be kind of their kids, the kids of the billionaires who will have the kind of computer teachers and the chatbot teachers. It will be poor people, right? Who don't have the money to buy access to private schools and private hospitals and those sorts of things, right? Even as, you know, especially in this moment, we see the challenges that a lot of governments are facing to fund public healthcare systems and the talk that, oh, we need more kind of private sector involvement in the healthcare system. And then what is that going to mean into the future? How do these things evolve? How does technology and AI then justify or help to justify, you know, a further privatization of these things because of how AI is used? But then even more kind of, I guess in a more hidden sense, in a sense that we don't see so much, as you describe in the book, there's also the way that governments, as you say, after having been cut so much, after having been subject to so much austerity, say maybe these AI tools can help us to, you know, administer our programs in a different way, right? Can help us to see who is a deserving welfare recipient and who is an undeserving welfare recipient. And what other ways can we use these tools to classify the people that we supposedly serve so that we don't need so many bureaucrats, so that we don't need so much labor, so that we can reduce the cost of delivering services, so that we can make them more quote unquote efficient. How do you see that aspect of this playing out as these technologies get integrated into the public sector and then have these consequences that are often not expected, but maybe people like us would say, hey, pay attention, this is going to happen, even though you know a lot of times those concerns are not listened to.
SPEAKER_00I think you portrayed a very convincing picture of what not only what will happen, what is happening. And that that is, although I'm trying to also raise warning flags about potential more extreme outcomes of these things, that is actually really my main concern because that's the stuff that's hitting people right now, and it hits all the people that are already being hit so hard by everything else, you know, that I've been digging into in the last couple of weeks a little bit more of what's going on behind the benefit system in the UK in this rather notorious benefit called Universal Credit. And actually, I have to say I have credit Human Rights Watch here for a report that I wrote a couple of years ago that I didn't read until recently. And reading through it, I mean, firstly, in somewhat sort of um affirming kind of way, it really made concrete a lot of the things that I'd written into the book without you know knowing these on-the-ground facts about how these things would affect ordinary lives under uh under pressure, if you like, you know, and feeling that these are simultaneously distanced, cold, thoughtless mechanisms of bureaucratic administration trying to achieve, you know, sort of impossible target given the ridiculous goals that politics has in turn set them, and at the same time sort of genuinely mechanisms of cruelty, you know, that they are actually forms of punishment. I mean, in the UK, again, that isn't really hidden. It's not said so much about universal credit, but for example, the the system that's in in the UK that's created to um uh deal with uh revealing phrase, you know, uh asylum seekers, is actually officially called the hostile environment. But I was you know chatting to somebody who's working with universal credit beneficiaries, and they were saying that one of those beneficiaries they were talking to referred to universal credit as a hostile environment, and I think that's absolutely right. I mean, it is in the way it it sanctions people, it it has this incredible sort of twisted logic of saying, well, we're only going to give you the absolute bare minimum, and if something happens, and this can be quite trivial, that somehow uh transgresses one of the you know many, many parameters we set up for how your behavior and life should be lived, then we're going to sanction you and only give you 70%, which is by definition 70% of what you need to survive. So, you know, it's it's real, really appalling. And I mean, there are many, many dimensions to this, and I think they perhaps reflect um a few of the things that I tried to put in the book that I did find quite early on, Hannah Arendt's concept of thoughtlessness very helpful in sort of guiding me through what's going on, because it was the capacity for such cruelty, such inhumanity, violences of various valences, you know, happening really from people to people, but through the mediation of these systems, is so awful that, you know, unless you've kind of seen it or or studied it or experienced it, I think a lot of people still don't really either know about it or sort of want to know about it. Again, these things all play into each other, you know. Sometimes we really don't want to know what's going on. That's already present, you know, that the things that you describe are already present. And yes, of course, the thing that you also sort of just described is that so much of that is now being absorbed into the private sector, which makes it, I mean, you know, what does that give us? That gives us perhaps a an incredible intensification of a single parameter, which is making a profit out of all this stuff. I mean, the fact that people, there's been newspaper stories in the last few days in the UK are comparing the food that you get in an old people's home with, you know, the stereotypical oysters that the CEO of that company is eating on the on their yacht. I mean, it's you know, it's cartoon-ish in its horror. Of course, the privatization makes it even less accountable. Not that local authorities or government or local government was accountable before in any meaningful way, but it makes it less accountable, makes it more opaque, makes it more difficult for those people who are trying to exercise some guardianship, journalists or whatever, or whoever else it is, or activists or whoever, you know, it makes it more difficult for them to get traction on it. It is the market ideology, of course, but it's the I'm thinking here of also our own experiences where I work, of that same kind of restructuring at the hands of consultancies who, you know, talk in terms of efficiencies and um optimizations. And I think that's the one of the other things I just like to sort of um you know uh drop in in a way that that's a consequence of this privatization, but it's kind of a giving much more of a hold to this single valued good as well, which is this idea that things must be optimized. And you know, optimization is a very, you know, combined with a capacity for cruelty, combined with the capacity to leave people without the the bare minimum that they need to survive, becomes again the term that I adopted in my jackdaw-like way, you know, and like picking up little bits as I go along. The term that made sense to me at that point was the idea of necropolitics, which is not meaning some massive transformation of politics. If anything, I think Mbembe, Achille Mbembe, when he wrote this book on Necropolitics, he was talking about the sort of still murderous colonialism that exists within post-colonial regimes. But actually that seems equally applicable to a lot of the systems that we live under, you know, depending on depending on which bit of the system you experience. And I wanted to emphasize the necropolitics because I would I just wanted to emphasize really that these systems, I guess, in a way, are literally a matter of life and death. You know, they aren't ordering anybody out to die or sort of automated executions or something like that. But what they are doing is seeping out of the system, you know, in a way that is justified by being somehow mathematized and algorithmic and computational, and you know, perhaps nobody really believes it's fully objective, but you know, it's systematized at least and it's authorized, just seeping out the sheer possibility of the continuation of existence for so many people. So that's much more everyday and very pervasive. And I think all that ChatGPT and these other sort of generative models, in a way, they're kind of generative, possibly of further sort of ideas about how to do this for other institutions that didn't yet think that they too could get in on this particular party, you know, that the the welfare benefits people and border forces. I mean, they're all over AI, you know. And now I think it will probably spread to the rest of them under the under the rubric of like, oh wow, maybe we could get, I mean, mental health is a particular concern of mine in general as a sort of interest area because it seems one of those areas which is a focus for the sort of uh fecosystemic categorization, the marginalization of the of the sort of more vulnerable statement about what society really means. How do we care for people who at that particular moment are not able to care for themselves? And at the same time, it speaks so much to our values of normativity and what we think is okay and what we think isn't okay, and who we'll accept because they're productive and who we won't accept because they're not productive. So mental health seems to encapsulate so many things. And of course, mental health is now, you know, I've I've been concerned about AI and mental health for many years, but right now I imagine that all of the people who are running underfunded programs for CBT therapy or whatever are looking at GPT models and going, maybe that's the answer.
SPEAKER_01It's very concerning, right? Especially when you go all the way back to Joseph Weizemum and Eliza in the 1960s and how he created the system that was basically an early chatbot, you know, and and it was supposed to be like a psychotherapist. And he was like, you know, people will talk to it, they'll never actually believe that, you know, the computer understands them. But then he was shocked to learn that they did, right? And and they did kind of buy into it and want to believe that it was real, and he spent his whole life afterwards saying, we really need to be concerned about this AI stuff, right? Uh, but we don't like to learn the lessons of the past, at least not generally, right? I think in the book you made a really good point in explaining this as you know, one of the benefits of of these systems of AI is that it doesn't feel ethical doubt, right? So if you integrate it into these kind of bureaucratic systems, it doesn't worry that you're giving someone only 70% of what they need to live on, for example, or you're cutting them off from benefits altogether because you flag them as someone who doesn't deserve it, right? You know, I'll just say for listeners that if they want an example of this, they can go back to episode 72 in August of 2021 with uh Dakshayini Surya Kumaran, where we talked about how this occurred in Australia with their kind of benefits system and how they implemented it to kind of cut people off, how it led to suicides and all these sort of things based on it was integrated in a in a way that wasn't appropriate, right? That didn't work. You know, whether these things are ever appropriate, I think is a question. But in this case, it was flagging people who were completely legitimate, right? I wonder, you know, going back to what you were saying about the kind of concerns about how this can lead us toward or or further empower a kind of far-right politics, a fascist politics that we're already seeing on the rise right now. I think I have two kind of questions on that. Do you think the views, the politics, the perspectives of the people developing these technologies matter in this sense? In, you know, when we look at these people, we know that there are people like Peter Thiel who are involved in this, who is very much, you know, I feel comfortable calling him a fascist, you know, and and he promotes the use of these AI tools in ways that are beneficial to the government, to the immigration agencies, all these sorts of things. People like Sam Altman, who is talking about artificial general intelligence, these billionaires in general who are supportive of things like long-termism and uh, you know, not wanting to address these social concerns because you know, we need to think about building uh artificial general intelligence and ensuring that the human species lives for way long into the future, and particularly people like them, not you know, the poor people who don't matter so much, right? Or so how much do you think that matters, the particular politician? Of these people. But then also, how did these systems in general kind of empower this way of thinking that leads us more in the direction of a far-right politics and that way of kind of seeing the solution to the problems that we very much do have in our society as we've already seen our political parties, our political systems shifting in that direction? You know, the center-left party moves to the center, the center-right party moves to the far right, for example. You know, everyone's kind of shifting in this direction. To what degree are these technologies kind of helping move that along?
SPEAKER_00I think, of course, it's important to it's important to do what you did and say, well, actually, this person is a fascist. I think, you know, the kind of meme that I I always like is the, you know, is is which gets repeated quite a lot at the moment, is is the variation that on the theme of saying, you know, when they tell you who they are, you should believe them. And there are many, many people at the moment who are mouthing and espousing and and and actively advocating for things that that I think are unambiguously fascistic. And so we shouldn't beat about the bush. So those people are present in the same way that there are actual fascists on the streets. You know, th those things need to be opposed and dealt with. But they are in a way, I don't think they're the determinant factors. You know, I don't think there's you know determinism in this. You know, it's a it's a complex unfolding of stuff, but it that doesn't need to leave us powerless in trying to analyze it. I don't think they're the most important factor, actually. I think it's perhaps no surprise that people like that find themselves drawn to or in positions of incubating technologies like this, but I think that's more a kind of resonance between their pre-existing worldviews and and what these technologies offer and what they're good for. I think what's for more concerning is the stuff that you have already been alluding to, which is the kind of systemic side of things. And, you know, we have a what I saw called a few weeks ago, which I've I've now adopted completely, the poly crisis. You know, we've got a plurality of overlapping crises going on, which are going to make life just, you know, obviously for huge parts of the world, they're already critically filled with crises and always have been, you know, ever since the perhaps, I don't know, perhaps since the beginning of empire, let's say, you know, there's always been the global south, has always been, you know, subject to mortal precarity. And that perhaps is simply just coming home a bit more now. I don't know, but whatever it is, I think it's pretty inevitable that all of us, in one way or another, are subjected to greater precariousness and and more sort of threats in some way. This is the crisis of legitimacy that the system itself is always experiencing because even those who are supposed to be loyal to it now are concerned about their own existence and the future of their families and so forth. There's a systemic set of circumstances in which AI emerges, you know, it's no accident that it's emerged at this time in this form, and to which AI forms a kind of feedback loop. And I think, again, from my sort of personal sort of probings into how AI actually works, it seems to me that again, that that in itself is no surprise in that what it does as a sort of computational party trick, you know, made meaningful in the world, if you connect it to actual world operations, is essentially forms of closing off. It is forms of enclosure, let's say. Or another way of saying that is forms of exclusion. And those kind of dynamics, that's what it offers. I don't think it really offers, I don't know what the phrase would be genuinely productive, but it doesn't really add anything new into the world. What it does is provide means of boundarying, dividing, valuing, as you say, particularly you know, ideas of worth, greater worth and less worth. And so that's what makes it create the prec the preconditions for more, you know, terrible people or more terrible uh outcomes by people who may not see themselves as actually um believing in any of these ideologies that the more extreme people propagate, but they just feel well something needs to be done, we need to sort this problem out. You know, too many people coming across the English Channel in small boats, and we just can't, you know, well, I can't hardly get a school place for my kid, and you know, when I go to hospital, there's a 12-hour wait, all of which is true. It's uh very easy to point the finger as always, you know, in scapegoating politics, which again is very much a far-right tactic as well, is to blame some other group, some outgroup, to find ways of systematizing, of algorithmizing, of sort of mathematizing those kind of operations as well. There's not this stuff will happen, it is happening. These kind of reactionary politics are happening, and they would happen if AI wasn't here, you know. But I do think that AI is very unfortunately uh resonates far too closely with these things. So I I don't say AI is fascist. I sometimes get sort of queried about that. I don't I don't say that because fascism is a political category. It's something that we do in social forms, but I think we do need to have a broader understanding of politics, actually. I think we need to have a techno politics, particularly in our time, but I think it's been true since the Luddites, we need to have a techno politics where we understand that what we understand as politics always rests on, is shaped by, is delivered through particular technologies. And that in itself is shaped by the character of those technologies, shaped by the particular things those technologies are good at doing and the particular sort of um opportunities they seem to close off. I fear that might already be getting a little bit vague-sounding. But essentially I see AI as you know forming a mechanism that's very able to extend processes of enclosure, extraction, and exclusion.
SPEAKER_01Yeah, no, and I think I just want to kind of emphasize what you're saying there by comparing that to what we hear from you know the people who are developing these technologies, right? Like if we think about the Sam Altmans of the world, for example, or if you we think about any of these other people, they are telling us that these AI tools are gonna have so many incredible benefits for the world. As I was saying, you know, they're gonna be your doctor, they're gonna be your teacher, and all these other things, they're gonna make opportunities and education and all this available to so many more people because these tools are just so powerful and what have you, right? And what we're talking about and what you're talking about is very much that it's very unlikely that those things are going to happen in the way that they are telling us. But then on the other side of things, these technologies can be used in really harmful and worrying ways when they're integrated into systems that are already kind of enabling you know, forms of exclusion and harm and all these other things that already exist in the world as it's happening, and especially at a moment where the crises that we face are escalating, and so there's a there's even increased pressure to like do something about something, and then that opens the kind of gates to say, okay, well, we just need to accept these harms or or whatnot because of of what's happening here, right? To close off our conversation, I wanted to ask about how you think we should respond kind of collectively to AI. You mentioned that you obviously started writing a book called AI for good. So is our response to think about how we can have good AI or what kind of a progressive AI looks like, or does it need to be a much more comprehensive response that challenges AI itself and thinks about how we address these problems in other ways? B.
SPEAKER_00You know, I definitely definitely. Yeah, yeah, yeah. Shock, shock hollow. That's uh you know, no, I definitely think it it should yeah, we should challenge it. I mean, because of the, you know, I think it's not the kind of thing that we'd hear in these parts on this podcast, but in general, you know, people still seem to feel the need to preface any conversation by no matter how critical, by saying something like, Of course, this had got huge potential for healthcare or huge potential for education or whatever else it is, as you say, you know, but and then go on to list you know some of the evidential and empirical harms. And it seems like such nonsense. I mean, why do we have that that is in fact complicit, you know, in extending the you know the range of harms that these things are likely to cause. All of the benefits. I mean, when it comes out of the mouth of the Twitter feed of someone like Altman, then you know, we know it for what it is. You know, it's supremacist ramblings mixed with you know self-interested um investment tactics and PR. But when it comes out of lots of other people's mouths, it becomes a lot more obscures, a lot more the reality of the thing, which is that actually I'd stand by this one, you know, in in a in a corner of a boxing ring, the sp the benefits are all speculative. All of the things that are talked about, how they're going to make society better, you know, are actually I would say fantasies. Whereas the harms that these things are able to do are already evidenced, even in the minimal ways that real AI has been deployed in the world. And one of the reasons why I to get on to your question when I talk about anti-fascist approach, for example, is simply that one of the principles of anti-fascism is you don't wait for fascism and then go, look, we're right, now we need to oppose it, because then you're in a really bad place to do that. And I think it's, you know, at least by analogy, at the very least, even if you don't think these technologies are fascistic in some way, at the very least, the analogy of not waiting for all the bad things to happen before you go, yeah, well, those things are bad. Now we need to really undo them. Once they've become sedimented in our infrastructures, once we are not training any doctors anymore because we relied on these systems or whatever else it is, that seems a very foolish measure. So much more on the plus side, I guess writing the book taught me is to try to think about um technologies in the same way as I would think about what's sometimes called prefigurative politics, which is that you're if you're going to envision and imagine, and I think this is what we should be doing, and envisioning and imagining better worlds, and I say that plural, better worlds which can be multiple and in different places different things. But if we're going to imagine those, and the first thing obviously is the need to imagine and believe it that those are as possible as the horrendous reality that we've got, then it's only consistent with that to think about if we're using tools of any kind and computation in particular, what forms of computation might be, you know, consonant with that way of living, with those kind of relationalities that we want. I mean, care is something that I didn't mention explicitly, and but it came out in so much of what you were saying, and and and I was echoing about the consequences of these systems is they're so uncaring and callous and cruel, forms of relationality that actually promote care and mutual aid and solidarity. I think those are the values that I would like to see much more prevalent in society that I have witnessed for myself, I have experienced and been part of. I think they offer a huge potential benefit for facing the challenges that are coming, actually, that are particularly relevant to times of crisis, then we should be trying to conceive of modes of technology that are consistent with those. And I think that doesn't need taking a step back. AI is a florid outcome of particular kinds of technosocial matrix. And I don't see anything actually in contemporary AI. I mean, I have had a couple of exchanges with Gary Marcus about this because you're sort of on the one hand, you know, really ready to go out there and denounce AI as it is, but mainly so he can wind it back to the form of AI that he thinks is better, which is a kind of hybrid with expert systems. And I'm just not interested. I don't think AI of any form that I've encountered, and I've got some historical understanding as well, not just contemporary. I don't see it, I see it all as either come out of neoliberalism or out of the Cold War. It expresses and instantiates those values. And if we go, I I'm not writing off computing. I think maybe we've already got more computing than we need, and maybe we should really roll backwards. I love technology, but I still hold out the possibility that there's ways of assembling kinds of technologies that are consistent with the values of the world that we want to live in, and that's the exercise that we should undertake. And at the moment, the way I'm thinking about that is a kind of, as my little shorthand, is a kind of a sort of shift from cyberpunk, which is telling us, you know, how the world looks when you actually put this stuff on the street in the hands of sort of already existing mafia, that's what looks like cyberpunk, um, towards solar punk, which is you know the idea of what might the world look like if we allow ourselves to be led by pro-social values of care and solidarity, mutual aid and and imagination and creativity and diversity and inclusion. If we allow ourselves to think about those kinds of societies, then we must allow ourselves to think about what kind of technologies might support them. And I think the potential is there. I'm not talking about a 50-year research project here. If anything, it's about recombining and taking apart and putting back together in a very different way some of the stuff that we already have. I do think this is an immediate thing. And the historical example, as you you would know, that I put in the book is one from the 1970s, where a bunch of workers in the Lucas Arms Company did exactly that. They said, we don't want to be in this arms company anymore for a variety of reasons. So we're going to use the existing skills we have. And they were all, you know, shop floor workers, but because they were in a high-tech industry, they had skills. You know, they had skills, they had engineering skills, practical engineering skills, design skills, and they designed, you know, early forms of um hybrid power, solar cells, and so forth. Because at that time it was the beginnings of the environmental movement. They were affected by that kind of thing, drew those ideas in and they turned them into real practical working possibilities. Of course, they were completely crushed, of course, at that point. But the possibility didn't die. That concept of social production didn't go away. It's still there and it's still to hand. It's to hand as much as OpenAI's you know, nightmare visions are to hand. And I think that's what we need to reach for.
SPEAKER_01Yeah, I think the example of the Lucas Plan shows how we can imagine a different way of organizing things. We can kind of deploy our productive capabilities in a different way, but the system, you know, kind of limits what we can do there. And I think one of the functions of these tech billionaires like the Sam Altmans is to kind of limit what we can imagine for the future and to accept, you know, their kind of visions for how things should work and how these systems should roll out and you know, what role we have in that, which is a very small one, to be more of an object rather than someone who uh kind of thinks up and has much agency in this.
SPEAKER_00I just wanted to say, supported also behind themselves by the kind of, you know, the apparently reasonable liberal establishment who also position themselves as people who are acting on everyone else's behalf, but they're a lot more sensible than these raving lunatics. I think that's also a problem. I think if the anything that Lucas Plan did, what that committee did said was, we can do it. You know, ordinary people can do it. And I think that's the ultimate lesson of AI is no matter how complicated this stuff is and how wizzy and sci-fi it seems to be, it's nothing compared to the insights, the capacity for insight and the capacity for organization in ordinary people.
SPEAKER_01I completely agree. And, you know, I think it's the perfect time to have a conversation like this as there's so much hype around the AI, around these tools, you know, so much imagining of what this is going to mean for our future, rather than questioning should this be our future at all. Dan, thank you so much for taking the time to chat. I really appreciate it. Thank you. It's been a lot of fun. Dan McQuillen is a lecturer at Goldsmiths at the University of London and the author of Resisting AI, an anti-fascist approach to artificial intelligence. You can follow Dan on Twitter at at Dan McQuillan, you can follow me at at ParisMarks, and you can follow the show at at TechWon't Save Us. Tech Won't Save Us is produced by Eric Wickham and is part of the Harbinger Media Network. And if you want to support the work that goes into making it every week, you can go to patreon.com slash tech won't save us and become a supporter. Thanks for listening.