Is the Open Web on Life Support?

Recently, at its big annual developer conference, Google announced the most significant overhaul of search in its history. It's rebuilding the search box around AI: conversational queries, an "AI Mode" that answers follow-up questions, and agents that gather information for you in the background.That announcement set off a wave of commentary about the death of the "ten blue links" — the idea that search is no longer a ranked list of websites you click through to, but a conversation with a chatbot and answers delivered directly. For many, it’s a fundamental reshaping of how the internet works.

So what does it mean for the people who actually make the stuff on the web—publishers, creators, anyone who depends on being found? They've already been bemoaning steep drops in traffic from AI-powered search that answers a user's question without ever sending them to the source. And with that lost traffic goes the advertising and the subscriptions that pay for the work in the first place. 

Predictably, many are waxing nostalgic about the good old days of the open web. But were those days actually so good? People have been griping about this bargain for years: that Google scraped their content without paying, that it kept users on its own page, that it became an ad-tech monopoly sitting in the middle of every transaction. And at the same time, the AI companies argue they could never have built these tools, now used by hundreds of millions of people, without access to the open web. For better or worse, ChatGPT probably doesn't exist if the web had been a closed, permissioned, pay-to-read place from the start. 

So what should the norms be going forward? When AI crawlers put enormous strain on a site like Wikipedia, who should pay for that? If the old handshake between the people who make content and the companies that index it is eroding, what should the new one look like, and how do we even get there? Do we write new rules, or does this whole thing just stay tied up in court for the next decade?

Evan is joined by Derek Slater, a founding partner of Proteus Strategies. He spent fifteen years at Google where he built and ran the firm’s information-policy team. Before that, he worked at the Electronic Frontier Foundation and Harvard's Berkman Center. He's an expert on all these issues, and has long advocated for the freedom to access and learn from information on the web.

Read some of Derek’s work here:

• “AI Training, the Licensing Mirage, and Effective Alternatives to Support Creative Workers,” in Tech Policy Press
• “Worried About AI Monopoly? Embrace Copyright’s Limits,” in Lawfare

Transcript

SPEAKER_00 0:00

And now we're entering the next chapter of Google Search. Where incredible AI features aren't just in search. Google search is AI search, through and through. Now we're taking an exciting step toward this vision. We will be able to create and manage multiple AI agents for your many tasks right in search.

SPEAKER_02 0:19

Welcome to the Center Edge. I'm Evan Swarstraber. Google recently had its developer conference I.O. And there the company announced the most significant overhaul of search in its history. It's rebuilding the search box around AI, conversational queries, and AI mode that answers follow-up questions, agents that go gather information for you in the background, and interfaces the system generates on the fly. The thing Google described as the entry point to the web for 25 years is now being reimagined. That announcement set off a wave of commentary about the death of the ten blue links, the idea that search is no longer a ranked list of websites you click through to, but a conversation with a chatbot with answers delivered to you directly, and AI agents even searching the web on your behalf. For a lot of people, that's a fundamental reshaping of how the internet works. So what does it mean for the people who actually make stuff for the web? Publishers, creators, anyone who depends on being found. They're not waiting to find out. They've already been bemoaning the drops in traffic from AI-powered search, where users' questions are answered without ever sending them to the source. And that lost traffic can also mean lost advertising or fewer subscriptions that pay for the work in the first place. Predictably, a lot of folks are now waxing nostalgic about the good old days of the open web. But were those days actually so good? People have been griping about this bargain for years, that Google scraped their content without paying, that it kept users on its own page, that it became an ad tech monopoly sitting in the middle of every transaction. The quote unquote open web that everyone's mourning was a constant source of complaint while we had it, and at the same time the AI companies make a real point in their own defense. They argue they could never have built these tools, now used by hundreds of millions of people, without access to that open web. For better or worse, ChatGPT probably doesn't exist if the web had been a closed, permissioned, pay-to-read place from the start. The same openness that people now are reconsidering is the thing that made all of this possible. So the real question is, what should the norms be going forward? When AI crawlers put enormous strain on a site like Wikipedia, who should pay for that? When a website says, please don't crawl me, and a company does it anyway, does that mean we need a new law? Or is a 30-year-old honor system good enough? If the old handshake between the people who make content and the companies that index it is eroding, what should the new one look like? And how do we even get there? Do we write new rules? Or does this whole thing just stay tied up in court for the next decade? My guest today has spent his entire career on these questions, these tensions between openness and control. Derek Slater is a founding partner of Proteus Strategies. He spent 15 years at Google. During that time, he built and ran its information policy team. And before that, he worked at the Electronic Frontier Foundation and Harvard's Berkman Center, organizations that focus on tech policy issues. He's an expert on all these things. He's long advocated for the freedom to access and learn from information on the web. Derek, welcome to the show. Thanks for having me, Evan. Great to be here. So I want to start with the history of how we got here, because scraping the internet is not a new phenomenon, right? As I mentioned in the intro, it's kind of how we have Google search. It's how indexing started. And maybe in your answer you can help define terms for the uh non-engineers among us. But my very oversimplified history of events, and then you can tell me why it's maybe more complicated than that, is that around the time that search engines were coming online, right, late 90s and becoming widely used, they essentially had bots that were scraping the web, right? Going all over the internet, seeing what was on there, and then indexing it so that when people typed in a prompt into Alta Vista or Yahoo, or then, you know, increasingly Google, uh, which became 90 plus market share, 90 plus percent market share, they would be able to find it. And it was never a perfect bargain, and there was always complaints in both directions. But at the end of the day, there was some level of symbiosis. Google got to advertise against those searches. And publishers, whether it's a newspaper publisher's cooking recipe website, or journals or whoever, they had the opportunity to either gain advertising from the visits to the website that Google would send them, or in theory, if they were subscription-based, someone would go and uh see a paywall and then sign up to read the article. That was essentially the handshake. Do I have that right? Or is there more to the scraping history of the internet?

SPEAKER_01 4:53

Yes. To both of those questions. Or rather, I think you get the broad strokes are right. I think it's important to underscore, as you have, this tension of, you know, I'm gonna put up my website online. What is that? What is the bargain? What do you get to do with it? What do I get to control? Has always been contested going back to the very beginning of the web, when uh people started to create bots, both for search engines and other sorts of data collection, to do research on the web, to analyze the web in various ways. And there were different problems that then came about from that. And we see echoes of that today. You know, one problem was these bots, you know, this was way before AI, right? This is 30 years ago, roughly, 25 years ago. These bots are sending lots of traffic and they're overwhelming my sites. And what should we do about that? And there, protocols were worked out to say, okay, tell me to back off. If you're overloaded, send me a signal to back off and I'll back off. And those sort of technical singles help mediate. Similarly, you know, some people wanted to be indexed and you know, crawled in various ways, including for search engines. Some didn't. And as you indicated, that's where standards like robots.txt came from. These are, you know, voluntary protocols that gave a site to say, hey, you know, you can crawl my site or you can crawl this part of my site, but not that part of my site. And again, with some of that issues of, you know, how frequently and so on and so forth. And that was always, there were always tensions there that had to do with competitive pressures or rather, how much of the value are we sharing? Are you getting too much of the value? Should I get more of the value from that crawling? But also real consider a sort of tensions around what should consumers be able to do with a site? Right? If I come and read your website, should I be able to, you know, use a assistant to help me find the little piece of it that I really want to read and block out the rest? Or should I have to read it with the interface that you provided full text? I say it that way because it's it's easy to jump from, and I think you did a good job avoiding this, jump from, well, there was search and now we're here with AI, and this is a total disjuncture. Along the way, there were lots of tensions to give, you know, a couple others to put into the mix. Um, there were a number of companies that worked um to try and crawl social media sites, both to analyze that content, but also to make it so that if I'm a user of Facebook and I want to read my Facebook posts with my Twitter posts in one combined interoperable interface, I could do that. And Facebook sued and said no, and Facebook won. And that's a that's you can tell the story of, well, it's their site, they have control, fine. Or you can tell it a story of they shut out competition in the market for social media interfaces leading to the social media world we have today, for better and and for worse. Another example, just to call out, that I think is an interesting sort of intuition pump is ad blocking. On the one hand, I think a lot of there are people who say ad blocking is terrible. You should not use ad blockers, it deprives news sites of their revenue and so on and so forth. But there are a lot of consumers who use ad blocking because ads are really annoying, and we all find ads really annoying. And ad blocking is interesting because there's been litigation, there's been standards, and you can think of it as sort of like how do we land at the right equilibrium? We're like, yes, there's ad blocking. It's just enough to make ads suck a little bit less than they otherwise would, but not so much that it erodes the revenue of news companies. And that's what I find really interesting of like, how do we land at that equilibrium? And it's always a mix of norms, tools, laws, market changes. Um, and that's where we find ourselves today with AI.

SPEAKER_02 8:25

Yeah, it's depending on how old you are, right? You go to a news website these days that relies on advertising, and it may say you have to disable your ad blocker to read the news. And I, because I, you know, value journalism and at this advanced age that I'm at of 34, I've learned enough. And I say, of course, I will disable my ad blocker. I want these journalists to get paid. There was a time when websites were almost unusable because of pop-up ads, right? And pop-up blockers were this incredible innovation that helped people avoid what was really awful web experiences in the late 90s and early 2000s. So, to your point, there's been a lot of evolutions and debates throughout this past 25 years. Just briefly, I want to get a sense of how much this honor system has worked in the pre-AI era. We'll talk about some of the controversies in the AI era, but there's never been a law that says that a website's directions to a bot, right, or or what they tell Google and Yahoo, et cetera, a Bing have to be followed, right? There's no law. There's separately copyright law, which we could get into later. But just if I say robots.txt, you're allowed to read my homepage and these sets of articles, but you can't read these other sets of articles, that is essentially an honor system. And much of technology policy and technology runs on honor system and protocols, that you essentially rely on a critical mass of people to be good actors. And then we ostracize and shame the bad actors. But would you say, given your expertise and your history working on this issues, maybe like what percent of the time robots.txt was was followed in the pre-i a AI era? Or would you say overall it generally worked that most actors in the ecosystem follow the rules, or was it more complicated than that?

SPEAKER_01 10:11

Yeah, I think um here two. On the one hand, there was an equilibrium, right? We had an equilibrium where robots.txt existed. It's you know, you could use it to state you can crawl me, but you know, you over here, I don't like you, you can't crawl me, or crawl this part of my site, but not others. And that worked relatively, you know, well, I think, in being able to give those signals. Um, you know, there were still, as you said, people who thought that doesn't go far enough. Um, you know, in the early days, there were websites that said, you should in their terms of service, they said, you should not be able to deep link to my site, right? So that means you may only link to the front page, not to inner pages. And they wanted to legally enforce that. So those people clearly didn't think it worked. And as you indicated, I think as search grew, um, certainly the news industry, among others, had growing concerns about how much of traffic was coming to them versus how much was staying on Google or other platforms. And that led to a set of a variety of different debates. So it, you know, it's been on the one hand, equilibrium and a standard that I think served people very well. On the other hand, always contested. And so when something then disrupts the nature of equilibrium, all of those contestations then come to the surface. I think it's also just sort of important to look at it from the other direction of, you know, the way robots.txt and the equilibrium did protect consumers, readers, people who use those websites, right? I didn't have to go and get your permission to say, you know, I'm gonna use this browser versus this browser, or I'm gonna use this assistant to help me clip out the ads, or just again, make the site something that is usable to me, accessible for me. Think if I'm, you know, um you're reading impaired. Um, I can use something to do a speak aloud function on the web generally. It's important to note that's not true with everything, right? If you buy an ebook from certain publishers, certain publishers or in Amazon give publishers the right to say, turn off the read aloud function, even for accessibility reasons. Right. So the freedom that we've had on the web, we didn't have in other domains. I'd say that's a good equilibrium, personally. Um, others disagree, and I say it that way because robots.txt and these other sort of standards are now being recast actually as um not just norms or you know, the sort of voluntary things, but things that should be legally uh that should be legally required that you follow, or in case they're saying, you know, it's already legally required. People are saying in in litigation in these AI cases, robots.txt is already an access control or an enforceable directive or something else that people must follow under the law. So I think even they would contest your description of sort of the state of robots.txt, and that's what's playing out in these cases.

SPEAKER_02 12:57

So let's talk about the traffic problem because a lot of companies, whether you're a copyright holder or not, they're now complaining about the stress that AI training and inference is putting on their sites. And training is more about what goes into the development of a model like a Claude, like a Chat GPT, et cetera. Inference is what happens when I type in, tell me everything about this topic, tell me everything about Derek Slater, right? And it pulls up a bunch of your articles and then it's inferencing the data set, it's inferencing the web, right? So that both of those activities can stress a website. And of course, websites have to absorb costs when they have lots of visitors, right? Uh listeners might be familiar with uh just any sort of instance, viral moments where some relatively unknown small website gets a glut of visitors and it and it goes down is because they weren't anticipating that traffic. They didn't have the servers and the cloud capacity to handle it. And that's what also allows people to do things like denial of service attacks, DDOS, where you just like launch a bunch of uh cheap, you know, essentially like hired hackers that you can get around the world to try to shut down a website, right? So these are maybe concepts people are familiar with. But now in the AI context, it's it's just bots, right? And you're you're seeing companies like Wikipedia, and which is you know, the parent company, I guess, is Wikimedia. It's a nonprofit, they are not even copyright holders, right? So their complaint here is not that you're not necessarily paying me. Their complaint is that essentially their traffic is just becoming overwhelmed and they don't get paid for that. And in theory, when I visit Wikipedia to look at something, they show me a nice thing saying, would you please donate? Right. At least there's a chance that a human being donates to them and that's contributing to their mission. There's no way that these bots are going to read that message and decide to donate. The bots are not as um bleeding heart, right, as maybe I or others are. So some of these stats are kind of staggering, right? They report that since you know January 2024, their bandwidth for serving multimedia, right? Their image catalog is up 50%. And that's driven by by scrapers. Uh, they say 65% of their most expensive traffic comes from bots, and it's it's you know, only 35% of it is page views because some of it is just bulk collection. Then you have some open source projects saying that close to you know 100%, like 97% of their traffic is is AI crawlers and and uh providers like iFixit, which helps people do DIY, you know, right to repair type stuff, they've sounded the alarm on this as well. So it seems like something in the social fabric, right, the the contract, the the protocols that have been developed over time that you referenced to try to prevent these sorts of stresses is breaking down. And these websites might look at this and say, these AI companies are worth trillions of dollars. They have virtually unlimited access to venture capital, even startups in AI, right? If you have a viable idea, you can go grab billions of dollars out of Silicon Valley. Not that difficult, it seems. And they're gonna put the the burden on me to pay for activities that they're engaged in that make them rich and I get nothing. Right? And I think you're starting to see a lot of frustration. Uh is there anything is it is it as simple as that? I mean, if if you're if you're Wikipedia, how is it not as simple as you're asking me to pay for you to become a trillionaire?

SPEAKER_01 16:28

Yeah, I think, you know, and I think you started to segment this into two different buckets. I think one part of this conversation is about generally about sort of commercial websites, larger websites that take a step back. There's long, as you said, you we've long had to deal with traffic of various types, some desired, some not, right? Some um that will behave well and follow robots.txt, some might be a DDoS attack. And so there've always been tools to try and you know deal with segment the different types of traffic. When it comes to, I think, commercial players, this traffic issue is bound up in a broader, again, value who value share discussion. Um, it's not unlike um your favorite subject, net neutrality, Evan, where you know, you're using my pipes. How dare you? Right. And you know, there's it's an interconnection issue in that way. The Wikimedia one, I think, is, and I'd say brought more broadly, the long tail of the web is um both very interesting and a bit more complex because I think when you look at a Wikimedia, they're all looking at being profitable, revenue generating. This is about sustainability, right? If you think about a small website or these open source repositories or OpenStreetMap or libraries, cultural heritage institutions, this isn't about, you know, you're a trillionaire and we want, you know, we want a quarter of your profits or something. It's we want to keep our servers up to keep fulfilling our mission of access to this content, right? Libraries, Wikimedia, they say our content is free, right? Wikimedia says our content is free, our infrastructure is not. And, you know, that's not just a Wikimedia problem. It's something that I think um, you know, a lot of different folks in the public interest community, in the consumer rights community that I spend a bunch of my time on, where generally, you know, it's about we want open access, we want open content to take this very seriously and say, but this has to be sustainable. How do we make sure this sort of information production and distribution is sustainable? And I think the paths forward there build a bit on the past. Um, you know, some of it is new tools for sites to manage their traffic. Sites have always, or you know, generally had ways to rate limit different folks or to block a DDoS attack, but new way, you know, new ways to sort of detect that traffic, rate limit it, move it around. And you know, there are companies like Cloudflare and others who are providing a range of tools to do that. I think it's also, you know, when you look at Wikimedia, one of the things they've done is develop services specifically for bulk and high-volume users to say, like, instead of just you know, fetching content directly from Wikimedia, use this API, which you're gonna pay for, but it's gonna have the data structured in a way that's better for you. We'll be able to manage the traffic together, it'll work better all around. And, you know, Google started, I think, in that a few years ago. A bunch more companies, including AI companies, came on board this year. Um, and I think, you know, it that is a direction to try and make all this sustainable. And groups like Creative Commons, Open Future, they're talking with other, you know, variations on this theme, cultural heritage institution, other repositories. How might that model scale? You know, and nobody's not everybody's gonna be Wikimedia and have Wikimedia Enterprise, but how might we create that sort of different lane for the bulk or high volume user? And the final piece of the puzzle that, you know, not without its complexities, is identity standards. So, you know, how giving people ways to say, hey, I'm a well-behaved bot. You know, maybe you go and get credentialed by some third party that says, yes, we agree, you're running a good bot. And, you know, sites can say, if we only are going to let you into this part of your the site or at this rate limit if you um if you you know signal this, or maybe some greater form of identity. These are things that are actively being discussed in the IETF, the Internet Engineering Task Force, which is the home for robots.txt and a number of these other sorts of relevant standards.

SPEAKER_02 20:15

I want to talk a little bit about Cloudflare before getting into some of these stealth crawler issues. I mean, they're they're related, of course. Um in your answer, if you could explain why Cloudflare is important and what they do, that would be helpful. But um essentially, as I understand it, they're this very important company that sits at the layer between, you know, a visitor to a website and the website, right? And they help websites essentially manage traffic, avoid cybersecurity attacks, avoid DDoS attacks. They're very important in the infrastructure layers of the internet that are not consumer-facing, which is why people may not be familiar with them. And they've become a very significant voice in in the debates that we're having on today's show, because controversially, right, they have this power, right? They have this huge market share. They're they're a very widely used company. And they see what's going on with the scraping, uh, which they which they view as a problem in terms of cost shifting, but they also philosophically seem to believe that people should be paid for for training uh uh on data and inferencing of data, which is of course a controversial topic that we'll get into the copyright debates and whether AI training should be quote unquote fair use, so non-compensated, or they should be forced to license content from copyright holders. But they decided to turn on blocking by default, which affects some huge percentage of the web. And many might see great, right? These otherwise powerless publishers who are being inundated by AI scraping now have someone automatically putting the decision in their hands to say the blocking is on, you can unblock. It if you want, and that gives you a point at which you can force a negotiation from the AI company if you believe your content is valuable, right? So maybe if I I publish one SoStack article ever, right? I've I've done one so far. I turn on AI training because for me, the idea of someone finding it on an AI bot is good for me, right? Because I have five so few users. But let's say I had a very valuable catalog with lots of subscribers. I don't want people summarizing my content without them paying for it. I might say, yeah, Cloudflare, turn it on. And now someday, hopefully, Anthropic or OpenAI comes to me and says, Evan, your content is so valuable. We'd like to negotiate. We'd like you to turn off that blocker. A lot of folks see this as giving agency to people who have no agency. Others have argued against this, you know, including my good friend Luke Hogg at FAI, folks at Public Knowledge, who I'm very friendly with, so you know, kind of bipartisan uh thinkers on these topics, have said this is creating like a gatekeeper effect. This is creating a toll booth, it's it's undermining the open web. So when you look at this, right, is Cloudflare taking this action? They view it as giving agency. Of course, it also helps them financially, right? Uh, because they provide this service, so it's good marketing for them. You look at this, how do you view this tension between, you know, giving agency to content holders versus the potential downside of closing off the web? Like, how do you look at this situation?

SPEAKER_01 23:25

Yeah, I mean, uh first and foremost, of look, we've always had paywalls, you know, or we always had ways of where you can put your site behind a password and block it off from people. There's nothing inherently wrong with a webmaster deciding to do that, right? Different strokes for different folks. Um, that's fine. Where things do get, I think, trickier, or uh you were suggesting, where these decisions get intermediated or where people have a significant amount of bottleneck power in the ecosystem, such that it's not that they're necessarily empowering the webmaster to make decisions, it's they're tilting the playing field to favor themselves, to hurt other with you know harmful unintended consequences in ways that hurt consumers. To be more specific there, right, like if somebody wants to turn on what you know the equivalent of a paywall to crawl their site, Cloudflare says pay-per-crawl, nothing inherently like in principle wrong. It's the in practice. If in practice um Cloudflare gets a cut of that and they have a significant market share, and they want to then steer people into pay-per-crawl, well, that starts to make things more complicated. If they set the defaults such that the Internet Archive or Common Crawl or other sorts of public interest repositories are you know blocked without the webmaster even really knowing, um, that's less okay. I mean, it's maybe some people want to block the Internet Archive, but there are lots of cases where people like, oh, my webmaster just turned this on and we didn't mean to block you. Um and finally, I think it leads to some hard discussions of yes, webmasters they need agency, controls, different sorts of business models, fine. But where does your right as a webmaster, as a publisher, end, and my right as a reader begin? Right? Again, we have we've we had a world before where you know, physical books where I own the book, I own a copy of the book, I can rip it apart, I can loan it out, I can sell it, I can use it as a pillow if I'm laying in the park. I can do, I can, you know, I can even privately perform it for my family in my living room if I want. Um, I can read it aloud. Ebooks don't work that way. That is a function of law, fundamentally, and technology, but law. And so these are choices we get to make about how we want the web to work.

SPEAKER_02 25:42

So getting to the stealth disguise problem, because I think this gets at the heart of whether codification is necessary, right? You brought up this notion earlier that some of these standards, some of these best practices over the years, some folks now want to start making laws about them because of what they see as bad actors who don't participate in in the voluntary system. And maybe some of these bad practices are being incentivized by the competitive pressure, right? The race to AI, this existential fight for market share to achieve artificial general intelligence, right? The superintelligence that's gonna rule the universe, right? That everyone's kind of acting crazy right now because of that, and that norms go out the window. And this is not, this does not apply to everybody. And I'm not even saying that these allegations are 100% because uh perplexity disputes them. But August of 2025, Cloudflare, right? They're gonna come up a few times on the show, right? They published an investigation essentially saying that perplexity, which is an AI search engine that that you know uses a bunch of different models, listeners might be familiar with it, but they're basically trying to compete with general internet search. And I've used it, it's a good product. Um they essentially, the the allegation from Cloudflare, right, is that publishers or content holders had had sent the robots.txt saying, do not crawl me to block perplexities, bot specifically. And perplexity then sends a bot that's pretending to be a human, right? That's disguised as essentially Chrome browser, right, which is used by humans on a Mac, right? Used by humans. And and they rotated IP addresses to dodge detection, sometimes straight up ignored the robots.txt, right, which requires a the bot to fetch it, right? To to acknowledge that its existence is part of how that works. And they said they essentially did like a sting operation to catch them, right? By by by saying block all bots and and perplexity still got through. Now, perplexity called the report a sales pitch, they denied the traffic was theirs, whatever. Um, and then Cloudflare said OpenAI as a counterexample, they did the right thing, right? They they listened to the instruction. So let's just assume that this is happening, whether in this particular case or not. Let's assume there are bad actors out there that are stealth crawling. And I've I've talked to publishers who say that this is a huge problem, right? That there are essentially stealth crawlers, maybe they're from China, wherever, that are that are going to sites to ingest the information for AI training and inference, not saying who they are, which prevents the website from deciding do they want to allow that access or not, right? To your point, maybe a lot of folks want the internet archive, right? These nonprofits that are dedicated to the preservation of information, they want them to go in. It's a nonprofit. They're not trying to compete with me commercially. Maybe they view a perplexity a bit different. They're, they view them as, you know, I'm a news website. My job is to provide facts to the world. That's also what you're doing in a different way. You're gonna summarize my content such that no one ever visits my website. You're essentially a competitor to me. So nothing wrong with that, but I'm gonna block you. And then use stealth crawl to get my stuff anyway. When that happens, it suggests that the standards may not be upheld, right? That all it takes is competitive pressure or, you know, the the profit motive for these things to collapse and that there really is no recourse, there's no enforcement mechanism, there's no penalty for noncompliance, other than maybe creative legal theories that you brought up earlier. So when that happens, is that not making the case for a law, right? If these are good standards that most people agree are reasonable, and the the quote unquote good actor companies acknowledge that they are good standards, what is the harm in passing a law essentially saying that these protocols need to be enforceable and that there need to be penalties for noncompliance? If these are, you know, it's the idea would be like industry is self-regulating, but in order to give that self-regulation some teeth, we need a an enforcement mechanism.

SPEAKER_01 30:07

Is there anything wrong with that from your perspective? There certainly can be. I mean, I think, or to take a step back, I think there have been a lot of valuable parts to robots.text effectively being voluntary, um even though it was broadly followed and so on, um, because it then creates the space for those public interest uses like the ones you were talking about to still happen and to not have, you know, law can be a blunt instrument with lots of unintended consequences. So, you know, to reference the the you know, the internet archive again. They generally follow robots.txt, but on some sites they don't because it's a government site or it's some other site where they're an archive, they're a library. It makes sense for them to be a historical record. That's different from the competitive situation you were talking about with sort of a CNN and a perplexity and um, or sorry, I say CNN because they f they sued perplexity just this morning, saying exactly the arguments you were just making. I think that's quite different from the sort of thing um, you know, with an Internet Archive or Wikimedia or whatnot. I think they're the path forward to me it would be not starting with the blunt instrument of law, because then you end up with all those unintended consequences. Um instead, I think we're in this period of, as you've said, you know, new technology reactions happening, right? It's and it is a bit of a cat and mouse game at first, right? Companies do one thing, Cloudflare and others come up with ways to block it and back and forth. But my expectation and what I see in the field is to some extent, despite huge headlines, some of that is leading then to constructive partnership conversations. Let me step out of news and the CNN example for a second. Talk about shopping, where sure there's litigation between Amazon and perplexity. Amazon says perplexity, you know, I don't like your bot coming to my site. Perplexity says, I'm just giving a shopping agent to your customer who has an Amazon thing and wants to shop at Amazon. What's the bad thing?

SPEAKER_02 32:00

And just to provide some context on this example, and hopefully I'm not throwing you off your train of thought, but uh, because this is this was something I wanted to bring up. So you read my mind. So perplexity has an agentic AI, when we say agent, right? An AI that is supposed to act on the user's behalf to achieve an end that the user wants. It's called Comet. And Perplexity, do I have this right? They were sending the bot to Amazon to shop. Were they checking out and actually literally buying things? Were they just comparison shopping? Like what were they doing that pissed off Amazon?

SPEAKER_01 32:31

Yeah, so let's yeah, let's sort of telecase from both sides. Amazon says, sure, I want you to shop here, but I want to control how you navigate the site because I need to collect data on how I improve my services. I want to direct your attention to some products versus others that you might like, plus the traffic issues and the engineering I have to deal with, blah, blah, blah, if there's other bots. Okay. On the other hand, Perplexity and folks like EFF and others who've supported them in this litigation note, Comet, which is Perplexity's browser, which has AI built into it, it doesn't cause Perplexity servers to access Amazon's computers, let alone protected areas of the site. Instead, Comet sits on it's my browser on my device. It communicates with Amazon servers on my behalf as a consumer, and then, and it's generally public pages, although when I go to check out, sure, it's behind, but it's my login. I get that data from Amazon, and then I share it with Perplexity. Say, hey, Perplexity helped me analyze this. And I'm the consumer. Why shouldn't I be able to augment my shopping experience with a bot of my choosing? And that's, you know, that's roughly that's perplexity's argument. And I, you know, I tend to think, on the one hand, all of these folks are companies. So I'm glad you called out like everybody says save the open web, and everybody can have a grain of their type of it. And it and they're all self-interested. So they're all like pulling at this value exchange issue. But, you know, personally, I have a lot of sympathy for the consumer rights angle in this. That shouldn't I be able to do what I want? Um, there's that litigation, but I I don't, but I want to be clear coming back to your question about where's the role of the law. I don't think law is the best option here, nor do I think it is the inevitable place where this ends. Because if we take a step back and we think about shopping, right? Shopping agents and shopping sites, there's a lot of alignment of interests, right? You're trying to sell stuff to consumers shopping site. I run a bot that will help them buy more stuff. We're aligned. And that's where we start to see lots of APIs and new standards coming out, AI companies and commerce sites saying, here's how the shopping agent works. Basically creating a front door for the agent to go through. And that might, those sorts of systems might be augmented by identity standards, which might say, hey, I'm a well-behaved bot. Maybe it's about like, hey, I am this company specifically, right? I'm not just saying, you know, comet is built on the Chrome browser, so it says Chrome, but it says I'm perplexity. There are some real privacy and anonymity trade-offs there to think about, but these will be in the stew, in the mix, and we should be considering all of them, as well as signals that are about particular uses. Maybe somebody says, you know, I don't want my content crawl for foundation model training, but I still want to be in Google search, you know, with the links, but not in the overviews, right? And that's another thing IETF is working on. So I do have some hope that people, right? We come, there's this disjuncture, people fight, they sue, that people do start to come to the table. And we do get new norms, new standards, new tools. With new trade-offs, we have to really think carefully about, but we can get a lot of the way there. And not every case will resolve that easily. So what I mean there is like it really matters where you set the default, right? Does Amazon get to say, CFAA, unauthorized access, you're done? Or is the default perplexity is let in, and then they negotiate. What's the background condition? To make that a bit more precise, let me give you an example around Google and I know you followed Google Search and the search anti-monopoly case and the remedies there. Right. So part of the remedies in the Google search case are Google needs to now create an API to share search result data, you know, and other other stuff related to the system with third parties. And the competitor Kaggy, that's K-A-G-I, you know, they're one of the ones that hope to be beneficiaries from this API. They're competitors. They've suffered and they say these remedies could be good, but today there's no API. So what am I doing just sit on my hands? No, what KAGI does is they go to a company called SERP API, which looks at Google search results, you know, takes it, puts it out of form where it's interoperable with Khaggy, and KAGI can then build on that. And Google has sued SERP API with a different unauthorized access claim saying they're circumventing an access control on this public website under what's called the the DMCA 1201. Generally, the thing that's used for ebooks, as I was talking about before, same sort of claim. So, like, here's the thing there's no antitrust remedy today. So if you shut down SERP API, KAGGE's screwed. And the question is, how much do you trust that antitrust remedy tomorrow? And so I think we need to think very carefully about how we allow what um Corey Doctor or an EFF call adversarial interoper interoperability. You know, where look, your front door, your API, it's broken. So I'm gonna go over here and crawl your site in this other way to help consumers and to help competition. And I think an answer can be not, you know, we need a law, but rather let them fight, you know, in in some terms, right? Amazon perplexity, play your cat and mouse game. And eventually maybe they come to the table because it's better for both of them to negotiate, right? And I think that's something we should keep in our minds of not trying to fix everything by law when we can leave that space for other ways to solve things.

SPEAKER_02 37:37

No, that was a very thorough and comprehensive answer. So I want to unpack a few things individually. So so getting back to the Amazon example, Amazon's a sophisticated company. They're obviously developing AI. They they give a ton of money to anthropic, they run cloud services, like they know where the future is headed, right? They're they're it would be difficult for me to imagine that Amazon thinks that human beings will never employ agents to shop. Yet what Perplexity says about Amazon, right, at least now, is that their main motivation for blocking the or not even blocking, right? But but trying to control Perplexity's bots and how they access Amazon is advertising revenue, right? With a very similar concern, ironically, to the news publishers, right? To the recipe website, right, to everybody, right? They all say the deal was when I put this stuff out there and a human being comes and looks at it, I get some money. Maybe it's not enough. Maybe it's it's uh I don't like the deal that I have, but it's something, right? And what perplexity is saying is Amazon is not just a company that makes money when you buy something and they take a cut. They also make money when you just look at the screen, right? And you see sponsored battery, sponsored third-party charger, whatever, whatever stupid thing you're looking at. And so it's interesting allegation, given that Amazon must know the where the world is headed. Just curious, Derek, like briefly, not to linger on this too much, but do you think that that is Amazon's motivation, right? That they're essentially trying to preserve, to squeeze some juice out of this lemon while it's still there, even though they know that eventually they're gonna have to let the bots buy things and that human beings are not gonna want to sit there and scroll through a thousand third-party battery chargers over and over again? Is this just like while the getting's good, let's get some more? Or do you think they have another motivation for the aggressive tone they've taken with perplexity, both tone and legally?

SPEAKER_01 39:31

Yeah, I think it's a good question. And and and again, just sort of say, and look, I use Amazon, I have an Amazon router, and when I use my Amazon router, there's an ad blocker in there. So Amazon is perfectly happy in some cases giving you some AI to go change somebody else's website, but not here. And I the not here is I think, yes, it is sure, their self-interest. They they want to, it's not just the ads, they want to rank things in ways that can maximize their profits or maximize their long-term profitability from you as a consumer. On the one hand, nothing inherently wrong with it, that's business. On the other hand, market power, concentration, right? These are the self-preferencing and open marketplace the debates we've been having around Amazon and search and Apple. It's very similar. I mean, you can draw a pretty clear analogy between, you know, Amazon's desire for control here. I need interface control. That's what my customers want, that's how I manage security. It sounds a lot like Apple in the App Store. And so your intuitions about the right answer here, I would imagine, probably track more or less those. What's what's strange here? I note it that way because for some people it doesn't for reasons I don't quite understand, except AI is this sort of, you know, you put AI on it and people take a totally different view, even within the same company, right? But yeah, I mean, I think it's not that different from Apple saying, I need control over the App Store, I need to control how things ranked, et cetera, et cetera. Or, you know, or Google with a Play Store for that matter.

SPEAKER_02 40:58

Well, and because you mentioned a lawsuit, I guess, that I was unaware of this morning. We're recording this on May 28th, just to orient the viewers and listeners. Um, and and Derek mentioned there's a some new suit, I guess, CNN against perplexity. So we'll see how that plays out. And there's also a chance um that by the time folks listen to this, that new antitrusts or you know, a version of an old antitrust bill will be reintroduced that actually gets at some of these self-preferencing issues. Now, I don't know if that bill will apply to AI at all, uh, but that'll be something interesting to look at. Now, getting to Google, right, because they have a unique position as what has been ruled, right, as a search monopoly by a judge. And you brought up a couple of interesting questions, both SERP and the monopoly problem. So one of the issues that's come up in the scraping wars, right? If we if we're gonna call it that, maybe that's the title of the episode. I haven't figured that out yet. But is that Google has this unique market power and ability, even in the context of good actors, right? So let's assume that I'm a publisher of whatever it is, doesn't have to be news, and I put up the do not crawl and everyone listens. Google doesn't have to listen for a very important reason, is that Google bundles its search crawler and its Gemini AI crawler. Meaning if I say no to the AI training, because I view that as a competitive threat, I also then get removed from search entirely. And the way I've described this personally is you're giving certain publishers a choice between a slow and an immediate death, right? The slow death is you let us continue to use your content for AI overviews, which sends less and less traffic to you over time. Or you get an immediate death where you say no to AI training and I just cut you off entirely because I have 90 plus percent market share in search, and not being on Google search for many is a death sentence. Now, the interesting thing here is that this also creates a competitive issue potentially, right? So if I'm a perplexity or if I'm an open AI, publishers can tell me to screw off, and it doesn't maybe cost them much, right? It the numbers show that AI bots are not referring traffic to to publishers, right? So I think one of the stats is you know, one in 14 pages that were crawled, right, by a search bot, there's some there's some traffic. The numbers for AI, it's one in several thousand, right? So just orders of magnitude, less traffic, according to this data, is being sent to publishers. So AI is not a good source of traffic for publishers, at least as it is today. And so they rely on that general, that old school Google search. So they can tell perplexity and chat GPT, you're not worth my time. Like block robots.txt. And if they're good actors, they agree. They can't tell that to Google because of Google's market power, and maybe that gives Google an advantage. So, one, do you think that that's an antitrust? Problem that essentially the judge should be updating his decision on. You brought up the remedies. Two, do you think that that advantage incentivizes companies like Perplexity to hide their bots, right? To essentially violate the code of conduct that had upheld for so many years because they're worried about falling behind competitively. Right. If Google can just get everything because they basically point a gun at the publisher and say, let us train or you're done. And the others can't do that because they don't have search monopolies. You know, how do you how do you how do you look at that situation? And do we need an antitrust remedy? Because we're not only, it's not only a competitive problem, it could also incentivize bad behavior.

SPEAKER_01 44:43

Yeah. And just to clarify one thing, you know, I think that is the way Google worked at when, you know, sort of the AI boom first, uh the generative AI boom chat GPT first started, they have sort of uh moved slightly where you can say, I don't want to be crawled by Google extended, which is their thing for Gemini and a bunch of other stuff, without disturbing being in search. The thing that you can't do today, though, is say, I want to be in what many of the publishers have called traditional search, a link, a snippet, that's it, versus AI overviews, right? And that's that's the so it's different than the Gemini part, but still quite you're you're you're spot on in that.

SPEAKER_02 45:20

But that's that's that's really the issue. And that we care more about it.

SPEAKER_01 45:25

And there are processes that you know that the competition markets authority in the UK has been moving forward with getting Google to create an option to say, I want to be in search, but not in AI overviews. And you know, that process, so that competition process, that antitrust process is ongoing. And this has been something in the IETF too, where how how might we create a signal, not just for Google, but for search generally? We can say, you know, don't crawl my site for foundation model training, but yes, I want to be in search where search is defined as blank and not blank, right? And not this AI overviews thing. And I do think that you're right that, yes, Google is different. Or I think there are, you know, in lots of these cases, we could treat some version of big tech as a distinct problem from the general problem of, hey, you're getting content from my site, but not sending my traffic. How do we how do we make sure that this is good for us and we can both be sustainable? There's a particular problem or challenge with people with more significant market power. And where Google does, yes, have greater visibility of the web and 40 million books that they can train on, you know, that you know, basically effectively nobody else can, and YouTube and and and that creates a different thing. And I think that um to circle one thing, uh, you know, I'm not an antitrust scholar, and my instinct is yes, competition and other parts of media policy are the right lens rather than copyright, which is, you know, copyright is the main event. I don't think necessarily think I don't think they're necessarily good arguments when it comes to some of these news cases, both on the foundational model training, but also on the, you know, you're inferencing my site and creating a summary. You know, perplexity's response to CNN was a little bit too terse of just saying facts aren't copyrightable. And that's true, but more generally, let's remember, newspapers every day copy and summarize other third-party sources without getting a license or paying them. So it's another notion of, you know, fair for me, but not for thee. We had the Amazon version earlier. This is the newspaper version. They quote and summarize people all the time, they don't necessarily send back traffic. We need other ways of dealing with the sort of the market power issues and get it out of the copyright space. I don't think copyright is the right lever. And I think we need levers that don't say, hey, consumer, you're not allowed to use an AI to summarize my article anymore or to read aloud, right? Do what we did with ebooks. I think we need something that is more protective of consumer rights than that. And yes, we need to come up with systems where news publishers publish, where they are readily accessible, where they can get value, and they can the people who help distribute and make them findable get value too. And the more we can focus on that rather than copyright, which is really it gets the attention because in the US, statutory damages are astronomical. So, you know, if you want to come sue somebody for several billion dollars, you use copyright. You don't use a lot of these other uh causes of action. I think that when we move beyond the copyright debate, we can get to solutions much quicker.

SPEAKER_02 48:23

So let's talk about that. Because right now I think it's very understandable that if you believe that your content is valuable and you're not getting paid for it and AI companies are making a lot of money off of it, you're going to turn to the tools that you have, right? And the tool that exists is copyright law. And that's why you're seeing these lawsuits, because what other lever do you have? If the AI companies wanted to negotiate ahead of time for the content they would have. Now, to be to be clear, there are a lot of actual licensing deals between AI companies and publishers because in some cases the AI companies have acknowledged that, you know, Reuters or Condi Nast or News Corp, whatever, some of these large publishers have a lot of content that is that is worth paying for while still arguing in court that they shouldn't have to pay for it or if they chose not to pay for it, they shouldn't, you know, a lot of folks view it as a hedge against litigation outcomes so that they don't like get cut off, you know, if a judge just decides to go like crazy and say like delete everything, right? They're not they're not they're not totally cut off. But at the end of the day, Derek, whether it's copyright or not, people need to get paid, right, to work, right? And this is one of the narratives that that's in the AI debate. It's not just about disrupting entry-level white-collar work. From the creative community, they're gonna say, what is the incentive? Now, a lot of folks, you know, including folks in your community, have said, well, the individuals are gonna, you know, if you force AI companies to pay, they're gonna get 50 cents, right? It's not, it's just not gonna be when there's billions of images out there, the person who took one photograph is just is just not gonna get paid. But at least advertising has historically been a shot that folks had at making money if they hit it big, right? Like if you create a website that that blows up, that does really well, you can make a lot of money off of advertising. Or if you decide to paywall, then you can make a lot of money off of subscriptions. It comes down to humanity, right? Like human beings are the reason that those content are valuable, and it's not valuable to have a bunch of bots scrolling around your site that then don't pay you at all. And and a lot of folks in copyright, like the the copyright skeptic world, I guess I'll call it the copy left, right? If we're being if we're being oversimplistic, right? You have the copyright, so to speak, which is like strong IP, like aggressive IP, and then you have the copy left, which is like, you know, we need to reform the reform these laws, the copyright's too strong. The copy left doesn't like things like putting up toll booths, like requiring AI companies to pay every time they access a site, right? They don't necessarily like what Cloudflare is doing. What is the alternative? Is my question. If it's not copyrighted, how do you get people to get paid? Because these companies have money to pay people. I I find it absurd when folks in big tech land say this would be a burden for AI financially to pay creators. They're spending hundreds of billions of dollars a year on data centers, right? They can pay an artist for their work. They can they can do it, and there's nothing stopping the licensing market from emerging other than their willingness to participate. If they sent a signal out to the world that said we are willing to pay for content, you bet your ass there would be people trying to build those marketplaces because it would be very lucrative to be the person that comes up with the mechanism to pay creators for their work, whether that's in bulk, whether that's as a consortium, whether we need to give them an antitrust exemption, whatever. So my question to you is kind of an existential one. If not copyright litigation, what is the mechanism to get people paid so that people have an incentive to do stuff other than just being hobbyists? Right? We want people to get paid.

SPEAKER_01 51:58

And I think, you know, I think you point to some of it in the way, part of the way you are characterizing it, which part of this is going to be business model evolution, that is already ongoing. Or rather, you're right. There are different deals going on and different marketplaces being set up and so on and so forth. And to drill into that a bit more, in a world with strong fair use, where training is free, et cetera, et cetera, there will still be the markets that we already see today, where people are paying for access to materials that they could not otherwise get, right? Stuff that's behind a paywall or in the archives of a newspaper that you know needs to be digital, whatever. Those deals are already happening for access. And we're seeing people cut deals for you for uses that, again, even where it would be there are some uses that are permissible under copyright law, others aren't. So to give an example, right? There have been these deals between newspapers and various different companies, Perplexity, OpenAPI, that allow use of content to display like full content of articles more than a quote, a snippet, right? The thing that might be fair use. That is consistent with deals that have been done in the past. It's consistent with the story of YouTube, right? YouTube, they said people are mad. All the stuff is on your site. I didn't give you permission. I don't want to send you all these takedowns. YouTube didn't just say, well, you can block it or not. They said, you can block it or you can leave it up and run ads and we'll share the revenue. And that's what we're seeing with, you know, Suno, Udio, you know, Sora was a version of this. It didn't last, but it's that version of business model of, oh, let's unlock ways for people to remix content, to reuse content in ways that wouldn't be permissible as fair use, like training should be and is. I think those deals are gonna come about. And I think they have a lot of promise. Then the question becomes okay, what problem are we solving for? I think where people should rightly be skeptical of copyright is is copyright going to be a solve for a crisis of automation where AI is so good that it can replace creators? No, it's not. They're the best people can expect is a trickle down of pennies. Most of the money will go to the major media companies. If we're thinking about competition, an anti-monopoly policy, the worst thing you could do is require that everything be licensed, because then that is creating a new barrier to entry to everyone not named Google or Meta, right? And again, they already have huge bodies of data that they foreclose from others, like YouTube, right? Where they can use YouTube, nobody else can. So, you know, if that's what we're solving for, can't can't get there from here. I also want to understand one last thing, which is yes, there will be deals, these will be a growing market. And so, fair use, what it did, what it is doing is it opened up all these new markets where people will also make money. It is also true that the vast majority of data will never be available to license in any form, period. The vast majority of all works ever created are not actively managed at all. Right? There are over a billion websites. Most are not actively managed. Most books are out of print, out of commerce. You know, most of the 20th century is in copyright, but not actively managed. So, and that's you know, that's most of the training data. When people say, well, you can just license your way, if you look at like Llama 3, so now a generation ago of data, it would take 316,000 years of New York Times publishing, 316,000 years to get the amount of data that Llama 3 was trained on, right? That's not, you're not gonna get that. So um, you know, as Matt Sag and his co-author put it in looking at this, like a bunch of cherries doesn't make a Sunday. Like, yes, there will be these training, these training deals or these licensing deals or data deals, but those things don't scale. They don't add up to the amount of data you need to train these high-performing foundation models. And so that's still going to be something where we need a copyright exception to fit in.

SPEAKER_02 55:48

I want to break up some of the arguments that tech companies have been making that some have characterized as hypocrisy and maybe undermining some of the middle ground potential areas that that could lead to win-wins here. So you brought up, right, the the Amazon case of the Computer Fraud and Abuse Act. You brought up the Google and SERP AI case, which was not CFAA, but kind of similar principles of, you know, Google scraped the whole web, didn't ask for permission. This is all in the SERP AI blog, which I read before uh today. They said, you know, they did all this, right? They scraped the whole internet to build their product, and now they're mad that we're scraping their scraped product, right? The product that was created through scraping is being scraped, and now they're like, you can't do that, right? It's scraping for me, but not for thee. And to many, that might look like hypocrisy from a company like Google. It's like you created your product by just doing whatever you wanted. Now you're mad that someone else is doing the same thing. Similarly, we've seen this other argument against distillation and a brief description of distillation. You can tell me what I got wrong in your answer. But we saw OpenAI and Anthropic go to Congress, you know, get go to the White House and complain that China, Deepseek, a Chinese AI company, has essentially been misusing American AI products to try to distill or steal their innovation. And the way they do this is they violate the terms of service of the product, right? The terms of service of the product say you can't set up like thousands of bullshit accounts and blob lob queries and prompts at me and then try to learn from those. Like you can't do that. If you and I tried to do that, we would get kicked off, right? They would delete our account. But DeepSeek was doing this. And they essentially successfully, right, in in late April, the White House said, we see this as a huge problem. And Michael Cratzios, the head of the Office of Science and Tech Policy, said something like, There's nothing innovative about stealing these American innovations. Now, if you're right, again, a publisher or someone, you might be looking at it and be like, Are you kidding me? Like, that's the exact same thing that I'm saying about what people are doing to me. All right. So it's it's when when a Chinese AI company comes in and lobs queries at my product to try to learn from it, it's distillation and it's wrong. But when an American AI company goes to an American copyright holder and scrapes everything, then it's learning and it's fair use, right? So some folks are going to look at that and say it's hypocrisy. Now, one, I guess, do you think there's merit to that hypocrisy argument? But more importantly, I want to ask you, as someone who's advocating for sensible middle grounds, right? Potentially antitrust as a remedy, interoperability requirements as a remedy, do you think that it hurts the potential for grand bargains if you have companies talking out of both sides of their mouth, right? Saying that what you're doing is unauthorized access, but what I'm doing is authorized access, right? You know, CFAA is a problem over here, but it's not a problem over here. Right? If everyone's just going to be hypocrites, which I guess companies are wont to do, does that narrow the window of opportunity for sensible policy making here?

SPEAKER_01 59:04

I certainly share your worry. Or rather, I spend a lot of my time on policy and trying to think in these principled ways. And I also recognize that politics has a lot of these hypocrisy, contradictions, things people say as a negotiating posture, and so on. And so I try to roll with it. But yeah, my intuitions, I think, are similar to what you're stating. I really recommend to people two things on this score. One is Nathan Lambert at the Allen Institute for AI, AI2, did a really long piece on the role of distillation in research and in the improvement of systems and why we should be really careful before you know maligning it writ large. Second, um, Mark Lemmley and Peter Henderson co-authored a paper about the enforcibility of these terms of service in general, where they note, I think rightly, consistent with the copyright office, that these outputs that you know you're you're doing a prompt and the AI system outputs, they're not copyrightable. And you can't use a contract to then lock up this public domain copyrightable thing, particularly for this like pro-competitive reason. It doesn't hang together. And I would be really worried about a world. Again, if I think about this, not just from the competitor standpoint, but as a consumer, I would worry about a world where publisher has a unilateral right to take a contract. That publisher could be an AI company or news publisher or whatever. They have a unilateral right to say, sign this contract, and here is granularly what you must do with this thing you created or what you or the book that you're buying, right? I mean, this goes back a hundred plus years now, where the Supreme Court faced a case where somebody had a book where they put a sticker on it basically that said, not for resale. And the court said, you can't do that. You know, the the the law gives the rights to the consumer. That's their property. They can alienate that property, they can sell that property, loan it whatever they want as they see fit. You don't get to override that with a contract. And I think courts have started to come around more and more to a posture I hope holds, which is different from what we saw in, say, Facebook v Power Ventures. There's a case where um Bright Data, you know, another company that does scraping and crawling and fetching of content for not just AI, but a bunch of things, was sued by X, by Twitter, um, for, you know, capturing public data. And the court said, wait a second, you you can't override the public's rights and copyright with a contract. That's not fair. You get certain rights as a publisher, but the public gets certain rights too. And you can't deprive them of that just with a stroke of a pen. So I do worry about um how these arguments get stretched too far. I do worry about that hypocrisy, and that's politics. So um I try and make the best of it. Um, but I I do think that at least the people where people can be honest about that is helpful. And to be really clear, this is not an argument for do nothing about China, right? So if you want that middle ground, that middle ground could still be how do we ensure that companies can share threat intelligence signals to protect their servers, whether it's for distillation or exfiltration or whatever. And how do we do that in a way that is good for national security, doesn't violate our antitrust laws? I'm open to that. I'm kind of skeptical that that's gonna be a big solve in fighting China, but I'm open to that. I'm less open to this, you know, it's fair for me, but not for the. And that's the distinction we're making. All right, final question.

SPEAKER_02 1:02:16

You've been very generous with your time. So there's a lot of reporting suggesting, right, that this contract is breaking, right? That the open web is is gonna die, and that there's there's gonna be, you know, bad outcomes from that perspective, right? Creators are gonna go behind paywalls, creators are gonna stop creating, right? There the symbiosis, however imperfect it was, between Google and the artist, Google and the cooking website is is fraying, and whatever replaces it, even if that doesn't mean everyone loses their job or their livelihood, is just not gonna be great. Now, you said you're optimistic, so paint me a picture. Five years from now, is there a symbiotic relationship between artists, creators, individuals, news publishers, et cetera, people who create content, human beings who create content? Is there a symbiotic relationship between humanity, the creative humanity, and AI companies? And what does that look like?

SPEAKER_01 1:03:11

Yeah. So things I am hopeful about, optimistic about. I do think we will be able to develop standards around, you know, preferences for foundation model training and search and ITF, that process has been going along. I think there will be evolution of identity and standards and APIs in various ways that help like reset the contract and the norms of what we're doing. I think a lot of that will move forward. Where I would say, and I think the business models as a whole for acquiring data, use of data for new uses that are beyond, all that is evolving and will continue to evolve. And I'm optimistic because I, again, I we watched some of this for the last 20 years, where we went from they said YouTube was only kids on skateboards and then it was a new medium, and now it's turning out the biggest, you know, horror movie directors of the last 15 years, all right. And like, you know, I think we're seeing that vibe shift in Hollywood in general on AI. And so I I guess I'm optimistic because I'm in the weeds on some of these solutions and I see them moving forward. They're not totalizing. And I say that in particular, the things I do worry about is not how are things going to shake out between these competing companies. Where's the user? Where's the consumer in this? Where are their rights? How are we protecting them? And how are we sustaining true public goods that the market won't serve, right? I I think news companies are going to do fine. They have, yes, they have valuable content. They will put up on a paywall, they won't, they will make deals. I worry more about the Wikimedias and the open access repositories of scientific journals and things that the market doesn't necessarily do with as well, including, again, local news, news that serves a community's needs, not big media companies. Those are things we need to take a very hard look at. How are we actually going to fund and sustain those? Is it going to be tax subsidies, tax credits, whatever? I wish there was more focus on that and less on the, you know, the the sort of the sky is falling of it all. The faster we get to those practical solutions, the better.

SPEAKER_02 1:05:07

We will leave it there. Derek Slater is a founding partner of Proteus Strategies. He spent 15 years at Google, where he, during that time, he built and ran the information policy team. Uh, before that, he was at two organizations that think a lot about these issues, uh, Electronic Frontier Foundation and Harvard's Berkman Center. Derek, thanks so much for joining the show. Thank you. Happy to we will link to Derek's writing. There's a lot of it, so I'll pick a couple examples in the show notes uh for today. Uh, but I had a lot of great conversations with Derek over the past few months. I think he's really thoughtful and I appreciate him coming on the show. Uh, this is the Center Edge. This podcast is sponsored by Digital Progress Institute. It is produced by Vulgate Media. If you're an AI company and you want to train on my content, please do. Just tell people about it. Um, you can find us on YouTube, Apple Podcasts, Spotify, wherever you get your podcast, please leave us a review. It will help others find the show. You can shoot me an email at Evanetcorepointstrategies.com with any suggestions, feedback, ideas for guests, you want to pitch yourself, whatever you want to say. Otherwise, we will catch you next time.