Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 25 - Privacy in the Time of AI
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Once upon a time we thought we had privacy. Then came credit cards, which captured the card owner's location and activity with each transaction. Then came the Internet, which made connecting all the dots easy and cheap, and the erosion of privacy accelerated.
Large language models, LLMs, like the Generative AI system ChatGPT and its ilk have the potential to make the cost of connecting dots vanishingly small, thus eliminating even the illusion of privacy, especially because large databases are irresistible to LLM developers as training data for their models.
In this episode of Exploring AI Matters we are fortunate to have as our guest Jane Horvath. Jane is comparatively unique in having earned academic degrees and practiced in both Computer Science and in Law. Before law school, she wrote software that may still be running on the International Space Station.
Jane is further distinguished by having served as Apple's Chief Privacy Officer, Google's Global Privacy Counsel, and the DoJ's first Privacy Counsel and Civil Liberties Officer, among other roles.
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap, Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that, in addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.
SPEAKER_03Once upon a time, we thought we had privacy. Then came credit cards, which recorded the card owner's location and activity with each transaction. Then came the internet, which made connecting all the dots easy and cheap, and the erosion of privacy accelerated. Large language models, LLMs, like the generative AI system ChatGPT and its ILK have the potential to make the cost of connecting dots vanishingly small, thus eliminating even the illusion of privacy, especially because the large databases are irresistible to LLM developers as training data for their models. Welcome to Mind the Gap, Dialogues on Artificial Intelligence, Jane. I am Mark Donner, a computer scientist.
SPEAKER_04And I am Roland Trope, a national security lawyer. We are your hosts for this episode of Mind the Gap, Dialogues on Artificial Intelligence. In addition, we have two more hosts.
SPEAKER_02Hello, I'm Charles Palmer, a computer scientist.
SPEAKER_00And I'm Mama Adams, a national security lawyer.
SPEAKER_03Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves them. Today on Mind the Gap, Dialogues on Artificial Intelligence, we are fortunate to have as our guest Jane Horvath. Jane is comparatively unique in having earned academic degrees and practiced in both computer science and law. Before law school, she wrote software that may still be running on the International Space Station. Jane is further distinguished by having served as Apple's chief privacy officer, Google's Global Privacy Council, and the DOJ's first privacy council and civil liberties officer, among other roles. Today she is a partner in the Washington, D.C. office of Gibson Dunn, where she co-chairs the firm's Privacy, Cybersecurity, and Data Innovation Practice Group.
SPEAKER_04Welcome to Mind the Gap Dialogues on Artificial Intelligence, Jay. You have credentials of both a computer scientist and a lawyer, which gives your comments particular relevance to this discussion. Looking back, how did your education in computer science inform and influence your career as a lawyer?
SPEAKER_07Ah, I could probably take, well, first off, I want to thank you for inviting me to this dialogue today, but I could probably take the rest of the podcast talking about how helpful my degree has been and how it's guided me throughout my entire career. When I graduated from law school, I there was no such thing as a privacy lawyer. Didn't exist. And so I very closely aligned my career at that point with a corporate practice group that was really focused on licensing deals, data deals, anything to do with data and computers. And I was fortunate enough early on to have been introduced to one of the first general counsels of America Online and joined America Online as the fifth lawyer. America Online was the internet. And so it was very, very helpful to me to understand the bits and bytes, the databases, et cetera, when we would get into product discussions, licensing agreements. I mean, when you're when you're looking at esoteric things like data, it was incredibly helpful. And then as the privacy profession started to grow, it's been hugely helpful for me as I've focused and specialized in privacy, because privacy, it really is about data and it's about data sitting online, and it's about software systems that process that data.
SPEAKER_04Well, you know, throughout your career at DOJ, Apple, Google, and Law Practice, you've had an opportunity to see not only the rise of the internet, but also now the emergence of AI into the public awareness. Have you observed recurrent themes involving AI and cybersecurity as those two areas have intersected?
SPEAKER_07Absolutely, absolutely. I was just, I spoke at another conference last week, Navigate 2025. And one of my one of the things I told the audience, I hope that we would do at some point is solve the recurring issues that always come up. And it's the same, it's the same issues, it's just the technology gets better. And those issues are terrorism, child safety, particularly child sexual abuse material, have driven a lot of movements. Terrorism has driven a lot of regulation, a lot of concerns. And then, of course, we're looking at big data and tracking and surveillance more writ large. Those are issues that recur and to my knowledge, we haven't solved yet. But the technology is only getting better. If you look at AI, and we'll talk about this more in depth later, but AI is just a much stronger tool, and it can be a tool for good, but it can also exacerbate all three of those areas.
SPEAKER_04Machine learning systems train models by ingesting or having data from large of large quantities put in. We know that models trained on inaccurate or biased data can produce flawed predictions, as you would expect. How serious and widespread is that problem, especially in the publicly available generative AI programs?
SPEAKER_07So, you know, I think that this is this is an ongoing set of concerns. And there was a recent article last week looking at some of the major large language models, asking them in particular about China, about Tiananmen Square. And you look at their responses, and all of them range from it was an incident, it was something, you know, very vanilla answers. So, you know, I think as as we look at these models, we know they've been trained, we know that we are a very, very large world in which there are tons of different issues, tons of different sensitivities. And these models are being trained and modified and their weights adjusted to deal with some of these very sensitive areas. And I think this will be an issue that we're going to see recurrent. How when you are looking at certain issues where an answer can be result in penalties or result in your model being banned, do you grapple with that when you're rolling out a product worldwide?
SPEAKER_04Can I ask just a quick follow-up because this is so much in the news? Several weeks ago, there was this bill that the House and Senate were looking at that would have imposed a 10-year ban on states regulating.
SPEAKER_07Oh, it's it's still in play. I hate I I wouldn't have to be able to do that.
SPEAKER_04Now they hammered out a compromise last night and they're reducing it to five years. I think suspending states regul any state efforts, although I know there's exceptions and I haven't been able to read them yet. Do you think that's a good thing or a bad thing, given that you know it's unclear what's really driving this?
SPEAKER_07Well, I have I have mixed thoughts here. I am concerned, and again, we're going to talk a little bit more about the potential harms that can come from AI or any technology, but I also have am living right now through a world in which we have privacy laws that the states have enacted. They're not all consistent, and we're up to, I think, 20 to 21 states that have enacted inconsistent privacy laws. And from a business's point of view, it's very hard to put a governance process or a compliance process in place when you have all of these varying laws that are not consistent. That being said, the federal government is going to need to step in and do something, I think, to set some sort of level playing field, level set of regulation, level set of response if there is a safety issue, because it can't just be that we tie the hands of the states. And it we have seen over the last 10 years, the states are able to enact laws while the federal government has a very, very difficult time doing that. And that's why I have mixed feelings. It feels like we probably need something. I mean, what we've seen in the past also, if we don't think about what should be put in place, we've seen reactive legislation, like the Video Privacy Protection Act. It was put, it was enacted in, I think, two weeks in response to senators getting very concerned that their video rental history could be disclosed as well when they heard the testimony of Bork in his hearings. So I think that if we're going to go ahead and put a moratorium on states, Penn needs to be put to paper to figure out what a piece of regulation looks like that could be applicable across all 50 states that may give consumers some feeling of comfort that that the technology they're using is safe, particularly in light of the stories that we've already started talking about.
SPEAKER_03So on a sort of changing changing the focus here, recently the machine learning community has been faced with AI models that have been trained on bad data. Data may be deemed bad by being unauthorized, proprietary, poor quality, biased, of unknown provenance, or collected in violation of someone's rights. In this discussion, we'll use the term tainted models to refer to ones trained on bad data. Industry executives generally offer two rationales for why they have to use bad data. First, bad data constitute only a small fraction of the training data. And second, it's impossible to remove bad data from the completed model. How do these industry rationales resonate with you?
SPEAKER_07Well, I think they they they raise valid points, but they also warrant scrutiny. The idea that tainted data represents a negligible quantity is difficult to evaluate without transparency, something that's often lacking in the current landscape. Even small amounts of biased or unauthorized data can disproportionately impact model behavior, especially in edge cases or underrepresented groups. You know, I think it's not an excuse to just say that it's too hard to find it. As for the claim that it's impossible to remove tainted data once a model is trained, that may be technically challenging, but not necessarily impossible. There is definitely ongoing research. I mean, we are in a nascent field right now. So there is research into machine unlearning, the inverse of machine learning, and model auditing and dismissing the problem as unsolvable, risk discouraging innovation in accountability mechanisms. So I don't accept that premise. I think it bears a lot more examination.
SPEAKER_03So would you comment on efforts to protect against or regulate the use of tainted models? I think you mentioned in our earlier conversations a new South Korean law.
SPEAKER_07Yeah, so South Korea's law that mandates deleting an entire model if any tainted data was used in its training is a really aggressive regulatory stance on data provenance and model integrity. The problem is, again, and I will have to give you a disclaimer up front, it's very easy to criticize regulation. It's a whole lot harder in this space to write regulation. You know, my area before I started focusing on AI more was privacy, but privacy laws are all based on this the fair information practice principles. They've been around since the 70s. AI, we're really looking at a harms-based set of regulations. I mean, the EU is the only one that's really gone ahead with an omnibus law. But I think the problems that we're seeing right now when we're looking at writing regulations is definitions. So when you're looking at AI and South Korea or any other proposed model, it feels to me like we're trying to do too much, you know, regulating AI. AI is software. Are we regulating the developers, meaning, or are we regulating the deployers, the people that use the software? And I think as people are looking at writing smart regulations and legislation, they need to start distilling down and maybe looking at a developer set of regulations and a deployer set of regulations. Because in software, when you're doing software licensing, usually the person licensing the day the software doesn't want to take on any liability. But with AI, where does it begin and end? And I think these are all really valid things that we need to be looking at as we're looking at regulations. I also think it's quite interesting that back in November 2022, we had a huge group of people calling for regulation. Please regulate us. This could be an area there's that you know that could be harmful. We really need regulation. Well, the EU sat down and they regulated, but no one likes what the EU has put out. And the EU is in fact, you know, looking and debating whether they should delay rolling out their regulation because it's hard. It's hard when you're trying to regulate AI writ large. And I would urge that lawmakers might want to take a smaller set of items and regulate those. I think we might come up with smarter regulations.
SPEAKER_02Hi, this is Charles. I can't let this one go by. When I was on the DP DPIAC, the Data Privacy and Integrity Advisory Committee, one of the folks we visited with was from Germany, the EU, right after they did all their privacy stuff. And being a troublemaker, hacker person, whatever you want to call me, I asked her, so have there actually been prosecutions? And she said, Oh yes. And I said, have there been convictions? And she said, Oh yes. I said, Have has anybody served time? And she said no because of the issue of trying to figure out who done it. Yes, yes, because data can go anywhere. Well now, AI is going to be the same thing. Given any arbitrary piece of software, I mean you know this. If you ask someone, is this an AI or is this software? Well, okay. That's getting harder.
SPEAKER_07Oh yeah. I don't know if you saw the New York Times quiz yesterday. They had 13 different videos up. I, who specialize in the area, only got three of my and you had to answer whether it was AI generated or not. I only got three right. So it is getting, in fact, very, very good.
SPEAKER_02So so how is regulation? Regulations are great if you can enforce them.
SPEAKER_07Yeah.
SPEAKER_02Yeah.
SPEAKER_03We have a history. I'd just like to sort of, you know, comment on what you just said, Charles. We it is not unknown to give a product the death sentence. Kodak introduced a an instant camera in competition with Polaroid and in violation of their patents. Polaroid took them to court one, and the the remedy that Kodak insisted on was the ending of the entire product line, death sentence. In some sense, what the Korean law that Jane talked about a minute ago was is also a death sentence for a product. So we we you know, given that we're not talking about killing humans, we actually have a a model that we could actually apply.
SPEAKER_02Well, my question is how can you know that it needs to be applied with AI? I mean, with that, it's a physical product. This is this is thought, essentially, or fake thought.
SPEAKER_07This is gonna be this is the question this entry because also there are existing laws out there. I mean, that's the other thing. You have people, oh well, AI isn't regulated at all. I had a client that had asked us to do some research on whether they could use an AI agent for a particular thing, and they ask us to look under existing law. They were a regulated entity, and it turned out that existing law required decisions are made by a person. So right now, I think the the onus is to look at existing law. And then I think, Charles, your point is very, very valid. When you look at existing law, does existing law need to be changed to determine who is actually operating the AI? Is it the end user? Is it the developer, et cetera? Who is actually responsible when the AI does something wrong? And I think that is really the heart of the matter. I wish I had an answer there. I think it is going to be probably the most difficult question. It's working its way through the courts right now with respect to copyright, you know, models that were trained using copyrighted material. And just last week, the copyright holders did not get the answer they wanted. And, you know, it will go through the appellate process. But, you know, these are all very interesting questions on how does AI impact existing laws as well? And where do existing laws actually need to be updated to take into consideration that we've got a new artificially intelligent thing that is making decisions? It's no longer human decision making if we ever reach AGI.
SPEAKER_00Jane, I wanted to go back to a term or a phrase you were using that caught my attention, sort of smart regulations, right? Concept in this idea of creating smart governance or regulations around AI. You know, from my vantage point as a very technical international trade, national security lawyer thinking about risk-based rules in governance.
SPEAKER_07Yes.
SPEAKER_00You know, part of the challenge is here as I see it, you know, not having the technology hat to wear the dual-hatted as you are, is part of the challenge seems to me that who really is smart about AI, right? Sort of the pace and the evolution of the technology is, you know, seems to be broader and greater in scope and deeper than anyone would have appreciated even three years ago or five years ago. And as you know, sort of legal regimes move slowly.
SPEAKER_07Right.
SPEAKER_00Right, right. Yeah. Technology that's moving so dramatically and significantly. I was really interested in what you were saying around like the focus around definitions, right? Sort of if you were to have your your your wish list of how to break this down and distill it, sort of what would those components be like? Because I don't know if there's a way to really govern this in its totality because it's so broad.
SPEAKER_07Well, I think as you're as you know, we've been looking at various legislative proposals. I think distilling it into a whole lot more subsections. So when you were talking about AI, you need to have a deployer section. You need to have a developer section. Then even under developers, you've got open source developers in which there, as they deploy software, the questions around open source are incredibly difficult because they'll put out a model, and that model can then have its weights modified. I mean, it's very hard for them to foresee. So I think you're going to have to parse the entire ecosystem and look at it. And to me, as I as I represent clients, I really talk about building governance processes inside. So, you know, what kind of data are you using to train, you know, auditing functions, transparency functions. So you know what is going into your data, or you know if you've bought a model, you've looked at the model cards, et cetera, and you are documenting everything. But I think even without AI regulation there, really the onus is on companies right now to put in place self regulatory governance structures. You know, bias, I know we'll get to. We'll we're going to talk about that probably later, but you know, that's going to be a really critical thing as you're looking at your models. How have you tested all the extremes? I mean, when you're a software developer, you do test extremes, but AI has a lot of other different extremes, it can have a lot of different harms.
SPEAKER_03I've been wondering in this whole question of governance, if part of the problem is kind of our obsession with writing it down in incredible detail in advance. Okay, we tend to write, you know, these huge stacks of paper that then either get gamed or are are uh vague or incomprehensible or whatever. And you know, I was intrigued we interviewed a European ethicist several quite a few episodes ago, and she talked about one of the models that that they've been experimenting with over there, where they put uh union uh representatives on corporate boards. Uh which sort of establishes sort of a stakeholder voice in sort of key forums. I'm not sure what how one would apply that kind of a model, but I could see it as a you know, as a way of saying, hey, you know, there's there's various players. There's the people who build things, there's the people who operate them, and there's the people that the systems act on. And in some sense, those who the systems act on tend not to be represented in any of the decision making. The builders and the operators tend to be there, but the the objects tend not to be.
SPEAKER_07You know, it's what you're proposing and what was discussed in that context is is also quite similar with what's required under European law. Technically speaking, European businesses have to have a data protection officer who is independent. And I think, you know, while the aspirations of the regulation are in fact there, and and the the aim is exactly what you're saying, is it would be great to have someone who is a user and who can look at this from an independent standpoint. Independence in a company where you've got confidentiality, you've got decisions made every single day. I think it becomes very hard to insert someone that is not to maintain that true set of independence. It's difficult, I will say.
SPEAKER_04You mentioned that part of the problem of writing regulations is definitions. Yes, I've noticed a rather sharp divergence in the kinds of definitions that are offered or crafted about AI. Recently, in a long negotiation I've been involved in, I I can't discuss except the two kinds of definitions that are out there are reflected in that debate. One is the very what I find overly complicated, and so complicated that I don't understand how the parties work with it, which you find in the EU Act and in the California Act. They basically are the same language. And they have, you know, AI system means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that for explicit or implicit objectives, well, why do you need to state both? Infers from the input it receives how to generate output such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. By the time I get done reading that, I don't think I could ever recognize an AI system.
SPEAKER_07And it to me would offer maybe every system is an AI system that's a computer system at this point. That's the other. It's so broad that it actually includes too much.
SPEAKER_04The other trend is in the UK and New Zealand, where their judiciaries have opted for really simple, readily comprehendable definition. They may not get more technical and they may not be overly precise, but at least people sitting around the table can say, okay, if that's what you mean. Artificial intelligence, according to the UK judiciary and guidance they issued in April of this year. Computer systems able to perform tasks normally requiring human intelligence. Well, hold that in my head.
SPEAKER_07You can hold it in your head, but I mean, think about applying that in an engineering standpoint. I mean, Charles, I'd be interested in your thoughts as a computer scientist. That that I think that definition may be short and sweet. I think it's very difficult to figure out.
SPEAKER_04That's the problem, is you got those two poles.
SPEAKER_07Yeah.
SPEAKER_02Yeah.
SPEAKER_03All right. So couldn't add much to that. AI researchers were able to train an AI, Alpha Go, to play the board game Go by having it play against itself. That approach does not always work. Researchers have recently observed that the search for more training data probably cannot be met by using the output of the model they are training, nor the output of other AI models. Are you hearing anything about that situation?
SPEAKER_07Absolutely, absolutely. What we're talking about is synthetic data, and it has, it's, it's been, you know, we're what we're looking at in privacy and AI. Privacy is one of the risk areas of AI. And one of the solutions that's been put forward is this idea of let's use synthetic data instead of personal data to train models. But unfortunately, researchers, and not in all areas, I mean, it needs to be fit for purpose, but researchers have found that training AI models on synthetic data, meaning data generated by other AIs, leads to a phenomenon often called model collapse or data poisoning through feedback loops. And so it can result in the loss of diversity and originality. When models are trained on AI-generated content, especially from other models, they tend to repeat patterns, biases, and errors already present in the original models. So they will actually make those issues bigger, and it gradually leads to a homogenation of the data. It also is degradation over generations. Each time a new model is trained on model-generated data, it inherits compounded distortions like making a photocopy of a photocopy. Over time, the performance will degrade, especially in complex, open-ended tasks like reasoning, creativity, or language, fluency, and then false confidence and hallucinations. Synthetic data often lacks the nuance, edge cases, or real-world messiness. And that is such a key thing when we're looking at whether AI will ever be able to reason like a human, that messiness of human-generated data. Training on it can create models that sound fluent but are actually factually hollow.
SPEAKER_03So there's hope for us humans yet.
SPEAKER_07Yeah, but the problem, you know, from a privacy standpoint, it's it, you know, we're always looking for privacy-enhancing technologies. You know, these are the technologies that risk the risk that lower the risk on using the data. But synthetic data at one point was looked at as it was going to be the solution. Now, it does work for some things. And, you know, there's a lot of research ongoing that maybe using synthetic data combined with other data can mitigate it. Again, we're in the early days of this, but it is not going to be the magic bullet that everybody was thinking it might be.
SPEAKER_04Well, going back to the challenge Mark described about what happens with this and your mention about model collapse, is there anything that AI vendors are doing to cope with that challenge? And as another prong to that, since so much of the data that I understand that AI models are trained on is pulled from the internet, and the internet now includes increasing quantities of synthetic data, do the model developers know when they're using internet data that's synthetic as opposed to human-generated?
SPEAKER_07That's a very good question. I mean, the model developers, I would say model developers may or may not, but if you are a company that's put in place a governance process and you have required the people that are training your models to look at the, you know, where where the data came from and to document that data, you would hope that they would be looking and know the provenance of the data, is this synthetic or not before they start training the model. So, you know, I AI is an arms race right now. So I would argue that some of the more responsible vendors know what data they're using and are abundantly aware and are very careful. Other AI developers are not so responsible and are taking every bit of data they can get off. I mean, we are reaching a point in which we might not we might not have enough data to actually train models. So, you know, I think everybody's looking at different options here. I mentioned earlier the hybrid data sets using synthetic and other data. But, you know, these they it's still early, and there's pressure from regulators, civil society, and enterprise clients that want to deploy the software to actually have some understanding and transparency around how the models were created. So, you know, I think we'll slowly see the ones that are commercially viable putting in place better governance structures.
SPEAKER_04Well, before I ask the next question we were planning on asking, you've suddenly made me worry about something. I'm not sure I understand well enough whether AI models just need more data or they need better quality data, which would explain why copyrighted works are so important for them. Is it quantity, quality, or both?
SPEAKER_07It's both. It's both. I mean, good data, good data in is good data out, you know, bad data in, bad data out. And, you know, I think it's been really interesting as we've been looking at the rollout of AI, and there's been growing concern that the West, China, we're all moving rapidly on developing AI systems. But you look at the global south, and their data is not being used to train AI models. And are we going to end up having a divergence where we have certain cultures that are the and certain languages, et cetera, that are their data is not used to train models? Or in Europe, there were certain restrictions on using European data to train models. And were those cultures, et cetera, going to fall behind? Because if you don't actually train the model on you with that data, how is the model going to be able to give answers? I mean, my husband's Hungarian, and the Hungarian language isn't spoken in too many other countries. But if you couldn't then train it on that data, then you're going to have a hard time having a model that can, you know, output relevant, correct outputs.
SPEAKER_03Right. Translate to Hungarian, for instance.
SPEAKER_07Yeah.
SPEAKER_03The Hungarian speakers is do they want their language in these systems or do they not want their language in these systems?
SPEAKER_07Yeah, and there's nuances. It's not as easy as just uploading a translation software and teaching. I mean, our language has so much nuance in it.
SPEAKER_04Is there any reliable way for users or developers of a Gen AI application to know that the models they're using are not tainted, either by bias or by bad data?
SPEAKER_07That's a great question. And again, it's one of the most pressing in AI ethics today. And frankly, right now, most users of Gen AI applications have little to no visibility into whether the models they are using have been trained on tainted data. That would include data that's unauthorized, biased, low quality, or otherwise problematic. So the current state is very opaque. There's a lack of transparency. Major AI providers typically do not disclose the full details about the data sets used to train their foundation models, citing trade secrets or security concerns. Now, the EU AI Act, if it is a high-risk model, you know, as defined under the EU AI Act, that does require more disclosures. But again, that's a certain subset of AI models. And there's also a user interface limitation that Gen AI apps do not currently provide tools or labels for users to assess whether a given response is based on trusted or tainted sources. But we are seeing some developments. If you use the research version of ChatGPT, it tells you where it was looking and it gives you some resources. Proximity AI does the same. So, you know, it's things are moving quickly. You know, model cards, some MetaSlama and hugging face models have model cards and data statements that describe the training data in broad terms, but few go into sufficient detail to that. But I do think that we were talking about regulation. I do think most regulatory proposals do have some form of transparency requirement. It is one that seems to work across at least all developers, including open source and those that are more closed.
SPEAKER_04Because you've worked so long in-house, in government, and now again outside, you're one of the few people who can say they've talked to corporate boards from different perspectives like this, and executives in the C-suite. What's the greater risk for boards and executives? And I realize they look at these things differently. The risk that cyber exploits pose to the development of AI systems, which could damage the model and degrade the predictions, or the risk that AI will enhance the effectiveness of cyber attacks, making it easier for them to create at low-cost, deep fakes to find vulnerabilities and to move faster than a target can to defend its system and deprive them of situation awareness.
SPEAKER_07So both are a problem, but I would say if you're looking from the corporate board's perspective and corporate executives, not just your CISO and CTO, I would say the greater immediate concern is that AI will enhance the effectiveness of cyber attacks. That's because of sophistication and scale. AI enables attackers to automate, personalize, and scale attacks like phishing, deep fakes, vulnerability discovery, and malware generation. I mean, the attacks we're now seeing, it used to be that you could tell when you're being attacked because there was, it was grammatically incorrect. It didn't use the right name, et cetera. Now with AI, these attacks are very sophisticated and they're very hard to make out. And I mean, they're they're also able to clone voices. And so some of the attacks you see coming in actually sound like a person you know, and it's actually AI cloning. There's lower barriers to barriers to entry now. Even low-skilled threat actors can now use tools like Chat GPT to craft convincing phishing emails or fake identities. And then finally, and that's probably why I would say this is the most important concern, is that the business impact is direct. Executives worry about ransomware, IP theft, brand damage, financial loss, and regulatory exposure. I mean, a data breach is a massive headache for a company. But, you know, you did raise cyber exploits impacting AI development. And that's one of the more interesting things about AI. It used to be when you're looking at software and cybersecurity, you're really looking at outputs. But right now, there are input attacks on AI models that are of a growing concern because as we were discussing earlier, bad data in is bad data out. So, you know, you're looking at train a data, training data poisoning, weights exfiltration, and modern inversion attacks absolutely matter. But again, as I said earlier, they're of general concern right now to the RD people, the CISO, the CTO. But the the amend the minute these models that have been subject to these attacks are rolled out commercially. So if you're a developer, you're very worried about this. But if you're a consumer goods corporation that's rent, you know, that is not actually deploying AI or using uh developing AI, you're probably less worried about the second one.
SPEAKER_03There's this treasury report that I'm quoting here. It appears that even live video interactions with a known client may no longer be sufficient for identity verification because of the advances in AI-driven video generation technology. I think this is driven by the incident not that long ago where a Hong Kong bank was scammed out of, I don't know, $25 million, in which the money transfer person at the bank was brought into a Zoom connection with a bunch of different people, all talking, people that person knew and recognized, all of whom were deep fakes. And then was you know persuaded to send a vast amount of money, well, to me a vast amount of money, to uh you know, off into the into the ether. How are corporate boards and executives reacting to this caution?
SPEAKER_07With concern, uh very similar to the earlier one. Um many are urging their compliance and security teams to explore more more robust identity proofing technologies. I mean, it's it's how can we have a second factor? We've been pushing two-factor onto everyone. And at one point, voice print, video print was a gold standard. Now it's not, because it's an arms race. You develop a technology, the bad guys find out a way around, and AI is enabling the bad guys to work a lot quicker.
SPEAKER_03But this is ultimately going to be crippling businesses because it'll limit their ability to move, it'll it'll limit their satisfaction velocity, which is ultimately how they make money.
SPEAKER_07You are you're not wrong. I mean, I think coming up with that second factor that is that is secure is is you know, it's if you can do it, it you've you're really sitting on uh an amazing piece of technology. But as of now, I think we're just in an arms race.
SPEAKER_03So after years of hand-wringing over the future of artificial intelligence by SF writers and futurists, the future seems to have arrived. While it may be unevenly distributed, the role of AI seems to be growing rapidly. Given that our efforts to govern it seem to be largely non-existent, what is likely to go wrong?
SPEAKER_07Well, I mean, I think we've talked a lot about what could go wrong. There are a lot of potentially harmful things that could happen: disinformation, automated cyber attacks, deepening social inequalities, and embedding of bias at scale. Without robust guardrails and smart guardrails, as we were saying, I AI could undermine institutions, markets, and public trust. Problem isn't the technology, it's the vacuum of accountability standards and enforcement. And it's this one is a particularly tricky one because I think it's very hard to figure out what that smart regulation looks like.
SPEAKER_04You mentioned, and it was I was struck by earlier, the lack of transparency in these systems. And when we talk about you know lawyers trying to advise corporate board members on how to make better decisions about AI in the next decade, and focusing perhaps, you know, as you just did, on the cybersecurity risk being the larger one. If you don't know enough about these AI systems, it's very hard either for the boards to make good decisions or for the lawyers to advise on them. And I was I'm reminded of something that the philosopher Hannah Arrent wrote back in a bicentennial essay where she stated when the facts come home to roost, let us at least make them welcome. Let us not try to escape into some utopias, images, theories, or sheer follies. The problem is do you think the facts about AI will come home to roost or will remain elusive because of so much of the secret sauce and other things being proprietary or trade secret protected?
SPEAKER_07I still think that transparency is probably. The most probable area of regulation to be included. And we did we do see it in the EU AI Act. Of course, that's only for high-risk systems, but I think, you know, as I said earlier, I'm counseling my clients to be transparent, to document, to put a governance framework in place. What data is being used? Is that data lawful to use? All of those kind of things. I think that I do think that we will we will reach a point that we understand more the systems that are coming out. Right now, everybody's at a throw it, you know, move fast and break things. But and it's it kind of reminds me of the dot-com boom and bust. You know, everybody was a very a move online, and advertising placements were massively expensive, and things settled out. And so I do think that there'll be a level of understanding. I do know the kids are becoming natives in these technologies. And it may not be us that have deep understanding, but as they grow up with these technologies, there'll be a much greater understanding of the risks and benefits of them.
SPEAKER_04Given how almost agnostic that answer was, do you think that the answer to the question really would be different for in-house counselers compared with outside, since you've worn both hats?
SPEAKER_07See, I would say, from my experience, is in-house counsel, you're embedded in everything. So you are working from inception of a product, all the vanilla counseling sessions, when they're designing, when they're whiteboarding, when you're working through risk. As outside counsel, you are given the higher risk, the areas that are, I don't understand. We think this is going to happen. Do you think this is correct? Or the law is unclear here. So I think outside counsel really, really has to be on the lookout and aware of everything that's going on right now from a regulatory and policy standpoint on AI, because inside they're looking at it from a very micro level, but they're relying on us to keep up with what's going on and to be that more expert figure that can sort of sort through all of the noise and determine where there are really true risk. And if there are true risk, how do you either get around them or change the product?
SPEAKER_04Well, since we're going to have both lay persons and lawyers in-house and outside listening to this podcast, and lawyers at least have the ethical obligation to keep abreast of new technologies, but the pace at which AI is developing and the pace at which articles about it appear both in scientific journals, in system cards from developers, and in the popular media, let alone AI and blogs. How do you select from all of that which signals as opposed to all the rest as noise to focus your attention on to keep abreast of this?
SPEAKER_07I mean, I read a lot of the different uh journals. I I try to stay on top of politico from a policy standpoint, the information, Axios, a lot of the tech publications. And then worldwide, I, you know, I have a network of people that I talk to and meet with and really just and then I sort through which conferences might be relevant to the issues that I'm focused on and what speakers. But you know, this is a fast moving and it it does take, I set up alerts, you know, uh news alerts uh it on the space so I can read if something comes up. And then in particular, if I have a client that's particularly concerned about a certain area, I'm much more focused on that as well because even I can't digest at all.
SPEAKER_04Well, I think I I speak, I hope reasonably comprehensively in saying this for all of us. This was a wonderful discussion. We were we're privileged to have had you given the the background you had and how pressed you are for time in your work. And all I could say is a deep thank you from all of us.
SPEAKER_02Indeed. Thank you.
SPEAKER_07Well, thank you so much for inviting me. And I I think if we had this discussion in a year, I am hoping we have some of the answers to those issues that keep coming up over and over, but I'm not sure we will.
SPEAKER_04Well, we would be interested in having you back in a year if you would be willing to absolutely be fun. We'll do that. We'll have the recording as a record to it. Look back at very good.
SPEAKER_06Wonderful.
SPEAKER_03All right, thank you, Harry.
SPEAKER_06Thank you so much.
SPEAKER_03We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. We welcome questions and comments from listeners. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you were writing about a particular episode, please do mention the specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap, Dialogues on AI.
SPEAKER_01Thank you for listening to the ABA Business Law Section's podcast series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit ambar.org slash bizlaw. That's B I Z L A W.