Episode 19 - The Intersection of Cybersecurity and Artificial Intelligence

This episode of the ABA-funded podcast Exploring AI Matters was recorded at a Continuing Legal Education session.  Our subject is, "The Intersection of Cybersecurity and Artificial Intelligence."

Our two guests, Candace Jones and Jon Garon, are distinguished lawyers who have written extensively on artificial intelligence and cybersecurity.  Some of their writing are included in the materials for this program and will be linked in the podcast web page about this episode.

Transcript

Copy Transcript Copied

SPEAKER_06 0:01

Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap, Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that. In addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.

SPEAKER_00 0:41

This is a special recording session for the ADA funded podcast, Mind the Gap Dialogues on Artificial Intelligence. Please put your questions into the chat and we will take the time to answer as many as possible throughout the session in the last part of our program today. Our subject today is the intersection of cybersecurity and artificial intelligence. Candace Jones recently retired from the Federal Reserve Bank of New York. She was previously a partner in a large law firm in Cleveland and a former chair of the ABA section of business law's cyberspace law committee. She is now a member of the ABA Cybersecurity Legal Cap. John Garin is a professor of law at Nova Southeastern University, Shepherd Broad College of Law, and is the current chair of the ABA section of business law's cyberspace law committee. Both Candace and John have written extensively on artificial intelligence and cybersecurity. Some of their writings are included in the materials for this program and will be linked in the podcast webpage about this week's episode. So special good afternoon to John and Candace. I'm Ama Adams, a national security lawyer.

SPEAKER_05 2:01

And I'm Charles Palmer, a computer scientist. We are your hosts for this episode of Mind the Gap Dialogues on Artificial Intelligence. In addition, we have two more hosts.

SPEAKER_07 2:12

Hello, I'm Roland Troop, a national security lawyer.

SPEAKER_06 2:14

And I'm Mark Donner, a computer scientist.

SPEAKER_00 2:16

So to help set the stage for this quick discussion, I wanted to offer just a little bit of context before we speak with Candace and John. In response to Executive Order 14110 on the safe, secure, and trustworthy use of artificial intelligence issued by the President in October of 2023, the U.S. Department of Treasury issued a report on March 27th of this year entitled Managing Artificial Intelligence: Specific Cybersecurity Risks in the Financial Sector. As reflected by the Treasury Department's report's title, the intersection between cybersecurity and AI now has significant national security implications.

SPEAKER_05 2:59

Cybersecurity as discipline probably dates from Project Mac in 1963, the first multi-user computer system. When do you think did AI first present cybersecurity challenges?

SPEAKER_01 3:15

AI has always presented cybersecurity challenges. If you think about how AI works, and I'm I'm excited to hear Charles, you and Mark talk about the technology. If you think about how AI works, it involves the ingestion of information, the processing of information, output that may or may not be reliable. It's a prediction. You have to understand what it's doing, what it's not doing. And all of those input and outpoint points give bad actors an opportunity to interfere with the workings of artificial intelligence. So cybersecurity should always have been a part of the design of artificial intelligence. And it should be there's an increased focus on security by design. And so I think it's going to get more of a focus, both at the outset of the development and throughout the use in organizations. But it always should have been there.

SPEAKER_05 4:12

Yes, I would certainly agree there. Security is not always the first thing on the feature list. So, John, what are your what are your perspectives on the challenges too and benefits for cybersecurity?

SPEAKER_02 4:23

First of all, thank you very much for inviting me to participate in this program. Very excited to be here. Just reflecting for a moment on what Candace already said, I think it's very helpful to think about the cybersecurity issue both from an integrity of the AI system and then the use of the AI system against cybersecurity protections of other institutions. There have been attacks on the data sources. There have been poisoning attacks to disable AI systems. And there, because a lot of the AI data has been unclear or very opaque as companies are building these models, there is a threat vector that disinformation and misinformation is being fed into the systems. So that these otherwise very robust systems are in fact using misinformation as part of their foundational basis. And then there's the second side, which is that AI magnifies the power of individuals to distribute vastly more content than they otherwise could. And so we're seeing examples of robot farms coming from threat actors in Asia, threat actors and Eastern Europe that are able to take what was once a very expensive process and through the use of bots and chats and the like, magnify their distribution of disinformation much more rapidly and use that as just one of the many vectors and as well to denial of service attacks and other more uh computer specific kinds of interference.

SPEAKER_01 6:04

Charles, I think it's worth taking a step back and for anybody who's on the program, you listened to day one yesterday. Rena started off with what is this, what is security, right? What are we concerned about with cybersecurity? And John mentioned a couple of those. It's confidentiality, it's the integrity of data, and it's accessibility of data. So if you think about the data that's going into these systems, there's a cybersecurity risk that we have to think about organizationally, how are people using the data? So that's not just the developers of the tool, that's the users of the tool as well. The integrity you can see as the output in considering what is going in, both the validity of the information going in and the validity of the information coming out, and how we assess whether or not that's correct and useful for the purpose for which it's intended to be used. All of those things roll up into the broader category of cybersecurity and information management.

SPEAKER_05 7:06

Thank you. So let's continue on this track here. Are the new cybersecurity challenges presented by this cool new thing, generative AI, is it the same or the is it somehow making things, well, different, worse, than regular AI systems? Candace?

SPEAKER_01 7:25

So I think it's it's more of the same, but again, I think it's affecting organizations differently. And so different people are having to respond differently to the to the both the benefits and the risks that come with AI. It was there was certainly lessons and training for the entire workforce about watching for phishing emails and and securing their computer and turning things off and protecting their passwords. And so you have all of those things. But AI is showing up as tools in so many of the systems that people are using every day. It's hard to keep people from going out to some of those AI-assisted tools, search, copilot, chat GPT, and taking your information, trying to be helpful, trying to do their jobs, trying to be efficient, and going out and maybe compromising information because they've used a tool that the organization has not properly trained them not to use. Or they're using a personal, those kinds of tools on their personal devices, which have different licenses and compromise information and use information differently than the way the information would be handled if they were doing the same kind of use of those tools on their devices where there's a negotiated license agreement that keeps more of that information confidential. So there's a lot of education in organizations about using these tools. That's not to say, and I don't want to make this a AI is bad for organizations, there are lots of great uses for machine learning tools. AI is great at what it can do in processing lots of data, in coming back, recognizing anomalies, lots of cybersecurity benefits for some of the tools that are AI assisted in identifying anomalies, bringing those to the attention of the organization of the cybersecurity organization so that they can be addressed and get ahead of the bad actors as opposed to being on the back end. So there are definitely benefits, and those are not generative AI tools. I think it's important for people to understand that AI is a broader category than generative AI. And there that's an important distinction that people have to make, and they have to dig into these tools a little bit to understand what they're doing, to understand the risks, and then how to manage the risks in their organizations.

SPEAKER_02 10:01

If I could add, in addition to the number of different kinds of technologies that are infused with AI, the predictive AI, the robotics, and other kinds of systems that are not generative AI, generative AI itself is also much broader than just chatbots. We use generative AI for image generation and video generation, for voice remodeling and audio files, for computer software, for engineering designs, as well as for chat and for conversational AI. Each one of those use cases, particularly as generative AI has evolved in just the last 24 months so rapidly, the use cases are getting stronger in each of those areas, but that also means that the sophistication of misuse is getting stronger, and the steps that companies need to take to protect themselves from misuse is getting stronger as well. So it really has ratcheted up dramatically. In 2017, we saw the rise of deep fakes and the use of deep fakes for harassment, non-conceptual pornography, and the like. Today we're seeing an increased use of those same videos targeting individuals as young as 13 and 14 with sexual content where false and false videos are being sent around. Well, that same strategy is being used to create pressure on individuals who work within organizations and the risk of basically blackmailing people within institutions. So we're seeing a lot of different ways in which these technologies can be misused. When it comes to software, we're seeing issues of malignant code being introduced into repositories, open source repositories and the like. And then that in turn gets impute imported into the AI tools that are being used. So each of the areas where AI, generative AI exists, have the same kind of concerns that Candace raised for the broader AI landscape as well.

SPEAKER_06 12:14

Beyond uh what Don and Candace have both said, there's a subtle vulnerability that we all suffer from that we're not fully aware of. And that is intuitively, we've been trained over many, many years to equate linguistic fluency in our native language with education and intelligence and competence. With these large language models generating vast quantities of very fluent text uh and very fluent speech quickly and easily, it's now the case that the cues that we have used to trust people when we listen to them are no longer trustworthy. And that's a that's a particular hazard that I don't think we fully understand. It's not necessarily that these things are being used by bad actors, though they would may well be, but that they they get past our sort of intuitive filters to say, I don't trust that person.

SPEAKER_02 13:18

And if I could have a particular issue in education, right? Coming from a university setting, you read a first draft of something from a chat bot, and it is really nicely presented. And you have to read much more carefully to realize that it may be stringing utterly unrelated concepts together and creating absolute gibberish, but it sounds really good. And if you're a novice in a field, you won't have the clues or the information to know that this needs to be edited or challenged in a way that if you are an expert in the field. And this is one of the reasons that the use of Chat GPT or generative AI is particularly a concern for pro se litigants because it sounds correct. And without the legal knowledge, pro se litigants are really put at risk of using in tools that sound good but don't say anything, or worse, misrepresent the law.

SPEAKER_07 14:18

I want to just underscore Mark's point about our intuitive inferences and the defenses we've traditionally relied on and give the example of that many of us have had when we go to a foreign country where we may not speak the language. If we ask directions of different people and we finally find someone who can respond in English, we assume, because they're speaking English, that they're bright, helpful, and giving us accurate information. They may have no such intentions at all. And that's what generative AI also plays on, and I'm just offering that. But I also want to raise a disconnect that I perceive, John and Candace, and maybe I'm misperceiving it, but when I look at the guidance that has been issued by the Florida Bar, the City Bar of New York, California, and most recently by the ABA, while they talk about the need to watch out for client confidentiality and what you enter into a generative AI prompt, there is no serious discussion about the likelihood that adding generative AI products to a law firm will increase its cybersecurity risks, whereas it used to be traditionally understood that if you added new software products to an environment, you were increasing the attack surfaces and vulnerabilities. Am I misperceiving, or is there a disconnect between those guidances and the need that and the idea that other than deep fakes, there is not a lot of discussion, at least in the law firm's areas, about the cybersecurity risks that may be on the trailing edge of the adoption of Gen AI tools by lawyers and law firms?

SPEAKER_01 15:55

My sense of what the courts have done and the guidance that have been issued is they're responding to the to the circumstances that have first presented themselves. And those are about the reliability, those have been about the reliability of the output of the generative AI tools. We saw those cases last year where lawyers were citing non-existent opinions. And so I think the courts were were obviously focused on dealing with those problems. And you know, technology is technology. Somebody is in the background doing the cybersecurity work, right? I mean, I think that's the assumption with a lot of people that these are that any organization knows that when you bring a new technology in, you should be thinking about the the risk of bringing that technology in and how you secure it. And that's that's something that every organization needs to. I think the speed with which some of these tools have emerged as and caught on and gained hype in the commercial space, right? In the market, everybody, AI is great, everybody should be using it, we use it and uh we sell it and it's wonderful, right? So the hype is outrunning as typical, the organizational ability to respond to it. And if you're the lawyer or the cybersecurity, the CISO and that organization, you're you're constantly the naysayer, you know, slow trying to slow people down. And so the risk management needs to catch up. And there's and people are looking at their risk management frameworks, they're they're being challenged to say, how is generative AI different? Right? So let's leverage the risk management frameworks that we have in place. We have those, we have those channels, we have those processes, we have those policies, but does Gen AI fit neatly into what we're already doing, or do we need to do something more? And organizations need to pay attention to holistically how you bring any new tool in and where it could come in intentionally and where it could come in unintentionally into the organization. And for the law clerks who are listening in, I think it's important when you're going in, you know, from an academic setting, you're going into you're employed by somebody, somebody's telling you what technology you can use for your day job, right? What you do at home on your own time, in your own writing, that's up to you. You do it on your own devices and read the terms, figure that out. But when you're at work, you follow the guidance that's given by your organization about what technologies are permitted. Um, you may have questions about why they don't, or bring bring this idea of this could be useful, but there's there should be vetting of technology before it comes into the organization, and there should be guidance about how it's properly used and what it's what is not properly used for, understanding what what you're gonna get out of the technology. So and then I think law clerks are also gonna see, as John said, law clerks are gonna see material pleadings coming in uh that have been AI assisted. They're not necessarily gonna be labeled AI assisted, but you have to recognize and think about if you get that gibberish, might it be that somebody's really trying to leverage this technology to do a bit of their work? And you're gonna have to check, you'll be checking those citations anyway, but you really need to pay attention to those citations. And then you're gonna face like who's liable when this stuff goes south, right? There are really interesting legal questions about who has liability when they take this Gen II tool, drop it into this tool, and guess what? This guy up here disclaimed that they have any responsibility for that thing working. And so somebody's gonna get stuck with some bad result that they're gonna be trying to find out who can compensate them for it, whether it's lost funds, whether it's a process that went poorly and they ended up with poorly made stuff. There are gonna be really interesting legal questions that the that the clerks are gonna encounter. And so I think there's a whole range of issues for for law clerks to look positively at the coming AI influence in practice.

SPEAKER_02 20:14

I I would like to just quickly add a couple things because I think Roland's point is really well made. Right? The ABA took on cyber going back in 2012. It published the 2020 report. And even in that report and the subsequent changes to the model rules, there is very little attention paid to data integrity. You know, confidentiality is discussed quite a bit, but data security and information security of those systems is barely touched on, right? There is an obligation to supervise for attorneys, and that includes the staff. And so theoretically, under the supervision provisions of the model rules, you can reach professional competence in that way to the attorneys who are supervising the people who are running their IT departments, but there's very little actual attention that's being paid. And we know over the last 15 years at least, law firms have been considered soft targets when it comes to real cybersecurity infrastructure and protection. And law firms are not particularly good. Senior partners to overgeneralize tend to think that rules don't apply to them. And so there is a tendency to create additional vectors that of access, and we think about deep fakes and creating a deep fake of the partner in the corner office who says, just turn off the firewall because I'm traveling and this isn't working for me. That's a real threat. But when we look at the 512 report and the state reports that Roland referred to, I think there's also an assumption that these Still written for going on to a third-party vendor site and putting your information into somebody else's site and never introducing the technology in-house in your own firm. In practice, we're seeing more and more law firms, particularly law firms of size, trying to develop these products and tools in-house or at least bring them entirely onto their server systems. And so they're trying to deal with the confidentiality issues by bringing them onto their systems. And with that, as Roland highlighted, you are adding new vendors, you're adding different contracts. The contracts, as Candace pointed out, are often as is with severe limitations on liability. So if something goes wrong for the law firm, they have very little recourse to the vendors. And going back to what happened to Target when its HVAC system resulted in a massive data loss, having a vendor that you don't understand what they're doing and how they vetted your software creates a whole new level of potential hurt if the law firms aren't being thoughtful about how they're engaging in this material.

SPEAKER_01 23:12

Even if there's not a breach, right? Understanding the data flows and how things and how things work and how how these tools can be almost unintentionally incorporated. Oh, you want to use this. We have this great new feature, right? That looks like a great feature. Let's bring that feature in, right? Let's bring that feature in, let's let people use it. We're not, you know, it's just a new feature from our existing vendor. We don't need to go back and look at the terms and see how that works. And how that works is suddenly, you know, in the fine print, you see that, oh, for this feature to work, instead of your data actually has to be exported out to our cloud processor to do that analysis and then put back in. So you've suddenly moved data that you thought was contained somewhere, you've given permission to take it out, put it somewhere else. And oh, by the way, if you didn't check the box ahead of time, the default rule is that data gets to stay there and we get to use it for training, right? And once it's guess what, once it's in that training set, you can't get it out. So it's it's extremely important with all technology to understand the data flows, follow the data, right? To understand where the data is going, what's coming back to test the results of what you're getting back, to figure out where, you know, summaries. People want to use stuff. AI it would be great to do kind of that wrote any kind of stuff that you, you know, you get no value from, you get no intellectual value from, right? So summarizing this, except that I've read some AI summaries and they they summarize, but they don't, they don't like the weighty piece, the really material piece of what happened in the meeting, it's the same coverage in the AI summary as the chit chat at the beginning and the end. Well, guess what? That's not really a helpful summary. So people have to go back, look at the output, understand where it's helpful, do what training they can to fine-tune those tools and make them useful. And that's, I think, again, an important part of bringing any new technology, but particularly this technology into any organization, including bringing it into the courts and how the courts are going to see it, how the courts might experience it as in their roles of for the courts.

SPEAKER_00 25:41

So, Candace, you know, you're you're you're touching on some of the challenges, and uh, John was also touching about some of the things that law firms have to be dealing with and considering. And I think we touched upon it very, very briefly, confidentiality issues. And I kind of want to go back to that because I think that's a really important issue, right? Privilege, confidentiality, that's something that lawyers themselves still get themselves wrapped up around. You know, what is privilege, what is confidential, what are some of the other legal obligations that we need to be considering. John, you know, can you help, you know, break down or describe kind of the difference between communications that are protected by attorney client privilege versus other legal obligations that lawyers have to protect client information, whether that be around confidentiality, privacy, trade secrets, and then how you see that intersecting with how perhaps AI-assisted discovery might be restricted or constrained to comply with or adhere to protective orders.

SPEAKER_02 26:41

Yeah, I'm happy to start that conversation. I'm sure Candace has a lot to add, as did the other folks, because there's a lot there. When we are in law school, we talk about the duty of confidentiality to a client, and there's quite a bit in all of the opinion letters that focus on the importance of client confidentiality, that to use any of the generative AI tools, it's likely best practice to get client uh consent. And the consent, of course, should be meaningful. But that's that confidentiality that you can't share information about your client is separate from the privilege issues of what is discoverable and not discoverable. But beyond those legal frameworks, we also have the regulatory frameworks. So, first, there is a wide range of state and federal laws that reflect that require privacy of information, and law firms may, in certain instances, have that obligation attached to them. In other instances, they may not. So, if we're talking about healthcare privacy in HIPAA, if you are employment counsel and you have health care documents for the employees of your client, HIPAA continues to apply. If you are litigation counsel and you are suing a company and you obtain records that are healthcare records as part of that litigation, that was not part of the treatment process. And so HIPAA would not apply. So it depends on how you have the information as to which privacy rules will apply. But student records under FERPA, minors under 13 under COPA, HIPAA, financial regulations under GLDA, all of these are important for law firms to respect if that confidentiality attaches. In addition, there's also obligations. Say, for example, you're representing a publicly traded company in a transaction. That's security-sensitive information that could move a market and create a securities violation if that information were inadvertently released to the marketplace. If you are representing a company who's developing a patent, you can destroy the patent if that information is inadvertently released. And most of our companies have trade secret data that again destroys the value of the data if that information is released. Those are all outside of the legal duties of confidentiality, attorney client privilege confidentiality, and go to the broader range of confidentiality. So there's a wide range of things that that trigger that duty. I'll just reiterate what Candace has already said, I think twice, which is your enterprise license today are generally pretty good at restricting the collection of confidential information. And so that data is not retained by the cloud service provider who's doing your generative AI work, if you have entered into an enterprise license and correctly configured the license. But if you're using a personal account, all that information goes up. If the system did not tick the box, you know, as one of uh the chat GPT folks said to me, if you don't tick the box, even with the contract, we're not going to respect that confidentiality. So you have to configure the system correctly. And again, to just highlight what Candace said, terms and conditions are constantly evolving, which means that whenever the terms and conditions are updated, there may be a risk that that protection has been diminished or changed, or a new procedure has to be undertaken to reassert that protection. We see this on the consumer side over and over again, that consumers may select more private relationships with their uh devices, but then an update occurs. And unless the consumer comes back and reasserts their privacy, the privacy is lost. The same thing is inevitably going to happen to the thousands and thousands of actual generative AI providers. It's not just OpenAI, Microsoft, and Google. There are hundreds, not thousands, of companies that have different products out. Every contract is different, every obligation to insist on those protections is different. And there's so many kinds of privacy that can be lost if one doesn't follow up correctly. It's really essential.

SPEAKER_01 31:11

So one, as the thousands and thousands and developers, these technologies are can be layered on, right? So you have provider A using the platform of provider B. Those terms layer on too. So it's not always good enough to look at the terms that you've got with provider A without understanding what they have, what what commitments they have from platform B, right? So so there's some density there. And the other thing I would say for again back to the law clerks in the audience, think about this technology as it relates to protective orders, right? There's there's as John was saying, there's all this information. And when people come in with their requests for discovery, right, there's usually a protective order that that's right up front. And those protective orders put obligations on the law firm that are that should be making them go back and think about what tools am I using to process the documents that I'm going to be getting. And have I properly secured the stuff that's coming from the other side? Because it's not just your client's stuff. If you're processing discovery information from another source, the opposing litigant or a third party, you have obligations there as well. And so I think there's needs to be broad thinking about where you're getting confidential information in the context of legal representations and litigation. Roland wants to jump in.

SPEAKER_07 32:42

Yeah, Candace, I want to build on that. There's two things that law firms are tending to ignore right now. And one of them is precisely the thing you just mentioned, the protective orders in discovery. If the protective order doesn't explicitly prohibit the receiving party from feeding the information that they get for analysis into a Gen AI product, that information then may be compromised by the vendor reviewing the inputs and outputs. It may end up in the model, and it you need to block a party from using a Gen AI system for the material you turn over in discovery. Right now, most firms do not put that in their protective orders, and I think that's going to be an important thing for them to check in the future. If you go outside the litigation context, there's even more non-disclosure agreements that have the same issue. If you are disclosing information and it happens to be trade secret protected and you're hoping later to make it patentable invention filing with it, but you've not prohibited the parties that are receiving it under an NDA from feeding it into an AI or generative AI application. Again, it may be going into a model, it may be released inadvertently by the model, because we know that sometimes these things generate out stuff that they thought wasn't going to go to a different user. So again, under the there's a need to review the standard, very standard MDAs and put in prohibitions of feeding the information disclosed under it into an AI product.

SPEAKER_02 34:20

And that raises a really fascinating issue for me that was tipped up by the ABA 512 report, but law firms have been very slow to think about. And that is the lateral cross information, right? As a cut client, we go to a law firm because they have developed expertise in a particular field. Expertise, of course, is skills, but it's also a knowledge base. And so if you litigate the same kind of case over and over again, over time, you accrete a lot of different information about that kind of litigation. Same thing is true as transactional. But the duty of confidentiality also means that you are not allowed to use client confidential information from client A on file B, even if it's beneficial for file for client B, with at least without the express permission of client A. And so if you are, in fact, feeding all the data from your clients into a combined model, you are violating a very clear limitation on duties of confidentiality by cross-contaminating the information. Databases need cross-fertilization to make them work. So there's an inherent disconnect between the rules of professional responsibility and that each client should be siloed and the way data aggregates and we operate. And that's an issue that's become more and more relevant as clients are saying, you know, the biggest clients are saying, I am spending millions of dollars for legal services, and you are using my information to train for other clients on my back. And I think that's also going to become a bigger and bigger issue as we go forward about what do we mean by confidentiality in the future.

SPEAKER_01 36:16

Well, and to build on that, I'm thinking about this. I don't know if this has happened, but you can see you can see a law firm taking, you know, wanting to figure out how to better price their services, right? Clients are always wanting to have a better pricing model than billable hours, right? So you take a law firm, you feed in lots of financial data from lots of clients and similar work and similar engagements, and you come up with some model, right? You feed, you start feeding those kinds of data into a generative AI tool. What can the generative AI tool, or excuse me, AI tool, not generative AI, start feeding that into a machine learning tool, and and what kind of information can the gen the AI discern? That may well be information that discloses confidential information. I can look at this big pile of data and not necessarily find the the data points that tie to client A, but an AI tool might be able to do that and inadvertently, unexpectedly to the to the person who deployed the tool, expose information. So again, going back to basic risk management principles, you need to understand what know what you know, know what you don't know, and think about the the prospects that there could be results coming from that unknown that are not helpful.

SPEAKER_00 37:47

So I want to dig down into a sort of a specific scenario, but as we're kind of talking about a lot of different things, I would you know encourage those who are listening if you have any specific questions to please go ahead and put that in the chat and we'll try to get to those. But Candace, for you in particular, you know, in February of 2024, Frosters used multiple deepfake videos to impersonate this chief financial officer and other senior employees, and actually ended up tricking an employee to transfer $25 million to the Q unauthorized account. The March 2024 Treasury report that I you know referenced at the outset of this discussion on managing AI-specific cybersecurity risks in the financial sector warned, right? You know, even live interactions with a known client may no longer be sufficient for identity verification because of because of advances in AI-driven video generation. So we've been talking a lot about what law firms should be doing, what lawyers, law clerks, all the things that we should be thinking about and the risks issued. Candace, you know, what measures do you think clients and counsel should be taking, should take in response to is really what is sort of a long-standing comfort that we all have on sort of relying on that saying, seeing is believing.

SPEAKER_01 39:03

It was interesting. Yesterday, the Times had a quiz on AI. They put up 10 videos. Were they AI generated or not AI generated? I got four out of 10. So hopefully you all could do better than that. But security procedures have always evolved with technology. And I think what these examples are teaching us is that we might need to take a step back out of the loop, you know, out of band, break that loop, and step back and figure out ways to validate that don't rely on somebody else set up a video call, jump into this video call, right? Whether that's whether that's picking up the phone, you know, they tell you for consumers, right? You get those calls, your account's been compromised. What should you do, right? Before you go spewing out your account information, you hang up, you go to the the your back of your card, you call the fraud number, and you get a person and say, was there really fraud on my card? Right. So they're out of band. You have to break that loop and and get to get to a number that you can phone, whether it's a phone number, whether it's you walk down the hall, if it's somebody in your organization to validate, organizations are going to, and people are gonna have to come up with security procedures that may not rely as heavily on the technology. And for global organizations, that's gonna be a real challenge. But whether it's whether it's cause and you know, they they tell with the grandparent scams that people are are facing, give your grandkid, you know, agree with your grandkid on a safe word. And and so that's that's clearly a non-technology solution, but it can work, right? So it used to be kind of complicated. Call trees were part of a security procedure before, you know, before transactions were approved, right? A organization A, one person calls organization B, organization B calls a two-person, right? So you have three people in the loop, and there's a call chain that's an agreed security procedure. People are gonna have to look at their security procedures and figure out where there are vulnerabilities and and and take those vulnerabilities out or break that break that loop and get outside with some other validation.

SPEAKER_07 41:26

Can I give an example that Fortune magazine reported in July of precisely what you're discussing to show that this isn't a sort of trivial suggestion to get outside the technology loop? A Ferrari executive was brought into a video call and asked to authorize a transfer of funds, supposedly by the CEO of the company, Benedetto Vigna. And the Ferrari executive said, Sorry, Benedetto, but I need to identify you. What was the title of the book you just recommended to me a few days ago? And the fraudster in the video said, We're checking on that. Immediately he realized there was a problem. Now, what was in fact the title of the book in Italian? I'm not going to translate it into English. Decalogue of Complexity, acting, learning, and adapting in the incessant becoming of the world. So that was a terrific impromptu test. And you need to realize that sometimes a pre-planned test isn't as good as one that nobody else would know of that relied on a unique interchange you had recently with that person. And everybody's capable of generating it if they don't fall for a smishing attack, meaning words of urgency that are asking you to do something without taking care to be, especially when money transfers or other kinds of interests are involved.

SPEAKER_01 42:42

And people should embrace it. They can be annoying, but people should embrace the phishing tests and the things that help that help us build our recognition skills for those traps that we can fall into. Urgency is urgency is one of them. Urgency is a big one. Financial, family. It's it's sad that we can't, you know, just you know, sad that there's a there might be an emergency, sadder yet that people play on on those emergency situations to get people to give them money, right? Or or whatever. So pause, right? Part of part of acting efficiently is taking the time to figure out the right next step. So if, again, if you're a law clerk and you get some sort of message from your judge that looks like it's uh give me this and and and and it looks off, right? Go go ask you know your judge or his assistant, did is this for real? And and how do I know, right? So I don't know that you're gonna counter those. I hope you don't, but those are the kinds of things we have to. If something comes with a link, don't click on it. If something comes with an attachment from somebody that you be very cautious and and study those uh rules for for.

SPEAKER_02 44:00

fishing social engineering people are people are a huge vector for for cybersecurity breaches and that's because people feel rushed to respond or rushed to do something or intimidated or frightened by something bad has happened and so think pause think critically about those things that you're going to be coming coming your way if you look back to as as early back as the Enron scenario and Sarbanes Oxu that came out of it what we learned from all of those events was that we need to routinize security procedures and we need to as much as possible eliminate ways for particularly organizations to end around the steps and procedures that create the double checks. And so right having multiple signatures on things having things be vetted quite frankly not using email the way we use it and moving into secure portals there's so many things that corporate America was forced to do. And because it's a pain quite frankly we have spent most of the last 15 years trying to get around the obligations that the government put in place under the SOX regulations. And so rather than embracing the fact that these are essential to protect our companies most employees spend a lot of time figuring out where the shortcuts are and how to make their lives a little bit easier. All of the steps that Candace talked about checking the phone number on the back always calling back your already identified secure lines and the like those are really simple steps but they're also requirements. And so that's an area where more training in law school with our judicial clerks and the law firms because again law firms are particularly vulnerable to these attacks those are things that we really need to reinvest in is the kind of training it takes to make sure that the fundamentals are still good because the AI is simply magnifying a threat that's already there in this area it's not a new threat but we know the threat is getting faster and faster. The number of people who are being impacted are being magnified each week because of the technology leverages the communications I've always said that people are the ultimate critical infrastructure.

SPEAKER_05 46:33

I mean that's really what you're saying. It comes down to us. So let's let's pick on the law clerks here in the audience. Throw your comments into the QA there. What should you be most mindful of in security cybersecurity and AI during your next year of service on the court? What what sort of things are on your mind? What do you think you should be on the lookout for?

SPEAKER_01 46:55

I uh actually want to pivot a little bit for the law clerks to again what I think is the the intellectual challenge that that I would look forward to as a law clerk which is these cases about AI and how AI has messed things up are going to be coming. And putting that together with the law everybody you know 20 years ago 30 whatever 25 years ago everybody thought we needed a whole new law of the internet and you know after some some period of time to shake that out really we didn't need a whole new law of the internet we needed to figure out how existing laws apply and and that's we're there now with AI and those cases that are going to come where somebody lost $25 million or $5000 in a small firm or an individual those cases are going to come and they're going to be really sympathetic parties and you're going to want them to win right you're going to want them to be made whole but the law might not support that right because they didn't follow the security procedure as John said those are requirements right you have requirement to have a reasonable security procedure and to follow it. So there are going to be tough cases because the party you're really rooting for is not going to have the law to back them up for their recovery. And and and rarely will the person in front of the judge be the bad actor right it's not going to be the bad actor it's going to be the victims in the chain of who's going to bear that loss. And so I think for the law clerks follow the technology rules that you have be skeptical of the technology use it right understand what it is and how it can be useful. Don't use it when it's not useful but when those cases come I I think that's where you'd really get you really get some charge out of you know we have this new technology and how does the law apply so enjoy that part of it.

SPEAKER_02 48:58

If I could reflect on the the the the invitation in a slightly different way present company accepted because we have some experts on this panel who have been doing this for many many decades. Judges are generalists as I was reminded by a judge I was speaking to the other night and judges generally speaking did not get on the bench because of their technological savvy. There is a generational break to a certain extent and again many of us on this call are in the wrong generation but there's a generational break on how people respond to these technologies. And the reality that we're seeing in law schools is that our law students are exposed to these technologies in college they are quicker to be willing to explore the technologies and as a result when you move into your clerkships it is very likely that you will be called upon by very senior people to become the subject matter expert in in the chamber. Embrace that right it is it is an opportunity for our junior lawyers to really get a jump and add value in their law firms and in their clerkships by learning the technology, embracing these issues it's not hard to learn what we talk about. It's a great area of practice but it's also just the the the fundamentals are pretty straightforward. But if you've never looked at these issues after 50 years of practice it feels quite intimidating. And so it creates a real opportunity for our junior lawyers to become the leaders in the field and gain a lot of respect by their peers in their chambers and their firms and really make a difference for their clients and for their work environments.

SPEAKER_07 50:52

So let me share a comment with the clerks that comes from the Guidance on Gender and AI, not from the American Bar Association, but published last December December 2023 by the judiciaries of England and Wales. And they pointed out on a very granular cybersecurity level two things that I would encourage clerks to consider because as digital natives they're probably you know already installing Gen AI apps on their phones. And the first advice was you should disable the chat history in AI chatbots if the option is available. The option is currently available in ChatGPT and Googlebard but not yet in Bing chat. And the reason is because otherwise they keep those things and feed them into the training models. Second be aware that some AI platforms particularly if used as an app on a smartphone may request or have as a default various permissions which give them access to information on your device. In those circumstances you should refuse all such permissions and make sure you review the default settings to find out if in fact they are giving themselves access to your phone's information because as responsible clerks if you're trying to keep abreast of the technology you don't want to in fact be an inadvertent source of a breach.

SPEAKER_00 52:13

So I I want to put one final question to Candace and John and then obviously you want to open it up to anyone in the audience but you know we've talked a lot about a lot of different issues a lot of different concerns challenges and risks you know how would you or how should cybersecurity concerns affect the development and deployment of AI going forward? That's a really big question but would love for your succinct thoughts on that going forward.

SPEAKER_01 52:40

So again I think I think organizations and and people right individuals in in our own lives need to be open to the benefits of new technology we we need to be skeptical of the hype right we've seen a lot of hype and a lot of law firms in particular being pushed to demonstrate that they're being more efficient by using new technology there's a big difference between the AI tools that have been around for many years and the the machine learning tools and and some of the other tools that have been deployed in in understanding large bodies of information right large bodies of documents large bodies of images right there's a difference between that and generative AI generative AI has is having a moment it's getting a lot of attention I think it's really yet to be determined how much more efficient or how helpful generative AI is going to be to law firms right so you have to you have to look at the benefits you have to understand the risks look at the terms right look your lawyers the terms are always very illuminating right what what the sales engineers do not tell the technical people you can sort of discern that there are issues when you read the terms I don't know how many times I've read the terms gone back to the technical people and said but are you sure it works that way because why do the terms say this? And so there's there's always a process of bringing in new technology and in in that respect generative AI is indifferent. There are going to be opportunities we should take them we should be open to them we should we should follow a a standard risk management process to bring new tools into the into the organization or into your homes.

unknown 54:34

John?

SPEAKER_02 54:35

Yep I mean first I agree with everything that you said I do think however that generative AI in particular is improving at a rate that we have rarely seen in technology. It may be because quite frankly it shouldn't have been released to the public as early as it was and so the first round of improvements we saw in live version when it shouldn't have been there. But anyone who recalls Windows 1.0 knows that we've been there before with products that never should have been released to the public and it took 2.0 and took 3.0 and it wasn't till 3.1 that users went wow this is a paradigm shift right with 4.0 Omni with perplexity AI with the integration into Copilot and the discussions that we're hearing about ChatGPT 5.0 we are getting to a fundamentally different set of tools but even as those tools get more and more effective at what they do they're limited in what even their vendors think they're able to do right none of them are cognitive machines they're all predictors of content. And so the critical thinking the strategic process that lawyers use to shape their information that's not changing and there's nothing that suggests that there's going to be an AI product on the landscape anytime soon that actually replaces the critical thinking that lawyers need to provide to their clients or the strategic thinking. They do provide a great deal more information about how to be strategic. And so lawyers need to balance those two things. We need to apply our critical thinking approaches and include the information we learn from them to harness the tools while taking much greater care to understand that privacy and security are always at risk with new tools.

SPEAKER_00 56:34

Thank you that is all the time we have for discussion I would love to thank our guests Candace and John for joining us this afternoon We thank the business law faction of the American Bar Association for their generous sponsorship of the production of this podcast. Send email to comments at mindthegapdialogues dot com. We read all comments and questions and we try to respond in the letters section of a future episode.

SPEAKER_04 57:17

If you are writing about a particular episode please do mention the specific episode number please do also include pronunciation tips to help us properly say your name when we reply in a subsequent episode see you next time on Mind the Gap Dialogues on AI Thank you for listening to the ABA Business Law Section's podcast series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section visit ambar.org slash bizlaw that's B-I-D L A W