AI Hot Take: U.S. Attorney ends 30-year Career by Using AI Hallucinated Citations

Sam Altman, CEO of OpenAI, predicts that in the near future all "knowledge" -based careers - what he refers to as "bullshit" jobs - will be replaced by AI.  Recently, an AUSA in the Eastern District of North Carolina, was sanctioned for using AI in the drafting of his brief, where the "hallucinations" of AI created false citations and made up legal principles.  Needless to say, the judge was not happy.  Mr. Shrum discussed the impact of this case from a broader societal context as he introduces a new "sister" podcast to Music Row Dealmakers.

Transcript

SPEAKER_00 0:05

Welcome to Music Row Dealmakers, where we explore our world of making deals from Nashville's famed Music Row in the heart of Music City. We are the deal makers, from composing to closing. Now, here's your hosts, Barry Neal Schrubb and Dennis Disney.

SPEAKER_01 0:29

Hello to all you dealmakers out there. We so appreciate your faithfulness and your patronage. Thank you for joining us for this, the uh 19th episode of Music Row Dealmakers from Composing to Closing. Can you believe it? We've been hosting this podcast for almost so. I want to thank each of you for listening and for supporting us and for joining in. I'm Barry Neil Schrom. I'm the founding partner of Schrom Disney and Associates. And together with my co-host Dennis Disney, we are the dealmakers of Music Row Dealmakers. Dennis, unfortunately, was unexpectedly called away today. He he had to leave early to go visit his mother in Ohio. She took a nasty turn or a nasty fall this week and is being cared for in the hospital. We here at Shram Disney want to extend our best wishes to Dennis, to his mother, and and to all of his brothers and sisters in this emergent time. We hear she is getting better and improving, but still we'd appreciate all of your positive good wishes and prayers as well. We are working toward our next goal here at MRD, as we've talked about, to surpass over 2,500 listeners by June. So please help us do that. Share this link to the podcast with your friends. Share the notice on uh social media. And if you haven't already done so, hit the subscribe button and spread the word. We really appreciate it. And don't forget, we'd like for you to interact with us more. We also welcome your questions and comments about anything we discuss here. So please leave either of us a message at 877-7 Dealmakers. That's 877-7 DealMakers, or send us an email at questions at musicrowdealmakers.com with your insights, and we'll take anything you have under consideration. In our last episode, Dennis and I took a look at the uh Ninth Circuit appeal involving the photographer Jeffrey Sedwick and his lawsuit against Kate Von D. That involved the uh viability of the Ninth Circuit's uh two-part test for substantial similarity. We were pleased to to uh find that after that episode, um Jeffrey Sedwick actually listened in. And Jeff, we appreciate you listening to us, and he emailed us and gave us some further insights into the case, and and we are planning to have a nice conversation with him. But and we were planning to do that for today's episode, but as I said, uh certain things came up with Dennis, and uh life happens sometimes. So I didn't want him to miss out on that conversation. So you're stuck with me flying solo, but uh we'll we'll try to do the best we can. So instead of our usual diatribe between me and Dennis, I wanted to take this time to talk a little bit about society's reliance on artificial intelligence and how we comprehend it and how we use it. As you may or may not know, I've done a lot of research and written several articles on the topic, several places, and I'll I'll I'll give you a list of those later. And I've also presented this to CLE requirements, and I've presented it to various audiences across the country related to AI. My main topic of concern, of course, when dealing with AI is the impact it's going to have on the creative industry, particularly the songwriters and the producers and the people who make the music here in Music City, the ones that uh affect our deals that we talk about on Music Role Dealmakers. And in my mind, music is part of what distinguishes the human race from all other forms of animals. And more specifically uh in this context from artificial intelligence. What made me think of this topic is recently where AI has impacted something different that is dear to me as well, and that is the practice of law. A couple of days ago, on the 10th, there was a hearing involving a 30-year veteran, assistant U.S. attorney, Rudy Renfer. He worked for the Department of Justice in the Eastern District of North Carolina. Renford somehow filed a document with the court in a case that he was appearing for in front of the I figure which court, but but he it was drafted by artificial intelligence. In this case, the U.S. Attorney's Office, and and Mr. Renfer for them, was representing the U.S. Defense Department who is being sued by a pro-se litigant, a retired Air Force Colonel, Darrence Fivehouse, who was challenging or is challenging a governmental policy limiting the availability of GOP1 weight loss medications for participants in the Army's uh TRICARE, which is, I suppose, a pro post-retirement medical plan. The offending pleading was the DOJ's response to Colonel Fivehouse's motion to supplement the administrative record. Mr. Fivehouse, by the way, is also an attorney. Now, when this AI brief, drafted brief was was filed, Fivehouse responded and pointed out in his response some fabricated quotations and misstatements regarding the holding of several cases. And I'll I'll read from the order in a few minutes. Of course, AI conglomerates would say, okay, well, that's okay. Those glitches are they use the anthropomorphic word hallucinations. In that world of AI, apparently, these kinds of misstatements of fact and even downright fictions do not create any obstacle to commercial success, obviously, because consumers, at least at this point in history, are so enamored with the technology that they readily accept it. And accept these frequent hallucinations is just par for the course, and we we go on and do our thing and use it like we all do. However, in the world of legal proceedings, attorneys are bound by a different set of rules and expectations. They are officers of the court. They owe a duty of candor and honesty to the court. The rules that we follow, their fundamental obligations, ethical rules, for every attorney that practices law in the United States. Many of the rules are based on the American Bar Association's model rules of professional conduct. One such rule is Rule 3.3, states that a lawyer, quote, must not knowingly make a false statement of fact or law to a tribunal, end quote. Furthermore, Rule 3.3 requires lawyers to disclose to the tribunal any authority that they know is directly averse to their position and is not disclosed by the opposing party. Then, furthermore, federal rule of civil procedure 11 also requires attorneys to verify the accuracy, accuracy of their court filings. When they put their signature on a document, it is warranting that to the court that they have done that. And that, of course, includes anything you prepare using AI. You may say, what's the harm? Well, violating these ethical rules for an attorney can lead to severe consequences, which include disciplinary actions, disbarment, not only for the lawyer who is doing the offending, who drafts the offending AI drafted brief, not just for her individually, but also for the firm or organization that he or she works for. The judge in this case, Fivehouse versus DOD, his magistrate Judge Robert Numbers. When he saw this reply brief from Five House, he immediately issued a show cause order and scheduled a hearing in the matter. I have the show cause order here in front of me, and I want to read to you from page two of the order where it lists some of the hallucinations, shall we call them, or in the words of the court, ethical violations. They list, I think, five. First of all, uh, there were fabricated quotations from where this case uh resides, and they included citations which were fictionalized. They also cited rules of appellate procedure incorrectly, and then cited fictional cases that supported their position in the opposition brief. Fabricated quoting or these quotings actually asked the court to take judicial notice of certain matters that AI had fictionalized. And, you know, the list just goes on and on and on. And then to make matters worse, he responded to this. Renford didn't clarify that he had used AI and made misleading statements, basically claiming that he had submitted an early draft and had not intended to. So all of these things led the court to ask for a hearing to show cause and ask the for briefing in the matter, which occurred. This order was done on March 2nd. Of course, the turt the court turned this around pretty quickly, and the hearing was actually two days ago on the 10th. And in the order by the judge, um Judge Numbers, he he indicates the sanctions. He said this can include monetary sanctions, including a payment of the fine of a fine to the court, payment of reasonable attorney's fees for any conduct that that generated those attorney hours on the other side's behalf. It might include uh mandatory CLE. More urgently, it will likely and could include referral for disciplinary proceeding according to the court's local rules. But more impactful for the attorney, referral to the North Carolina State Bar for disciplinary procedures. He could even be suspended from practice either before this court or in general. And then finally, the court could hold them in contempt. And at the end of the order, the court further requests that a representative of the U.S. Attorney's Office for the Eastern District of North Carolina appear at the hearing in addition to Renford. So if they determine that Renford violated Rule 11, as we noted earlier, the U.S. Attorney's Office could be held jointly responsible. As you might imagine, this is a big deal. And so in the hearing two days ago, testified and said he had inadvertently included incorrect citations to case law from this circuit, and attributed those errors to inadvertent filing of an unfinalized draft document. Ostensibly, Renford had AI prepare the draft and then passed it off to a lower associate who then filed it. Now, there's nothing wrong with that in general, but that doesn't, I mean, that we do that all the time. But that doesn't alleviate Renford from the responsibility of making sure that the law he's citing is valid, is current, and has not been overturned, much less being fictionalized. Judge Numbers pushed back a little bit on Renford's characterization that this was an uh just an inadvertent error. He said, It sounds to me like you intentionally used AI, intentionally filed it to the court. And so he chastised Renford, and he said that conduct was disappointing, particularly the lack of candor and taking responsibility in accounting for the errors. Finally, he he pointed out that a violation like this by Renford of Rule 3.3, quote, calls into question any other statements, end quote, that he has either made or makes in the future to the court. Did he intentionally do this? But he he sort of destroyed his 30-year reputation in the process. In the midst of that hearing, Renford basically said he's made the, quote, personal decision to separate from the office, end quote. In other words, he he's quitting his position, and that that pretty much ended the career he's had since uh 2009. And while it may be the end of his career as a lawyer, it's not necessarily the end of the story. The judge may very well find cause to impose some of those sanctions I mentioned earlier. And as of right now, no sanctions have been imposed. But stay tuned. If that happens, we'll let you know. Now, in truth, Renfer, Renfer is not the only lawyer who has fallen victim to the siren song that is AI. It's very tempting to use AI. I use AI for some purposes, but a lot of lawyers use it. And a lot of lawyers have been in the very same situation Renfer's in right now. In July of last year, in fact, the uh U.S. District Court for the District of Colorado was some lawyers from from that district were sanctioned for a brief that uh, according to that show cause order, contained nearly 30 defective citations, misquoting cited cases, misstating holdings of cases cited, citing cases for legal principles that aren't even discussed in those cases, misidentifying the court for some of the cases cited, and then finally citing non-existing cases. I mean, it's not that difficult, folks. You can Google most of these cases, right? Just last month, a lawyer in the Tenth Circuit was sanctioned $1,000. Same kind of thing. She used AI and drafted a document that had misquotes, misstatements, miscitations, hallucinations in it, brought up on disciplinary charges in Maryland. And of course, this has been happening since the beginning of AI. In June 2023, Judge Kevin Castle ordered the offending attorneys Peter Luca and Stephen Schwartz, together with their New York law firm of Levito, Lavito and Oberman, to each pay $5,000 as sanctions because they quote unquote abandoned their responsibilities when they submitted an AI written brief in their clients' lawsuit. Then, according to the court, they quote, continued to stand by the fake opinions after, even after judicial orders calling their existence into question. He made those attorneys uh apologize to the judges who they had incorrectly and falsely quoted. And then finally, in the fall of 2025, California State District Court ordered Amir Mustafi to pay $10,000 and referred him to the disciplinary board as well in California when he used a series of AI engines, Claude, ChatGPT, Gemini, Groc, to write an appellate brief and a reply, in which, again, all of the opinions, all of them, were fabricated and not published opinions. Folks, when will we learn? Sam Altman, CEO of OpenAI, is famous for his controversial remarks about how much of today's workforce is built around what he calls repetitive or bureaucratic task. Let's face it, in law, there are some of those. There are routine matters that are often handled with formulaic templates and forms. Altman calls these bullshit jobs. And he says AI will someday advance to the point where these humans who are in these bullshit jobs can be replaced by AI. And among these are what he calls the knowledge jobs. And in one interview, he describes that as like this. He said, I've never seen a technology or technological revolution quite like this. He says, unlike past innovations, AI can perform knowledge work, cost $10,000 a year ago, for as little as a dollar or 10 cents today. He gives an example of what he's called talking about with knowledge work. He gives the example of customer support. I find this one a little bit humorous. And he says that's an that's an area, a job market that's going to disappear when AI gets proficient. He said it's going to be totally, totally gone. My God, I hope he's wrong. And he points out the fact that many companies are using AI customer service even as we speak. He says, you call one of these things, and AI answers, it's like super smart. It's a super smart, capable person, he says. There's no phone tree, no transfers. It can do everything that any customer support agent at that company can do. Now let's face it. Well, first of all, let's ask the question is Is Altman correct that AI is like a super smart or or even capable person? Altman is a billionaire. Altman has never had to pick up the phone and call Amazon and deal with Rufus to fix a problem. You and I have been there. You and I have tried to resolve issues with a delivery using Amazon's AI. It simply does not work. It cannot comprehend the problem. Now, let me be frank. None of this means that that I am anti-AI. I use AI in both my professorial and and and my legal responsibilities. I use it to draft parts of day-to-day correspondence, letters, or even glossaries in books I'm writing. You know, even to you know spell check certain things. I I recognize it for the powerful tool that it is, but let's be honest, it is just that. It's a tool. By that I mean it is not and can never be a person. In that regard, it cannot and will not ever replace humans and human knowledge. To elevate it to the point of humanity is not just a categorical mistake or even a logical fallacy. It is a betrayal and a denial of who we are. We are humans. And that humanity makes us unique. And I've talked about this in various articles. I've talked about the humanity. And absent that quality, absent that set of conditions that exist in the human, AI will never be able to reason, discern good from evil, discern truth from fact like a human can. AI sees things simply in zeros and ones, binary code, on and off. And that's all it's doing is turning pixels on and off according to the algorithms it's programmed with. Now, there's nothing wrong with a lawyer using AI. Let's be frank about that as well. We all do. But it simply cannot replace the intuitiveness and the logical reasoning, which are key components to making uh legal arguments, to uh persuading. AI does not comprehend the difference between a case citation that is good law and one that isn't. More importantly, it cannot tell the difference between a case citation that is real and one that is made up. It cannot read the case. It does not read the case, it doesn't read anything it puts in its learning language model. It simply puts the data in there and is programmed to process it according to certain that are discerned between those zeros and ones. It doesn't have the ability to understand the reasoning of a court and how a court derives an opinion in a case. It doesn't have the ability to understand the difference between a concurring opinion and a dissent or an original opinion. It has limitations, and until we as a society come to a place where we understand those limitations and treat it as the tool it is, then we're going to continue to make the mistakes that these lawyers are making. It's as simple as that. The reason we have an issue with it. Is that we refer to it like it's human or even like it's one of our pets. It does not have the capacity to love. We talk about it in anthropomorphic terms, and it's hard to talk about it without doing that, primarily because the AI conglomerates have set it up that way. Learning language model. It is not learning, it is accumulating data. And that is done through algorithms programmed by humans. But we talk about it like that. Even the word hallucinations I mentioned earlier, which is what the AI industry calls these lies, these incorrect things that come out of AI. The real debate is not whether AI will reshape society. It already has. The real debate is whether our humanity will get lost in the process. One of the things I plan to do in the future is to begin a podcast, sort of a sister podcast to music role dealmakers, maybe a subset or something of that nature, that will discuss the issues of AI, but particularly how AI relates to the creative arts and how it relates to copyright. I've got some ideas about what we'll call that, and I may solicit your help a little bit later. Now, ironically, talking about Altman, one of the areas where he says AI will not have as great an impact is the creative arts. Now, that's cautiously encouraging. I'm not real confident in the things that Altman says, let's just be honest. But in my mind, creativity is one of those things that will keep AI from being treated as humanity. And it is, and creativity is the thing in my mind that distinguishes the human race from any other form of life. So, in this regard, it's important to realize that since it's not a person, AI can't interact with the world in the same sense that a human does. It can't interact in a philosophical sense, or in any sense for that matter, and is thus not capable of, let's call it, existential knowledge, the knowledge of existence. Recently there was a country music song created through AI, using AI, an artist called Breaking Rust. And it hit number one on Billboard's country digital charts in November 2025. It had at its peak two million followers on Instagram. Two million people sings, and I put that word in quotes, about experiences and concepts of which it has no experiential or no existential awareness awareness. It talks about losing friends, understanding truth, and ironically perhaps, quote, selling my soul, end quote. Now, if those lyrics were crooned by a Vince Gill, that would engender empathy. But when I hear, when I first heard this song by Breaking Ross, I it it sounded good enough, it sounded professional enough. But when I hear it talking about selling its soul or talking about losing its friends, it rings hollow because it's being quote unquote performed by an algorithm which takes human creativity and repackages it. When it speaks of living in the past, losing its friends, those are empty empty symbols. Singing that song, it's kind of like having a conversation with a corpse of a deceased family member. The lack of empathy that's in that interaction with the empty vessel that was once embodied as a human, it just feels wrong. And as humans, we understand the corpse is nothing like the vibrant family member that was once filled with life and more specifically humanity. And that's why I don't think customer service will ever fully be engaged through AI. We know, right? You know when you call a customer service and you're dealing with this issue. So rather than beat this horse to death here, I'm going to put a pen in it right there. And I plan to do this, like I said, in the future. Uh, if you'd like to read the articles that I referred to, I I have an article at the University of Southern California's law review website. Uh that's at uh www.uscspotlight.com. And if you just do a Google search for my name and University of Southern California Law School, you'll get the article. It's called That Something Irreducible, and it talks about the DNA of copyright and the human element that's that's in our creativity. That'll kind of be the theme of my new research. And then I also have one coming, and that was uh that article appears in the uh March uh 2025 edition of Spotlight. I also have another one coming out in the University of Southern California Law Journal, an article called iRobot 2026, again talking about the human recipe and dealing with the issue of AI. So be sure to check those out and read those. As for the uh aforementioned podcast, uh the sister podcast uh to music role dealmakers, I'm potentially going to call that Ghost in the Machine, Music Without Humanity. I'll give you more details about that later. However, uh, if you have any ideas about other names or designations uh about the podcast, I'd love to hear from you. Send them via email to questions at musicrolldealmakers.com and I'll take a look, or leave us a message again at 877-7 Dealmakers, or if you have any thoughts or questions about that. Also, in the meantime, uh show me and Dennis a little bit of love on social media, help us spread the word about the podcast, and again, help us hit that goal, help us uh get our subscriber base up over 2,500. We really appreciate it. Read more about our law firm at shramdisney.com, and you can find all of our past episodes of Music Row Dealmakers from Composing to Closing at MusicRowDealmakers.com. Thank you so much for listening. I will talk to you next time.