The Myth of 'Human-in-the-Loop'

"Don't worry — there's a human in the loop." It's become the reassurance phrase of AI governance. But what if the human in the loop isn't actually doing anything? A 2025 EU Joint Research Centre study of 1,400 professionals found that human reviewers showed no tendency to choose fair AI recommendations over biased ones — and consistently deferred to what they perceived as organizational interests. In this episode, Dr. Marilyn Carroll dismantles the myth of human oversight and replaces it with something more honest: the four conditions required for meaningful oversight, the documented reality of automation bias, and a new standard — not "human-in-the-loop" but "human accountable for the loop." The difference is everything.

[ Hashtags ]

#HumanintheLoop #AIOversight #AutomationBias #AIGovernance #ResponsibleAI #TwoDoctorsAndATwist #AIAccountability #ExecutiveLeadership #EthicalAI #FutureOfWork #AILeadership #DrMarilynCarroll #AIPolicy

Stay connected with 2 Doctors & A Twist – Just What the Doctor Ordered!

• Subscribe on Apple Podcasts, Spotify, or wherever you listen.
• Follow us on LinkedIn for clips, insights, and upcoming episodes.

Transcript

SPEAKER_00 0:01

Hi guys, welcome back. I'm Dr. Marilyn Carroll with Two Doctors in a Twist, and I'm doing a limited series on AI governance and leadership. So we're on episode or podcast number five. We're going through there are 12 of these. And I really, really do hope that you're paying attention, that you're taking down the questions. I try to put the questions in as each episode airs. I try to put it in the description of the questions you want to ask yourself. And I really hope you're taking time to ask yourself those questions as well as listen to the podcast. So we'll see how this really goes. We get the data. Um we're in an AI-driven world. And so we get the data and everything, and we're able to see what happened, how many people are listening, where they're listening from, uh, what did they react to? Yes, we get to hear all of that or see all of that. And so our podcast is on various um in various uh 17 countries. I think it's 18 now. We're in 18 countries. And um we're also in now, I think it's up to 400 different cities or uh yeah, cities that we're in. And so we're spreading, somebody's listening, and I hope that's you, and I hope you're learning and taking a lot away from it. Uh, some of the things that we've heard so far, people really wanted this. They wanted this so much so, and questions that I get on LinkedIn as well as Facebook. They wanted to know more about it. I'm very much interested in this, coming from banking and finance and uh all the other areas I've come from, accounting and all of that, and my experience and my years of experience. This was just a natural for me to question it, look into it, you know, that research a background. You look into things and you're wondering like what's going on, what's really happening, and where are we, where are we going with it? So today, today I want to talk to you about the myth of the human in a loop. I remember when I first heard this, I was like, come on now, guys. Yes, I got what you're saying about the human in a loop, but I think this person that brought it that I was listening to had just finished uh their dissertation. Uh, and and which is a good point I want to make here. Just when you just finish your dissertation or your theses or your DBA, whatever it is, you still got a long way to go. Don't think that immediately you know everything that's going on, you're the top expert in it. You're not. It takes time. I finished my dissertation in 2009, yes. And with that, what came more learning? Because I knew as I was going through my dissertation, that human in that loop, I was already uh working as a senior level manager. So everything I was doing, I was able to use that information on in my class in with my teammates. As a matter of fact, my team said, stop using us as your experiment lab. You come to us with something different. I remember um I after uh going as I was going through uh my PhD program, I wanted to study how leadership impacts organizations and what that impacts uh looks like, uh, especially uh positive uh uh uh psychological capital, emotional intelligence, things of that nature. How did these leaders impact that situation? Or any leader, especially transformational leaders, uh different types of transformational leaders. And at that time, it was the first time I was introduced during my studies, adaptive leadership. It was a book, it was a lot out on adaptive leadership at that time, but I really got involved in that and really honed in on it and learned a lot of things in my team. I would take that back to my team. I started a management class and teaching them how to be better managers. I wasn't waiting because you educate one in a tribe, you should be educating everybody in that tribe. Yes, I saw my team as part of my tribe. And as the chief of the tribe, it was my responsibility to ensure that they had the knowledge, the skills, and the uh uh uh wherewithal to be successful. So, back to the myth of the human in the loop. Um, don't worry, that's a human in a loop. You've heard that before, right? You heard it. You may have said that yourself. It becomes the most comforting phrase in AI governance. As a matter of fact, at work, they said, Don't forget the human in the loop when I was doing uh uh writing up my presentation uh on a presentation I was given on entrepreneurship. They said, Don't forget the human in the loop. Well, listen, I don't know if they understood what that meant altogether. The people asked me that, but it becomes the most comforting phrase in AI governance, really, because of that. And in many organizations, it is also the most misleading because what leaders believe is oversight, it is not, it's often nothing more than a checkpoint, and that's the difference, is where risk lives. So I want you to understand something that don't worry, that's a human in the loot. That's become the reassurance phrase for AI governance. But what if the human in the loop isn't actually doing anything? So I'm gonna tell you this a 2025 EU joint research um center study on 1400 professionals, 1400 professionals found that human reviewers showed no tendency to choose fair AI recommendations over biased ones, yeah, and consistently deferred to what they perceive as organizations' interests. So throughout this episode, in the time we have together today, I want to dismantle the myth of human oversight replacing it with something more honest. The four contributions required for meaningful oversight, the documented reality of automation bias, and a new standard, not human and a loop, but human accountability for the loop. Get that. Human accountability for the loop. That difference is everything. And I want you to understand that, okay? So I want you to reframe this. So to be clear about something, a human again, being present in a process does not mean human oversight exists. Presence is not authority, guys. Review is not judgment, and a click is not accountability. Okay, so today I want to challenge a that phrase that sounds responsible, but in many organizations it's really quietly felony organization, that human and the loop. Yeah, human in the loop. That grounds this, let's ground this in research, okay? Uh, as I stated a few moments ago, a large-scale study for the UNPN Commission Joint Research Center looked at 1,400 professionals. I gotta say it again. Here's what they found. Humans did not consistently correct biases, AI bias, did not correct bias AI recommendations. In many cases, they accepted them and especially under pressure. Now, let that sink in. They accepted them, especially under pressure. The human was in the hoop, but the bias still passed through. Another body of research, a meta-analysis of dozens of studies on automation biases, shows something similar. Where people rely on AI, cognitive effort, when people rely on AI, their cognitive effort decreases, confidence in their own judgment declines, and they are more likely to defer to the system. That is not a character flaw, guys. Household Maryland. Let me tell you, we become complacent. And I wrote about this the other week. Yes, I write a lot. You can't become complacent in thinking uh that the system is right and not think. You have to be, I think, and I've said this to some of my colleagues, you actually have to be a little bit smarter than you were before now. Uh so the higher level you go, the more intelligent you need to be. And the more uh train on culture and ethics, all of those things we train for in grad school and going to get our uh doctoral, going through doctoral studies or whatever the case may be, we have to go through that even more so now. So what's going wrong, you may ask. Marilyn, what's going wrong? Most organizations have confused two things: checkpoints, a moment where a human is present, oversight, a moment where a human is empowered to make real decisions. Those are not the same, guys. A checkpoint looks like this: a dashboard is reviewed, a recommendation is seen, a button is checked. Okay. Oversight looks like this: the human understands the context, has time to evaluate, has authority to override, and is accountable for the outcome. And in most organizations, what they call oversight is actually a checkpoint. Now in my class, I do, I have these thick notebooks. Yeah. Every time I teach a class, I have these notebooks because I'm taking documentation, because no class, no student, no quarter is ever the same. Whether it's in the um a structured academic institution or in the own training I do myself. Nothing is ever the same. Yeah. I I learned that. And so documentation notes and things we give students before they even enter my class, whether they are uh uh a person in my course or a person taking a training and development course from me, I give what we call a learning archetype assessment. From that, also I give a behavior assessment before people get it started. Because I want to know who I'm working with. Yeah, I want to know. They don't have to take it, but I suggest that they do take it. Because once I know who I'm working with, a little about them, the more then I can understand how I need to work with them. That this is not something that uh I can rely on AI to completely work with them on without putting some guardrails and information into that system so that it actually accurately gives me information. Now, once I get information at the end, I'm comparing that against things I had before to see if the system that I built is actually working. Okay. There is a, it's not, oh, it works, don't touch it anymore. That's not true because things change every day. People are changing their systems every day. So no more can you build something and just leave it right there. It's how many times do you get your updates, for example, on if you have an Apple or Android that your apps are constantly updated? I think last week I had like five updates, seems like, from various things that we're updating. Where as those updates happen, you have to then check to see if the processes or procedures you put in place still work. So why human in the loose fails? Here, here you go. Here I'm gonna tell you. There are four conditions required for real oversight. And if even one, even one is missing, the loop breaks. Okay, time is one, if the decision must be made quickly, humans default to the system. Number two, information. If the human sees a summary, they cannot truly evaluate the decision. They can't, they just see in a summary. Authority. If overriding the AI is discouraged, it won't happen. Number four, accountability. If the human is not responsible for the outcome, they won't challenge the system. Those are four things. Okay? The four time, information, authority, and accountability. Okay? Remove any one of those, anyone, and the human becomes symbolic. Yeah. Not supervisory. So here's my leadership decision moment for you. Let's walk into a real scenario. Okay, we're gonna go healthcare this time. I have some health care clients. A healthcare system uses AI to prioritize patients. The system flags who is high risk, who needs immediate attention. I talked to you about a health situation of my own situation there. A doctor reviews the list. Patient A is ranked high, patient B is ranked low. But something doesn't feel right, right? Now the doctor has a choice to make: trust the system or override it. But here's the reality: the doctor is under a time pressure. The system's the system has more data, the organization is expects efficiency. This is real stuff here, guys. So what happens? The doctor follows the system. The human was in the loop, right? Because the doctor followed the system. He looked at it and said, okay, let's go. But the system made the decision. The real issue, guys. Here's the real issue. This is what most leaders are missing. They're missing it. Human in the loop has become a comfort phrase, right? It signals safety, compliance, responsibility. But in many cases, in many cases, it provides none of those. Okay? Because the human is present, but not empowered. The system was empowered, not the human. And without empowerment, there is no oversight. Because the doctor wasn't part of making the system. And who knows who was part of making the system? The placement standard. Now here's the shift. We need to move from human and a loop to human accountable for the loop. That's move from human and a loop to the human accountable for the loop. This means, check this out now. Check it out. A named person owns the decision, they have the authority to override, they have the capacity to review, and they carry the consequence. Make sense? All right, good. Again, the human accountable for the loop means the name, a named person owns the decision, they have the authority to override, they have the capacity to review, and they carry the consequence. This aligns directly with the core principles of human govern AI. AI can assist decisions, but cannot hold responsibility. It cannot, guys. So the human must. Not symbolically, but structurally. So let's bring this all together. All right, you ready? Human in the loop is not a guarantee for oversight. It's a design choice. And many organizations, and in many organizations, really, it has been a do it has been designed poorly. So the real question is not, do we have a human in the loop? The real question is, is that human actually governing the decision, the decisions? Because if not, the system is. Now, before the next episode, audit one process for me. One. Here goes. Here's the process. Okay, you ready? Are you ready for this? Is the human reviewing or just approving? Do they have time to evaluate? Do they have full information? Can they override without consequence? And are they accountable for the outcome? Okay. If the answer is no to any of those, you don't have oversight. You have a checkpoint, guys. I'm gonna say them again. Is the human reviewing or just approving? Do they have time to evaluate? Do they have full information? Can they all excuse me? Can they override uh without consequence? And are they accountable for the outcome? If the answer is no to any one of these, you don't have oversight. You have a checkpoint. So on my next episode, we shift again. Yes, we're shifting. But I'm trying to get you to see the bigger picture and bring it all together. And really, you really do need to listen to the end to understand how it all comes together. Because right now, you may be a little confused. You may be thinking, Marilyn said we have to have somebody check this before it goes out the door. That's not what Marilyn is saying. Marilyn is saying that the human must be part of the loop that was designing the system. So a system built for Chick-fil-A may not be the system for McDonald's. Because once you understand governance accountability and oversight, you're left with a deeper question. What does leadership actually do that AI cannot? And that answer may matter more than anything we discussed so far. I'm Dr. Marilyn Carroll. And this has been a special episode series of Two Doctors and a Twist. I want you to take care, and I want you to remember all the great information we covered today. And I'll see you on our next episode of Two Doctors and a Twist. Take care, guys.