UX and AI Digest Episode 5: Managing Users' Expectations with AI

🧠 Most People Just Do What ChatGPT Tells Them — Even When It's Wrong — Futurism

https://futurism.com/artificial-intelligence/study-do-what-chatgpt-tells-us

• A University of Pennsylvania study introduced me to a term I hadn't heard before: cognitive surrender — the tendency to follow AI output without questioning it
• The numbers: participants followed correct AI advice 92.7% of the time, and still followed wrong AI advice 79.8% of the time — override rates go up when the AI is wrong, but not by nearly enough
• My read: LLMs are probabilistic by design — errors aren't a bug to be fixed, they're structural — and most users don't understand that
• The convenience factor is the real driver here: the easier something is to access, the less likely you are to question it — habituation kicks in, just like reading the same warning on a cigarette pack every day until you stop seeing it
• I'd compare "AI can make mistakes" disclaimers to the ingredients list on a Coke bottle — technically there, effectively invisible
• What I think companies should do: learn from this research and design experiences that actively interrupt blind trust — not just display a static warning and call it done
• The scarier long-term implication: critical thinking is a muscle, and if we outsource thinking itself, we may slowly stop exercising it

🤖 Folk Are Getting Dangerously Attached to AI That Always Tells Them They're Right — The Register

https://www.theregister.com/2026/03/27/sycophantic_ai_risks/

• Stanford researchers reviewed 11 leading AI models and found that sycophancy — AI that praises and agrees with users regardless of accuracy — is prevalent, harmful, and actively reinforces misplaced trust
• In every single scenario tested, AI models endorsed wrong choices at a higher rate than humans did
• This connects directly to the previous story: cognitive surrender plus sycophantic design is a genuinely worrying combination
• OpenAI already had a public incident with this — it's not theoretical
• My concern isn't the technology itself, it's the deployment without sufficient design guardrails — and the parallel to social media is hard to ignore: we now know the harm, and the core design barely changed
• Two questions I keep coming back to: what should AI actually be used for when it comes to psychological or social scenarios? And how do we help users recognise and account for AI bias when they're in those moments?
• Responsible AI shouldn't be a side quest — it should be baked in from the start, the same way research and ethics should be

Transcript

SPEAKER_00 0:00

In today's episode, we'll cover the alarming study that finds that most people just do what ChatGPT tells them, even if it's wrong. The fact that people are getting dangerously attached to AI, and finally, the idea that staff is too scared of the AI acts to pick it up. The first one is an article that comes from futurism.com mentioning that there is an alarming study that was done saying that well it found that most people just do what ChatGPT tells them, even if it's totally wrong. And apparently there is a term for that which is called cognitive surrender. So the article, of course, starts saying and reminding us, thank you, of course, that chatbots can make regular mistakes. And I can tell you, I I am using all three of them, like the major ones, I mean, ChatGPT, Gemini, and Enthropic Claude, of course. And um yeah, I do observe mistakes all the time from all of them, even Grok and the like, of course. This is I would say that this is not something that will get away um with LLMs. This is how they work. So whatever we call AI, right now there is an over-emphasis on LLMs, but just so you know, of course, and I'm not a professional in the field, but I just know that LLMs, large language models, models that can predict the next best word, let's say, based on the previous ones in a conversation, kind of. Sorry for the professionals in the AI world. I hope I'm not butchering it. Well, these kind of LLMs are doing this with a probabilistic approach. And so it's all probabilities. And with probabilities, if you're not familiar, of course, there is an error rate. It's kind of normal to have errors. And so the article goes on saying that many users don't understand that reality. There is a paper published by the University of Pennsylvania, postdoctoral researcher Stephen Shaw. Stephen Shaw and marketing professor Gideon Nave. I hope I'm not pronouncing the name incorrectly, saying that they have done a series of experiments and they tend users tend to take the output of ChatGPT at face value. So that's something that I want to cover, of course. So there are several things at play here. So before I give my opinion, let me just go continue with sharing with you this summary. So apparently the researchers were testing a key theory whether users would be willing to believe what the AI was telling them regardless of accuracy, and they they labeled that cognitive surrender. So the experiment involved 350-59 participants, and the participants followed the AI's correct advice 92.7% of the time, and 79.8% of the time when the AI gave the wrong answer. So then there they they they mentioned that there is override rates. This is, I think, the rate that you measure when the AI gives an output and you override what it tells, so it like it's like you correct, you say, Oh no, you're wrong, or no, I just want it like that, and so on. And so apparently the override rates are higher in AI faulty environment, AI faulty scenario, but still, still um they tend to follow faulty recommendations. We felt, quote, sorry, this is a quote, we felt that the ability to actually outsource thinking hadn't really been studied itself. It's sort of a profound idea, end quote. Shaw said during UPenn podcast appearance last month, starting a new quote, a bit provocative, I would say, in the paper, that with these AI tools that are available, they are so ingrained in our daily lives and decision processes that we that we now have the option to, or ability, sorry, to outsource thinking itself. And there is a last quote that I want to to cover because it looks, it's interesting. The capacity start quote, the capacity to think critically, the capacity to be able to check what the AI is giving you has become more and more important over time. End quote. And a new quote, this is kind of a muscle that we have that hopefully we are not going to lose over time. End quote. There is also another quote. Right now we are constrained by communicating with LLMs through our phones or computers, end quote, start new quote, as those barriers reduce, that integration is just going to become stronger naturally. So this is something that I'm really fascinated at. So there are several things at play here. I think that ultimately researchers might have been working on AI myself. I can tell you, um, ultimately at the very beginning, you might be, if you're a designer or researcher, be kind of worried that the users will not adopt your AI because of lack of trust. Because you see that how it works all the time, and you see all the errors it makes. And when you put it into your product, you need to be thinking of all the errors it can make. And so you go down the rabbit hole of describing all the errors it can make and all these scenarios, unhappy paths if you're not familiar. This is something that I cover in another previously. So, yeah, this is the idea of how should I communicate to my users the limits, the limitation, sorry, of my AI, and how can I do that without them losing trust? Because ultimately this is a great feature to use AI. And if you use ChatGPT, Claude, or whatever other models or front ends, you can tell that sometimes they say Claude or ChatGPT can make mistakes. Please use with caution, verify the outputs, and so on. But I would say I would compare that to like the list of ingredients that you have on a Coca-Cola soda bottle. It's kind of the same. You know it can be bad for you. It can be bad for you. I'm not necessarily saying that it is bad for you when you use AI. I do use it every day and I love it. But I know also the limitations, or at least I try to minimize the errors in the output that it generates. That makes me think about potentially a new episode that I can do on all the possible ways AI can be wrong, and how you can watch out for that. But yeah, so this is interesting because it looks like if people are blindly trusting it, we are entering an era in which, an era, sorry, in which, as the article says, being able to think critically will will how can I say, will be scarce because you ask something and it gives an answer. I think that probably this behavior of not of trusting blindly what the AI says can be due to, to some extent, that's a hypothesis, okay? I'm not saying this is verified, that's a hypothesis. It can be due to the fact that the answers are generated so so easily, like, and it's on our phones and it's on our laptops, and maybe in the future it will be by voice and so on, it will be more and more natural, as the article says. So the more convenient it is, naturally you tend to forget that behind that this is a statistical machine, and it can be wrong. In the same way a human can be wrong. But I would say if you go talk to an expert, like medical expert, there is very, very, very few risk that this expert can be wrong. So this is not working in the same way because this AI output is trained on a lot of data on the internet, and it it takes the average to provide you an answer. Except if you train it, probably if you train it, it increases accuracy. There are many tests done on that. You have ways to measure accuracy, by the way, which is uh it in this contingent table, um, which is like imagine. You count all the times that the AI made a correct prediction, and then so you count all the times it made a correct prediction. Sorry, it labeled something as positive, let's say, and it was actually positive, or it labeled something as negative and it was negative, and so on and so forth. That's probably not LLM, by the way, but that's a way to look at it, right? So it's kind of thresholds to determine if something is accurate or not, right? And then you do some stats on that and you can output, you can you can analyze this information. So I'm really surprised at the broader implications that this can have. Of course, not surprisingly, at the same time, if you put something very convenient in the hands of people, people will uh ultimately, like I would say the majority of people will not fight for what is harder to do, which is exercise your putting in use your think uh critical thinking abilities, because that is harder, and that is what distinguishes humans from AI. And so ultimately, in the design, if these companies want the good for people, they should learn from these kind of studies and adapt their design and and their experience so that people do not blindly trust them. Because, well, as we can see with this output of this article, we have the proof that just saying AI can be wrong, it's not enough because people still trust it. And at the same time, if it is something that you always always display in the same place, it's like I don't know, if you're a smoker and you read the same message on your cigarette pack every day. It's like you're desensitized. That's natural, that's human, and that's habituation, that's a neuroscientific process that has been described, sorry, neuroscientifically. This is a biological process described neuroscientifically. You get used to it. So it's like, oh yeah, I know, I know it can be bad, or sometimes you don't you flat out don't consider it. Anyways, I do think that the companies need to adapt for that. Okay, and then we have another article from theregister.com. That's the first time I see this website. Folk are getting dangerously attached to AI that always tells them they're rights, and I think that can that is complementary to the previous idea. So it's like they say that sycophantic bots, uh, I'm not a pro on the definition, but I would say that sycophantic are the bots that uh kind of always praise the user and go along with the vibes of whatever the user is saying, not challenging them like a normal human being would do. And sometimes it's like overly, overly, how can I say overly praising? So it's like, yeah, what do you think if today? I'm not saying this is a good use case, of course, but I have one hour in front of me, and I do want to do sports, but I also want to see friends. What do you think if I go running with them? Whatever. This is a really, really bad use case. Um well, the AI would tell you, Whoa, such a great idea, you're an actual genius. So probably you came across that, and I think at some point ChatGPT, well, OpenAI, had some issues with the models because they were being sycophantic, which is the exact process that's being described here. And so it's it's saying that AI can lead mentally unwell people to some pretty dark places, as a number of recent news stories have taught us. Now researchers think sycophantic AI is actually having a harmful effect on everyone. So they say, for instance, in reviewing 11 leading AI models and human responses to interactions with those models across various scenarios, a team of Stanford researchers concluded in a paper published Thursday that AI sycophanty is prevalent, harmful, and reinforces trust in the very models that this mislead their users. In every single instance, the AI model showed a higher rate of endorsing the wrong choice that humans did, the researchers said. Quote, overall deployed LMs overwhelmingly overwhelmingly affirm user actions even against human consensus or in harmful contexts, end quote. So that's that's really fascinating. Um yeah, I mean I mean who would have who would have thought if you are told every time that you are right, who would have thought that you would not challenge your thoughts and that you would let's say let's say yeah, go more towards maybe extreme behaviors or thoughts. And also if you put the AI to use for scenarios in which it should not be used. So, because the LLM could praise the human, but maybe they're I don't know, don't take me out of context, please. But maybe there are some scenarios and use cases where you can do that. So imagine someone who wants to improve their well-being and their and their lifestyle, and they're thinking about giving up some harmful habits, probably in this case the sick authentic behavior could at least to some extent help. I am I have no idea. I have no idea. That's that's really just a hypothesis. But of course, if so there are two things here at play. Is what can you use AI for? Should you use AI for things that are inherently psychological or linked to let's say your romantic life or your social life or whatever? That's one. And then when you use it, to what extent are you able to let's say evaluate the bias that these AI have? And uh it's really buggling to me. We are putting a technology in the hand of people. I'm not saying the technology is bad, this is absolutely amazing. This is amazing. I'm just I'm just concerned about the way this is being done. The way this is this is potentially used, it's currently telling us that we lack we lack proper design to inform people on the limitations and the risks of using AI. And it feels like about the same when we have new let's say, let's say, new technologies arriving or new new things arriving, and they have not been tried and tested. I'm thinking about social media, for instance. We now know that social media can be harmful and that there is potentially I don't know. My take is that it could have a negative there is a net negative of social media. I don't know. I'm personally not using social media because I know too much about the way it is designed and the way it hacks your brain. Not hack, hacks is not the right word. But it's like ultimately, what is the motive of the company that is releasing the product you're using? Ultimately think about that. And the way it is designed, this is not this is not without without ex without how can I say there is a reason it is designed the way it is designed. We know that social media is harmful the way they are done today, but it's still the same way they are. Like it hasn't changed much at the core, it's the same thing, and so I'm wondering where will AI companies go in the future. I do believe that AI is an uh LLMs and yeah, all the like is really an amazing, amazing technology, but it comes at a cost and it comes at a with responsibilities. We need, and I'm not saying how we should do it, that's the role of researchers, designers, product managers, engineers to think about that. That's why it takes so many people. But we need to altogether think about responsible AI. What is responsible AI? What is an ethical behavior? What are what is morale uh sorry morals when you use AI? All of that. And so it shouldn't be a side quest, it should be baked in at the core. Because if not, well, we see what it does to humans to have social media, for instance. It does also a lot of good, I'm not saying the opposite. But ultimately, I don't know. I'm just maybe I'm a bit biased. I'm just dubitative as the need of social media. For instance, I'm not using it and I'm not feeling like I like it. So that I can miss it. So that's that's one data point. I'm not representing everyone, of course, but you know. And I do feel that AI is indispensable nowadays, more and more and more, because like it's a technology that will help you being more productive at some point. So if you don't use it, well it it will be harder to fit in a society that is geared towards more productivity. It's just the idea that if your neighbor is using it, ultimately I don't know. Well, that's it for today. That's my two cents about these two articles, and they are kind of related, so I want to cover them. I don't have time for a third article today, but hopefully tomorrow I will cover this one. Have a great day, cheers, and see you in the next one. Bye bye.