AI Gets It Wrong Sometimes

We've all had someone confidently give us the wrong directions.

Artificial intelligence can do the same thing.

In this episode of AI AI Oh!, Mac explains why AI sometimes gives answers that sound completely convincing—even when they're wrong.

Using real-life stories about planning a vacation and creating radio trivia questions, you'll learn how to recognize AI mistakes, ask better follow-up questions, and know when it's time to double-check the answer.

Try This Today

Ask your favorite AI assistant a question about something you already know well.

Then ask:

•  What assumptions did you make? 
•  What should I verify? 
•  What part of this answer are you least certain about? 

Building that habit will make you a much better AI user.

Learn more at mc3.media
Contact: mac@mc3.media

Transcript

Copy Transcript Copied

AI Gets It Wrong Sometimes

Mac 0:00

Hi and welcome to AIAIO. Less tech, more life. I'm back. This is the show for people who are curious about artificial intelligence, but would rather not earn a computer science degree before lunch. Last time we talked about treating AI less like a search engine and more like a conversation. That's still one of the biggest keys to getting good results. But today I want to add something just as important. AI gets it wrong sometimes. Now don't panic, that doesn't mean AI isn't useful. I use it every day. But just because it gives you an answer doesn't mean you should immediately assume it's the right one. I learned that the hard

Two Real-Life Examples

Mac 0:41

way. My wife and I were planning a trip and I asked AI to help me with hotels and flights. It put together a real nice organized list. Everything looks good. Then I noticed it had planned the trip for next year, not this year. Now to be fair, I probably could have been clearer when I asked the question, but that's the point. AI made an assumption, built a really nice answer around that assumption, and if I hadn't caught it, I'd have been looking at hotel prices and flight schedules that had absolutely nothing to do with the trip I wanted to take. That taught me something. AI can build a very good answer on one bad assumption. And if the assumption is wrong, the whole answer can be wrong. I've seen the same thing in another part of my life. On my morning radio show, we occasionally do trivia contests. AI is pretty good at helping me come up with ideas. It can suggest categories and write questions and keep me from asking the same old things over and over, but I learned pretty quickly that I can't just copy and paste a trivia question and put it on the air. Every now and then it'll give me something that sounds completely believable. And it isn't true. Not kind of true, just wrong. If I used it without checking, I'd end up giving listeners a trivia question based on something that never actually happened. So now I verify them. Not because AI is bad, because it's smart enough that when it's wrong, it can still sound convincing. And I think that's the biggest lesson for today's episode. Just because AI sounds confident doesn't mean it's correct. We've all known people like that. Maybe you've asked somebody for directions, and they answered immediately. They never admitted they weren't sure. Fifteen minutes later you're wondering why you're driving through the middle of nowhere and up somebody's driveway. AI can do something similar. Sometimes it misunderstands what you meant. Sometimes it fills in information that isn't really there. Sometimes it simply doesn't have current information. None of those mean it's useless. It just means that you have to stay engaged.

Pause. Ask. Check.

Mac 2:41

Here's the habit I've gotten into. Whenever the answer actually matters, I do three things. First, I pause. Before I act on the answer, I ask myself one question. What happens if this is wrong? If I'm asking for dinner ideas, honestly, who cares? If one recipe turns out to be lousy, we're gonna order pizza. But if I'm planning travel or spending money or fixing something around the house or sharing information publicly, now it matters. Second, I ask another question. What assumptions did you make? What year are you using? Where did you get that information? Is there another possibility? Sometimes that's enough for AI to realize that it misunderstood me. Sometimes it even corrects itself. But and this is important, having AI check itself is only the first step. You're still asking AI. If it's something important, I still go verify it somewhere else. I check the airline's website, or I look at the manufacturer's instructions, I read the original document, or I find another reliable source. Think of AI as the first stop, not always the last stop.

When AI Makes Things Up

Mac 3:48

People call it an AI hallucination. Personally, I think that sounds more mysterious than it needs to. Here's the plain English version. Sometimes AI simply makes something up. It isn't trying to fool you, it just produced an answer that wasn't based on reality. The good news is that it doesn't happen nearly enough to make AI useless, or nearly enough as it used to. It happens often enough, though, that you should stay awake while you're using it. Today's myth, if AI sounds confident, it must know what it's talking about. Nope. Confidence is just part of how AI writes. It isn't proof. The answer can sound polished, logical, even authoritative, and still be wrong.

Try This Today

Mac 4:30

So try this today. Ask your AI assistant something about a subject you already know pretty well, and then ask, what assumptions did you make? What should I verify? What part of this answer are you least certain about? And finally, go check one important fact yourself. Not because you're trying to catch AI making mistakes, because you're building a habit. The goal isn't to become suspicious of AI. The goal is to become smarter while you're using it. AI is an amazing helper. It's just not the boss. Use it, learn from it, let it save you time. Just don't hand over your common sense. That's still your job. I'm Mac. Thanks for listening to AI AIO. Less tech, more life. We'll talk again next time.