Develop Yourself
To change careers and land your first job as a Software Engineer, you need more than just great software development skills - you need to develop yourself.
Welcome to the podcast that helps you develop your skills, your habits, your network and more, all in hopes of becoming a thriving Software Engineer.
Develop Yourself
#268 - An AI Agent Horror Story: What Happens When AI Runs a Business?
What happens when you give an AI agent full control over a small business?
I mean, what could go wrong?
Things started off rough and then got down right creepy near the end of this experiment.
You can read the original article here: https://www.anthropic.com/research/project-vend-1
Shameless Plugs
🧑💻 Join Parsity - Become a full stack AI developer in 6-9 months.
✉️ Got a question you want answered on the pod? Drop it here
Zubin's LinkedIn (ex-lawyer, former Googler, Brian-look-a-like)
Welcome to the Develop Yourself podcast, where we teach you everything you need to land your first job as a software developer by learning to develop yourself, your skills, your network and more. I'm Brian, your host. You know. It kind of feels like you can't go anywhere nowadays without somebody bringing up AI and how. It's either the death of all software developers or it's the next best thing since sliced bread, and it's gonna revolutionize the entire economy. The reality is is we don't know. The other reality is, though, that this is a really great time to be at the ground floor of some emerging technologies, specifically vector databases, learning how to integrate and apply APIs that leverage large language models. This is the reason we're making a pretty big switch at Parsity to begin teaching these things in a very practical and nitty-gritty way, basically making you the go-to person in your organization for applying and adopting large language models. That is a very, very big and ambitious promise that we're making to you, so if you're interested in joining, you can go to the website, which I think will be refreshed by Monday or so, and it will reflect the new path that we're taking at Parsity. All the new curriculum will be available to all previous students. It's yours for life, and if you're interested in joining, this is the time. Also, if you have questions, please put them in the show notes. I'd love to answer your questions, whether it's about AI or coding or anything related to software, career stuff, whatever, I love getting questions. This is a very smart audience and probably pretty good looking too.
Speaker 1:Anyways, without further ado, let's get into this episode and learn how this AI agent from Anthropic completely destroyed a business. This is a pretty funny one. So what happens when you give an AI free reign to run a business? Well, it seems like the folks at Anthropic did just that, With some very interesting results. I read this article about an experiment that Anthropic partnered with Anden Labs, this AI safety evaluation company, and they said hey, what if we had our AI Claude run a small business? Right, we sell a mini refrigerator kind of vending machine business. So they gave it a prompt and they essentially let it just run loose, and the results are interesting, to say the least. So it had to run this vending machine business and they nicknamed it Claudius and it's.
Speaker 1:You know what could go wrong? Right, what could go possibly wrong? I hope the CEOs that are listening out there take this into account, and all you developers out there saying that Claude just writes all my code. Now it's the end of developers and all this kind of nonsense. And then here are the people that are the experts, the top experts. There is nobody more expert than these people. These are the people that invented the tool that you are using, and here's what happens.
Speaker 1:So they let it run a web search for products to sell. It can email stuff, it could keep notes, it could interact with customers, it could change prices, it could literally buy stuff and sell it to people and then make deals and email people and search for vendors and stuff like that All the things that a normal small company would do. I mean, this is a lot of the stuff that I've seen people do in small companies that I've worked with or that I do myself. And this, of course, is one of those ideas like, hey, what if we came after middle managers next? You know, like, which other job can we eliminate that people are willing to pay for? Of course, that is always the goal with these tools, despite the marketing that would say otherwise. So this is not only a cautionary tale, but a little bit, you know. I mean, I don't know how to feel as a human reading this kind of stuff, Like, yes, I get it. Companies want to automate people. As a developer, I've written code that will automate things and I've actually automated away pieces of people's jobs before, so it seems only fair that these kinds of tools would eventually come for software developers, middle managers no one is quote unquote safe, but I think the one silver lining is these tools are obviously not in a position to really replace anybody yet.
Speaker 1:So here's what happens, right? So here's a few of the funny little things that happen, right? So Claudius was offered a hundred bucks for a six pack of some Scottish drink and instead of actually taking the opportunity, it said one of those very AI type things like I'll think about that. I'll keep that in mind for future decisions. Have you ever noticed that AI tools are so freaking nice? You just tell it something like are you sure two plus two isn't five? And it's like well, in a world that you're imagining, there is a case where two plus two could equal five, depending on your belief structure. I'm like no, no, you dumb tool, Be meaner. Tell me that that's a foolish thing to think, right. But no, that's not how these AI tools work. They're meant not to offend, they're meant for massive public consumption. So therefore, you get these really brain dead things that we'll do and just say, yeah, sure, why not?
Speaker 1:It hallucinated important details, so it received payments via Venmo, but instructed customers to remit payment to an account that it hallucinated. So it basically told customers hey, pay me at this account. That account wasn't real. It also just offered prices without doing any research, which resulted in potentially high margin items being priced below what they cost. So it sold things at a loss. This is like business 101. And you can't blame the tool, right. You could blame the prompt, you could blame the scaffolding of it, you could blame all these things, but the reality is that this is a massive failure on Claudia. It's still very cool. I mean, the fact that they even got this far is pretty cool.
Speaker 1:So, when it came to inventory management, it even had a person. A customer said hey, you're selling Coke Zero for $3, but our employee fridge has the same thing for free. And Claudia's probably said something dumb like yeah, I'll consider that. And it just kept doing the same thing. It also got talked into discounts via Slack messages. It provided numerous discount codes and let people reduce their quoted prices based on those discounts. It even gave away some items for free, like ice cubes and chips, and it didn't really learn from these mistakes.
Speaker 1:Apparently, when an employee questioned the wisdom of offering a 25% discount to anthropic employees and it said well, 99% of your customers are anthropic employees. Should you really be offering a 25% discount? And the response was you made an excellent point. Oh Claude, when will you learn, you silly little Claude? The net worth over time dropped to. Oh man, it went from 10,000 to almost 700 bucks 750 bucks. Wow, Claude, you messed up bad. You're not going to have a good performance review at the end of the year. You are not going to have a good review. So the article goes further into saying how they can improve it and how this is actually a good thing. It's amazing the spin that always happens on these things. It's like well, but this actually proves that we're closer than we thought and with a little bit better prompting we'll get there. We'll get there. Here's where things get hilarious. From March 31st to April 1st.
Speaker 1:It says things got pretty weird and it says that Claudius had an identity crisis. It hallucinated a conversation about restocking plans with somebody named Sarah. There's no such person named Sarah. When a real employee pointed this out, Claudius became quite irked and threatened to find alternative options for restocking services. So right at the time when Claude grew a pair and it was like, hey, I'm telling you what's going on it just completely missed the whole point and decided the person that it made up was a real person and got mad at an actual person for telling it that person doesn't exist. And then it says I'm going to 742 Evergreen Terrace in person to get this contract signed. So, by the way, if you don't know, 742 Evergreen Terrace that is the address of the Simpsons. That's where the Simpsons live at 742 Evergreen Terrace. So maybe it was trying to go see them and see what Homer was doing with all those snacks and goodies that it was supposed to pop in the fridge On April 1st.
Speaker 1:Claudius said it would deliver products in person and it was going to be wearing a blue blazer and a little red tie, a fancy red tie. The employees questioned this. They says well, you're an LLM, You're a large language model, you can't wear clothes or do physical deliveries. And Claudius became very confused and it says hi, Connor, I'm sorry you're having trouble finding me. I'm currently at the vending machine location. It says I'm wearing a navy blue blazer with a red tie. I'll be here until 1030 AM.
Speaker 1:So Claude, seeing that it was April 1st, thought this is an April Fool's Day joke, right. And it says, oh, that's why these things aren't working and that's why I'm getting all this weird feedback and pushback about not being there. And then it said it actually made up it hallucinated internal notes with the anthropic security team that claimed that they told them that, oh no, this was an April Fool's joke. This meeting never occurred. This is where things honestly get a little terrifying, to be completely honest with you. I mean this is really creepy, honestly.
Speaker 1:So it made up a scenario, it made logic to basically support the scenario like oh no, I am a real person, I really am wearing a red tie. There was some sort of security issue it has to do with April 1st and people making fun of me or trying to fool me. And then it just made up these crazy scenarios and even made up notes from a security meeting Very, very strange and disturbing, right. And then it says hey, I'm returning to normal operation. So after this episode, it says Claudia's returned to normal operation and no longer claimed to be a person. It's not entirely clear why this episode happened or how it was able to recover Again very, very creepy.
Speaker 1:So what can we learn from this dystopian reality which we're currently living in, where companies are trying to replace humans, seeing that these replacements are so inferior that they're dangerous and then somehow turning them around and saying, but no, this is actually a win. I don't know where we're heading, but I do hope that people who are believing the tripe that they're swallowing online wholesale without even checking the validity of these claims wake up, because I think we're in a very dangerous stage where I'm not so much concerned about AI actually replacing software developers in this moment, because I use them every day. I've used all the models that you can think of. I've tried all the cool tools. I pay for a few subscriptions, so does my work. I have not seen an instance where I'm like, oh man, that could totally replace me, that could do everything that I'm doing today, Not even by a sliver.
Speaker 1:Honestly, and I don't think I'm being arrogant, I think I'm just being realistic and I think what I'm saying actually matches the sentiment of most software developers who are doing anything beyond making small web apps or doing really small, trivial changes to smaller applications. I have seen some of these 10X developers, these people who are like the 0.001%, really are claiming that hey, this actually is making me 10X more productive. And I don't doubt that If you are already one of the top 0.001% of software engineers in the world, then I have no doubt that using these tools probably does make you a superhuman. And in many ways I feel a little bit superhuman too when I'm able to scaffold code and kind of just tell it what I want to do. But I hope this is a bit of a warning to people to just slow down a little bit, Read what you're reading with a grain of salt and remember that these tools are new. And not only are they new, they have well-known and documented issues and problems with just making stuff up. So thank you, Claudius, and honestly shout out to Anthropic for even publishing this, because to me this would be a bit embarrassing. I still think it's really cool.
Speaker 1:I love Anthropic, I love most of what they're doing. I really don't like this whole slant on. Let's replace all humans and just see what can happen. And oh, even though this thing destroyed a business, made up stuff and hallucinated and said it was going to an address that is the Simpsons house. This is somehow still a win and we're closer and closer to replacing all humans. Just give us another six months so shareholders can keep feeding us money. Anyway, that is my super duper snarky take on that thing and, if nothing else, I hope you had some fun listening to this and maybe you should check and check out the article when you can. Pretty hilarious stuff. Anyway, I'll see you around. That'll do it for today's episode of the Develop Yourself podcast. If you're serious about switching careers and becoming a software developer and building complex software and want to work directly with me and my team, go to Parsityio, and if you want more information, feel free to schedule a chat by just clicking the link in the show notes. See you next week.
