Building AI Agents that actually work

AI agents are everywhere—but building one that actually works in the real world is a completely different challenge.

In this episode, we break down the essential principles behind creating reliable, intelligent, and production-ready AI agents. From choosing the right models and tools to designing effective workflows, handling failures, and improving performance, you'll learn what separates impressive demos from AI agents that deliver real value.

Whether you're a developer, founder, or AI enthusiast, this episode provides practical insights to help you build AI agents that are dependable, scalable, and genuinely useful.

🎧 Tune in to discover how to move beyond prototypes and start building AI agents that actually work.

Transcript

SPEAKER_01 0:00

I think AI is confusing. There, I said it. I think there's a lot of terms, skills, MCPs, agent harnesses that are difficult concepts to understand. So I had my friend Remy come on the podcast and explain it in the most simple terms possible. In this free course on how to master AI agents, he breaks down exactly what each piece is, how they connect together, and the simplest ways beginners can start using them today. Enjoy the episode. I beg them to come on. Remy Gaskills on the pod. You've structured your company where you basically have these folders and.

SPEAKER_00 1:33

All right, let's do it. Sweet. So one of the reasons why I really wanted to make this episode is because I feel like the AI landscape is moving into like stage two from chat to agents. And most people are getting left behind right now, just using the chat models. And uh the founders and employees that are utilizing agents are like no word of a lie, 10 to 20 times more productive in their day. And when you stack that up over days, weeks, years, you're gonna just be miles ahead of the competition. So I really want to make this episode today to help bring everyone up to where the AI landscape is at the moment and to start using agents to manage every department of your business. So the key thing to understand here is chat models versus agents, because the word agent is thrown around lots online. I'm sure you've seen it, Greg, like AI agents this, agents this, use this agent for this, and it's kind of lost a lot of meaning. So I wanted to give start by giving a really clear definition of what an agent actually is. So the way I think of it is a chat model is question to answer, but then an agent is goal to result. So moving from just like uh you asking AI replies, then you do the work, to you giving the agent a task, it planning out the task and then executing and then delivering you a result. Does that make sense?

SPEAKER_01 2:58

Crystal clear. I mean, the way I think about it is chat is kind of like ping-pong, back and forth, back and forth. Yep. And agent is uh, you know, you're giving it a goal. I mean, the best way is yeah, you're giving it a goal and you're hoping that over time it gets better and closer to that goal. Exactly.

SPEAKER_00 3:16

Yeah, that's exactly it. And I just think that's a nice way to lay it out in your head is chat is question to answer, agent is goal to result. So when you chat to an agent, you might give it a task like build me a website for XYZ, and then it goes away, it does its work, and outputs this wonderful website to you. But it's really important to understand what's actually happening in this step here. So inside this agent step, we have what's called the agent loop. So you give it your prompt or task, and it goes through these three steps here, which is observe, think, and act. So let's just say, for example, um, we're actually gonna do this demo after this, but if we gave the agent a simple task like build me a minimalist portfolio site for Greg Eisenberg, it's gonna start by like you've loaded in that prompt, it's gonna check if there's any files in the workspace that it can work with, like maybe you've got some information on Greg Eisenberg. Um, and then it's gonna think about what to do next. It's gonna act, and then it just keeps going through this loop. So for that actual example of building the portfolio site for Greg Eisenberg, let's just say it was a blank agent, we hadn't given it any context. The first thing is it's received this prompt to build the website. And the first thing it's gonna be thinking about is okay, well, I need to build this website about Greg. Who the hell is Greg Eisenberg? So it's gonna then decide to do some research into Greg Eisenberg. It's gonna research everything about Greg and then feed it back into this observe step. So then it's gonna think to itself, okay, so I've got this prompt to build a website. I've now got my research here, so I know exactly who Greg Eisenberg is. And then it's gonna start thinking, what is the next step? And the next step is probably to write up a plan to build the website. So it might write up that plan, feed that back in. Now it's got the research, the prompt, the plan, and it will think, all right, what next? I should probably write the code. It just keeps going through this loop as many times as it needs until it can conclude that the task is complete. And how it concludes that the task is complete is based on the parameters that you set in your prompt. So, you know, if you're giving it a research task, you might say compile 10 sources and then create a report as a PowerPoint. And then once it's compiled 10 sources and built the report as a PowerPoint, it can conclude that the task is complete and then give you the output as the user. The agent itself is made up of these four components. So it's the LLM, which is the brain behind it. So think like you know, Claude Opus 4.6 or GPT 5.4 or Gemini 3. It's the model. Uh it's got the loop, which means it just keeps going until the task is done and doesn't stop after one response. So you're going from ping pong to like it continuing to go rather than you having to sit there babysitting it. Uh it connects in all your tools and then it connects in all the context. And a platform that facilitates this process and basically facilitates this loop to happen is known as an agent harness. And all of the popular AI agent platforms on the market that you'd be familiar with are just agent harnesses. They're just applications where this loop is facilitated. And I want to actually run this little prompt I prepared earlier. I want to open up codecs, claude code, and anti-gravity. And I'm going to show you this loop actually happening in action. So I've uh nicely prepared before the episode these three demo folders which we're going to run in. So I'm going to open up demo one to work in Claude Code. And the way these folders work is if you've used, if you're familiar with like any of the chat models like Claude and ChatGPT, there's a projects feature, which is where if I open it up actually. Try not to get dizzy with me switching tabs so much. But you know, if we create a project here, it contains all your chats in one place. It allows you to upload all your sources here, which is your context. And then you can even add custom instructions, which tells it how to behave within this project. And that's also known as a system prompt, which we're going to dive into how to do this with agents as well later. But it's a similar concept that you'd be familiar with if you've used projects before. But instead of the project being here on the cloud, we're actually working within projects that are local in our computer. So I've just selected this demo one for now. Then we're going to run build a minimalist portfolio site for Greg Eisenberg. And then this little bit here just tells it to actually spin it up, like to publish it on the web in a preview mode so we can see what it's done. So I'm going to run that. So this is this is um Claude Code. Yes. Yeah, right now we're in Claude Code. And this is just accessing it through the desktop app for Claude. Um so I'm just gonna run that. And then I'm also gonna give the same prompt to Codex here. So this is the Codex app. And you can see the same concept. It says, let's build, we can choose a folder on our computer to work in, like demo two. And then we're gonna give that a prompt as well, and we're gonna tell it to host it on a different one. And then also in anti-gravity. So you can see the same concept. We're going in, selecting a folder, and then we will give it the prompt as well.

SPEAKER_01 8:52

How should people think about security and these different products?

SPEAKER_00 8:57

I like to think of security as in just like scoping what they have access to. So by default, anti-gravity, cloud code, and codex, they're very, very secure because they're built by these massive companies that have a lot on the line to protect. And I just, you know, if you're if you're building out these agents to manage different elements of your business, like the other week I built one that does manages meta ads. And obviously, that's quite a risky thing to give an agent control over managing ad budgets. So it just comes down to like what you feel comfortable giving the agent, and also you can control what privileges or you can control what um tool permissions has access to, so that if it was compromised for whatever reason, the worst case like isn't that bad. And that means you know, just giving it like read-only access to certain important platforms and stuff like that. Does that make sense?

SPEAKER_01 9:50

Yeah, totally. I mean, comparing it to like OpenClaw, which is like way.

SPEAKER_00 9:56

Which I want to touch on at the end as well, because that's the same thing, just another harness, but it's just like the Wild West.

SPEAKER_01 10:02

Cool.

SPEAKER_00 10:03

Uh and one thing, like a nice little analogy to think about these harnesses is what we're gonna learn today is we're gonna learn to drive. So we're gonna learn about how to you know steer the car, like how the pedals, the brakes work, the accelerator works, the handbrake. But then once you know how to drive, you can kind of jump in any car, whether it's like an old Toyota, a Range Rover, and you inherently sort of know what to do. And that just comes down to understanding all these key concepts that we're gonna go through today. And you can think of the agent harnesses like different cars. And some of them will have better features like seat warmers and cruise control, but it's all once you know how to drive, you can pretty much jump in any of them and use them. So we've just got our thing over here building tar, uh building the website for Greg. And it's going through this agent loop right now. So you can see here it's actually decided that it's gonna launch um an agent to go and research Greg Eisenberg. And I've connected it up to Perplexity, so it's now using Perplexity to research Greg. So it's going through its first step of the loop. And I imagine that uh Codex has also done something similar here. You can see it's still working, but it's gone. And um started to build this out through the loop. I think Clawed Code does the best job of actually displaying that loop and allowing you to see what it's thought about compared to anti-gravity and codex, but it's all just going through the same sort of loop process that I described earlier.

SPEAKER_01 11:32

And I think when you say you hooked it up to perplexity, didn't it's not like you asked it to hook it up, right? It just sort of did it.

SPEAKER_00 11:42

Yeah, because I've um I've given clawed code perplexity as a tool via MCP, which we're gonna get into um very, very shortly, all about MCPs, which is just connecting tools up. So we can see that in anti-gravity, it's gone. You can see this thinking process. It's gone, um I'm now examining the current directory to figure out if there's an existing project or if I build one from scratch. It's then going, um, I'm now gonna start to build this thing. And then it's built the website and it's given us a little local host preview here. So it's created this nice little portfolio site for for you, Greg.

SPEAKER_01 12:22

What's interesting is like it's super minimalist, and I mean it it did its job, right? Like it this is I would totally launch something like this. It actually looks really nice.

SPEAKER_00 12:37

Did it did it scrape your email address correct?

SPEAKER_01 12:39

That's that's not my email address, and I don't live in control canada anymore. So, but yeah, so there's a few copy things, but other than that, uh Yeah, it's done a pretty good job. It did.

SPEAKER_00 12:50

And if we go, so that was anti-gravity. Um, if we go into codex as well, you can see here it's finished doing its website, which is somewhat similar. I think I prefer Gemini's. Yeah, I agree. And if we check out Claude as well, um it's still going. But you can see this loop, right? It's gone. Okay, first off, who is Greg Eisenberg? It's gone and researched Greg, then fed it back into that observe step, and it's gone, all right, what next? Now I need to create the HTML file. So it's written the code. And then now it's gone, okay. So he wanted it spun up on this local server. So now I'm gonna spin it up on the server. And then the last iteration of the loop is to check that it's actually done and can conclude the task is complete. It's opening it up and screenshotting the website and then reviewing the screenshots to check that the website is complete. And you can see here it's done another pretty good job. This one's very similar to the Gemini one, hey. It's true. Um, but yeah, that's just like demoing how that loop is actually working in real time.

SPEAKER_01 13:57

Yeah. I mean, what comes to mind just by watching this is like how many people on the planet would benefit from a very clean website? And like, yeah, how how do you set up these agents so that like, you know, maybe it's like a cold email loop, right? Like you're sending cold emails, hey, I built you this website, so-and-so business, do you want it? It's gonna cost $250.

SPEAKER_00 14:20

Yeah, yeah, that's actually a great idea. Um, pre-making websites for companies. And it's like an off-the-shelf thing. It's like, hey, I made you this website if you want it, like if you want to own it, it's $250. You can just do a mass cold email thing. Uh cool. So I think that's like pretty much illustrated that agent loop example. So I'm just gonna go um back to our trusty board over here. But you can understand that it's just like all of these apps are just different flavors of the same thing. And then what we're gonna be working up to today is my workspace looks something like this is I have you know a big like a folder for each company or client that I'm working in. And then I'll have folders underneath with all my heads of departments. And then within those heads of departments, I'll have skills and MCPs, which we'll get into, and context. And then I've got like an overarching one at the top to just to sort of manage them all. But we're gonna be focusing today on building out this executive assistant to take care of just your manual day-to-day tasks and free up at least one to two hours extra per day. Um, cool. So to build this out, like we did uh with our demos, it's running off your local files. So we're gonna create a folder here called executive assistant. And also through building out this assistant, it's gonna allow us to clearly explain each of the concepts of building an agent in real time. And the way I like to think about building agents is onboarding them like a real employee. So if you took on a real executive assistant, you couldn't expect just for them to come into the office and you'd give them a task with out explaining your business first, your clients, what you do, the tools, um, because they just would not be a very good executive assistant. So that's the first step that we need to go through when we're building out this agent. So uh I'm actually gonna work uh in cowork at the beginning. So cowork is just another agent harness to do the pretty much the same thing as all the others, just that loop connecting in your tools and the context. So you can see here, um, this was my little uh previous session where I was building some diagrams. But we can go um and you can follow along in Claude Code or Codex or Anti-Gravity or whatever agent harness that you want to work in. But I just think that cowork has really nice simple UI for people to just understand really well what's actually going on. So we're gonna open up this executive assistant folder, and you can see here that if we ask it, write me a cold email and send that off.

SPEAKER_01 17:05

So people are gonna ask how how did you transcribe? You did like a voice to text.

SPEAKER_00 17:10

Yeah, so that is um, I use one called Monolog, but there's a lot out there on the market. Whisperflow is another popular one, and it just allows you to hold a little button on your computer and just yap away, and it will just transcribe it neatly into text. And I find that it looks good.

SPEAKER_01 17:27

Monologue looks really good.

SPEAKER_00 17:28

Yeah, I think it's built by the team at every every. Um, it's a cool product, but uh so what it's asking, so it's it's straight away we we've it's got no context here. So it's working out of uh that folder here on our computer, but there's nothing in the folder. And it has no memory of our previous sessions. Um, and it's asking like, what like what do you even sell? Um, and then we've got to kind of give it like who do you target? What tone do you want? This is all things that our executive assistant should know. Um, so I'm just gonna stop the response there. And one thing that's really important to know, which might be a bit of a shock moving from chat to agents, is that these agents memory work a little bit different. So if you're used to using chat models like ChatGPT and Claude, if you open up a fresh session in one of these chats, you don't give it any context, you don't upload any files, and you just say, Who am I and what do I do? It's gonna know a scary amount about you. And that's because with these chat models, they have memory built in automatically. So every time you sort of say things that are important, the chat model saves it to its memory in the cloud that you can't see and you can't control. And with agents, you have to set up memory and control exactly what you give it. And I think that's actually uh a benefit, not a limitation, because what happens is if you're using ChatGPT and it's got the auto memory, you're having conversations about three different companies, maybe you're asking for relationship advice, and then all of a sudden when you ask it to write a landing page copy, it's pulling in context from all these other places that you don't really want in there. So with these agents, um you need to actually set up that context in memory. So as you can see, when we asked it to write a cold email, it just had no idea about anything. So we need to give it a context file. And the way you do this, right? So you can see this example here, it doesn't know anything about us. And that's because we haven't populated what's called an agents.md file. And an agents.md file is just like a system prompt. Just like if you've created any custom GPTs before, you have that field for custom instructions. Or in the project, like I just showed before, you've got that field for custom instructions. And it just gives it this context that's kind of always there, always on. And you put in there things like its role, context about you, um, your preferences for working. And then what happens is every new session before it answers your query or task, it loads in all this context to its brain as part of that observe step in the loop. So I have pre-prepared, pardon me. So I've pre-prepared an uh agents.md file here. So if we drag this in over here, this uh when you're working within claude code, it's called a claude.md. When you're working within Gemini, it's called a Gemini.md. But when you're working in within Codex or OpenClaw, it's an agents.md. But it's all the same concept. So we can drag this into our folder here. And if we open up this file for a little preview, we can see here, I've got in here um all about me, what my business does, my working preferences, like the tools that I use and what for, like notion project management, stripe. Um we've got you know all the information, my item customized loaded with context here. And I pre-prepared this, but if you want to make one of those, you can just use claude chat or cowork, whatever, and you can ask it to help you build out this uh agents.md file and to just ask you interview style questions to extract all the context from you and then build the file. So if I jump back in, now if I go to a new task, same folder, and we say write me a cold email, it's gonna have all that context.

SPEAKER_01 21:35

Yeah.

SPEAKER_00 21:36

That's what we hope. That's what we hope. There we go. And it knows, you can see this files over here. It knows automatically to load in this file if you title it correctly. Yeah.

SPEAKER_01 21:52

It's basically just like a remote a reminder file.

SPEAKER_00 21:55

Yeah, pretty much. It's just like loading it in so it has all this set context before you even start working. And one of the other big shifts to make, which comes with moving from chat to agents, is prompt engineering used to be the big thing. It was like, here's the ultimate prompt for going viral on social media, or use this prompt for this. And now it's all about context engineering. It's about how well you can load up your agent with all the information about your business so that your prompts can be stupidly simple, like write me a cold email, and you're still gonna get an amazing result. Um, you can see already here, it's already asking like, um, is it a brand or sponsor, potential partner, or consultant client? So it's already got that context. Um, book a call, um, you know, it's it's it's loaded in everything that we've given it from that agents.md file. And then now we've got a pretty decent cold email there, ready to go. So that's basically agents.md files for you. And you want to create one of those to onboard your agent with all the context it needs. And if you have lots of context, without getting into too many advanced concepts here, sometimes what I will do is I will create like a um a folder called context, load that in, and in here it's got different files about me, brand voice, idle customer profile, et cetera, et cetera. And then in order to keep this smaller, I will then just say in this claude.md file, um, before answering any questions or before doing any tasks, read my context folder to understand about myself and my business. Because by default, if you just have this context file in here but no claude.md, it won't load all that into the session by default. But if you tell it in this file that it always loads in to then check this file, you can start to like string all your context together. And a lot of people have done that with obsidian. So they'll have like in their claude.md file, they'll tell it to go check their obsidian vault for their second brand to go and find context. So that is agents.md files explained. So that's how you actually, when you're onboarding your agent, like our executive assistant, you can train it up on who you are and your business. And then, as you can see here, you know, I've got folders for all these different roles in my business. And in the head of marketing, that client.