.png)
Deep Learning with PolyAI
PolyAI's CEO/co-founder Nikola Mrkšić and team invite guests to candidly discuss trends and tech in AI, voice throughout the enterprise, and nailing the customer experience.
Deep Learning with PolyAI
Can you build a custom AI agent in 10 minutes?
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Ten minutes to a working AI agent, tested and ready to deploy.
In this episode of Deep Learning with PolyAI, Frank Ferro sits down with Ruari Phipps, Senior Forward Deployed AI Engineer at PolyAI, for a live demo of the Agent Development Kit — and a look at what it actually means to give developers real control over how they build AI agents.
They explore why enterprise AI platforms so often force a tradeoff between speed and control, how the ADK changes that equation, and what it looks like when a developer pairs the toolkit with an AI coding agent like Claude Code to go from a blank canvas to a fully tested, branch-ready agent in real time.
Before the ADK existed, you were sort of locked into the platform. And everything you had to do was through there. So if you wanted custom scripts or be able to integrate with the other ways you want to build, it was impossible. The ADK allows you to sort of really interject and sort of build these custom scripts, these extra logic, these other tools to really improve your workflow that you weren't able to before.
SPEAKER_00Today, I'm joined by Rory Phipps, senior forward deployed AI engineer here at PolyAI. Rory works at the intersection of engineering and deployment, helping our enterprise customers get the most out of our platform. He's been one of the engineers closest to our agent development kit, or ADK, since day one. Today, we're going to dig into what ADK is, why we built it, and we're going to do a live demo writing real code against the PolyAI platform. This all speaks to a bigger theme in agent development right now. How do you give developers the control and flexibility they want to build? The ADK is our answer. And to prove it, we're going to build a working agent live in about 10 minutes. So, Rory, thank you for joining me today. Great. Well, let's touch on some of the time savings that we get from ADK. Before this tool existed, why was it so much more difficult for developers to ship quickly?
SPEAKER_01Before the ADK existed, you were sort of locks into the platform. And everything you had to do was through there. So if you wanted custom scripts or be able to integrate with the other ways you want to build, it wasn't possible. The ADK allows you to sort of really interject and sort of build these custom scripts, these extra logic, these other tools to really improve your workflow that you weren't able to before.
SPEAKER_00There's a spectrum of control in enterprise AI platforms from fully managed to fully open. Where does ADK sit? And from your point of view, why do we design it this way?
SPEAKER_01The ADK was designed initially for our internal teams to sort of speed up how we're building these agents and sort of delivering it sort of as a service thing. As we sort of want to open up the platform and give more people access to the tools to build their own agents, it sort of felt intuitive to allow them to do this. The ADK gives them a loss of control in how they do this. Move fast, write functions, write topics. It really enables these developers and sort of opens it up a lot more.
SPEAKER_00It's been pretty incredible to hook ADK into some of these tools like Claude Code and Cursor. What's your experience been using those? It's been really good.
SPEAKER_01I think they work really well if you can just sort of present the information as sort of stripped down as easy as possible. And the ADK allows you to do that. We've spent a lot of time writing documentation and sort of learning how to use these tools with new prompts and features to sort of be able to get them to work really well with the ADK and really construct some new bots and sort of features a lot faster than we ever would have before.
SPEAKER_00Yeah. Maybe an example of where before you were writing in code and now we're able to use something like Claude Code. How quickly were you able to do that?
SPEAKER_01Especially for like the simple sort of boilerplate stuff, it's excellent. You know, you want to add sort of the knowledge-based stuff, and it's like, oh, what's some example questions? Ping his 10 straight away. Even for more complicated features, it works really well. A lot of the time, stuff that you just have to spend time sort of making sure you've configured properly, it can spin up in the matter of minutes and have everything linked together much faster than we would have before.
SPEAKER_00Yeah, I've been using it myself and been able to spin up multilingual agents in languages that I don't even speak, running it through iterations, tests, and simulating conversations, which have led to really great experiences when um sharing those agents with others. Yeah.
SPEAKER_01I think one of the best things about a lot of these tools is they're able to sort of spin up multiple versions of themselves. So if you want it to do 10 different tests instead of you having to manually sit and and call or sort of type away in a chat widget like used before, it could just go out and do them all at once and get back to you really quickly. It's sort of a real time saver and can get you really good insights much faster than you would have before.
SPEAKER_00True, true. Not only on building the agents, but the testing piece, I think, is one of the major unlocks here as well. The simulated tests from an omni-channel perspective. So now being able to both test in text through chat or through actual simulated voice has really unlocked some speed and given us the ability to test more edge cases as well.
SPEAKER_01Yeah, absolutely. If you sort of can give these bots when it's testing, just some license to just, you know, have some fun as much as they can, just play around, come up with new wordings rather than static wording for your tests. It could really make these testing much more dynamic and sort of, especially when you're putting these voice agents in the world, nothing's ever static. And so it's like having that flexibility where it can really sort of push your your agent and make sure it's up to sort of production ready is really useful.
SPEAKER_00Yeah. And I think the ADK really enables that with these tools. I've been able to create multiple personas and to test different cases that I, as an individual QA, would not have been able to catch myself. I think that's been really exciting. You feel like when you're going live, there's less of a gap between your first launch and a really great agent.
SPEAKER_01Yeah. Being able to go like just crate from nothing to 70, 80% with these coding tools is really, really useful. And you can then sort of use it to tune, or you can sort of get into the weeds yourself and start working on it. But that's sometimes magic.
SPEAKER_00Yeah, I think that's a perfect segue into doing the demo where we can show people what 10 minutes really does, because I do think we can get from zero to 80% very quickly and then give users the ability to fine-tune to get themselves to production.
SPEAKER_01Yeah. Absolutely.
SPEAKER_00So let's do it. Let's pull it up. For those of you listening to the audio, we'll narrate this as we go, but head over to YouTube to see the code and agent in action. And if you're as excited as we are and want to share with your development team, send them to poly.ai slash developers for everything they need to know. Let's do it, Rory. Let's pull it up. There we go. All right, perfect. Rory, want to give a little bit of background to our listeners and watchers out there on what it is we're actually showing today.
SPEAKER_01So we're going to show how you can sort of use the ADK and sort of a coding agent like Claude to sort of spin up an agent from scratch, really. So I've got here sort of a pretty much empty workspace. I've set myself up with a little sort of demo hotel that we're going to sort of try and build out. This is just sort of a spec that you sort of may just get from sort of a client or your own business. And it's just like just covering the basics. With the ADK, an agent can use this and really sort of build something sort of surprisingly fleshed out from sort of very, very little detail initially.
SPEAKER_00Yeah. And this is something that the platform allows you to pull from a website, from different documents. So it's pretty simple to get the information in there. And I think it only really needs what every business already has created.
SPEAKER_01Absolutely. So I'm going to quickly set up the ADK. It's incredibly simple to install. It runs on the latest version of Python. So we're just going to set myself up here with a quick Python environment and sort of activate that. It uses Python 314 as this is sort of how we run our agents and the functionality we use. So polycape. And that should be that. And we can sort of get it there. Once the ADK is installed and all set up, we can easily view all of the commands it can do. The ADK is just a CLI tool, pretty much. And it works quite similar to Git. You can pull, you can push. We have branching, so you can work on different features at once. And there's lots of different ways we can interact and sort of edit locally to make sure everything's going well.
SPEAKER_00Yeah, and that's quite important, I think, as an enterprise ready offering the software development lifecycle and what we provide when it comes to branching, versions, the ability to push to sandbox, pre-release, and live really makes users feel comfortable with the changes that they're making. None of this is actually going to hit live until you're comfortable, you've done tests and feel like it's ready to push forward.
SPEAKER_01Yeah, absolutely. I've already set myself up with uh an API key that I can generate on the platform. And once that's there, I can just initialize the project and sort of pull in this blank M2 projects that I sort of created for this hotel. In it is just all of the default parts of the projects. There's sort of nothing filled in. We don't have any functions or tools, the knowledge base is empty. The sort of main agent prompt is just the default thing. It's all sort of unspecified. So it's sort of really a blank canvas for us to sort of get in and make some changes.
SPEAKER_00For the users out there, just so you know, nothing here is set up beforehand. It's all from scratch. So it's just a brand new project.
SPEAKER_01Yeah. We'll open the project and we're just gonna open Claude and get it to have a look. We sort of set ourselves up with a small Claude skill just to sort of teach it how to use the platform. The CLI has sort of tools and information that sort of can really teach these agents how to do it. The Claude skill is quite simple. It's just sort of showing, oh, use these commands, use this to learn and sort of how to do the workflow. But once it knows that, it can easily use the ADK and the documentation provided to build. So we can sort of get a setup here. We'll say use poly hotel PDF and set up my agent. And I think what we want here is to use the PDF example, and we'll just set up initially some knowledge-based topics because that's fairly simple. And create knowledge base topics. Now it's you'll start thinking and sort of exploring everything we have here with the ADK. We've sort of given it permission to run the commands and have a look. And look, it's just read the PDF and now it will start looking through everything. Here we go. It is starting to read the documentation and sort of understanding what it needs to write. Yeah. Within our internal teams, we often are working on several projects at once. And we find Claude and other agents, especially when you have a good knowledge base of information, how to set it up. It's referencing other projects, it's looking at other things. So this is something that the more you tune and the better you build your agent when you're adding new things, it can really sort of like learn the best practices based on what you're doing. Coming from scratch, with the documentation, we do have sort of some of our best practices and some of our ideas on how to set up a good agent, which we hope would really help getting you set up initially.
SPEAKER_00Yeah, I think that's a really important part here, Rory, is that a lot of this is possible because of the work that we've done here at Polyai over all the years to get here. So the best practices, the frameworks, the foundation, the architecture that allows for this to be so simple.
SPEAKER_01Yeah, absolutely. And here it is, it's starting to set up some topics here, and we can start seeing what it's creating. So it's starting to build out these knowledge-based topics, say, oh, this is sort of what people might ask to trigger it. This is the information, and sort of telling how when the agent is running, how it should answer these questions. And it's just looking through the PDF we gave it, and it's working this out themselves, using its own knowledge of like, oh, what a customer agent should be. And it's really able to produce this a lot quicker than I would have been able to type it out.
SPEAKER_00Yeah, before this, Rory, it would have taken you definitely a day to have done something like this.
SPEAKER_01Yeah. It's sort of all in the platform, you're sort of manually typing it out. Maybe you can sort of come up, you've pre-written stuff, so you're copy-pasting it in, even before all of the ADK, all of the clawed tools, and we just had the ADK. This was much such a bigger thing because you can start doing automations without these AI tools. Maybe you want to pull from a spreadsheet or different sources of information and sort of inject it directly here. You don't have to use these clawed tools, but they're really good at sort of setting this up.
SPEAKER_00That's a great point. It's really up to the user and what they would like to use if they want to use the ADK directly or if they want to use one of these AI coding tools. Yeah.
SPEAKER_01So here it should have set this up. It's run some validation to make sure everything is configured properly. It's sort of checked everything and made sure it's all okay. And I think we're going to ask it to push it up to Agent Studio, because currently everything's locally. Say uh create a branch and push this, please. You can push. That should all be done correctly. And now, so once it's there, we can start to chat with it. Uh chess and test, please. So it's set up as about 10 or so different topics, and then we're just going to get it to start running some test conversations.
SPEAKER_00It's really you've gotten that control too. So not only do you have the speed, but you also have the control and comfortability that what you've done you can test right away.
SPEAKER_01Yeah, and that's really important. And you can test it on your branch so that you're not affecting anyone else's changes. If someone else is working on the same project, so you can easily just push onto your branch, test there, iterate and make sure everything is working as expected. Here we go. It's sort of outputted all the tests, everything is working correctly. If we want to sort of check this ourselves, we can easily do this. We'll go into the project and we can start a chat here. Like, oh look, it's got the correct utterance at the beginning. Can we cancel booking? And yeah, it's all there. So it's, you know, Claude isn't making it up. It does actually all work. It is able to tell them about the about the cancellation process policies. So it's all good. And you can interact with it here as well.
SPEAKER_00That's great, Rory. And we're not going to get into it today, but it is possible to just very easily set up integrations here as well. So to actually create a booking in a system is quite possible with the ADK as well.
SPEAKER_01Yeah, it's definitely it's using all its knowledge, especially Claude and stuff, is trained to make code and building out these API integrations and these complex sort of ways of interacting with sort of your systems. It can do that. You just tell it, it goes, okay, I'll build it like this, and it's able to do that itself. I was also going to quickly show building something more complicated, flow for this. This would be a booking flow. We're not going to build the integration, but it's able to do something more complicated than just like one-turn features for the agent.
SPEAKER_00We're using it to build live production agents. So it goes much deeper, but obviously in 10 minutes, we want to just show you the scaffolding here of what a flow would look like.
SPEAKER_01We're telling it to build a booking flow. We'll say collect all the information, confirm with the user, then handoff. Probably here in a booking flow, you'd want it to call the API to do it itself. But if this is the situation you want, then handoff, that's sort of we'll do here. Flows and functions are a lot more complicated. You have to stitch together a few things. But this is something that the ADK has been created to easily validate, make sure everything it's set up is correct, and sort of teach these sort of coding agents how to do it well and sort of present everything in a way that it can easily see.
SPEAKER_00Yeah. And flows are our way of introducing control to give you that fine-grained ability turn by turn to do exactly what you want the agent to do.
SPEAKER_01Yeah, absolutely. Tell it it can make these changes. So it's sort of here, it's quickly produced this entities file. So this is everything that it wants to collect. Sort of these are used throughout the flow. So at the different points, it would be like, oh, at this point we need to make sure we collect all these things. And having these entities allows the agent to sort of analyze the user speech and extract the information that we need to use for these API calls and sort of persist this information correctly throughout the call. So here we go. It started writing functions as well. So this is a function that will start the booking process and trigger that.
SPEAKER_00So, Rory, before this, you would have written this by hand in code.
SPEAKER_01Yeah, with the ADK and without the claw, we'd have to manually create all of these step by step, or we would have been on the platform, dragging out these boxes, typing out these prompts ourselves, and sort of doing it there. Internally, now we have like a whole library of how these work. And we've sort of tried to get build the ADK so it can represent that the best and add onto our docs how it should construct these flows.
SPEAKER_00And we're making that library available to users of the ADK so that they don't have to write their own code. They don't even have to go drag and drop, grab templates. It's all accessible through the ADK and easily implemented through a coding tool like Claw here.
SPEAKER_01Uh definitely. So here it's produced this flow, it's sort of correctly fixed some of the issues that it sort of did on the first thing. And now we're gonna get it to push and do some tests. It's not even asking me, it's just starting to test itself in the background. And yeah, this is what is two minutes, and then we've got a booking flow, which it doesn't look like it's got a huge amount of steps, but it's just making sure everything is sort of set up and correct and filled in, would have definitely taken an hour or so beforehand. And on the scale that we're building agents where the flows can be super complicated or it's just the number we're doing, having this speed up is really good. One of the best things about some of these agents is just the way it can input data. So here we've gone from a PDF, but especially for flows, which are a lot like if you've seen like flow diagrams and sort of decision trees, those sort of stuff, you can just show it a diagram and go, this is what we want it to do. And it's able to sort of produce it. So if there's other places that you want to have this clawed code and these other tools able to really read this, this well. So here we go. It's sort of telling the end-to-end thing, it's and to talk about everything's done and all pushed. And yeah, it says everything's pushed on the branch and it's ready to merge. Let's do a test of ourselves and sort of just verify it there as well. Uh oh, I want it to book updates this weekend type of room. Would you prefer a classic room? And we guests just be me. And there we go. I mean, I think for example, Friday, 11th, Saturday, 12th. What day is it? Yeah, we'll say that. Yes. No. It's once a specific days. Friday, 11th, and 12th. So there is some hiccups with the initial version, but this is where you can easily jump in and sort of go, oh, you know, this is not right, and manually tweak it or get Claude to tweak it, and it is really good in sort of being able to interact with it that way. And then it's all all done. We say, yep, all good, ready to merge. And the ADK should merge this all in. It's gonna show me a quick diff before it happens, just to make sure everything's correct. You can merge and it will just quickly run the merge. And now once it's done, it's ready to deploy straight away, and that's all ready to be used by, you know, if we want to start pointing agents to or customers, they can go straight away now.
SPEAKER_00Yeah, it's incredible, Rory. And so this agent would actually power our omnichannel experience. So you'd be able to call into this agent, chat with this agent, receive SMSs from this agent as well if we set that up on one of these flows.
SPEAKER_01Yeah, absolutely. At the moment, this is just configured for voice, but if you wanted to have it via web chat, the configuration is there. If you want it to be able to do both, it also gives you that flexibility and it can make those edits through the ADK. Yeah, it's incredible.
SPEAKER_00So from here, you could also build in multilingual functionality as well, right off the bat.
SPEAKER_01Yeah. And as you said, you can build it out for I don't speak Japanese, but if we wanted the agent to speak Japanese, I'm sure we could get uh Claude to do some translations and make sure it's all set up correctly.
SPEAKER_00Yeah, absolutely. And those are some of the great tools that Poly AI and Agent Studio provide to users. All that's inside of the platform and already there for you. Yes, absolutely. Roy, this has been incredible. Seeing an agent go from zero to actually demo ready and not. Too far from getting ourselves to production with a little bit more tweaking, adding some integrations, et cetera. I just wanted to thank you. Is there anything else you want to share with our listeners and viewers out there?
SPEAKER_01Not too much, really. The ADK is sort of constantly growing and expanding to the whole of sort of the poly toolkit and sort of really evolving as the product evolves. It sort of really helped us integrate with our systems and build bigger testing frameworks and our own sort of methods that weren't possible without this sort of being able to interact directly with the platform and sort of add these sort of tools and facilities that just having it all sandboxed away is just it wasn't possible. But but now it is, which is is really good to have.
SPEAKER_00Yeah, it's really exciting. I'm mixed and I'm looking forward to seeing what everyone's going to be able to build with this new tool. So reminder to everyone out there please review and subscribe the podcast and go out and check out the ADK. It's at poly.ai forward slash developers. Thanks, everyone.
