Code with Jason
Code with Jason
323 - David Yanacek on 20 Years of Innovation at AWS
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode I talk with David Yanacek about his journey from operating Amazon's web server fleet to revolutionizing DevOps at AWS. We discuss AI in software development, spec-driven development, and universal testing techniques. David shares insights into operational excellence and the Amazon Builders Library.
Links:
- Amazon Builders Library
- Nonsense Monthly
Snail Mail Newsletter Pitch
SPEAKER_01Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Email newsletters are a really nice way to keep on top of what's going on in the programming world. Except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more. That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right. I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like? You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming. The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says, thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary. I'm sure that's what you were going for, so just wanted to let you know that you succeeded. Looking forward to more. Drew Bragg from Philadelphia says, Nonsense Monthly is the only newsletter I deliberately set aside time to read. I read a lot of great newsletters, but there's just something about receiving a piece of mail, physically opening it, and sitting down to read it on paper that is just so awesome. Feels like a lost luxury. Chris Sonnier from Dickinson, Texas says, just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it. Something about holding a physical piece of paper that just feels good. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsense monthly dot com. That's nonsensemonthly dot com. I'll say it one more time nonsense monthly dot com. And now without further ado, here is today's episode.
AWS Veteran Meets The Host
SPEAKER_00Hey, thanks for having me. Very excited to be here.
SPEAKER_01Excited to have you here. So you've been at AWS for about 20 years, if I have that right.
SPEAKER_00Uh yeah, that's right. Uh I guess the split hairs a little bit. I guess Amazon for 20 years, um, like the vast majority of that has been AWS, though.
SPEAKER_01Got it. Okay. Um and and we were talking a little bit pre-show, and you and I have some geography in common. You you've spent some time in Michigan.
SPEAKER_00That's right. I guess about half my life here in Seattle, and then about half my life uh within a you know hour and a half or so of where you are in a couple locations. But yeah, uh, in fact, just outside in Seattle, right before this, I was very excited uh that I heard some uh thunder, which is a very rare thing here, but I know in the in the Midwest, you know, you can tell the people who I work with who grew up kind of around thunder, like you get in the in the Midwest. Uh because it's just it's you get the nostalgia here in Seattle, it happens only like once uh oh, there it is again. Uh yeah, it only happens uh once a year, maybe. Maybe and so when I hear it, I get this nostalgia and this feeling of like nap time growing up and hearing the rain be so calming. Uh so there's other core from like from El Salvador, also kind of you can tell who runs to the window because they they had had that kind of experience growing up.
SPEAKER_01Interesting. Um, it's it's always funny the things you miss. I I moved down to Austin, Texas. And by the way, I don't know if we said uh I live in uh West Michigan, just outside of Grand Rapids. Um I moved down to Austin, Texas, partly to get away from the cold of Michigan, and I found that I missed the cold and like the overcast skies and stuff like that. I'm like, this kind of sucks. It's like beautiful every single day. I want it to be like crappy out sometimes.
SPEAKER_00You can decide to have all four seasons.
SPEAKER_01Yeah, yeah. And uh last question for you on on this topic: where exactly in Michigan did you live? Like what town?
SPEAKER_00Oh, I grew up in Midland, Michigan. Uh and then I went to school in Ann Arbor. So U of M. Yeah, U of M, University of Michigan, yeah. Um but yeah, Midland, Michigan, yeah, it was a kind of a it's a large suburb. Uh I guess it calls itself a city, about 40,000 people or so.
SPEAKER_02Mm-hmm.
SPEAKER_01Yeah, I believe I've been through Midland.
Why AI Needs Specifications
SPEAKER_01Um okay, so getting into the meat of it, I listened to a little bit of an episode that you recorded for a different podcast. Um, and you talked about AI and testing and spec driven development and stuff like that. Um this is right up my alley. Um and and I think it's kind of funny um because it's like the industry is rediscovering these things that a lot of people were doing all along, um, like test-driven development. And and it's so funny to watch because it's like, hey, if you just like decide what to do before you do it, and then write tests that represent what you intend to do, and then write code to fulfill those tests, everything turns out to work a lot better. And it's like, yeah, like uh we we knew that, but it's it's good that people are rediscovering this. Anyway, I'm I'm curious to hear your take on this.
SPEAKER_00Yeah, I mean, I think uh I think that's that's actually quite a bit true around the transformation that teams I'm seeing software teams go through to be so to get so much more done uh than before. It's actually doing things that like we not necessarily new things, like just doing things that maybe we've been meaning to do and haven't gotten around to it yet. Um things where you can just increase the autonomy and and let the agentic coding loop just go and be super productive where um yeah, where before it was kind of on the backlog. Oh, yeah, that would make us a little bit faster if we had really good, you know, bet a little bit better unit tests that kind of would fail sooner if there was a problem versus an integration test, which is free further along in the pipeline or that kind of thing. So these things that teams like would always like to be able to do um just become that much more useful because you just get that much more done if you can let an agent loose with it. And and so spec-driven development, just as a starting point for all these other practices, if you want to talk about more beyond the spec, but um, you know, we it's something where we found um internally that when we were using vibe coding tools, agentic coding tools, um, we just found that they were, we could see the potential. They were so powerful and in how much it was so impressive that they could generate so much. Um, but we found that they would wander off. Maybe uh you'd come back and look at what it was doing, and it was doing some refactor that you didn't really want to do right now or ever. Um it would uh you know cheat on the test, it would kind of maybe comment out uh tests or you know, say, oh yeah, that seems about right, and and just not really do what we agreed on, or do, or maybe not even complete the total assignment. Um, and so we found that what we could do is is introduce specs, like you're talking about, the spec-driven development. And and we actually made a whole uh agentic IDE, a coding environment called Kiro, um, around spec-driven development to be able to uh to just do this, to get to be more useful for to have a more useful workflow for the agent so that it could do production grade coding, where it would actually get everything done. Um with all the resiliency and everything built in.
Treat Agent Drift As Defects
SPEAKER_01Yeah, I want to talk a little bit more about the ways in which vanilla AI with no special guardrails will fail to do what you might want. Um, because there are a lot of different uh software development methodologies. Some are better than others, and the AI, you know, it it like knows everything, but it doesn't know how you want it to behave. It's like, what, you want me to behave smart or dumb? Like I'll I'll be dumb if that's what you want. Um and and out of the box, it's not gonna follow this like I don't know, Kent Beck style of development that I personally might want. But it can, it's it's not like it doesn't know. It's just that it's not gonna do that by default, and then it's like it if if it does something that I think is stupid, and I'm like, oh, like don't do that. Use like the dependency inversion principle in instead of all these conditionals, and it's like, oh yeah, of course, I'll do that. But it's funny because it it won't do that on its own.
SPEAKER_00Right. I think where people spend like the the the most productive time in in getting an agent to run um just better, is when they when they view everything that the agent did that they didn't want to do as a defect. Like as a defect that you can do something about. Like imagine like uh uh it's like uh if if you do like a code review, like you you actually have all the code ready and it's time to look at that code and see like, okay, if I have feedback on that code review, like obviously, okay, let's address that, but like let's figure out how to remember that for next time. And so if you kind of embrace this the this workflow of viewing every everything is okay, how do we make sure as I address this code review feedback that I add some steering, some skill, update my skills, update the steering with the project, or maybe so that the team uses so that it remembers next time when it's either generating a new spec and following it that way, or whether it's just going off and doing something that it would remember, remember how to address that thing that went
Context Windows And Written Specs
SPEAKER_00wrong last time. Because you asked what uh what can go wrong. I mean, it's it's uh I guess I yeah, I just see it wander off and start and start uh you if it if it r actually, okay, so one class of things comes to when it um either hits a uh like a sometimes it'll hit a context window limit, like it'll it'll it'll just have been running for a long time and it kind of starts losing track of what it had done and what it had already learned in that session. And so that's why having having this like a spec that lays out this larger project that we want to do that might take a couple hours. It might take more than a couple hours. And that could potentially overflow context window, need to reset it, need to compress it. Um, and so having that all the whole plan written down helps it pick back up. If you say, okay, actually, let's just we've we've achieved a few tasks so far. Um, like we've done steps one through five. This is part of the spec. The spec is essentially a uh a requirements doc that you work with the agent to develop. It's a design that you work with the agent to refine, and then it's a task list. And so these three things that make up the spec end up being a really useful thing to fall to deal with the fact that the agent, yeah, it can run out of essentially memory, context window memory. And if it does, it can just pick up which whether after it can look at what it tasks it was going to do and had already done and re-just kind of re-get its bearings and start over. So that's one thing, like the context window overflow.
SPEAKER_01Um yeah, and if I if I may interrupt, um whenever I learn something new in programming, um I try I try to extract principles out of it. Um, because we're we're always presented with new tools and stuff like that. Um, but it if you can extract the principles out of it, the principles are much more portable and durable over time and stuff like that. And and it's there's a lot of profit there if you can extract the principles. I've been trying to do that with AI. Um, and I'm I I'm I'm kind of noticing multiple layers to the way the ways that people are harnessing AI. Um one is to just like tell it what to do, say like behave in this certain way. And that can work. It it seems that as the models improve, they're getting better at that. That's like my anecdotal perception on that.
SPEAKER_00Act as like you are a professional senior developer, like uh that kind of thing.
SPEAKER_01Yeah, yeah, just like in general, uh, don't mix refactorings with uh behavior changes, that stuff like that. Um it it seems to be getting better at that, but it's that is like a um it's like a what's how do I want to put it? It's not a very tight harness, you know? It it won't necessarily follow that. So you can do things like uh I now add pre-commit hooks that run the whole test suite before any commit can be committed. And that's like a uh physical barrier.
SPEAKER_02Yeah.
SPEAKER_01And I I I I think those things, whenever you can put in like a physical barrier, I think of it kind of like a woodworking jig or something like that, like a a jig that makes it impossible to make a mistake, like maybe a hole drilling jig where like you can't not drill the holes in the right place because you have this physical barrier there. So that's another thing. And then the the kind of thing that you're describing, um, where you just save these files, and it's like, okay, I have this spec, which is kind of the authoritative source of truth for the project I'm working on, and that's durable, and you can always refer back to that. Do do you do you think about it in terms of like principles that you can apply to to harness the agent? How do you think about this?
SPEAKER_00Yeah, I think though uh the the types of barriers that you can uh it's sort of actually partly about the agent harness that you build of like that it that it will the the program surrounding the agent won't let it succeed and proceed, excuse me, proceed until it uh until it uh does the thing and proves the proves the previous step. So it just enough workflow kind of along with the letting the uh LLM uh be the sort of Ouija board, uh if you will, that's like just kind of ghost that's in the machine.
Property-Based Tests As Guardrails
SPEAKER_00Um and so that one of those things uh by by breaking down the project into tasks, um those tasks would include uh actually some pretty interesting testing, uh testing barriers that you're talking about. Um one of them is uh a technique, again, another technique, like you say, that's been around a long time is is called property-based testing. It's just a type of testing, it's sort of a mindset. Um it there are frameworks out there and have been for some time around this testing technique. And this is one where, because you have a spec with really well-written um directions with like caps, like you shall, the program shall do this when in caps will this happens. And from those, it turns out those are pretty useful inputs to generate property-based testing. Um, property-based tests for just for the for everyone are things that um that test exhaustive um properties of of the program versus just boundary cases. And so you could let's take, let's say you're you're implementing a uh a traffic light uh system. You of one really important property invariant of it is that at most one direction has a green light at a time. Um that has to be held, that's its only job, really. And it's everything else is just goodness. Um and so you can just that would be described in the spec as a requirement. Um, and then a property-based test would uh generate input, they generate inputs like with a just a bunch of combination of the inputs to the program as sort of like a test driver. And they're testing along the way in every step of this that the program maintains those invariants. It's sort of just it's just a test framework, right? Uh but a good technique that is particularly useful when it comes to these requirements documents, that it can just be turned into all of these verifications. And so the coding agent won't continue until it has written the property-based tests and that they pass. And so it just it can't proceed until it has done that.
SPEAKER_01I've never done property-based testing and I never I heard the name and that's it until I listened to that other podcast of yours where you where you talked about it. Um it the way it works, will the tool typically like generate some test cases? Like, if there are 17 different permutations, then it'll generate 17 tests and you do it that way, as opposed to like there is a chunk of of the test that like loops through and and does that kind of stuff, or maybe both. How does that work?
SPEAKER_00Yeah, the input generate, there are a bunch of different input generator strategies that they that the frameworks supply. Some of them are include some amount of your randomness, um, looking at certainly making sure that the important combinations are all exercised. But yeah, input generator that just comes up with all the different there are different input generator strategies in the in the frameworks. Uh yeah, it's uh it tries to exhaust all possible uh inputs, uh as many as many as are practical.
SPEAKER_01Um yeah, yeah, that's that's really great. Um, and that's you know, that seems to go along with this idea of like a jig that that makes it physically impossible to do the wrong thing.
Mutation Testing And Better Coverage
SPEAKER_01Um I have found sadly that LLMs tend to write fairly fairly poor quality tests, at least at this present moment. Maybe they'll get better. I think they have gotten better uh in the last couple years or whatever. Um, but at the moment, not great. I found that I if I give it a fairly thorough uh suite of examples, if I drop it into a program that already has a lot of well-written tests, it's good at extending that and writing more tests that resemble the existing ones, uh tests that I'll be pretty happy with. I found that when I do a greenfield project with Claude Code, for example, the tests are not always great. And it seems to depend somewhat on the language. There's kind of a testing culture base baked into each uh community, and and some communities have better testing cultures than others. Anyway, I was writing a program in Rust. I I don't know Rust at all. I was just letting Claude Code write the Rust, and I was really unhappy with the tests that it was writing. They were very just like perfunctory, um tautological. Just if I put in five, assert that it's set to five. It's like, well, that's that's kind of pointless. So I hooked it up to mutation testing, and so I said, write the tests and then do the mutation testing so that um it's it's a much more rigorous, thorough way of testing it. And I found that to to work out a lot better. It kind of reminds me of of this property testing idea. Have you also applied mutation testing to this this at all also?
SPEAKER_00I'm actually not familiar with the term mutation testing.
SPEAKER_01Oh yeah, it I'm new to it. Um so the idea is that you write the test, you write the code to make the test pass, but then who's to say uh that that everything is as it should be? Like, could it be that your test gives some sort of false positive um And maybe you wrote too much application code beyond what you really needed to satisfy the test, or something like that. So the mutation testing library will mess with your code and it'll say, okay, this passes, but what if we tweak it like this? Do the tests still pass then? Oh, because if it does, then like, gotcha, these tests are invalid and we need more test cases and blah, blah, blah. Um I I I found that to be a really powerful
Formal Methods For Distributed Systems
SPEAKER_01technique.
SPEAKER_00Yeah, I think like these verification and testing techniques that have been around, I think it's it's really their time to shine. Uh and so one thing we do uh at Amazon a bunch for these um um for we build a lot of distributed systems like in AWS, uh, you know, we have things like DynamoDB, things like S3, where we have to prove that the replication, like these distributed systems around failover during a network partition while bootstrapping another replica, like all these the this is the this is the stuff that personally I like I love so much is all of these uh distributed systems algorithms. And in order to prove these, um we do a lot of formal verification methods, like TLA plus. Uh, it's a really good verifier of a model. It's like I have this model, uh, and and this the TLA just it it beats it up. You have your your here's how my system will behave, and it beats it up with introducing like whatever latency at the critical moment, and to see if if you still maintain an invariant, like only one uh replica of a replicated database is the is the leader at any point in time. Like all the it just is very good at these kinds of synthetic uh tests around all the different cases that you would care about in a digit distributed system. And I think these this is they are kind of hard to write. Um they can be. Um, these models, and then you write your code, which isn't the model. It's actually your code is a rep is a is a manifestation of that model. And so I think it's these methods kind of time to shine, because you can now you can now write generate, write the model, and then generate the code and and have the LLM reason about and prove differences between the model, which already proved your algorithm, and your code to see does it match the model. There are a bunch of techniques around this, but I think it it my point is just that it is it's the time for these formal methods and and just testing methods to shine.
SPEAKER_01Yeah, yeah, I totally agree. Um I I have to take us on a small digression. Um, my friend uh Stephen Baker mentioned the other day, uh he he made an analogy with oil. So uh back in the I don't know, mid-late 1800s, second half of the 1800s, um oil wasn't used for gasoline and automobiles, it was mainly used to make kerosene for lighting, and the market size was limited. Um Standard oil was was dominant at that time, but their their main market was um kerosene. Um but then uh industrialization came along, uh World War One happened, um battleships converted from coal to oil, um, people started buying cars, the the adoption of automobiles just totally exploded, and all of a sudden it it was oil's time to shine, you know. It it had been there this whole time under our feet for millions of years or whatever, uh and and then used for this limited application, just for for lighting and and stuff like that, but then the world changed and it got so much more useful, indispensable. Um and so these these testing techniques techniques and such uh could be compared to oil. We've had them for decades, and they've they've already been great. Uh but now it's like okay, there's there's I think two two things happening. One is a greater necessity for these techniques, because if you don't use them, then AI will just go off the rails. Um and another is that the the bar has been lowered. Uh I I don't want to confuse my metaphors, but I've I've always talked about AI kind of like a mental lubricant. You know, it these these things that have friction, um, like writing tests, for example, there might be some mental friction there. Uh it's AI is like WD40, which is not a lubricant, but um it can unstick your mind. Um and it makes these things which were previously difficult much easier. So those two things, uh I think, you know, like you're saying, it's it's their time to shine. Those those are uh hurrying that along.
SPEAKER_00For sure. I think uh the it's just you know, I've spent the last 20 years just with uh almost singular purpose in mind, and that's just to make developers' lives easier. That is the pattern that I have followed ever since I started at Amazon and had to operate a database in order to automate server operations because I needed to keep track of all the servers. I needed to operate a database. Operating a database is harder than operating the servers. And so then it's like, okay, how do I net how can I never have to do this difficult database ops again? Oh, I heard we're let's make this dynamo DB, let's make NoSQL, so then we don't have to do database ops from then. It'll just be managed. That sort of rinse and repeat with like I've worked on Lambda then for managing servers, the building serverless, so you don't have to manage servers, API gateway because API routing is tedious and and easy to get wrong. Um so just I've been chasing this this like constant. How do I build the next abstraction that's gonna make my life easier? And the really fantastic part about LLMs is that they can do that just they're so flexible. Like they can, they are that uh sort of this missing ingredient to be able to do that. Well, how do I how do I do the next part that I find tedious? Like server ups, okay, now I don't have to do that. But what about uh writing tests? You know, right? Like that is it is something that people we could have been writing property-based tests and and uh mutation tests this entire time. But it's just the it's been tedious uh to do all the time and spend all of our time. It's not as fun. And so great, let's have let's have the agent do that. We have to now the work is to prove that it actually wrote good tests. You know, and and these frameworks can help help do that, provide that evidence or the the enough of that guarding guardrail, just like the spectrum development provides some guardrails and and and a path to success. Um, these tests can be a part of that too.
SPEAKER_01Yeah, yeah, it's a really exciting time. Um and and uh uh uh what you say is exactly true. Um on the developer experience side, like I've invested a lot in my own developer experience because it's um you know you you pay a small price to get hopefully a large benefit, and then you can go faster, and it's kind of a positive feedback loop.
Developer Experience Through Abstractions
SPEAKER_01Um so maybe if we if we can get into that a bit, like what kinds of stuff uh, and and it doesn't even have to be only AI, but like what kind of stuff have you done to make uh developer experience easier, especially things that people might not like most companies might not do, that kind of stuff, just anywhere you want to take that.
SPEAKER_00Sure. Uh I'd say a lot of it stems from the that the first team I was on on Amazon.com, where I was on the team that ran Amazon.com's web server fleets. Um we had just a lot of servers that needed to get provisioned. We had to forecast how many to physic to actually buy. This was pre pre-EC2, pre-AWS cloud. We had to buy servers every year. So I had to manage keeping track of how many we have and how many we need to buy, what their utilization is, forecast the efficiency uh that we're going, like, is the website going to get less efficient or more efficient? Probably less efficient because that's the way that entropy works. Um, and so we had all these things. And so one of the things that might be a little bit uh should it seemed like it should be easy, but it wasn't, was uh calling other services. So I would make this a forecasting tool that would figure out how many servers to buy every year, but I would need to call and get the utilization data, the observability data about how like CPU usage, requests to the website, all that stuff. And that was in this other web service, um, the monitoring web services as it happened to be called, later became CloudWatch. Um but calling that service was actually kind of annoying because they it was a sort of a new paradigm around REST, where it was like, okay, uh you don't have to generate a client library. It's like, but I'm writing a Java program. It's like, no, just like, you know, use what just make HTTP requests and fish around in the return to XML or JSON for the response. Okay, but I've I have a Java program here. Like I do need to generate types or write types. So I just found it sort of uh tedious to call some of these services. Whereas before we had some frameworks at Amazon that just generated strongly typed stuff, but it wasn't REST under the hood. And so it was there was this just kind of a new way that people were writing and new frameworks people were bringing in. And so I found it those made it maybe easier for the service author to write a service because they could do it in the newer, newer way and newer tools and newer frameworks, but those had left the clients behind. And so I found that tedious of like just calling services. It's supposed to be easy. And so I learned that there was a team making a new framework. Um, we call it Coral. It it its goal was to be a modern framework that that could speak REST, but also speak all the other protocols that we have, all the binary formats, all the everything to be back so that it would fit into whatever tooling that clients wanted to use. It would sort of actually just make everybody happy, gener be a nice modern like web service framework, but also support any kind of client that people want. Like also basically generate clients as well. And that kind of became like the AWS SDK generation in a way. Um so I found that this that's just an example of one of these that I've chased down over the years. That's it's like subtle. It's like, okay, a web services are are tricky and uh and and building the service is hard, but also making sure that you are paying attention to your customers so that calling the service is easy too. And so uh this coral sort of actually in a way became this API gateway uh thing to because frameworks are are also kind of hard.
SPEAKER_01Right. Um okay, so this might be the the point in the show when we uh wildly wander off the trail as inevitably happens.
Hub And Spoke Versus Standards
SPEAKER_01Um but I I'm curious about something. So like if you have a large system with a bunch of different services and stuff like that, um there's a lot of different ways you can have them talk to each other. Um something I've seen that I haven't loved is when each service basically talks to every other service. Um and I I did some research because uh uh most of my career has been at very small startups that haven't had this situation. Um but I I I did some research once I started to work at a big company, and I'm like, okay, like what's what what are the different ways this could be architected? And it sounds like what I was looking at was point-to-point architecture, and I was seeking something more like hub and spoke. Um because the the thing that I the thing that makes me uncomfortable about the point-to-point architecture is every service had its own custom, unique way of talking to every other service. And then what happens when you add a new service? Now you have to like start from scratch and incur the cost of building ways to talk to all the other services, and you have to have all the credentials and all this stuff, and it's just like this this can't be the way to do this. With a Hub and Spoke, it seems like you just talk to the hub, everybody just talks to the hub, and it's like, hey, I want to send a message. I I'm not gonna integrate with Slack and tell Slack to send a message, I'm just gonna notify the hub that something happened, and the hub can say, Oh, okay, this thing happened. Uh I'll send a message to Slack and or send an email, whatever, whatever I, the hub, thinks need to happen. Um I'm just curious, any commentary you might have on that whole thing is somebody way more experienced than myself and those kind of things.
SPEAKER_00I've seen so many patterns of this um over the years, like what work to different degrees. I think there are different problems that that are it's helpful to have more of a of this kind of hub and spoke. Uh there was one uh like looking at the um, so I worked on the Amazon.com web server fleet. That means the thing that ran that rendered HTML, ultimately. And it needs to make a lot of service. Get data about if you're looking at a product page, we need to get data about the product in order to show decide how to display it. Um turns out the information about what is a product with when you have when you can sell any kind of product, uh, whether it's digital or physical. And you know, when we started, we only had books, and then we had physical products, then we had digital products, it just evolves a lot. And over time, yeah, we built uh one of a kind of a dedicated around products. We built some of what you describe as as a as one of these hub services that would kind of aggregate all of the and and deal with figuring out like, okay, given this request to look to display this product, it could it had logic in it to figure out where to get that from the different other services that had a little piece of this information here, a piece of information there. Similarly, with like the order processing pipeline that would because placing an order for something that is uh there's so many different types of products, and so that you need to be able to handle different products differently and the workflow changes on the fly as things get um depending on who's going to fulfill that order and everything. So it's uh we've seen a lot of it. I don't think there's any any one size fits all, but there is definitely like a lot of people will do what you described with uh with API gateway because it can you can just have that is the API layer and it handles all the different types of API requests, but um behind the scenes, some paths go to one service or another, and you don't have to think about point integrations like that. So that can be nice. Um, I do see think that one thing that worked particularly well that kind of works no matter which, whether it is point-to-point or hub and spoke or event bus or whatever is just common frameworks. If people are speaking the same, having the same identity, like the company uses the same identity system for what is a service, what identifies a service, and what authorizes one service to talk to another. Like in AWS, you you can that's one that's this identity access manager. IAM just is the policy that just and sort of a signing language that that describes you with with one set of credentials, you can scope those to be able to call any service. You don't have to figure out how to speak a different identity provider system or anything. Um internally, there are others. So by having some standardization, um, you can uh I guess uh having standardization is is really helpful because it because you don't have to uh it doesn't you can actually choose whether hub and spoke works for you or point-to-point or workflow or or anything because you uh but the standardization makes it so that you don't have to pay the price of of integration 10 times. You're just okay, that's the advantage of this framework that I looked on. It's like the and it's still around. Like uh pretty with a few exceptions, like every AWS API call is is run through this framework, even though there is no common API layer. Like you still talk to different endpoints, but it's all the same framework. And so the SDK is all generated off of the same service distribution, service definition library, uh their language. It's called Smithy, it's actually open source now, um, that defines services and what shapes and APIs and signing methods for identity. Um, even though they aren't, we don't, as all of AWS have one API layer, this like Hub and Spoke. It's still we don't have to reinvent the wheel every time we make a new service or make a new client for that service.
SPEAKER_01Yeah, that's very interesting. I hadn't considered that possibility. Um I'm trying to think of an analogy to understand it better. Um I I like to buy all the same brand of power tools so that I can take a battery out of one and stick it in another tool.
SPEAKER_00Um I know your podcast is probably isn't sponsored by any one of them, but what is the what is your go-to? Skill. Skill? Okay, okay. I've I I kind of the sorting hat put me into the uh Mikita line at one point. Uh so that's to mix to mix some metaphors there.
SPEAKER_01But yeah, well, when my wife and I got together, she already had some skill tools. And and she has an insistence, which I agree with, on having everything uniform. So I was like, well, skill is what we have, and I think their tools are pretty good, so that's what we're gonna go with. Um yeah, okay, so the the the takeaway for me there, a takeaway, is Hub and Spoke isn't the only answer. You don't need to have the hub as the intermediary in order to get some efficiency gains. Um you can save the cost of those uh unique uh uh point-to-point integrations by having something in common between them that allows them to talk to each other.
SPEAKER_00Yeah, exactly. I think identity and uh and uh especially framework for for protocol help
Observability Instrumentation That Scales
SPEAKER_00a lot. Um another kind of cool aspect of this was uh basically like make enough the the key with the framework was to make it uh so um is to just solve a lot of the problems that people would have with point-to-point in a standard way, like metrics and observability, for example. Um, if you are going to call some other service and you have some other framework for that client, you'd have to add your own instrumentation. Um, where just to say, okay, when I call a service, I need to record whether or not it succeeded, what kind of error it gave back, specifics about the error it gave back, how long the latency was. I need to bucket the latency intelligently. Like if I just measure the latency, I could see my latency get really good, really low, but actually all those requests are failing. So that I shouldn't be happy that the latency is better. It's like it, so I need to actually have the right dimensionality of the observability to say successful request latency. And what does success mean? Okay, well, 201, 200, like what HTTP stuff is happening under the hood. So having the right observability is actually pretty tough. Now, open telemetry exists now. That is a nice framework that that instruments with tons of framework. It's an observability framework. I think that's fantastic, and it's changed a lot for the for the industry around being able to get this just correct automatic instrumentation for any client server, any database, any whatever anybody has come up with um to have just like great instrumentation that way. But that was this was before that, and it it just kind of out of the box gave everybody instrumentation that was compatible with our observability tool we had internally, which is Cloud Watch. Um so it just like that's just making that framework just do a lot for you, but also be flexible enough so that you don't feel like suffocated by like a framework that's overbearing. Uh, that's the dance.
SPEAKER_01Yeah, well, a framework can give you um leverage. Um okay, so here's here's something I've been preaching lately to people I work with. Um and it's it's taken me a while, but I've gotten the message across, at least to some people, I think. Um I talk about the concept of universality, and some tools, some things possess universality and some don't. Um, for example, uh Grafana. Uh it it lets you plug in data sources and it gives you visual graphs. It absolutely does not have universality. Um you can customize dashboards and stuff like that, but like no matter how you slice it, you're gonna end up with a dashboard of graphs. Um whereas if you build, for example, a Ruby on Rails app. Application or a JavaScript application, just something using code, it does have universality. Anything that can be built, you can build. And there's a trade-off there where you might have to do some things completely from scratch. And then there are tools which give you like both at the same time, which is it it gives you the leverage, but it doesn't pin you down so that you can only do what the tool lets you do. It gives you a base to build on top of rather than something that uh that that squeezes you in and limits you.
Universality And The DevOps Agent
SPEAKER_00I think that's such a cool and and actually uh this universality is such an like a applicable to like just to full circle this to to to agent uh stuff is is actually very cool because universality is is exactly why I'm excited about and why I joined from like from being on on like uh you know infrastructure service teams to being in working on agents because I see this problem that I've always been trying to chase around improving operations, like improving DevOps, like every every annoying part about building and operating software that's like somewhat tedious, but so important. I've been I've been trying to just solve that. And so because the universality of agents, um, I think we we've figured out can be applied in some pretty amazing ways here around that. Like you mentioned Grafana, like as or like observability tool. Like you can, it's pluggable to so many things. That's one of the things that's so great about it is pluggable, pluggability to so many data sources. Um, but when I'm trying to solve, like observability is a is a means to operations. It's a means, not an end. Like you don't do observability for observability's sake. You you do it so that you can operate um well. And and so we tackled that when the thing that I'm working on currently that I'm the most I'm just so excited about, like over the last 20 years, is is we've made an agent that does DevOps for you, the AWS DevOps agent. And it is um DevOps means a lot of things to a lot of people. Um to us, it means just what the DevOps team does, right? No, no, to us, like I thought I've done DevOps this entire time, like because it developers do the ops. There is no, yeah, exactly, no DevOps team. Uh and so um, and that's what what I I've always liked wearing all the hats. Like I also want to pay attention to security. I want to pay attention to what customers are saying, uh, and and where and so I can make the product better and support them. Like I want to wear all of the hats and I love that. And so with DevOps, like that's that my that lifestyle of wearing all the hats.
SPEAKER_01And yeah, and I just want to say, um in in case not everybody caught that that joke of ours, um, you know, people put those two words together, DevOps team, which is like the antithesis of of DevOps. DevOps is where you put the dev uh the development and the operations together, that idea, as I think of it, of like you build it, you run it. Um so that's why the term DevOps team is so funny.
SPEAKER_00Yeah, I the nothing against specialization and everything. Like for people who do uh who are like are DevOps engineers and everything like that. I think that that specialization has been has been really powerful. Even even at Amazon, where you know we're we we just we're all software developers, there are some who are just a lot better at at automating ops. And that's just their kind of flavor and focus. And and like, and I think I'm one of those, like compared to whereas I'm not as good at other like maybe abstractions and frameworks and stuff like that.
SPEAKER_01To me, the significant thing, obviously, individuals vary in their strengths and inclinations and and backgrounds and stuff like that. Um, to me, one of the significant things has to do with incentives, because if you have all the developers over here and then all the SRE people over there, um the the developers might have needs that the SRE people don't have any natural reason to care about, uh other than like hopefully out of the goodness of their heart, but that's not like why people do things at work. Um and and so that creates really uncomfortable situations. Whereas if you slice that up differently and put some devs and some SRE people in the same team, then the the interests are are more aligned.
SPEAKER_00That's right. And there's this a natural like back pressure and reacting to pain, uh, you know, which is which is I think so important. So instead of like negotiating a contract and everything like that, which is you know, and there are a lot of philosophy, there are a lot of these that work. I've talked to so many customers of AWS over the years, like back and forth about, hey, what do you do? Oh, what do you do? And like, and we just kind of there are a lot of different places and models and everything that work for different different companies. I of course have my I'm you know, I'm very biased and and opinionated about DevOps being this what we're describing as DevOps, like to be the true Scotsman fallacy of like you know, defining well, you're not doing real DevOps, yeah. Okay, so but uh I'm a fan of this model. I think I I think it works with how I like to to work and and I think it results in really good outcomes for customers because the developer teams are just you know, ultimately ops is a and observability are a lens with which to understand the customer experience. And I uh and I really like that. I like being so connected to it to how how are our customers doing? Well, let's let's look at how the service is operating. Oh, yeah, okay. Like, well, I don't know how if customers are happy based on my observability data. It's like, well, then we're missing some observability data. Let's let's measure customer experience better.
SPEAKER_01Yeah, and that's a whole podcast episode in itself, at least. It's probably many, many podcast episodes, is like um, how much of the story uh do metrics tell? Um it's of course not the whole thing. Um the I I had an experience where uh numbers were being cited for months and months, and then finally uh we were like, hey, let's like go and actually talk to some people. Um and it turns out we had totally the wrong idea because we were just looking at the numbers.
SPEAKER_00Yeah, measure it from the right place, got to talk to people, constantly check your assumptions about whether that's one of the kind of cultural things that's
Weekly Ops Reviews And Culture
SPEAKER_00just so important. And I think what helps with DevOps helps a lot is that you know it at AWS teams uh get together every week. Like, and actually all of AWS gets together every week. Every week, like for two hours. And we talk about ops. We talked about what's going on with of things things that are size of formula, like every what what were the wins? What did somebody do that they want to just like tell the world about? Hey, we just used this new tool or we made this new tool, we had this big efficiency gain in the service, or or whatever.
SPEAKER_01And what do you mean? What do you mean everybody in AWS gets together like on a call in person?
SPEAKER_00Uh yeah, every uh we used to be in person, but every every team is represented by at least one person. Um, a lot of people are on uh like a live stream uh watching. Anybody can participate on like we joined these days, kind of we were kind of forced this way, uh, but I think for the better, um, with COVID and everything, we went we went virtual for this, and we've stayed that way. We find that it's easier for everybody to participate uh without the kind of intimidation of being in a large room where you're trying to yell from the back and you don't necessarily want to. So um we have a channel, a Slack channel where we're all talking and uh about it during the meeting. Uh the kind of the side conversations are very interesting and ongoing. But yeah, there are people from from every team, um, every single like not just every service, but every team within every service um who attends uh and and watches and uh and and discusses like what what what went well, like what maybe we'll talk about retrospectives of things that that didn't go well that we wanted to share with everybody and and share how we think. Here's how I think about things that didn't go well and how to improve. Uh like uh and then uh we'll the agenda varies, but sometimes we'll actually what we used to do, especially more was we would pull up a random services dashboard, like in with a metric, a metric dashboard, and say, well, and then we would we would ask each other, like there would be uh a line that would show, say, uh no, here's uh oh, here's an error graph. Uh but these are just the 400s, these are the four XX HTTP status code responses. So those aren't those aren't server faults, those are clients calling us wrong. And people would ask, well, okay, like if that spikes though, like somebody isn't happy, they're not successful. Like, is there some some sharp edge in the service that's causing that? And we'll discuss that. Oh, no, those are just people exceeding their rate limit. Well, okay, like are they happy that they exceeded their rate limit? Maybe you could just increase their rate limit for them. And then we're we have good discussions about what like what limit, you know, about limits around input validation and everything about and that really do get to the question of, okay, is is this the right customer experience? And we kind of challenge each other on that.
SPEAKER_01Yeah, yeah. I so I I like can't help but think in analogies and something okay, so I've I've been like re-studying calculus lately, and I found this really nice video. Um there's this YouTube channel that's just called Math and Science. It seems to be this one guy. Um, and he did a really nice job of explaining what calculus was all about. And he used what I guess is a fairly common example of like here is a graph of um a particle's uh position over time. And then if you take the derivative of that, you get the um let's see, there's there's a position, velocity, and I don't remember what. Sorry? Exactly, yeah. And and so I like to take these things and take like the the derivative of each one. It's like okay, we have a culture of culture of firefighting and and we need to turn that around. We need to like be more preap proactive and have better monitoring and alerting so that we don't have that reactive culture of firefighting. It's like, hang on a second, why do we have a culture of firefighting? One of my favorite quotes is things are the way they are because they got that way. And so, like, if this is the way, like, why are we just looking at this now? Like, why didn't somebody look at this five years ago and straighten this out five years ago? Like, why now and not back then? Unfortunately, it's not very common to dig into that.
unknownYeah.
SPEAKER_00You're describing kind of one of these ops meetings pretty well. Like they this gets second and second derivative pretty fast of like, okay, it's like, and people say, oh, okay, why there's this, you know, something happened in a service. Oh, like well, why does that happen in services? Why do like why isn't this just out of the box? So oh yeah, we didn't have this uh alarm on on like file descriptors or something. Oh, okay, well, why doesn't why aren't those already always the case? And so we do talk about that whole that whole like shift left aspect of it of how do we just get everybody to be better, which which like back to that universality point you're talking about. Like this is like agents are really that you go, there's a lot of the culture you cannot bypass the culture around of Roundups about making things better. There is no magic wand. There's no compression algorithm for experience. But gosh, agents are pretty, pretty universally adaptable in both root-causing issues and plugging into and understanding any system. Doesn't matter what framework it has, doesn't matter what observability provider it has, there's an MCP server for it. It'll figure out how to get the right metric out that it's looking for. Um, for for the shift left aspect of it, scan through an application, look at all the observability, look at all the incidents, look at the code for it, match it against things that we know are are like not good, things that opportunities to improve, and just do those. So this like the both the React and also the improve, the proactive part are what we've built into DevOps Agent uh last sales pitch on DevOps Agent. But it it's like that universality and like being able to adapt to to to really any learn an architecture, learn an application just by looking at everything it can tell from it, uh the whatever tools you throw at it, whatever MCP servers. Um it's just it's working better than I expected almost. Like it's it's pretty incredible the universality of uh of agents.
SPEAKER_01By the way, this is by far the fastest uptake I've gotten on on the term universality and having it make its way into the conversation. Um yeah, I got it from this book, um, The Beginning of Infinity by David Deutsch, or maybe it was The Fabric of Reality by David Deutsch. Both those books are are excellent and they've they've changed the way that I think about programming and just like the life in general. Um life-changing books, especially the beginning of infinity.
Postmortems With Five Whys
SPEAKER_01Um where we're about at time, um there's there's one more thing that I have to ask, but let me know if you get to where where you need to have a hard stop. Um I want to ask about postmortems. Um something that really frustrates me in po postmortems is when a large number of people are brought together for a short amount of time, and the the incident is analyzed maybe superficially, and then out of the post mortem come uh several action items. It's like here here are the seven things we have to implement immediately to make sure this never happens again. And and I'm like, uh, is is this really the the way to do it? I think maybe we should have fewer people for a much longer time instead of like an a one-hour postmortem. You know, some postmortems maybe maybe they call for 30 minutes, some maybe they call for like three days of of picking it apart because different things call for different levels of thought and stuff like that. I don't know to you what what makes a good postmortem?
SPEAKER_00I one of my favorite things to do is to is to write a postmortem doc. We call them COE correction of error. You can take any name. Um, but we have a pretty pretty standard formula for these, but I love writing them. They're a document um that a team will write. Like a generally like a maybe one or or two people will spend a lot of time with the pen, but then that they kind of just can keep riffing on it until with the like as a team and as then a larger team if it again, if it needs more scrutiny, uh talk about it. But first, what makes up a one of these COEs, we call it. Um, it's a description that we we write them for anybody in the company to be able to read. Like to to and and so we want to make sure we're not using too much jargon about oh, the the uh the BSF service called the you know, MVP service called that we do, you know, we don't want to use too many acronyms, or we certainly will put a like a description of it in it. But we say, here's what it is, here's what happened, here's the customer impact, like pre- uh here's the timeline of like what break it down really granular. The timeline includes uh when we learned about something, when an alarm went off, when the impact happens, when we took any action, every when everything. Um, and then the meat of it is the the whys, the five whys. My understanding is this is a Toyota method for for kind of looking back at when something happens. Say, well, uh service fails. Why did it fail? Uh it ran out of something. Well, why did it run out of something? Uh well, it ran out of something because we didn't have an alarm. Why didn't we have an alarm? So you just keep asking why. You don't ask five. And and these answers, it's not just a one-sentence response to the why. Like, that's where I put all the thinking of like, well, okay, gosh, well, how do I really keep this from happening again? And then invariably you think, like, well, how would anybody else like I have people have that kind of, I think as engineers, you really maybe something just built in as being a programmer, but culturally, of you really want to help other people, like other teams, other people not have not face that same thing. That's why people really like writing frameworks, I think, is because you're like, oh, let's help other people do that thing. Uh, similarly with this, like, how do how would I make it so that uh that this like that everybody has an alarm on high CPU? Exactly this way. It's like I had an alarm, but I've alarmed on the median host CPU instead of really, it should always be the well, you should have that, but you should also have the the max, the P100, hundredth percentile host CPU to see like because maybe you're because it's really a distribution across your fleet. So all these little nuanced things. How do I make it so that and who do I talk to to who might have some some leverage, some tool that could be uh could apply this to everyone kind of out of the box? What is the abstraction that we could have? And just I love just unpacking and writing one of these documents that um where you're just digging into the data for a long time. Like this isn't a is a short thing. Um and uh it's a team, like I said, so in terms of review, then um some of these are like it they're always be reviewed as as a team, like uh whether that team's like 10 people or something, but then and also as a as a larger like a group of people, uh you know, maybe a larger team, maybe you bring in outside, help to be, oh, I wonder what this person thinks about how I got here. And so I'll just invite other people to review it. And and some of them, like this is part of that that Wednesday ops meeting I was talking about. Some of them uh we'll actually review every in a given week for for a while. Um, and and as a and talk about it and talk about our own the patterns that we've noticed over time. People will weigh in with, oh, I've seen this happen a bunch. And and uh and so like what do we do to what do we do to keep this from happening for any service? Um we also are analyzing.
SPEAKER_01I don't know, David. This all sounds like it takes a lot of time. That doesn't sound very efficient.
SPEAKER_00Oh, I don't know.
SPEAKER_01I I'm I'm speaking as a character right now.
SPEAKER_00Of course, yeah. Oh, there's no compression algorithm for experience. Like the thing is with like, and this isn't you know, maybe it there is just a a there are different businesses with different incentives. Like I like that ultimately that is totally totally the case, different different requirements. Um, I mean at AWS, like everybody is counting on us to be like we people to be perfect, right? So we we have to that that is the product, like is that we are doing ops so you don't have to. Like over the last 20 years, like that's like that's what we are doing. We are we are your ops team in a way. Uh not not not your application ops team, obviously, but like for the infrastructure, you're relying on so we spend an enormous amount of effort on this, obsessing over this. Um and uh yeah, I mean, we even have I didn't even describe everything. Like there are also teams. If you are a team, when I was on that framework team, for example, or when I was on the observability team, I would look at every one of these uh COEs, like we have it's a database of them, I can mine them and you know learn, like actually with LLMs, now it's even easier for me to mine them for for like patterns. And I'm like, well, if I own CloudWatch, that's the service I'm building, how could I make that just so much more out of the box and easier so that everybody has the alarm that they need all the time? Like, what do I need to build into my service? Like, how can I improve my service by seeing how how people maybe struggle with it or when some things don't always go exactly right? Um, every team I was on, it was like that. Okay, like, you know, when I was on on Dynamo DB, people would say, well, yeah, I had an outage because I like hit a throttling on my table. It's like, okay, how do I how do we add auto-scaling to DynamoDB? Or like make it so we we started DynamoDB with this notion of provisioned throughput because people wanted super predictable performance. So we would pre-provision capacity and like you could because it was elastic, you could call an API to add more or take away database capacity, and we would just do that behind the scenes, but um, we didn't have auto-scaling at first. And so when we saw people run into uh, oh yeah, like it's it's hard to actually monitor this all the time and then dial it up. So let's just build that into the product, this like real-time elasticity into it. So, like by understanding where things go wrong for your customers, you you make your product better. Uh whether it's whether it's going wrong because the service has a rough edge, or whether you know we need to just be better ourselves. Uh, and both both cases apply. So these these these root uh retrospectives, these uh COEs, they we get so much out of this process uh because it is, it takes a lot of time, but we we we really get a lot out of it from different angles, from the team itself, from other teams not running into the same issue, to just making the underlying services that we build for all of you better by looking at how we use it ourselves.
SPEAKER_01And I imagine that the patience and rigor pay themselves back many times over.
SPEAKER_00I hope so. I mean, uh yeah, I hope I hope that yeah, I mean, I hope that people, you know, I hope that we are having the operational outcomes that our customers need. That's all that matters to me. So it I uh we're we're always looking And and wishing we could do more, spend more time on it, uh it'd be better. So uh always looking for that next next tool to give our leverage uh that will make things better. So uh always looking to to to figure out how to how to keep keep that bar going up.
Links And Closing Thoughts
SPEAKER_01Mm-hmm. Well well David, I have about a million more questions for you, um, but unfortunately we don't have unlimited time. Um I have to say that this has been this this episode has had among the highest uh nuggets of wisdom per minute uh uh of of any episode I've done. So I really I personally have gotten a lot out of this.
SPEAKER_00And that's partly because different directions. It's been it's been a lot of fun just visiting different things.
SPEAKER_01Yeah, yeah. Um so yeah, I really enjoyed it. Uh before we sign off, anything that you want to share, links you want to send people to, that kind of stuff.
SPEAKER_00Uh oh sure. I mean, certainly. Um I guess okay, two two things. Of course, AWS DevOps agent, uh, you know, that that's a uh I think it's I I think it's gonna change a lot in terms of where as you code more and ship more so much faster with agents, the the the the bottle the code just piles up unless you can get it shipped and run it. So I think that's very important. The second thing, um, when we talk about what we've learned as AWS over the last 20 years, um one thing that we did a few years ago kicked off. Um we we were trying to help like just capture this like to for ourselves of like, well, what have we learned? Like how what is the right way to build a service and operate it? So it was like a like we we were just doing that for ourselves. We had been a lot of um talks over the years that we give every week that that so how would how do we kind of distill that? So we wrote um we launched this thing called the Amazon Builders Library. We wrote these really long articles, honestly, like they got a little longer. It's like, oh, just write down write down what does it mean to do instrumentation of a service. It was like, oh, wait a minute, that's gonna take me like 12 pages to to touch on, right? And so we wrote we published this Amazon Builders Library, um, uh AWS.amazon.com slash it yeah, AWS.amazon, AWS.com or slash builders library, just just Google it, I guess. Builders Amazon Builders Library. And it uh it has a lot of articles in depth about everything from fairness in multi-tenant systems to how we do CI C D. Um, a lot of ways to go about that. I think we have three articles about CI C D to uh to shuffle sharding of like how to make multi-tenant systems at large scale be um uh seem like single tenant systems. Uh so yeah, I would I would recommend that folks check that out. It's uh uh something we've put a lot uh of in into, but uh you know there's all we're always learning something new, so we will add articles from time to time.
SPEAKER_01Wow. Okay, that is gonna be a gold mine for me and my team. So I already Googled that on my phone just now. I'm gonna be digging into that later. Um Yeah, I I certainly will. Um again, this has been extremely educational. And David, thanks so much for coming on the show.
SPEAKER_00Thanks a lot. Uh oh yeah, and I guess last link is uh yeah, I'm on Twitter, LinkedIn, etc., uh the Barrett Blue Sky, whatever social network is in the fragmented current world. Find me on there, David Janichek. Uh, and uh happy to chat with anybody anytime. So uh but yeah, thank you so much for having me on. It was this was a really fun conversation.
SPEAKER_01Thank you.