Confidently Incorrect — with Caitlin Colgrove

Caitlin Colgrove is the CTO of Hex, the data workspace for building and sharing data projects using SQL and Python that somehow counts a Sweetgreen chef as a power user. She joins Michael, Hadley, and Isabel to talk about what AI agents actually get wrong in data work (it's not the hallucinations, it's supreme overconfidence), why data teams aren't going anywhere, and how she thinks about building products for humans and agents at the same time.

What's inside

• What Hex's Context Studio does, and why it's a data team's new job
• More code is now written in Hex by agents than by humans
• "My job is to vouch for the correctness of the answer" — redefining the data team
• The vibe-coded CEO PR is coming for your data team (if it hasn’t already)
• Soulsborne games as couple's therapy, aka, the Elden Ring co-op report

Transcript

0:02

Welcome to the Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Dig into what makes them tick, plus the insights, experiments, and OMG moments that shape the field. Michael's traveling at the moment, so you're stuck with the producer's voice over this week. On this episode, the Test Set crew jams with Caitlin Colgrove, CTO and co founder of Hex, the data platform that somehow counts a Sweetgreen head chef among its power users. Caitlin joins Michael, Hadley, and Isabelle to make the case that data teams aren't going anywhere. But right now, their job is changing from writing SQL to vouching for the answers. We hope you enjoy. Caitlin, welcome to the Test Set. Thank you. Thanks so much for coming on. Just by way of introduction, you're Caitlin Colgrove, the CTO of Hex. And I'm Michael Chow. I'm the host of the Test Set. And I'm joined by Hadley Wickham, chief scientist at Posit, and Isabelle, software engineer extraordinaire at Posit. And we're so excited to have you on. I feel like Hex is everywhere today, and I'm constantly trying to explain to people what Hex is and why they should use it. I know Hex is included in Claude.ai now. Maybe do you mind just explaining a little bit about what Hex is and what it does? Yeah. Totally. Hex started when we when we started as as founders, we were basically we'd all left Palantir and we were kind of out in the world looking at how people were working with data. And what we discovered was that there are a lot of new patterns for working with data that were emerging. People were a lot more technical. There was a lot less clicking of buttons and a lot more writing of code, SQL, Python, all of that. And the tools out there were Jupyter Notebooks, those are very popular, this is twenty nineteen or so. But in order to actually sort of string together a full enterprise data workflow, like to actually do something useful with your data, people would be doing these ad hoc analyses in Python, SQL, do SQL in a notebook, still had to do the backticks, you know, it was like really not native. And then, you know, that was in a notebook, and it was a locally running notebook with your own Python environment. And, okay, so maybe you do this whole analysis and, like, what are you what are you gonna do? Are you gonna email that notebook to the CFO? Like, that that, you can't do that. So they would take these notebooks and they would okay. They're like, okay. We're gonna turn it into a Python script, and then we're gonna run the Python script on a schedule on AWS, and then we're gonna dump the output into Looker so that the CFO could look at it in Looker so that they could take a screenshot of the chart and put it in the board deck. And that was the workflow. And so Hex was really born of this idea that there just has to be a better way, that there needs to be first class tooling for this new era of of data work. And that was the original product that we built. And then obviously like, you know, in the past couple years AI has become a huge part of that. And I think even though some of the technology has changed, the underlying story is still true. People who are working with day in and day out, what are their workflows? What are the best tools for them? And Hex should be that. I've seen so many engineers in Python use like pandas read SQL, that Yeah. Classic like pandas.read_sql with a string of SQL in it Yeah. Which feels like maybe a little bit unnatural. And then More than a little bit unnatural. Yeah. Yeah. Yeah. In that scheduling dynamic, just all these bits of duct tape kind of like in the process. I'm really curious. I know, like, with the Anthropic integration, Like, what what does that look like for a person to use Hex? Yeah. So I think this is one of the big ways that work is changing. Pre AI, you basically had these big heavyweight vertical SaaS tools, and you depending on what you wanted to do, you would navigate to HubSpot, or you would navigate to Salesforce, or you would navigate to Hex because you're like, I wanna do something with my marketing data, I wanna do data science, data analysis, and that would be where you would start. And so you'd have these big full featured front doors to these vertical applications. Today everybody just starts by like asking Claude or ChatGPT a question. And and we've actually had some really, really interesting conversations with the Anthropic team and the and the OpenAI teams about how they're seeing the, like, the changing and and sort of the emergence of these of these new workflows. And and they really view a lot of these chat applications as a front door. They're not gonna go and build, you know, a first class data science agent or a first class, you know, customer service agent or all of these things. They wanna be the platforms on which those things run. But still today, like, it's very sticky, these habits of just ask your question to Claude. And so what Anthropic released, and ChatGPT has sort of its own version as well, is this plugin architecture. And it's based on MCP, as you might imagine, but they've enhanced that with, you know, sort of more UX around it. So now if you hook up your MCP to Claude, you actually get a interactive Hex experience in Claude. And Claude, you know, the models are actually smart enough to know how to route. So if you go and you have Hex wired up as your sort of data science back end, and you ask a question, Claude knows that the Hex plugin is the plugin to use to answer that. And so it'll actually go and it'll go kick off sort of a sub agent architecture where it kicks off an agent in Hex, which will go look at all your data, it'll go look at all your semantic models, it'll do all this stuff, and it'll come back. And it's actually really nice. You get to, like, sort of see the thinking tokens, you can get an interactive set of charts back. And then if you if you really wanna go deep or you wanna go build a dashboard or you wanna do something further, you can kind of click through into the underlying Hex project that it's been building up behind the scenes. What's a question I could ask that might produce like a Hex widget? Like is it like if I wanted to check the weather in different cities, or like, let's say we just had yeah, like Like what like what's something you have used, Hex, to answer about like, Hex? Well, today so so Hex is typically most most useful for the types of enterprise data workflows where it's all governed in the data warehouse. You've probably done some modeling on top of it. You probably have some dashboards, you know, maybe have your KPI dashboards or whatever. Those it's like sort of classic data workflows, and then you can start to pull in other sources obviously via MCP and other things. But it's very similar to, you know, like pre AI, the types of things that you'd be managing with dbt and Snowflake and all those things. And so it's usually business data. And if you're syncing other sources, those are those are to enhance the questions that you're asking about the business data. So these days I'm asking a lot of questions about how much we're spending on Anthropic. Where are those tokens coming How are people using these these products? Products? And so that's a very it's a sort of com I would call it combination kind of finance, product analytics type workflows. And so basically what we're doing is we're piping all of our logging around the usage of Hex's AI into into Snowflake and all of those, like, fine grained token token counting and, you know, attribution and where those are coming from. And so I'm spending a lot of time looking at, you know, where are we spending the most tokens, are there places where we could be being more efficient. We also have a lot of data around, you know, our internal evals and performance, and so are there areas where we, like, should be using Opus and we can't actually drop down to something like Sonnet or Haiku. So I spent a lot of time looking at that in Hex. We know that, like, models can be, like, pretty sycophantic. Like, how much do you worry about, like, when you come in with a question and there's, like, an sort of an answer that, like, you wanna hear, and then, like, the you know, do you worry about you kind of accidentally steering the model in that direction that just reinforces, like, what you wanna hear? Yeah. I think I think that's actually one of the bigger prob like, model behavior is one of the bigger problems in sort of agentic AI for data science today. And it's a it's a combination of that. It's it's I think it's less of the sycophancy side of things that we see, but we see a lot of confidently incorrect. So the models will, instead of doing what even a relatively junior data scientist would do, which is you start looking at the data, you start exploring, you get an answer, and then you check that answer. Maybe you look at some other, you know, sort of other sources to see if you can corroborate what you're seeing. Models don't do that at all. They find a table, they're like, ah, this table has a column that looks like it should be useful, and then they just go. And so that's one thing that we're thinking a lot about is like how do we give the models how do we push them to explore the space more and how do we push them to sort of self validate in a way that more likely will because often, and you see this even with humans, right? If you start at different points in the data and there's different ways and different assumptions that you might make about how to answer a question, you can end up with just wildly different answers. And you kind of want to look at all of those and understand, okay, what's what's the actual sort of truth here? There's a lot of knobs to turn when making sure that the model is, you know, being a little bit more creative, maybe thinking a little bit harder about the answers. What are maybe the knobs that you're thinking about in Hex to make sure that this is being solved? Yeah. I think the models these days are very there are the the reasoning steps are actually quite good and quite solid. And it you know, if you rewind a couple years and hallucinations and just like them not paying attention to the things that you want them to pay attention to are huge, huge problems and even just keeping them on task. These days I actually find that they're quite creative in taking the pieces of information that they have and putting them together into an answer. Like that's act it's actually like really impressive when you read their thinking traces, how much they're able to like hit a roadblock and overcome it and like cycle through that in order to get to an answer. Where I usually see the problems happening is, as I was saying before, in them having not sufficiently explored the space. And so they're very, very good at coming up with an answer based on what they have, but if they don't have all of the pieces that you need them to have, then that's where they'll, you know, that's where they'll kind of fall on their face. They may even have an answer that looks plausible but it's actually not what you want, or maybe they'll, you know, they won't even get you an answer. So what we're thinking about is that exploration phase. And see this in other modes too, like in coding you have like a, can have like a plan and design phase and then an implementation phase. That's something that we're thinking about how we bake into our models, where you don't necessarily just like start from the question and have them go at it. You actually kind of push them to look harder for things before they come back and start to actually then go and do the analysis. Just to be sure, I'm I'm gonna try to paint in exquisite detail what it looks like to use Hex just because it's it's such an interesting cool experience. But I also think some people when I talk to people, some people listening I think might have trouble. If you've never used Hex, like, conveying it that and and correct me for anything I get wrong, but if I understand it's kind of like you have often like data in the warehouse and say, like, an analytics engineer or some people have tried to transform and kinda curate what you need. And and you might even be a person who doesn't do a ton of data analysis. You know, you might be working in, like, finance or not no shade on finance or, like, another area. And essentially, you go into this notebook that's wired up to your warehouse. And right now, you're able to ask an agent essentially a question. And then what happens is it starts populating cells that run and produce outputs. And some of these outputs even are like filters. So it might be like a drop down to select like something you're interested in or to filter a company that you might want data on. And then essentially, like, you can adjust the filter and everything else in the notebook will kinda update. So you can sort of like build this notebook and explore your data at the same time. But then what I I think is also interesting that I just wanna make sure to mention is, if I understand, you also have an app mode. Is that right? Where you kinda shift from seeing the code in the notebook view to the app mode, where the code sort of, like, goes away or is more hidden? Is that Yeah. Yeah. That's that's all totally correct. I think you can think about it in Hex as the backbone of the product is this notebook. And and we were just chatting about this earlier. That notebook actually under the hood is essentially a graph of transformation nodes that we can run behind the scenes. And then you can take that notebook, which we call project in Hex, and you can do a bunch of different things with it. So you can manipulate it with the agent, so the agent will actually edit that graph, it will build up that graph, and you can do that just to do sort of some exploratory analysis. But often what you want, and often, and this is actually one of the huge pain points that Hex was solving in the early days, is, you know, data workflows don't end with getting a number. They end with making a business decision. And so the way you get from I have an answer from the data to we've actually decided to do something, there has to be sort of a collaboration and presentation and sharing step. And so what we've done with Hex, it used to be excruciatingly difficult to go from those analyses to that, know, screenshotting charts in a PowerPoint and things like that. And so what we've done is we've actually layered a presentation layer directly on top of the notebook. So it's exactly the same and, you know, that, you know, it's a typical more dashboard like experience I would say. We call them apps because they're more, they can be more powerful under the hood than what you would expect from a traditional dashboard. And then you can also really extend this. So this has actually like been such a versatile paradigm for us. So Hex has the notebook and the notebook agent, which is really geared towards technical people. It's people who want to be in the code. Even if the agent's writing it, they want to be looking at it and and and working with it. But there's a lot of people in the organization who don't want to look at the code and but also still want to be able to take advantage of the power of the the Hex agent. And so what we have is we have our our conversational agent, which we call threats. But under the hood, it's actually the same as the Notebook agent. And so what it's doing I love it puts a different hat on. Yeah. It's a skin. Yep. Yep. But it's actually building up a Notebook behind the scenes. And this is really really powerful because it means that if you get to the end of this conversation and then you wanna do something with it, maybe that thing is share it. Well, great. You're in a notebook. You can like make the same types of dashboards, things like that, that you could before. Or maybe you're like, hey, I don't know if this is right. I actually wanna send it to the data team to check my work. Well, the work's all there and then I'll put behind the project. Yeah. Hadley, I know you in R, it's super different. Right? Like, the analysis the kind of workflow, like, versus a notebook where you're sort of like, this workflow is so great. If I'm gonna say, it's kinda like, whoops, I made a dashboard. Like, you do your notebook stuff, but then you have a dashboard at the end. Yeah. I think it's yeah. I think it's really interesting because the sort of heritage of R is this, like, REPL, this read eval print loop where you're in the console trying stuff out, and that's kind of your primary, like, means of exploration and experimentation. And then as you discover things you kinda believe to be true, like, that's when you record them, like, into a script. And that's kind of what led to R Markdown and Quarto, which is, like, you've written this mingling of prose and code, like, much like a notebook, but you then you run the whole thing, and that produces your output. And that that has some, like, big advantages, like, kind of prereactive notebooks. Very easy in a Jupyter notebook to kinda get into this state where you've, like, run some cells in, like, different orders, and there's no way to reproduce what you've done. Like, no way to have that problem with R Markdown because you always run it from scratch. Like, there's a different set of problems, obviously, and a different set of trade offs, but, like, that's the one problem you never experience. Yes. And then I guess there's often a dashboard phase after Yeah. And so the the thing that you produce might be a document. It might be a dashboard. It might be a presentation. Like, I think that that same, like, underlying thing, and you're putting on, like, different hats for different people, that's just that's, like, such a valuable metaphor. I mean, we we kind of often joke in engineering, I guess. I'm like, oh, that's a DAG. That's a directed acyclic graph. Like, everything's, oh, that's an immutable DAG. That's a reactive DAG. Like, oh, wow. What a surprise. Like, we've created a DAG, and it turns out to be the right model. Like, it's just And sometimes at obviously, the Hex notebook, as we've been talking about behind the scenes, is DAG. And as the as the cells within the notebook got more and more complicated, we started to realize that the cells themselves needed to be DAG. And so then you have DAGs and DAGs and DAGs. It's DAGs all the way down. And, actually, you know, what is a DAG of DAGs? It's just A DAG. It's kinda like every it feels like every problem with an LLM, you solve with, like, another LLM. Like, you don't know if your LLM is doing a good job or not. Well, let's have an LLM a different LLM judge, but it's doing a good job. It's yeah. It's like turtles all the way down. And another thing just just to not to keep buttering up Hex, but OpenAI, they released their one of their reports was using Hex. Is that right? Like a state of data Probably Anthropic. Anthropic. Yes. Okay. Yes. I gotta be careful about my Yeah. Do you recall what that was? There was like a I don't know specifically what you're referring to. There's a there's a few times where we've seen Anthropic kind of use Hex in Pops up. Yeah. Because, you know, they're they're Hex customers. Yeah. And sometimes it's fun to like watch their live streams and like spot the Hex chart. Yeah. Yeah. For Hex. Yeah. Yeah. That's cool. And I know I know you mentioned Sweetgreen as an example. Oh, yeah. This is a recent one. Hex Hex is like the whole spectrum of customers. You it's like very traditional data science all the way to some of these use cases that you would just never imagine. So Sweetgreen just signed on as a Hex customer and this is the wildest one that I've I've heard in a while. They use Hex to monitor the temperature of their ovens so that they can correlate customer complaints about the temperature of their food to, you know, what's happening on the ground in their restaurants. It was wild. Every now and then you come up with you can't you hear some of these things from customers. It's actually one of the really fun things about building a product that's so flexible and so horizontal. Did the head chef have, like, a lot of data experience? Like, what's You know, I'm not I'm not sure. I I, you know, I I actually haven't met him him personally, but this is one that we heard kind of from our field teams. Yeah. Yeah. Like, one of the things I've sort of been, you know, thinking about, like, how like, as these, you know, people who never would have, you know, worked with data in this way before because they couldn't program, like, and now they can because they can write in English. But, you know, there's a lot of, like, vocab and, like, ways of thinking about data that you don't you you don't have and you can't express in everyday English? Like, how do we start to, like, train those people up to give them, like, the basic kind of like, that kinda gut feel for, like, I should be a little bit skeptical here or, oh, what I what I need right now is, like, a smooth or, like, I need tidy data? Or how do you all think about that? Yeah. So I think this is an unsolved problem broadly, in that I think building these agents to behave like a data scientist, truly, I think we haven't gotten there yet, and there's a lot more work to be done there. But I actually think that this is one of the big breakthroughs broadly in terms of, you know, AI for data and really enabling a much broader spectrum of people to work with data. Because, yes, there's, you know, the actual can you write code aspect of it. But if you look at sort of the prior generation of data tools, even ones where you don't have to write any code, people still don't use them. Like, the laypeople in the organization do not use them. Like, we talked to a Looker, an ex Looker customer now, they're an ex customer, and they said, you know, they bought Looker for everybody. And in practice, it was like they looked at the last ninety days and only two people in the company had made like new dashboards in Looker. And when you like unpack that a little bit, it's it's just buttons. Right? You don't have to write code. How hard can it be? It's actually that doing data work is just inherently very complex. And so none of these tools, code or otherwise, were walk up usable. There was just too much foundational knowledge that you needed in order to be able to be effective in the tool. And so one of the big breakthroughs besides the, you know, LLM writing code bit is actually imbuing the agents with that knowledge. And today, that's already a lot of what we're doing with the Hex agent, is like teaching it not just like, you know, how to write code, but you know, how to work with the data, you know, your tribal knowledge around the organization of what data is where, all of those things you bake into the agent. And so you're starting to see this pattern across, you know, across agents and general purpose models with skills and things like that. Again, I don't think we've totally nailed it because I think even with a skill, some of these more specialized domains like data and data science, the models haven't been post trained on it as, you know, as rigorously as they have been with coding. I think there's like, you know, still a fair amount of headroom. But it's like, even if it's, you know, sixty, seventy percent, it's still massively better than someone who's never really thought about how to work with data before. So our internally at Hex, some of our we have we have this feature that we call Guides, which is basically basically Skills. And some of them are like, you know, this is how this dataset is shaped and things like And some of them are, you know, this is how we do data analysis at Hex. These are the types of things that you should do. And the models are pretty good at adhering to it today. But again, as we've been talking about, if there's any kind of like judgment or anything that they would need to off road on, that that they're like much, much less good at. It is and, like, I think one of the things a good data scientist has always done, like, you come with a question, and then the data scientist, like, helps you turn that into a question that you can actually answer, like, with the data you have. And, like, you know, if you just go into the data warehouse and you just start asking questions, like, you you just don't even know, like, the right questions to ask, and that's such a valuable kind of and we're starting to get that a little bit with LLMs, I think. They can at least help you brainstorm and kind of say, oh, hey. You could ask this instead. But, yeah, it's it's it's interesting interesting time to be a data scientist. The other thing that we do in Hex that I think is is is very powerful is the agents have access to not just the underlying data warehouse, but the other things that people are doing with the data. And so some of this is gonna be garbage, just like, you know, people not knowing what they're doing. But, you know, you may have that company dashboard that gets looked at thousands of times a day or whatever. And that is actually a very rich source of context for the agent for how you should do something. And so when a user comes in is like, I wanna know something about revenue, they may not know anything about how we actually calculate revenue, but that's actually encoded in various places in your data workspace that the agent Which is yeah. Which is because that I mean, that's also how humans work. Right? Often that's not documented. You have to go to another successful project and, like, copy and paste that code. Yeah. Yeah. That was an interesting point I think I saw posted recently was, like, this idea of maybe, like, some of your best documentation is just the other projects around. And the the agents it's if I understand, y'all made it so the agent can essentially, like, look across projects, so that if it doesn't exist, like, in the documentation or in the models, it can sort of figure out how someone else maybe, like, went off road or use some kind of, like, knowledge in their head to do something? Yeah. Exactly. I mean, the the idea here is it's always been extremely difficult and basically impossible for humans to exhaustively and thoroughly document their entire data ecosystem and then maintain that and keep that keep them up to date. So what are the other places that you can go to understand, you know, how to use particular parts of data or how how to do particular types of analyses? And, you know, it's it's a range because when you have when you have the the documented context, for the most part, that's like highly trusted and that's something that the agents can just rely on. But then everything else is this range from you could go all the way down from like, oh, I'm just gonna like look at the SQL queries that were executed against this table, all the way up to, you know, you have a project that, you know, is highly used and maintained by the data team. And in Hex, you We have this mode called endorsements where basically the data team can put a tag on it that's like, this is Oh, funny. This is like a less Gold gold star. Yeah. Yeah. Essentially, yes. And and all of this is to help guide the agent towards you want it you want it to be able to take advantage of all the context that it's able to, but you also want it to be able to distinguish between the highly trusted parts and then the the less trusted parts. Because I think that that is, like, one of the risks of agents because they can be so much more productive. You just get stuck in this loop where they kinda get worse and worse and worse, and a greater and greater proportion of your internal dashboards and stuff are written by agents, and they're just learning from other agents who made mistakes, like, fifteen months ago. And, yeah, like that that kind of idea of saying, like, we also need to give feedback. Like, these are the really good things. Like, like, copy from that, not from. And and this, I think, is one of the key sort of new jobs to be done for the data teams, if that makes sense. And a big part of of Hex that we haven't really talked about yet, but is relatively new post agent, is what we call the context studio, which is basically about observability and management of all of the context for all of the agents. And so you can see what people, you know, what the agents are actually doing. For each thread, you can actually see what context the agent is using. And then we have a second sort of like a separate set of agentic workflows around managing the context, around extracting the pieces out of these things. And then putting that, feeding that back into the like highly trusted, highly governed context to kind of avoid this like spiraling problem of, you know, the agent just kind of deciding that that's the way to do something and then not producing sort of being a self fulfilling prophecy over there. You can actually like break that. You can see that it's actually doing something that's that's incorrect. And then update your human managed context, which is prioritized, you know, more highly above the kind of ambient context for the agents. I was gonna say, I'm really curious. I read recently that now more agents are deploying code on Hex than humans. How do you manage as someone who obviously cares so much about their users and, like, is giving people such a beautiful suite of tools? Like, what are you making for agents versus what are you making for people nowadays? Yeah. It's a really good question. I think in the early days of building the Hex Agent, it was very clear that the tools that we'd built for humans were actually also very good for agents. And so there was a lot of overlap. You know, we had this modular notebook structure, looks a lot like tools that you can give to the agent, and the agents just took to that very naturally. And that was like the first generation, I would say, where it was pretty pretty one to one. But now that we've had this sort of crossover moment of more code is being written in Hex by agents than by than by people, you start to think about what that implies for the rest of the system. And you still have, you still want the human interfaces because there are still times where people are gonna come in and tweak things. And you know, it's it's not it's not fully fully be a natural language. Like sometimes typing a whole sentence is just a very inefficient way to do something that's, you know, as opposed to clicking a button. But at the same time, knowing that over time more and more of this is going to get produced by agents, it actually gives you a lot of opportunities to build things that are useful for agents but less useful for humans but are very powerful for the agents. And so, you know, one thing that we've thought about in Hex is could could we give the agents like slightly different all all of our cells are built to be, you know, they're they're very sort of like easy to inspect. You get a little preview. You get all these nice UI features. Agents don't need that. And actually they're quite expensive. So if an agent is kind of building up a project, could you give it a slightly different set of tools that would have different characteristics? Maybe, you know, less data warehouse load, maybe it runs faster. Maybe it's just honestly, sometimes agents like to work in a different way than humans. They've been trained on a certain distribution of data, and it's not exactly what our human brains have been trained on. And so I do think that there will be some some divergence there. But the, you know, the power of Hex is that these things are, you know, shared across like, they're they're interoperable. And that way, I think think that will continue to be true for a while. One of the things I really like about the kind of context studio is it feels like Hex is sort of thinking, well, like, the job of, like, what a data team or data scientists do is changing. This is gonna be one of the things where you give value to your organization as opposed to, like, there's also a lot of startups out there today, which their pitch is basically, like, buy us and fire your data team, which, you know, as a data scientist doesn't feel very nice. Well, we were talking to a head of of data, and he said something that was really interesting. And I think when you frame it in this way, it it made it really clear to me, you know, why data teams are not going away. He said, my job is to vouch for the correctness of the data and the answer that you get. Like he was a he was sort of, you know, was the head of the data team and obviously like that's slightly slightly different role than ICs on the data team. But when you frame it like that, you're like, oh yeah, actually like in the abstract that is when you when you don't have people clicking buttons, that is the job of the data team. The job of the data team is is to make a system that produces correct answers. And today, a lot of that has to do with instead of, you know, being the person who writes the SQL that is correct, it doesn't have bugs in it, how do you set up the context layer such that the agents can arrive there themselves? I mean yeah. I mean, like, the depressing thing is, like, definitely in corporate America, there is, like like, having the person to fire when something goes wrong is, like, important. Well, I mean, even more positively. Like, someone has to own the stuff and be like, yeah, it's my reputation on the line that this is correct. Like, you can't just distribute that to a bunch of, like, nebulous agents that no one really understands. Yeah. And I think I think ownership is not just like, it's my reputation on the line, but it's like, you know, my my job is to make sure that this functions. And because it's, you know, in in organizations, like this data is being used for, you know, existential, answering existential questions, like, the business needs someone to care that that is correct and to, you know, build the system that is correct rather than just kind of assuming that everything's gonna be fine and figure it out. But yeah. But it does feel like data the data teams are gonna get more, like, unhinged analysis from execs. Like, that's also gonna have to be part of their job. Like, someone has done something that they think is really cool. Like, you know, that and and it's cool it is really cool Yeah. That more people in the org are, like, interacting directly with the data. But the consequence of that is, like, more people are gonna be making mistakes, and the data team is gonna have to, like, walk them back. Yeah. You already see this you see this in in software engineering. Like, you know, a CEO will come in and, like, you know, use Cursor or whatever to, like, vibe code some insane thing and then, like, make a pull request because, like, all the, like, traditional barriers to that happening and then the the engineers are kept to code and like review this thing. I I do think you'll start to see a similar pattern like that in in data where all of a sudden people have all these really powerful tools at their fingertips and they don't know how to use them. And I I think there's a couple things there. One is, you know, in Hex, we're thinking a lot about building review tools, both, you know, manual and also agent assisted review tools so that that process is as painless as possible. But I also think it's just another indication of, you know, where there will still be value. Like, there are there are times where you actually want an expert to do the analysis. There's many, many ways to do the wrong things with statistics. And if you if you have a very important decision, there will still be a a time and a place for for people to kind of do those things and not just have someone, you know, fire off an agent query and and, you know, not not be able to really think critically about the the result they're getting. It's it's funny you say that because our CEO got in trouble with our IT department for an app he vibe coded with HR. But I think it's so, like, maybe empowering and lovely to think about, you know, data scientists are coming with all of the the muscles for, you know, really productive AI usage. Like, data scientists do a lot of experiments. Data scientists know you look at something, say, wow, this is bad. Like, let's scrap it all and start again. So I think there's something really interesting. And especially with the aid of things like review tools built in the loop, you know, no better time, question mark, to be someone who's writing code to do data science. Yeah. And I think there's a lot of hype around AI engineering and and things like that. And we internally at Hex have an AI engineering team. But one thing that I've observed is sort of the lines between our AI engineering team and our data team really blurring and starting to overlap. And so, for example, a lot of things that we'll do internally for our ourselves to build our own product and build our own agent, We'll build, know, we have evals, we have various other things that we've built. You know, we'll we'll look at those and be like, oh, we should actually take those and put those into the product for our customers because these workflows of, you know, building a highly trusted agent actually, you know, converge over time. Yeah. Who runs Hex's internal context studio? Who's the person curating that? Yeah. That's that's our data team. Nice. Yeah. Okay. And they are, like, our number one dog fooders for obvious reasons. Yeah. And do you see, like, the lines blurring elsewhere? Like, it you know, not just because you know, I I guess, like, in our like, we are starting to see, like, designers who previously would create, you know, like, a beautiful thing and turn it over to engineers to implement. Now they can give a work deliver a working prototype or, like, product people who previously were like, hey, engineer. Can you implement this? They just go away and do that. Like, are you seeing that blurring of lines in your engineering org too? I would say absolutely. Our but I think it's a little bit less dramatic at Hex than maybe at other places because our org has always kind of been designed like this, if that makes sense. Most of our PMs used to be technical, like, actually hands on technical in some some capacity. All of our designers, we, like, give them an interview that's designed to be like, can you do light, you know, CSS and React work? Like, we don't want people to just be completely boxed out of a core part of the of the technology. You know, our front end engineers and our back end engineers kind of flow back and forth. Everybody has their spikes. You know, it's not like we're expecting, you know, a back end engineer to be, you know, deep in React. But we don't want people to have these really hard boundaries because that can often create just like a lot of really annoying inefficiencies in the organization. And so, yes, we are seeing that blur. We're sort of seeing that intensify, I would say. Like, you know, designers and product managers are like can go much deeper of in what they're able to contribute than than they used to be. But, you know, we hired a lot of people who are excited about that. So I think that's like just strictly a good thing. Yeah. Yeah. It's it's interesting and maybe this is a nice segue into to a less kinda AI more style topic, but Hex has incredible style. Hex is just cool. I don't know how to explain. Like you go to the Hex website and it's been like this for years and years and years. I feel like the website, cool. The the booths at conferences, unhinged. It's like In a good way. In a good way. I'm sorry. Like a a retro dining booth. Hex is throwing like magic shows. What I I don't know how to ask this. How did Hex acquire such immaculate style? Well, I'm gonna have to give some credit to some other people in the organization for that one. My cofounder Barry, the second person we ever hired was Adam, our head of design. And I think, you know, those two have been kind of the masterminds. Also Annie, our our head of brand design, have been the masterminds of a lot of these things. In some ways it's down to specific individuals and having particular individuals who have these kind of off the wall takes. But I think also it's about the culture that you build as an organization and and who you find and and us trying to bring in people who are creative, who don't take themselves too seriously. And when you have you have some of those things, it creates an environment where you can be generative and think think differently like this. And and yeah. So it's it's both there's like some very, very talented people that we have and also, you know, the team that we've built around them. Yeah. I really like the Hex. Like, the Hex color scheme is like so like, it really stands out in, like, tech. No one is, like, pink and cream and brown. Like, that's I mean, probably now that's the next next trend. But, like, you know, I I kinda love like, we we spend a lot of effort at Posit coming up with our logo and stuff, but just, like, blue and orange is just, like, such a, like, traditional kinda tech color scheme. It's, yeah, it's it's just fascinating to see, like, as we grow as a company, how these kind of forces of, like, unification and kind of corporatization, like, pull on you towards these just, like this is, like, the safe safe choice. And Yeah. And some of that just has to do with you have to scale your content and branding engine. And so early on, a lot of creativity can actually just come from particular individuals. And you almost always get certainly a more, like, unique result when you're when you're doing that, you're doing that in small teams. And then you get to a point where it's like, oh, you have to write all of this content and you have to put out things on Twitter and LinkedIn to get the, you know, SEO or AEO or whatever we're doing now. And you, the just the amount of stuff that you have to produce can't bottleneck through your main creative engines. And sometimes you can scale. Like, you you obviously scale your brand voice and and and your brand styling and things like that, but you can't do it perfectly. And so I do think that, you know, we're we're trying to fight really hard against this and trying to, like, balance the two. But you do sometimes sort of inevitably have feel this, like, regression towards the mean pull, just like the more and more stuff you have to do on the branding side. Yeah. And it's sort of also interesting just as the company grows, like, the the amount of visibility you have within all of the parts of the organization. And and this, like, sounds weird coming from a data person, but, like, you are forced to start developing, like, metrics so that you can get some oversight. And then, like, metrics are good, obviously, but they're also bad because they kinda, like, trap you into these, like, predefined like, how like, how do you, like, wrestle with that as, like, clearly a data driven company, but you don't wanna be, like, too data driven at the same time. Yeah. I think for me, a lot of it a lot of it has come down to being able to sort of shift the altitude at which you operate and then knowing when is the right time to be at what altitude. So obviously, it's extremely important, you know, as a senior leader of a company to be thinking about strategy and long term time horizons, and I spend a lot of time, obviously we spend a lot of time talking about AI. That's where I spend our AI strategy is where I spend a lot of my time thinking. And then, you know, you're intermediated through layers and layers of organization. And sometimes there's something down at the bottom that's like not going well. And if you've built a a good organization, you'll know about that most of the time. But sometimes you don't. And sometimes you just, you see things in the data or you see things, you know, that that don't feel right. And I think for me, cultivating that kind of like instinct of, hey, this section feels like it feels like there's something not working here. The vibes are off and being able to just do a deep dive on that. And in order to be able to do that, you have to know enough about the things that are like happening on the ground. And so for me, you know, when I first started really building up a bunch of our AI programs, spent, and I still do actually, like, I read the papers, like I spend a bunch of time in the technology and that's to both to like be able to see the big picture, but also when something's not right on the ground to be able to kind of go all the way down and like help debug, you know, where something is breaking down. Yeah. As CTO of Hex, I'm so curious. What is your what does that look like? Like, what's your day to day look like? It's a pretty big range of things. I think one of the funny things about being CTO is sometimes I feel like my job is more about humans than it is about technology, even though technology is in the title. And so a lot of it is about when I'm not sort of thinking about the AI side of the AI technology side of things, a lot of it is about like how how are we running a functional technology organization? And there's engineering, there's data, there's there's our AI research side, there's security, there's all of these things involved in that. And a lot of that day to day is actually not necessarily like making technology decisions. Like, I've hired smart people. If I've done especially in areas like security where I'm not the expert. If I've done my job right, I've hired people who are better at making those technology decisions than I am. It's about, you know, setting up the the humans to do their best work. But as someone has like come up from the kind of technologic, like the technical side, like how do you like, you still is there some amount of programming, like, you just want to do to kind of like scratch that internal itch? You're like, let me cook. Yeah. Yeah. For sure. A lot of it I try to do on sort of side less, you know, mission critical critical path things right now. Some of it is actually fun to get. So when I was in college, my concentration was then called ML, and then I did my master's in NLP. And so I've always really been interested in this side of the house, but this was over ten years ago. For for people who might NLP is natural language. Processing. Processing. Yeah. And it's all about It's basically all of it's like pre LLMs, pre generative AI, how you operated with natural language in, you know, in a statistical sort of modeling sense. And actually a lot of the techniques back when I was in college were sort of hilariously primitive versions of what we have today, if that makes sense. So But you know, it was it was pretty clear, this was twenty twelve, was pretty clear that this was like, the stuff that they were doing was like, kind of neat. And you could like see some interesting properties of language with it. But like, your ability you you couldn't generate text. Like, your your ability to actually do things, like machine translation was like a big area of study back then. And it was like, you know, comically bad. You know? It was like, you could get the gist of what the what the text was saying, but it it wasn't anywhere approximating real speech. And it was pretty clear to me that like, the techniques that we were learning about as state of the art at the time were like, not it. And and so I kind of like left that and went into just core software engineering because I like, I wanna do something real. Wanna do something that's like practical. And then sure enough, a few years later, Attention Is All You Need was published, and then it's kind like off to the races from there. But so so sometimes it's me getting hands on and building things. And sometimes it's me just like getting back into that has been really fun. And spending some time not just like reading blogs, but like reading the actual underlying papers and like under understanding understanding the research side has been has been super fun over the last couple years. Yeah. Oh, that's interesting. So you've been kind of diving back into NLP. It kind of like reignited. Yeah. I mean, well, now now it's just, you know, all generative AI. And it's like it's sort of a blurred line. Large language models do so many more things other than just NLP. But yeah, a lot of a lot of it is like, again, at at the foundational level, like, you know, similar techniques. It's all just matrix multiplication under the hood anyway. But it it is interesting, like, you know, like, named named entity recognition. Yeah. Used to be like this really challenging thing. You have to have a massive tech stack specifically for that, And now you just, like, throw it at any, like, random It's it does a good job. If I'm a barbarian, what's named entity recognition? Like, you know, if you've got, like, a blog post and you just wanna extract all the people's names that were mentioned in it. Like, that's you can't do that with a regular One of the hard problems naming things. Yeah. You know? Yeah. Extracting names. Extracting names from things. Like, I remember a very long time ago, Hilary Mason had this really cool blog post where she scraped, like, two hundred chocolate chip cookie recipes and then created, like, the average chocolate chip cookie recipe, which she then made. And, like, back then, that was, like, that was a lot of work. I was like, wow. That is so cool. And that is now something I could do today in, you know, like, thirty minutes worth of Or less than that. You just ask ChatGPT for a talk with university, and that's basically prevalent. Yeah. Ben Stancil mentioned something about this too where, like, and having to do with data work like support. If you have like thousands of support tickets, you know, as a data team you might have used to go in and try to analyze it and maybe like calculate a number from support tickets. He mentioned yeah. Like, nowadays, you might even just dump your support tickets into, like, ChatGPT or something and just ask, like, your question directly. Whereas before, we were trying to roll up numbers to try to kind of Yeah. You try to have to do some sort of like In between. Yeah. You have to like take the unstructured thing, and then, you know, run some models over it, maybe some classifiers, maybe some sentiment analysis, maybe you're like extracting specific signal from it. And then you get this like very, very limited structured representation of it, topic, you know, topic clustering or whatever. And then you like try to figure that out, figure out what's going on based on that. And and some of that actually is still very complementary I would say because, you know, LLMs are, you know, nondeterministic and they're not very precise. Especially for things that are like, if you're just dumping a bunch of unstructured data into them for things that are require, you know, numerical calculations and things, like marrying those two things can be challenging. But for things like, you know, what are the most common, you know, user complaints? I've done this in Hex, on Hex Data. And you can it's it's actually, like, very, very fast and very powerful to pull out that kind of signal. Yeah. Are you saying, like, user complaints you can get surprisingly far by putting a lot of the feedback in and just asking. Yeah. Yeah. Just get straight to the kind of where you're going. Yeah. And you probably want to move into the more structured well when you're getting to the point where you're like, trying to be like, okay, well, what fraction of users are asking about this? And, you know, you know, how does that compare to, like, time to resolve the tickets and and things like that, then you probably do want to start to shift into, like, a more classically structured data. But often you just want, like, imagine, like, someone reroll the support tickets, and they're just gonna give you the vibes. Like, that's super useful even if they can't tell you, oh, like, twenty five percent of the support tickets are about blah. Yeah. Yeah. I like I like it's like a alternative data science workflow. If you have this like, from like R for data science, you have this workflow that's like import, tidy, transform. I like the alternative, which is like copy paste. Yeah. Just try to get to Just give me the vibes, and then you, like, figure out then you can iterate. But I think I even find that with, like, some of my projects where I'm just like, hey, Claude, just Claude Code, just do this. Like, look at this transcript. Do this stuff to it. And then I'm like, okay. This this it feels like this has teeth. And then I'm like, okay. Now Claude code. Write the code that will do that for me, like, reliably and reproducibly. Like, that's such a just to be able to do that very, very quick initial prototype to say, like, is there anything here that I should bother exploring further? You have come from, you know, cofounder, your CTO. I'm really curious, like, what does success look for you look like for you, especially, you know, in the age of AI? Maybe how has that changed throughout your career? Yeah. I think starting a company has really forced me to like disentangle myself from classic indicators of success, if that makes sense. And there's a couple reasons for that. One is you'll just drive yourself absolutely insane. You're like, that company has how much revenue and they raise how much? Like, you just like, you just can't compare yourself. Like, a lot of us, you know, come up through school. You're basically like ruthlessly, you know, trained to optimize this like one metric and like compare yourself against all the other people to get into and like you just have to just let all of that go. Like, you just have to let the numbers go and and focus on, you know, what you can control. And I think the other thing about being a cofounder that I found is I found that the hardest part about being a cofounder is that you are, you know, the job changes every six months. And and so what that means is just as you're getting your feet under you, you have to do something different. And so you're always in this mode of like, I'm not good at the thing I'm supposed to be doing right now. I'm learning how to do this. And for someone who, you know, generally, you know, had relatively successful career prior to that, you're like, you're used to operating in areas where you're good and you're strong and things that you were hired for and you interviewed well for and you're like, great at software engineering. And being a cofounder is just like not like that. And so all of these like classic ways of being like, I am successful, just kind of go out the window. And I think for me, I just kind of keep coming back to a couple different things. One is, are we building something that, you know, is doing good for the world, that is a useful contribution? And then am I enjoying what I'm doing? Is this a good use of my time? And as long as I feel like and you know, you don't have to enjoy everything that you're doing, but like sort of in the macro. And as long as you feel like those two things are are happening, then I think that's like a pretty it's actually I feel like it's quite rare to really be nailing both of those things. And so that's kind of how I've been thinking about my own personal career success. Do you feel like there was one pivot that was like, this is the hardest thing or this is the easiest thing? Was there one pivot that felt like it hit harder than the rest? I don't know. I think in some ways, I kind of, from the beginning of Hex, had to make that pivot even just to start the company. And it was a bit of a sort of mental hurdle to get over because prior to that, my career had been extremely linear. It was like IC and then manager and then manager of managers and then like, you know, he's just kind of like walking up that and taking a step away from that. And and also candidly, like, you know, I I went into Hex being like, I don't know if this is gonna work. Right? You know, most startups don't work. We may all be back on the job market in a year or two and that would feel like failure. But I think getting kind of getting past that and being like, is, you know, is that the worst thing in the world? Like, we try something and it doesn't work out? Like, maybe you're like slightly embarrassed? Like, I don't know. Like what like what what is the actual negative outcome there? And kind of pushing through that feeling of like, what if it fails? Well, what if it does fail? So what? We all go, like, we're all software engineers. We all go get jobs. And and I think that was probably the biggest mindset change and even just like jumping off and then, you know, having to kind of repeatedly remind myself of that over the years. One of the frameworks I found really useful is, like, this sort of idea of maximizing versus satisficing. Like, it's I think I'll feel like, you know, data people, people have, like, had good grades at school. Like, your natural thing is to always, like, find the best, like, maximize. Like, I find this a lot when I'm trying to, like, buy a hotel somewhere. Like, I do so much comparison shopping because I'm trying to find, like, the best combination of location and price Hotel matches. Facilities. Is that Exactly. Whereas the this this alternative approach of, like, satisficing of just finding the first thing that's gonna be, like, good enough, like, that's such a powerful, like, mindset change. Like, you don't have to be like, not everything in your life has to be the best, just like finding the things that, like, make you where you'll be, like, happy and satisfied. Like, that's There's an interesting story of, like, the, you know, step one to two feels hard, but what was way harder was step zero to one. And, you know, what what mindset mindset shifts do you have to do to even get to a spot where you're at zero versus, you know, continuing to just go like, oh, well, I'm at one. I can go to two. I can go to three. It's like the same with, like, growing a team. Like, the first person. Like, going from zero to one is really hard. From one to two, that's not so bad, and then it just gets gonna get on the downhill slope a little bit. Yeah. This is a pretty common piece of coaching that I I give to engineers, which is it's it's basically telling them, like giving them permission not to be miserable. I love that. Yeah. They they often sometimes in in engineering sometimes, again, sometimes you gotta do what you gotta do. But, you know, sometimes someone like, we had an engineer who moved into an engineering manager role and was miserable. They hated it. And I was like, you can go back. Like, we actually intentionally have set up our org structure such that like senior engineer and engineering manager are the same, no pay difference, they're just the same level, just a different title, different role. And coaching people through their first moment of I tried it and I didn't like it and I didn't like make it all the way to something that feels like a success moment, but that's okay. I don't have to get there if I'm really unhappy with what I'm doing. I can just stop. And that, especially like we've been talking about for for people, high achievers, that can feel like a failure because sometimes you're in a you're in a spot where you're actually not doing a great job at the thing that you're supposed to be doing, and you feel like you have to push through to get to a point where like you've made it. But I don't know. We only you only live once. Right? Like, you know, what you do with your time matters, and there's only a certain amount of like pushing through. And and it really depends on like what the underlying reason is that you're not happy. There's like lots of different reasons, but usually I see this in a case where someone just like, they don't like the role. And that's not a failure. That's just like you would rather be doing something different. Obviously, like, there's differences where, you know, you just have you sometimes you do have to power through, like, tough times in the company and you just have to, like, make it out the other side. But that those are temporary things. Like, the role doesn't change. The role is still the role. And so if you don't like it, don't do it. You mentioned like the kind of uncertainty of starting a startup reflected in this person's shift to engineering manager. Like, mentioned, you know, the the fear of starting a startup and thinking like what if it fails? And then being able to say, like, so what? Like, I tried it, and I'll still have other things to work on all I've learned and growed. And I think that, like, mindset shift between, like, this experiment failed versus I'm a failure. Like, that's Yeah. Really Yeah. Really big difference. And to coach an engineer on that, that, like, jumping from to being an engineering manager is a similar, like, experiment as starting a company. Because it makes sense for that person, maybe it is probably as big and as scary of a jump, I'd imagine, to try this like new way of operating and that you might not know how you'll do. So it seems like honestly so, has gotta be so relieving or helpful to know, like, it's an experiment and you can switch back. Yeah. And now that we've had we've had this happen in various parts of the organization a couple times now. And again, it creates this permission structure for people to try things. And that obviously has, like, huge positive externalities when people are, like, less afraid to go out on a limb and try something new. I mean, again, being the first person to go from being an engineering manager back to senior engineer, it's, like, super scary. Like, if you're the fifth one, like, That that's a good point. It kinda, like, snowballs, I guess. Yeah. Yeah. But it does feel like right now, like, like, creating an engineering culture where people are excited to try experiments even though they're not gonna succeed. Like, that's like, particularly today with AI, just so, so important. Yeah. Okay. We went through Hex, life as a CTO. How do you like to unwind at the end of the day? What's your move? Mean, we do a lot of the normal stuff. I also have a a toddler, so not everything is unwinding at the end Winding in different ways. Yeah. Once he's in bed though, actually my husband my husband and I have been we've been playing through together all of the Soulsborne games, which might be a little bit weird for an unwind session. But it's like it's a really good way to like, you know, release some tension, you know. We gotta, like, start like, I hadn't played them before, but, I got started with Elden Ring, and then we, like, played all the way through Elden Ring. And then the and the rule is when you die, you pass the controller. So you get a little bit of a break. Like, die, you pass it, and then you get another turn. Do you do you have different specializations or jobs in Elden Ring, would you say? You and your husband, like? Yeah. I will say he is much better at sort of like learning the timing of the boss mechanics. So he's typically like he's he's gonna be a little better than that. I tend to be a little bit better at the like mobs because I'm less reckless than he is. He just, like, charges in and just get, like, totally annihilated. And I'm little more strategic in how I, like, do the placement of the yeah. My partner and I played a lot of Elden Ring, and I am very much so that avoid the mobs, run away, like, learning that you don't have to fight everyone, didn't realize that was an option in Elden Ring. Life changing. Life changing. It's an option in real life too. Yeah. I guess for the whole table of Elden Ring players, when you're playing with a person, how is this a collaborative peaceful act or Act of violence. You know what I'm saying? We actually stopped playing real co op games together because they got too intense. And we were like or it's just like we have just different play styles. Again, like, his is very, like, you know, just like rushed straight at it, and mine's a little bit like more sort of like careful exploratory. And so we were getting really mad at each other. Like, tried to play Baldur's Gate co op together and just like that did not go well. And so I actually think that this this model works a lot better. Like, one person is in control at any given time. I feel like that's a hot take. I love Baldur's Gate because split screen, you can all almost have a co adventure, like, or completely different adventure. Yeah. Oh, you can split off. You're saying you took the essence of playing separately, but within a cooperative game Yeah. Pretty much. You're like, see you. And then the screen just splits and you never reunite. Yeah. Sometimes. And then it's like, oh no, I'm dying. Can you come help? Yeah. Yeah. And then she'll pop in and have good Yeah. Love everybody for me and That's incredible. It's a good life. Caitlin, really appreciate you coming on the Test Set. I Thank you so much for having me. Yeah. No. It's okay. I feel like so much to think about. I I honestly I hate to admit it, but I feel like I have to I hadn't learned about Context Studio, but it is gonna live rent free in my head for the next You should check it out. Probably all my life. We'll hook you up. I I would love to I mean, always showing Hex to people. I feel like I don't actually, what you said about Context Studio really kind of blew my mind because I've I do like a lot of software engineering on data tools, but that strikes me as such a unique maybe not totally unique problem across a lot of domains, but for data people that need to like curate and stamp and endorse. It's honestly, like this moment in some ways is like a match made in head of heaven for data people for this workflow. Because this is the workflow that data teams have been doing forever, but it's just been so painful and laborious for, you know, relatively little impact. Like if you if you think or I mean, obviously there's impact, but like the amount of impact you get out of like all of this labor just sometimes doesn't feel worth it. Like, I don't know if you all remember, you know, Ari, I'm sure you remember like when semantic models first kind of became a thing, we just didn't see a ton of adoption of semantic models. And it wasn't because people didn't want them, it was just because it was so painful and took so much time to to build them out. And now to and and at the end, it's like, only two people are gonna be building your dashboards. Right? And now today, that's like totally been inverted. Not only can you take, you know, constructs like semantic models and make them available to everybody in the organization a way they can use, but actually also you can do the curation and you can do the governance much much more easily with sort of agents yourself. It is a pretty incredible that, like, both the cost has decreased dramatically and the impact has increased dramatically. Like, it's pretty unusual to get both of those at the same time. It's pretty cool. Yeah. Totally. Yeah. So a lot to check out. Yeah. Can't can't recommend Hex enough. I feel like also giving software engineers permission to, like, try things and fail, I think, is so relevant, especially with AI where there's a lot of experimentation going on. And then I'm putting in my pocket Elden Ring co op co op strategies. I'm gonna find peaceful ways to to play with my partner. But, yeah, thanks so much for coming on. It's been so nice and been great having you. Awesome. It's been fun. Yeah. Thank you so much. The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio, Adjy. For more episodes, visit the Test Set dot co or find us on your favorite podcast platform.