{"version":"1.0.0","segments":[{"speaker":"Ben Lloyd Pearson","startTime":7.902,"endTime":9.442,"body":"Welcome"},{"speaker":"Ben Lloyd Pearson","startTime":7.902,"endTime":9.442,"body":"to Dev Interrupted."},{"speaker":"Ben Lloyd Pearson","startTime":9.602,"endTime":11.142,"body":"I'm your host, Ben Lloyd Pearson."},{"speaker":"Andrew Zigler","startTime":11.842,"endTime":13.482,"body":"And I'm"},{"speaker":"Andrew Zigler","startTime":11.842,"endTime":13.482,"body":"your host, Andrew Ziegler."},{"speaker":"Andrew Zigler","startTime":13.742,"endTime":17.622,"body":"In today's news, we're talking about"},{"speaker":"Andrew Zigler","startTime":13.742,"endTime":17.622,"body":"how banks are capturing tech talent,"},{"speaker":"Andrew Zigler","startTime":17.852,"endTime":21.958,"body":"why your AI can't see this gorilla,"},{"speaker":"Andrew Zigler","startTime":17.852,"endTime":21.958,"body":"and how the skill of future is focused."},{"speaker":"Ben Lloyd Pearson","startTime":22.189,"endTime":26.189,"body":"Wait a second, did"},{"speaker":"Ben Lloyd Pearson","startTime":22.189,"endTime":26.189,"body":"you just say that AI can't see gorillas?"},{"speaker":"Andrew Zigler","startTime":26.559,"endTime":28.579,"body":"Yeah, well, you"},{"speaker":"Andrew Zigler","startTime":26.559,"endTime":28.579,"body":"ever hear that saying about not"},{"speaker":"Andrew Zigler","startTime":28.579,"endTime":29.964,"body":"seeing the forest for the trees?"},{"speaker":"Ben Lloyd Pearson","startTime":30.439,"endTime":31.189,"body":"Yeah."},{"speaker":"Andrew Zigler","startTime":31.924,"endTime":35.264,"body":"Well, it turns out if"},{"speaker":"Andrew Zigler","startTime":31.924,"endTime":35.264,"body":"you're an AI, you might not even see this"},{"speaker":"Andrew Zigler","startTime":35.304,"endTime":37.984,"body":"literal gorilla drawn in a scatterplot."},{"speaker":"Andrew Zigler","startTime":38.394,"endTime":42.914,"body":"My favorite article from this week was"},{"speaker":"Andrew Zigler","startTime":38.394,"endTime":42.914,"body":"one called Your AI Can't See Gorillas."},{"speaker":"Andrew Zigler","startTime":43.144,"endTime":48.454,"body":"And it explores a study that demonstrated"},{"speaker":"Andrew Zigler","startTime":43.144,"endTime":48.454,"body":"how when you're given a hypothesis"},{"speaker":"Andrew Zigler","startTime":48.734,"endTime":53.534,"body":"to prove or disprove from data up"},{"speaker":"Andrew Zigler","startTime":48.734,"endTime":53.534,"body":"front, it challenges the way that"},{"speaker":"Andrew Zigler","startTime":53.534,"endTime":55.274,"body":"you can explore that data freely."},{"speaker":"Andrew Zigler","startTime":55.644,"endTime":61.004,"body":"And basically, There's a literal"},{"speaker":"Andrew Zigler","startTime":55.644,"endTime":61.004,"body":"gorilla depicted in a group of data"},{"speaker":"Andrew Zigler","startTime":61.274,"endTime":64.274,"body":"that when depicted on a scatter"},{"speaker":"Andrew Zigler","startTime":61.274,"endTime":64.274,"body":"plotted, it shows that gorilla."},{"speaker":"Andrew Zigler","startTime":64.544,"endTime":69.927,"body":"But if you don't actually, plot the data"},{"speaker":"Andrew Zigler","startTime":64.544,"endTime":69.927,"body":"on a graph and look at it and explore"},{"speaker":"Andrew Zigler","startTime":69.927,"endTime":71.727,"body":"it, you're never gonna see that gorilla."},{"speaker":"Andrew Zigler","startTime":71.967,"endTime":75.237,"body":"If you just go in and look at the"},{"speaker":"Andrew Zigler","startTime":71.967,"endTime":75.237,"body":"columns and the rows and what's in"},{"speaker":"Andrew Zigler","startTime":75.237,"endTime":76.737,"body":"the cells, you're never gonna see it."},{"speaker":"Andrew Zigler","startTime":77.147,"endTime":81.497,"body":"For me, it really highlighted the"},{"speaker":"Andrew Zigler","startTime":77.147,"endTime":81.497,"body":"importance of intuition and curiosity"},{"speaker":"Andrew Zigler","startTime":81.777,"endTime":84.087,"body":"and how we as humans explore data."},{"speaker":"Andrew Zigler","startTime":84.487,"endTime":86.967,"body":"Because AIs, they totally"},{"speaker":"Andrew Zigler","startTime":84.487,"endTime":86.967,"body":"fail at this right now."},{"speaker":"Andrew Zigler","startTime":86.997,"endTime":93.557,"body":"Even in the article when the author would"},{"speaker":"Andrew Zigler","startTime":86.997,"endTime":93.557,"body":"take a screenshot of the scatterplot that"},{"speaker":"Andrew Zigler","startTime":93.567,"endTime":98.317,"body":"the LLM is so helpfully rendering for"},{"speaker":"Andrew Zigler","startTime":93.567,"endTime":98.317,"body":"it, that shows the gorilla plain as day,"},{"speaker":"Andrew Zigler","startTime":98.317,"endTime":101.557,"body":"it still doesn't want to see the gorilla"},{"speaker":"Andrew Zigler","startTime":98.317,"endTime":101.557,"body":"because it thinks it's looking at a chart."},{"speaker":"Andrew Zigler","startTime":101.927,"endTime":104.997,"body":"So for me, this is spelling strawberry."},{"speaker":"Andrew Zigler","startTime":105.057,"endTime":108.657,"body":"All over again, and it really"},{"speaker":"Andrew Zigler","startTime":105.057,"endTime":108.657,"body":"emphasizes why it's important"},{"speaker":"Andrew Zigler","startTime":108.667,"endTime":110.057,"body":"to keep a human in the loop too."},{"speaker":"Ben Lloyd Pearson","startTime":110.782,"endTime":111.032,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":111.032,"endTime":115.752,"body":"So I guess the generative AI comes"},{"speaker":"Ben Lloyd Pearson","startTime":111.032,"endTime":115.752,"body":"from a universe where scatterplots"},{"speaker":"Ben Lloyd Pearson","startTime":115.752,"endTime":119.442,"body":"are never used to visualize things"},{"speaker":"Ben Lloyd Pearson","startTime":115.752,"endTime":119.442,"body":"or to make pictures, you know."},{"speaker":"Ben Lloyd Pearson","startTime":119.762,"endTime":123.262,"body":"But so, all right, so if I understand"},{"speaker":"Ben Lloyd Pearson","startTime":119.762,"endTime":123.262,"body":"this, if you plot the data on a"},{"speaker":"Ben Lloyd Pearson","startTime":123.262,"endTime":127.017,"body":"graph visually, A human is almost"},{"speaker":"Ben Lloyd Pearson","startTime":123.262,"endTime":127.017,"body":"certainly going to immediately"},{"speaker":"Ben Lloyd Pearson","startTime":127.017,"endTime":129.117,"body":"understand that it's a gorilla, right?"},{"speaker":"Andrew Zigler","startTime":129.177,"endTime":130.327,"body":"Literally waving."},{"speaker":"Andrew Zigler","startTime":130.427,"endTime":133.987,"body":"Well, we'll make sure we share the link so"},{"speaker":"Andrew Zigler","startTime":130.427,"endTime":133.987,"body":"our readers, our listeners can see it too."},{"speaker":"Andrew Zigler","startTime":133.987,"endTime":135.697,"body":"It's, it's, it's quite hilarious."},{"speaker":"Andrew Zigler","startTime":135.777,"endTime":138.457,"body":"Literally waving, but it"},{"speaker":"Andrew Zigler","startTime":135.777,"endTime":138.457,"body":"can't see it, no matter"},{"speaker":"Andrew Zigler","startTime":138.457,"endTime":139.287,"body":"what you ask it."},{"speaker":"Ben Lloyd Pearson","startTime":140.002,"endTime":142.982,"body":"and even humans, I"},{"speaker":"Ben Lloyd Pearson","startTime":140.002,"endTime":142.982,"body":"mean, if a human just looked at the data"},{"speaker":"Ben Lloyd Pearson","startTime":142.982,"endTime":147.142,"body":"and never visualized it, like, they would"},{"speaker":"Ben Lloyd Pearson","startTime":142.982,"endTime":147.142,"body":"completely miss the plot here as well."},{"speaker":"Ben Lloyd Pearson","startTime":147.142,"endTime":150.672,"body":"So like, you know, in other words,"},{"speaker":"Ben Lloyd Pearson","startTime":147.142,"endTime":150.672,"body":"like, get off the command line and look"},{"speaker":"Ben Lloyd Pearson","startTime":150.672,"endTime":152.562,"body":"at some images for a moment, you know."},{"speaker":"Andrew Zigler","startTime":152.862,"endTime":153.117,"body":"Yeah."},{"speaker":"Andrew Zigler","startTime":153.117,"endTime":155.637,"body":"And sometimes it's even"},{"speaker":"Andrew Zigler","startTime":153.117,"endTime":155.637,"body":"applying common sense."},{"speaker":"Andrew Zigler","startTime":155.677,"endTime":160.127,"body":"And I think the biggest takeaway here too,"},{"speaker":"Andrew Zigler","startTime":155.677,"endTime":160.127,"body":"is it really highlights the difference in"},{"speaker":"Andrew Zigler","startTime":160.127,"endTime":162.347,"body":"which you and the LLM are processing data."},{"speaker":"Andrew Zigler","startTime":162.667,"endTime":166.407,"body":"This is strawberry all over again,"},{"speaker":"Andrew Zigler","startTime":162.667,"endTime":166.407,"body":"literally, because we don't struggle"},{"speaker":"Andrew Zigler","startTime":166.422,"endTime":170.067,"body":"with spelling a strawberry or knowing"},{"speaker":"Andrew Zigler","startTime":166.422,"endTime":170.067,"body":"how many R's are in it because we"},{"speaker":"Andrew Zigler","startTime":170.067,"endTime":171.487,"body":"look at each individual letter."},{"speaker":"Andrew Zigler","startTime":171.707,"endTime":173.607,"body":"As we all know, LLMs don't work that way."},{"speaker":"Andrew Zigler","startTime":173.787,"endTime":177.767,"body":"Similarly in this situation, we"},{"speaker":"Andrew Zigler","startTime":173.787,"endTime":177.767,"body":"learn from data and graphs by"},{"speaker":"Andrew Zigler","startTime":177.777,"endTime":179.847,"body":"looking at them with our eyes, but"},{"speaker":"Andrew Zigler","startTime":179.867,"endTime":182.567,"body":"an LLM understands them by"},{"speaker":"Andrew Zigler","startTime":179.867,"endTime":182.567,"body":"looking at the actual data."},{"speaker":"Ben Lloyd Pearson","startTime":183.282,"endTime":183.522,"body":"yeah."},{"speaker":"Ben Lloyd Pearson","startTime":183.522,"endTime":187.132,"body":"And for our listeners who aren't aware"},{"speaker":"Ben Lloyd Pearson","startTime":183.522,"endTime":187.132,"body":"of the strawberry phenomenon, there was a"},{"speaker":"Ben Lloyd Pearson","startTime":187.132,"endTime":191.792,"body":"while where Chad GPT would gaslight you to"},{"speaker":"Ben Lloyd Pearson","startTime":187.132,"endTime":191.792,"body":"convince you that there were, what was it?"},{"speaker":"Ben Lloyd Pearson","startTime":191.792,"endTime":193.472,"body":"Two R's in strawberry when"},{"speaker":"Andrew Zigler","startTime":193.787,"endTime":195.307,"body":"It would commonly change."},{"speaker":"Andrew Zigler","startTime":195.397,"endTime":197.017,"body":"It would really never get it right."},{"speaker":"Ben Lloyd Pearson","startTime":197.317,"endTime":199.267,"body":"Yeah, it never"},{"speaker":"Ben Lloyd Pearson","startTime":197.317,"endTime":199.267,"body":"gives you the correct answer no"},{"speaker":"Ben Lloyd Pearson","startTime":199.287,"endTime":200.737,"body":"matter how you ask the question."},{"speaker":"Ben Lloyd Pearson","startTime":200.987,"endTime":205.197,"body":"And I believe they had to like manually"},{"speaker":"Ben Lloyd Pearson","startTime":200.987,"endTime":205.197,"body":"fix that one use case to get it to work."},{"speaker":"Andrew Zigler","startTime":206.407,"endTime":206.787,"body":"Yeah, the,"},{"speaker":"Andrew Zigler","startTime":206.877,"endTime":207.057,"body":"the,"},{"speaker":"Ben Lloyd Pearson","startTime":207.087,"endTime":207.317,"body":"I mean,"},{"speaker":"Andrew Zigler","startTime":208.007,"endTime":210.462,"body":"strawberry news"},{"speaker":"Andrew Zigler","startTime":208.007,"endTime":210.462,"body":"was tragic and very viral."},{"speaker":"Ben Lloyd Pearson","startTime":210.762,"endTime":212.762,"body":"perception"},{"speaker":"Ben Lloyd Pearson","startTime":210.762,"endTime":212.762,"body":"is a hell of a drug, especially"},{"speaker":"Ben Lloyd Pearson","startTime":212.762,"endTime":217.002,"body":"when you combine it with pattern"},{"speaker":"Ben Lloyd Pearson","startTime":212.762,"endTime":217.002,"body":"matching like us humans commonly do."},{"speaker":"Ben Lloyd Pearson","startTime":217.642,"endTime":220.992,"body":"So were any of the GPT models"},{"speaker":"Ben Lloyd Pearson","startTime":217.642,"endTime":220.992,"body":"they tested like actually able"},{"speaker":"Ben Lloyd Pearson","startTime":220.992,"endTime":222.752,"body":"to detect the existence of this"},{"speaker":"Andrew Zigler","startTime":223.794,"endTime":224.774,"body":"not successfully, no."},{"speaker":"Andrew Zigler","startTime":224.794,"endTime":229.284,"body":"Even when given the picture, however,"},{"speaker":"Andrew Zigler","startTime":224.794,"endTime":229.284,"body":"it was noted that Claude was able to"},{"speaker":"Andrew Zigler","startTime":229.284,"endTime":233.724,"body":"kind of start to pick out that there's"},{"speaker":"Andrew Zigler","startTime":229.284,"endTime":233.724,"body":"some sort of depiction in the image."},{"speaker":"Andrew Zigler","startTime":233.734,"endTime":235.974,"body":"It doesn't go as far as to"},{"speaker":"Andrew Zigler","startTime":233.734,"endTime":235.974,"body":"understand that there's a gorilla,"},{"speaker":"Andrew Zigler","startTime":235.974,"endTime":237.714,"body":"you know, waving at it in the plot."},{"speaker":"Andrew Zigler","startTime":237.974,"endTime":242.474,"body":"but it does start to understand that there"},{"speaker":"Andrew Zigler","startTime":237.974,"endTime":242.474,"body":"might be visually something depicted."},{"speaker":"Andrew Zigler","startTime":242.664,"endTime":245.514,"body":"It goes one step further to"},{"speaker":"Andrew Zigler","startTime":242.664,"endTime":245.514,"body":"even say that because there's an"},{"speaker":"Andrew Zigler","startTime":245.524,"endTime":249.294,"body":"image depicted in the data, it's"},{"speaker":"Andrew Zigler","startTime":245.524,"endTime":249.294,"body":"highly likely that that data is."},{"speaker":"Andrew Zigler","startTime":249.439,"endTime":252.259,"body":"Generated or not authentic in some way."},{"speaker":"Andrew Zigler","startTime":252.269,"endTime":256.319,"body":"And so then Claude warns the reader"},{"speaker":"Andrew Zigler","startTime":252.269,"endTime":256.319,"body":"about the data they're using and"},{"speaker":"Andrew Zigler","startTime":256.319,"endTime":259.979,"body":"its authenticity, which is a fun"},{"speaker":"Andrew Zigler","startTime":256.319,"endTime":259.979,"body":"twist for the story to end on."},{"speaker":"Ben Lloyd Pearson","startTime":261.304,"endTime":265.014,"body":"it's almost like it"},{"speaker":"Ben Lloyd Pearson","startTime":261.304,"endTime":265.014,"body":"detect, it detects that like something is"},{"speaker":"Ben Lloyd Pearson","startTime":265.014,"endTime":269.734,"body":"waving at it, but it doesn't think that"},{"speaker":"Ben Lloyd Pearson","startTime":265.014,"endTime":269.734,"body":"it's important to know what that thing is"},{"speaker":"Andrew Zigler","startTime":270.009,"endTime":273.639,"body":"More importantly,"},{"speaker":"Andrew Zigler","startTime":270.009,"endTime":273.639,"body":"it knows that in its world that"},{"speaker":"Andrew Zigler","startTime":273.639,"endTime":277.039,"body":"scatterplots don't wave at you and"},{"speaker":"Andrew Zigler","startTime":273.639,"endTime":277.039,"body":"scatterplots don't have pictures in them."},{"speaker":"Andrew Zigler","startTime":277.049,"endTime":277.839,"body":"So the fact that this"},{"speaker":"Andrew Zigler","startTime":277.839,"endTime":280.299,"body":"one does both is something suspicious."},{"speaker":"Ben Lloyd Pearson","startTime":281.33,"endTime":283.45,"body":"now I want to"},{"speaker":"Ben Lloyd Pearson","startTime":281.33,"endTime":283.45,"body":"switch gears to this story that"},{"speaker":"Ben Lloyd Pearson","startTime":283.45,"endTime":287.51,"body":"you brought up about banks, you"},{"speaker":"Ben Lloyd Pearson","startTime":283.45,"endTime":287.51,"body":"know, in financial institutions"},{"speaker":"Ben Lloyd Pearson","startTime":287.51,"endTime":291.14,"body":"trying to catch up in the battle for"},{"speaker":"Ben Lloyd Pearson","startTime":287.51,"endTime":291.14,"body":"getting, for hiring developer talent."},{"speaker":"Ben Lloyd Pearson","startTime":291.82,"endTime":295.05,"body":"So I read this article and, you"},{"speaker":"Ben Lloyd Pearson","startTime":291.82,"endTime":295.05,"body":"know, it really broke down how"},{"speaker":"Ben Lloyd Pearson","startTime":295.07,"endTime":299.04,"body":"like financial services industry"},{"speaker":"Ben Lloyd Pearson","startTime":295.07,"endTime":299.04,"body":"as a whole is trying to seize an"},{"speaker":"Ben Lloyd Pearson","startTime":299.05,"endTime":300.83,"body":"opportunity to hire more top talent."},{"speaker":"Ben Lloyd Pearson","startTime":301.76,"endTime":305.45,"body":"from all these big companies that"},{"speaker":"Ben Lloyd Pearson","startTime":301.76,"endTime":305.45,"body":"are out there undergoing layoffs."},{"speaker":"Ben Lloyd Pearson","startTime":305.51,"endTime":309.57,"body":"you know, and I think, these layoffs"},{"speaker":"Ben Lloyd Pearson","startTime":305.51,"endTime":309.57,"body":"have become such a like a seasonal"},{"speaker":"Ben Lloyd Pearson","startTime":309.61,"endTime":313.68,"body":"occurrence that, many of these big tech"},{"speaker":"Ben Lloyd Pearson","startTime":309.61,"endTime":313.68,"body":"companies are kind of losing their allure."},{"speaker":"Ben Lloyd Pearson","startTime":313.84,"endTime":316.75,"body":"You know, it's like a certain"},{"speaker":"Ben Lloyd Pearson","startTime":313.84,"endTime":316.75,"body":"number of people are going to"},{"speaker":"Ben Lloyd Pearson","startTime":316.75,"endTime":317.863,"body":"get laid off every January."},{"speaker":"Ben Lloyd Pearson","startTime":318.0,"endTime":320.53,"body":"Like that's not a great"},{"speaker":"Ben Lloyd Pearson","startTime":318.0,"endTime":320.53,"body":"environment to work in."},{"speaker":"Ben Lloyd Pearson","startTime":320.913,"endTime":324.923,"body":"And, you know, traditionally, like I think"},{"speaker":"Ben Lloyd Pearson","startTime":320.913,"endTime":324.923,"body":"a lot of developers, particularly from"},{"speaker":"Ben Lloyd Pearson","startTime":324.923,"endTime":328.523,"body":"these big tech companies, they gravitate"},{"speaker":"Ben Lloyd Pearson","startTime":324.923,"endTime":328.523,"body":"towards those companies because they"},{"speaker":"Ben Lloyd Pearson","startTime":328.523,"endTime":331.013,"body":"just have, like, they want to work there."},{"speaker":"Ben Lloyd Pearson","startTime":331.013,"endTime":332.703,"body":"Like they pay really high salaries."},{"speaker":"Ben Lloyd Pearson","startTime":332.703,"endTime":334.053,"body":"they're great institutions."},{"speaker":"Ben Lloyd Pearson","startTime":334.713,"endTime":336.333,"body":"And they've not really viewed financial."},{"speaker":"Ben Lloyd Pearson","startTime":336.771,"endTime":338.451,"body":"Uh, institutions in the same light."},{"speaker":"Ben Lloyd Pearson","startTime":338.791,"endTime":341.941,"body":"But the reality is a lot of these"},{"speaker":"Ben Lloyd Pearson","startTime":338.791,"endTime":341.941,"body":"organizations are now modernizing"},{"speaker":"Ben Lloyd Pearson","startTime":341.971,"endTime":346.301,"body":"their, their tech stacks, their"},{"speaker":"Ben Lloyd Pearson","startTime":341.971,"endTime":346.301,"body":"increasing budgets into development,"},{"speaker":"Ben Lloyd Pearson","startTime":346.621,"endTime":350.261,"body":"giving them access to all the latest"},{"speaker":"Ben Lloyd Pearson","startTime":346.621,"endTime":350.261,"body":"and greatest tools like AI and machine"},{"speaker":"Ben Lloyd Pearson","startTime":350.261,"endTime":353.421,"body":"learning and, you know, all The cloud"},{"speaker":"Ben Lloyd Pearson","startTime":350.261,"endTime":353.421,"body":"technologies that we know and love."},{"speaker":"Ben Lloyd Pearson","startTime":353.876,"endTime":357.726,"body":"and compensation is also going"},{"speaker":"Ben Lloyd Pearson","startTime":353.876,"endTime":357.726,"body":"up with this, uh, as well as like"},{"speaker":"Ben Lloyd Pearson","startTime":357.726,"endTime":361.316,"body":"flexible work arrangements and the"},{"speaker":"Ben Lloyd Pearson","startTime":357.726,"endTime":361.316,"body":"benefits that they offer to be more"},{"speaker":"Ben Lloyd Pearson","startTime":361.316,"endTime":362.866,"body":"in line with the big tech companies."},{"speaker":"Ben Lloyd Pearson","startTime":363.566,"endTime":366.546,"body":"And there's actually been like,"},{"speaker":"Ben Lloyd Pearson","startTime":363.566,"endTime":366.546,"body":"there's an element of geography too."},{"speaker":"Ben Lloyd Pearson","startTime":366.673,"endTime":369.273,"body":"you know, US banking hubs"},{"speaker":"Ben Lloyd Pearson","startTime":366.673,"endTime":369.273,"body":"are a lot more dispersed."},{"speaker":"Ben Lloyd Pearson","startTime":369.273,"endTime":373.014,"body":"So I think as a result, it's,"},{"speaker":"Ben Lloyd Pearson","startTime":369.273,"endTime":373.014,"body":"easier for them to get into like the"},{"speaker":"Ben Lloyd Pearson","startTime":373.073,"endTime":375.773,"body":"remote work movement and to find."},{"speaker":"Ben Lloyd Pearson","startTime":376.098,"endTime":379.228,"body":"Developers that maybe would"},{"speaker":"Ben Lloyd Pearson","startTime":376.098,"endTime":379.228,"body":"work remotely for these big tech"},{"speaker":"Ben Lloyd Pearson","startTime":379.238,"endTime":382.638,"body":"companies, but live in a city where"},{"speaker":"Ben Lloyd Pearson","startTime":379.238,"endTime":382.638,"body":"these banks have a headquarters."},{"speaker":"Ben Lloyd Pearson","startTime":383.448,"endTime":386.898,"body":"But the main reason I brought this up,"},{"speaker":"Ben Lloyd Pearson","startTime":383.448,"endTime":386.898,"body":"because it reminds me a lot, actually, of"},{"speaker":"Ben Lloyd Pearson","startTime":386.898,"endTime":392.668,"body":"a conversation I had with Louis Vega from"},{"speaker":"Ben Lloyd Pearson","startTime":386.898,"endTime":392.668,"body":"Bloomberg LP back in May of last year."},{"speaker":"Ben Lloyd Pearson","startTime":393.003,"endTime":396.723,"body":"and he builds all of these really"},{"speaker":"Ben Lloyd Pearson","startTime":393.003,"endTime":396.723,"body":"cool internal tools for Bloomberg."},{"speaker":"Ben Lloyd Pearson","startTime":397.043,"endTime":400.613,"body":"I really loved learning about the"},{"speaker":"Ben Lloyd Pearson","startTime":397.043,"endTime":400.613,"body":"engineering culture they built, because,"},{"speaker":"Ben Lloyd Pearson","startTime":400.623,"endTime":404.043,"body":"you know, one of the coolest aspects"},{"speaker":"Ben Lloyd Pearson","startTime":400.623,"endTime":404.043,"body":"was how, he was describing how they"},{"speaker":"Ben Lloyd Pearson","startTime":404.053,"endTime":407.393,"body":"create these like unique brands for"},{"speaker":"Ben Lloyd Pearson","startTime":404.053,"endTime":407.393,"body":"each, for each of their projects."},{"speaker":"Ben Lloyd Pearson","startTime":407.393,"endTime":412.593,"body":"So like, A log manager that you call"},{"speaker":"Ben Lloyd Pearson","startTime":407.393,"endTime":412.593,"body":"logger, like without an E. And then you"},{"speaker":"Ben Lloyd Pearson","startTime":412.603,"endTime":414.943,"body":"go to a designer and get a logo for it."},{"speaker":"Ben Lloyd Pearson","startTime":414.943,"endTime":417.983,"body":"it gets added to this internal"},{"speaker":"Ben Lloyd Pearson","startTime":414.943,"endTime":417.983,"body":"marketplace for all the other"},{"speaker":"Ben Lloyd Pearson","startTime":417.983,"endTime":419.173,"body":"developers at the company."},{"speaker":"Ben Lloyd Pearson","startTime":419.173,"endTime":422.473,"body":"Like, uh, they really kind of"},{"speaker":"Ben Lloyd Pearson","startTime":419.173,"endTime":422.473,"body":"celebrate building cool things"},{"speaker":"Ben Lloyd Pearson","startTime":422.473,"endTime":424.213,"body":"and sharing them internally."},{"speaker":"Ben Lloyd Pearson","startTime":424.213,"endTime":426.293,"body":"So, if you haven't listened to that"},{"speaker":"Ben Lloyd Pearson","startTime":424.213,"endTime":426.293,"body":"episode, I definitely encourage"},{"speaker":"Ben Lloyd Pearson","startTime":426.293,"endTime":429.213,"body":"you to check it out after you read"},{"speaker":"Ben Lloyd Pearson","startTime":426.293,"endTime":429.213,"body":"this article or before you read it."},{"speaker":"Andrew Zigler","startTime":430.083,"endTime":432.093,"body":"Yeah, definitely"},{"speaker":"Andrew Zigler","startTime":430.083,"endTime":432.093,"body":"a good one to check out."},{"speaker":"Andrew Zigler","startTime":432.093,"endTime":434.783,"body":"I also really resonate with"},{"speaker":"Andrew Zigler","startTime":432.093,"endTime":434.783,"body":"what you're saying about what"},{"speaker":"Andrew Zigler","startTime":434.783,"endTime":436.053,"body":"you learned from this article."},{"speaker":"Andrew Zigler","startTime":436.243,"endTime":440.543,"body":"I do think it says a lot about how tech"},{"speaker":"Andrew Zigler","startTime":436.243,"endTime":440.543,"body":"is being more distributed everywhere"},{"speaker":"Andrew Zigler","startTime":440.603,"endTime":445.293,"body":"and technology is not something"},{"speaker":"Andrew Zigler","startTime":440.603,"endTime":445.293,"body":"relegated to the tech world per se."},{"speaker":"Andrew Zigler","startTime":445.533,"endTime":449.183,"body":"the entire world's undergoing"},{"speaker":"Andrew Zigler","startTime":445.533,"endTime":449.183,"body":"transformations with technology and AI"},{"speaker":"Andrew Zigler","startTime":449.183,"endTime":451.738,"body":"is Pulling that even faster forward now."},{"speaker":"Andrew Zigler","startTime":451.748,"endTime":456.568,"body":"So the result is, there's so many"},{"speaker":"Andrew Zigler","startTime":451.748,"endTime":456.568,"body":"opportunities to even be a tech"},{"speaker":"Andrew Zigler","startTime":456.578,"endTime":459.288,"body":"person within a non tech company."},{"speaker":"Andrew Zigler","startTime":459.348,"endTime":462.898,"body":"that's how I actually"},{"speaker":"Andrew Zigler","startTime":459.348,"endTime":462.898,"body":"started in technology, by"},{"speaker":"Andrew Zigler","startTime":462.908,"endTime":464.738,"body":"being that technical expert."},{"speaker":"Andrew Zigler","startTime":464.758,"endTime":467.718,"body":"for non technical folks that are"},{"speaker":"Andrew Zigler","startTime":464.758,"endTime":467.718,"body":"trying to execute something at"},{"speaker":"Andrew Zigler","startTime":467.718,"endTime":472.108,"body":"scale, you can actually find that"},{"speaker":"Andrew Zigler","startTime":467.718,"endTime":472.108,"body":"your insights can go really, really"},{"speaker":"Andrew Zigler","startTime":472.108,"endTime":476.328,"body":"far in places that are maybe non"},{"speaker":"Andrew Zigler","startTime":472.108,"endTime":476.328,"body":"traditional to where you work from."},{"speaker":"Andrew Zigler","startTime":476.658,"endTime":481.348,"body":"and this also reminds me of even some of"},{"speaker":"Andrew Zigler","startTime":476.658,"endTime":481.348,"body":"the insights that I learned from today's"},{"speaker":"Andrew Zigler","startTime":481.348,"endTime":486.031,"body":"guest, Vikram Chatterji of Galileo, who's"},{"speaker":"Andrew Zigler","startTime":481.348,"endTime":486.031,"body":"coming up here in just a few moments."},{"speaker":"Andrew Zigler","startTime":486.031,"endTime":490.811,"body":"And in order to implement AI at the"},{"speaker":"Andrew Zigler","startTime":486.031,"endTime":490.811,"body":"scale, you're talking about, In a bank,"},{"speaker":"Andrew Zigler","startTime":490.821,"endTime":495.151,"body":"get the prototype rapidly, but get"},{"speaker":"Andrew Zigler","startTime":490.821,"endTime":495.151,"body":"the measure that impact and ultimately"},{"speaker":"Andrew Zigler","startTime":495.151,"endTime":499.811,"body":"build trust in your organization, which"},{"speaker":"Andrew Zigler","startTime":495.151,"endTime":499.811,"body":"is probably going to be very rigid"},{"speaker":"Andrew Zigler","startTime":499.821,"endTime":502.171,"body":"against that kind of rapid change."},{"speaker":"Andrew Zigler","startTime":502.611,"endTime":507.041,"body":"But before you can get there, you have"},{"speaker":"Andrew Zigler","startTime":502.611,"endTime":507.041,"body":"to attract that top talent and catch up."},{"speaker":"Andrew Zigler","startTime":507.291,"endTime":508.031,"body":"PeopleWise."},{"speaker":"Andrew Zigler","startTime":508.071,"endTime":510.231,"body":"So this news article"},{"speaker":"Andrew Zigler","startTime":508.071,"endTime":510.231,"body":"makes perfect sense to me."},{"speaker":"Andrew Zigler","startTime":510.441,"endTime":514.061,"body":"It even relates to what we saw recently"},{"speaker":"Andrew Zigler","startTime":510.441,"endTime":514.061,"body":"with Goldman Sachs acquiring that AI"},{"speaker":"Andrew Zigler","startTime":514.241,"endTime":517.451,"body":"transformation head from a tech company."},{"speaker":"Andrew Zigler","startTime":517.691,"endTime":520.521,"body":"I think this is a trend we're going"},{"speaker":"Andrew Zigler","startTime":517.691,"endTime":520.521,"body":"to see over and over again this year."},{"speaker":"Ben Lloyd Pearson","startTime":520.826,"endTime":521.226,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":521.296,"endTime":525.936,"body":"and one thing I've actually seen, some"},{"speaker":"Ben Lloyd Pearson","startTime":521.296,"endTime":525.936,"body":"specific examples of is when a lot of"},{"speaker":"Ben Lloyd Pearson","startTime":525.936,"endTime":529.976,"body":"these more traditional organizations"},{"speaker":"Ben Lloyd Pearson","startTime":525.936,"endTime":529.976,"body":"launch like a more coordinated software"},{"speaker":"Ben Lloyd Pearson","startTime":529.976,"endTime":533.506,"body":"development arm of their company,"},{"speaker":"Ben Lloyd Pearson","startTime":529.976,"endTime":533.506,"body":"they're often set up as like almost"},{"speaker":"Ben Lloyd Pearson","startTime":533.506,"endTime":537.586,"body":"like an independent entity that gets a"},{"speaker":"Ben Lloyd Pearson","startTime":533.506,"endTime":537.586,"body":"lot of freedom that you don't typically"},{"speaker":"Ben Lloyd Pearson","startTime":537.586,"endTime":540.596,"body":"see within a financial sector company."},{"speaker":"Ben Lloyd Pearson","startTime":540.626,"endTime":543.446,"body":"So, that allows them to move"},{"speaker":"Ben Lloyd Pearson","startTime":540.626,"endTime":543.446,"body":"a lot more quickly to use new"},{"speaker":"Ben Lloyd Pearson","startTime":543.446,"endTime":547.256,"body":"technologies, so yeah, you can, you"},{"speaker":"Ben Lloyd Pearson","startTime":543.446,"endTime":547.256,"body":"know, banks don't write them off."},{"speaker":"Ben Lloyd Pearson","startTime":547.306,"endTime":549.806,"body":"if you're one of the unfortunate"},{"speaker":"Ben Lloyd Pearson","startTime":547.306,"endTime":549.806,"body":"people out there who's been laid"},{"speaker":"Ben Lloyd Pearson","startTime":549.806,"endTime":552.466,"body":"off, or even if you're not like,"},{"speaker":"Ben Lloyd Pearson","startTime":549.806,"endTime":552.466,"body":"you know, go check them out and"},{"speaker":"Ben Lloyd Pearson","startTime":552.486,"endTime":553.596,"body":"maybe they're a cool place to work."},{"speaker":"Andrew Zigler","startTime":554.633,"endTime":557.013,"body":"But you know, speaking"},{"speaker":"Andrew Zigler","startTime":554.633,"endTime":557.013,"body":"of your skills and where your skills"},{"speaker":"Andrew Zigler","startTime":557.013,"endTime":560.593,"body":"can take you, there's another article"},{"speaker":"Andrew Zigler","startTime":557.013,"endTime":560.593,"body":"I read this week that talks about how"},{"speaker":"Andrew Zigler","startTime":560.593,"endTime":565.303,"body":"the skill of the future is not AI as"},{"speaker":"Andrew Zigler","startTime":560.593,"endTime":565.303,"body":"everyone's talking about, but it's focus."},{"speaker":"Andrew Zigler","startTime":565.503,"endTime":567.583,"body":"And this one, I loved this article."},{"speaker":"Andrew Zigler","startTime":567.793,"endTime":571.433,"body":"it really highlighted how leaders"},{"speaker":"Andrew Zigler","startTime":567.793,"endTime":571.433,"body":"need to create a culture where"},{"speaker":"Andrew Zigler","startTime":571.503,"endTime":575.733,"body":"engineers can validate and understand"},{"speaker":"Andrew Zigler","startTime":571.503,"endTime":575.733,"body":"AI's inputs and outputs and seek"},{"speaker":"Andrew Zigler","startTime":575.733,"endTime":579.573,"body":"alternate solutions and understand"},{"speaker":"Andrew Zigler","startTime":575.733,"endTime":579.573,"body":"ultimately what's happening underneath."},{"speaker":"Andrew Zigler","startTime":579.733,"endTime":583.323,"body":"That way you can engage in a"},{"speaker":"Andrew Zigler","startTime":579.733,"endTime":583.323,"body":"stronger understanding of how"},{"speaker":"Andrew Zigler","startTime":583.323,"endTime":587.823,"body":"the AI is working, but also then"},{"speaker":"Andrew Zigler","startTime":583.323,"endTime":587.823,"body":"harness that understanding to get"},{"speaker":"Andrew Zigler","startTime":587.833,"endTime":590.953,"body":"really good uninterrupted work done."},{"speaker":"Andrew Zigler","startTime":591.303,"endTime":596.593,"body":"In sum, basically use AI and understand"},{"speaker":"Andrew Zigler","startTime":591.303,"endTime":596.593,"body":"it, but then harness that free time"},{"speaker":"Andrew Zigler","startTime":596.593,"endTime":598.843,"body":"and focus on the stuff the AI can't do."},{"speaker":"Andrew Zigler","startTime":599.703,"endTime":603.363,"body":"And it really talks about how you build"},{"speaker":"Andrew Zigler","startTime":599.703,"endTime":603.363,"body":"a team culture to harness that unlock."},{"speaker":"Ben Lloyd Pearson","startTime":604.763,"endTime":605.083,"body":"Awesome."},{"speaker":"Ben Lloyd Pearson","startTime":605.093,"endTime":605.273,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":605.273,"endTime":608.933,"body":"And it kind of reiterates a point"},{"speaker":"Ben Lloyd Pearson","startTime":605.273,"endTime":608.933,"body":"that I've been making for a while"},{"speaker":"Ben Lloyd Pearson","startTime":608.933,"endTime":612.273,"body":"is that, you know, a lot of these"},{"speaker":"Ben Lloyd Pearson","startTime":608.933,"endTime":612.273,"body":"GPT models are, They're very"},{"speaker":"Ben Lloyd Pearson","startTime":612.303,"endTime":614.583,"body":"accurate, but they lack precision."},{"speaker":"Ben Lloyd Pearson","startTime":614.583,"endTime":619.373,"body":"So what I mean by that, and a good"},{"speaker":"Ben Lloyd Pearson","startTime":614.583,"endTime":619.373,"body":"example of this is if you go out and ask"},{"speaker":"Ben Lloyd Pearson","startTime":619.703,"endTime":624.613,"body":"any of these GPT models to, to create a"},{"speaker":"Ben Lloyd Pearson","startTime":619.703,"endTime":624.613,"body":"picture of a waving gorilla, for example,"},{"speaker":"Ben Lloyd Pearson","startTime":624.973,"endTime":629.168,"body":"they're probably going to create something"},{"speaker":"Ben Lloyd Pearson","startTime":624.973,"endTime":629.168,"body":"that You know, if I took the output and"},{"speaker":"Ben Lloyd Pearson","startTime":629.168,"endTime":632.168,"body":"handed it to our producer, Adam, and"},{"speaker":"Ben Lloyd Pearson","startTime":629.168,"endTime":632.168,"body":"asked him what it is, he would probably"},{"speaker":"Ben Lloyd Pearson","startTime":632.178,"endTime":633.838,"body":"immediately say, it's a waving gorilla."},{"speaker":"Ben Lloyd Pearson","startTime":633.898,"endTime":634.338,"body":"Like they,"},{"speaker":"Ben Lloyd Pearson","startTime":634.348,"endTime":636.028,"body":"they do that quite successfully."},{"speaker":"Ben Lloyd Pearson","startTime":636.508,"endTime":640.348,"body":"but if you look closer, you'll probably"},{"speaker":"Ben Lloyd Pearson","startTime":636.508,"endTime":640.348,"body":"see a lot of like micro errors that it"},{"speaker":"Ben Lloyd Pearson","startTime":640.348,"endTime":644.678,"body":"makes, like, you know, to borrow a phrase"},{"speaker":"Ben Lloyd Pearson","startTime":640.348,"endTime":644.678,"body":"from Westworld, the answers in the hands,"},{"speaker":"Ben Lloyd Pearson","startTime":644.678,"endTime":648.358,"body":"like you can tell that it's fake or it's"},{"speaker":"Ben Lloyd Pearson","startTime":644.678,"endTime":648.358,"body":"AI generated because the hands look weird."},{"speaker":"Ben Lloyd Pearson","startTime":648.868,"endTime":650.908,"body":"and in that environment, it's"},{"speaker":"Ben Lloyd Pearson","startTime":648.868,"endTime":650.908,"body":"actually very difficult for"},{"speaker":"Ben Lloyd Pearson","startTime":650.908,"endTime":653.098,"body":"humans to find errors, right?"},{"speaker":"Ben Lloyd Pearson","startTime":653.108,"endTime":656.498,"body":"If, if everything on the surface looks"},{"speaker":"Ben Lloyd Pearson","startTime":653.108,"endTime":656.498,"body":"correct, but it's actually filled"},{"speaker":"Ben Lloyd Pearson","startTime":656.498,"endTime":661.158,"body":"with a bunch of problems, you need"},{"speaker":"Ben Lloyd Pearson","startTime":656.498,"endTime":661.158,"body":"focus to, to suss out those problems."},{"speaker":"Ben Lloyd Pearson","startTime":661.481,"endTime":665.431,"body":"and it goes to show like, we've been"},{"speaker":"Ben Lloyd Pearson","startTime":661.481,"endTime":665.431,"body":"seeing research and anecdotes of, the"},{"speaker":"Ben Lloyd Pearson","startTime":665.431,"endTime":669.211,"body":"difference in how junior developers"},{"speaker":"Ben Lloyd Pearson","startTime":665.431,"endTime":669.211,"body":"and senior developers adopt AI."},{"speaker":"Ben Lloyd Pearson","startTime":669.571,"endTime":672.801,"body":"one thing that's come up again and"},{"speaker":"Ben Lloyd Pearson","startTime":669.571,"endTime":672.801,"body":"again is that junior developers are"},{"speaker":"Ben Lloyd Pearson","startTime":672.801,"endTime":676.011,"body":"a lot more likely to just blindly"},{"speaker":"Ben Lloyd Pearson","startTime":672.801,"endTime":676.011,"body":"accept AI output and then try to"},{"speaker":"Ben Lloyd Pearson","startTime":676.011,"endTime":678.501,"body":"fix any problems after the fact."},{"speaker":"Ben Lloyd Pearson","startTime":678.521,"endTime":683.391,"body":"Whereas a senior dev might treat it more"},{"speaker":"Ben Lloyd Pearson","startTime":678.521,"endTime":683.391,"body":"like a sparring partner where they ask it"},{"speaker":"Ben Lloyd Pearson","startTime":683.391,"endTime":685.391,"body":"some questions, get some ideas from it."},{"speaker":"Ben Lloyd Pearson","startTime":685.421,"endTime":688.341,"body":"And then the developer"},{"speaker":"Ben Lloyd Pearson","startTime":685.421,"endTime":688.341,"body":"themselves goes out and."},{"speaker":"Ben Lloyd Pearson","startTime":688.701,"endTime":692.081,"body":"Creates the code themselves, you know,"},{"speaker":"Ben Lloyd Pearson","startTime":688.701,"endTime":692.081,"body":"asking for advice where they need it."},{"speaker":"Ben Lloyd Pearson","startTime":692.601,"endTime":696.141,"body":"in that environment, you don't have to"},{"speaker":"Ben Lloyd Pearson","startTime":692.601,"endTime":696.141,"body":"catch as many of those precision problems."},{"speaker":"Andrew Zigler","startTime":696.431,"endTime":697.011,"body":"Exactly."},{"speaker":"Ben Lloyd Pearson","startTime":697.656,"endTime":697.906,"body":"well,"},{"speaker":"Ben Lloyd Pearson","startTime":698.376,"endTime":701.706,"body":"some great stories we had today, Andrew,"},{"speaker":"Ben Lloyd Pearson","startTime":698.376,"endTime":701.706,"body":"why don't you let us know about who we're,"},{"speaker":"Ben Lloyd Pearson","startTime":701.816,"endTime":703.386,"body":"we're having on the show today, insert"},{"speaker":"Andrew Zigler","startTime":703.686,"endTime":707.986,"body":"Yes, we've talked about"},{"speaker":"Andrew Zigler","startTime":703.686,"endTime":707.986,"body":"how AI is changing the way engineers work,"},{"speaker":"Andrew Zigler","startTime":708.126,"endTime":710.436,"body":"but also the risks of blind dependence."},{"speaker":"Andrew Zigler","startTime":710.436,"endTime":714.976,"body":"And if we treat AI as an oracle instead"},{"speaker":"Andrew Zigler","startTime":710.436,"endTime":714.976,"body":"of a tool, then we lose the ability"},{"speaker":"Andrew Zigler","startTime":714.976,"endTime":719.156,"body":"to question and refine and truly"},{"speaker":"Andrew Zigler","startTime":714.976,"endTime":719.156,"body":"understand the solutions it generates."},{"speaker":"Andrew Zigler","startTime":719.906,"endTime":724.526,"body":"And focus is the real skill that"},{"speaker":"Andrew Zigler","startTime":719.906,"endTime":724.526,"body":"separates great engineers from the rest."},{"speaker":"Andrew Zigler","startTime":724.766,"endTime":729.166,"body":"So in today's conversation, we're talking"},{"speaker":"Andrew Zigler","startTime":724.766,"endTime":729.166,"body":"with Vikram Chatterji, the co founder"},{"speaker":"Andrew Zigler","startTime":729.166,"endTime":734.196,"body":"and CEO of Galileo, about how to future"},{"speaker":"Andrew Zigler","startTime":729.166,"endTime":734.196,"body":"proof your AI engineering investments."},{"speaker":"Andrew Zigler","startTime":734.456,"endTime":738.876,"body":"Galileo is building the system of"},{"speaker":"Andrew Zigler","startTime":734.456,"endTime":738.876,"body":"record for AI models that help teams"},{"speaker":"Andrew Zigler","startTime":738.916,"endTime":743.056,"body":"open the hood and understand what's"},{"speaker":"Andrew Zigler","startTime":738.916,"endTime":743.056,"body":"happening with their LLMs underneath."},{"speaker":"Andrew Zigler","startTime":743.486,"endTime":744.146,"body":"Stick around."},{"speaker":"Ben Lloyd Pearson","startTime":746.009,"endTime":747.499,"body":"Habits"},{"speaker":"Ben Lloyd Pearson","startTime":746.009,"endTime":747.499,"body":"are a powerful thing."},{"speaker":"Ben Lloyd Pearson","startTime":748.139,"endTime":751.529,"body":"Maybe you've read one of the many"},{"speaker":"Ben Lloyd Pearson","startTime":748.139,"endTime":751.529,"body":"books about the habits of highly"},{"speaker":"Ben Lloyd Pearson","startTime":751.529,"endTime":753.049,"body":"successful people out there."},{"speaker":"Ben Lloyd Pearson","startTime":753.499,"endTime":757.049,"body":"Well, LinearB is out with their"},{"speaker":"Ben Lloyd Pearson","startTime":753.499,"endTime":757.049,"body":"own book, The 8 Habits of Highly"},{"speaker":"Ben Lloyd Pearson","startTime":757.049,"endTime":758.499,"body":"Productive Engineering Teams."},{"speaker":"Ben Lloyd Pearson","startTime":759.054,"endTime":762.674,"body":"This practical guide offers advice"},{"speaker":"Ben Lloyd Pearson","startTime":759.054,"endTime":762.674,"body":"and templates to help you establish"},{"speaker":"Ben Lloyd Pearson","startTime":762.684,"endTime":764.444,"body":"durable data driven habits."},{"speaker":"Ben Lloyd Pearson","startTime":764.754,"endTime":768.757,"body":"It covers things like setting actionable"},{"speaker":"Ben Lloyd Pearson","startTime":764.754,"endTime":768.757,"body":"team goals, coaching developers"},{"speaker":"Ben Lloyd Pearson","startTime":768.757,"endTime":772.517,"body":"to level up their skills, using"},{"speaker":"Ben Lloyd Pearson","startTime":768.757,"endTime":772.517,"body":"monthly metrics check ins to unblock"},{"speaker":"Ben Lloyd Pearson","startTime":772.557,"endTime":776.417,"body":"friction, and run more efficient"},{"speaker":"Ben Lloyd Pearson","startTime":772.557,"endTime":776.417,"body":"and effective sprint retrospectives."},{"speaker":"Ben Lloyd Pearson","startTime":776.887,"endTime":779.597,"body":"This guide has something for"},{"speaker":"Ben Lloyd Pearson","startTime":776.887,"endTime":779.597,"body":"everyone within your engineering"},{"speaker":"Ben Lloyd Pearson","startTime":779.597,"endTime":782.757,"body":"team, so check out the link in the"},{"speaker":"Ben Lloyd Pearson","startTime":779.597,"endTime":782.757,"body":"show notes for the eight habits of"},{"speaker":"Ben Lloyd Pearson","startTime":782.757,"endTime":784.477,"body":"highly productive engineering teams."},{"speaker":"Andrew Zigler","startTime":789.381,"endTime":791.761,"body":"Hey everyone,"},{"speaker":"Andrew Zigler","startTime":789.381,"endTime":791.761,"body":"welcome back to Dev Interrupted."},{"speaker":"Andrew Zigler","startTime":791.961,"endTime":795.841,"body":"I'm your host, Andrew Zigler,"},{"speaker":"Andrew Zigler","startTime":791.961,"endTime":795.841,"body":"developer advocate at LinearB, and"},{"speaker":"Andrew Zigler","startTime":795.841,"endTime":800.311,"body":"joining me today is Vikram Chatterji,"},{"speaker":"Andrew Zigler","startTime":795.841,"endTime":800.311,"body":"co founder and CEO of Galileo."},{"speaker":"Andrew Zigler","startTime":801.021,"endTime":805.661,"body":"Vikram has been on the front lines of the"},{"speaker":"Andrew Zigler","startTime":801.021,"endTime":805.661,"body":"AI revolution for many years, from leading"},{"speaker":"Andrew Zigler","startTime":805.661,"endTime":810.171,"body":"product management at Google during the"},{"speaker":"Andrew Zigler","startTime":805.661,"endTime":810.171,"body":"birth of Transformers, to building tools"},{"speaker":"Andrew Zigler","startTime":810.171,"endTime":814.471,"body":"that help engineering teams confidently"},{"speaker":"Andrew Zigler","startTime":810.171,"endTime":814.471,"body":"evaluate and deploy AI systems."},{"speaker":"Andrew Zigler","startTime":815.371,"endTime":817.601,"body":"Here's the crux of today's conversation."},{"speaker":"Andrew Zigler","startTime":818.121,"endTime":821.941,"body":"Engineering leaders are stuck"},{"speaker":"Andrew Zigler","startTime":818.121,"endTime":821.941,"body":"between a rock and a hard place."},{"speaker":"Andrew Zigler","startTime":822.381,"endTime":827.061,"body":"They know they need to experiment with"},{"speaker":"Andrew Zigler","startTime":822.381,"endTime":827.061,"body":"AI to stay competitive, but they're under"},{"speaker":"Andrew Zigler","startTime":827.071,"endTime":829.821,"body":"immense pressure to justify those costs."},{"speaker":"Andrew Zigler","startTime":830.501,"endTime":835.161,"body":"All the while, AI evolves so rapidly"},{"speaker":"Andrew Zigler","startTime":830.501,"endTime":835.161,"body":"that today's wrong move could"},{"speaker":"Andrew Zigler","startTime":835.161,"endTime":837.051,"body":"cost them tomorrow's opportunity."},{"speaker":"Andrew Zigler","startTime":837.681,"endTime":839.171,"body":"Vikram, welcome to the show."},{"speaker":"Vikram Chatterji","startTime":839.887,"endTime":840.517,"body":"Thank you, Andrew."},{"speaker":"Vikram Chatterji","startTime":840.517,"endTime":841.577,"body":"Super excited to be here."},{"speaker":"Andrew Zigler","startTime":842.241,"endTime":843.921,"body":"Likewise,"},{"speaker":"Andrew Zigler","startTime":842.241,"endTime":843.921,"body":"let's jump right in."},{"speaker":"Andrew Zigler","startTime":844.291,"endTime":847.871,"body":"Starting with the biggest challenge"},{"speaker":"Andrew Zigler","startTime":844.291,"endTime":847.871,"body":"currently facing engineering"},{"speaker":"Andrew Zigler","startTime":847.871,"endTime":852.961,"body":"leaders, AI being that moving target"},{"speaker":"Andrew Zigler","startTime":847.871,"endTime":852.961,"body":"and experimenting, feeling risky."},{"speaker":"Andrew Zigler","startTime":853.441,"endTime":856.731,"body":"It's a big barrier to adoption"},{"speaker":"Andrew Zigler","startTime":853.441,"endTime":856.731,"body":"and the fear of making the wrong"},{"speaker":"Andrew Zigler","startTime":856.731,"endTime":858.381,"body":"investment is ever present."},{"speaker":"Andrew Zigler","startTime":858.921,"endTime":863.591,"body":"How do you think leaders can take the"},{"speaker":"Andrew Zigler","startTime":858.921,"endTime":863.591,"body":"first steps without putting themselves"},{"speaker":"Andrew Zigler","startTime":863.611,"endTime":865.621,"body":"or their teams in a bad position?"},{"speaker":"Vikram Chatterji","startTime":866.464,"endTime":868.474,"body":"I've always"},{"speaker":"Vikram Chatterji","startTime":866.464,"endTime":868.474,"body":"thought about AI as, you know, it's"},{"speaker":"Vikram Chatterji","startTime":868.474,"endTime":870.554,"body":"another tool in your arsenal, right?"},{"speaker":"Vikram Chatterji","startTime":871.014,"endTime":875.465,"body":"even before generative AI became a"},{"speaker":"Vikram Chatterji","startTime":871.014,"endTime":875.465,"body":"very big thing with, machine learning"},{"speaker":"Vikram Chatterji","startTime":875.845,"endTime":880.685,"body":"and with NLP, It was never about"},{"speaker":"Vikram Chatterji","startTime":875.845,"endTime":880.685,"body":"like, hey, you have to use this thing."},{"speaker":"Vikram Chatterji","startTime":880.905,"endTime":882.325,"body":"It was more about what's your use case."},{"speaker":"Vikram Chatterji","startTime":882.365,"endTime":885.195,"body":"And based on that, is this a"},{"speaker":"Vikram Chatterji","startTime":882.365,"endTime":885.195,"body":"good fit for your use case?"},{"speaker":"Vikram Chatterji","startTime":885.875,"endTime":888.695,"body":"now I guess the difference is"},{"speaker":"Vikram Chatterji","startTime":885.875,"endTime":888.695,"body":"what we're seeing with a lot of"},{"speaker":"Vikram Chatterji","startTime":888.705,"endTime":891.975,"body":"engineering leaders that we talk to,"},{"speaker":"Vikram Chatterji","startTime":888.705,"endTime":891.975,"body":"there's a lot of top down pressure"},{"speaker":"Vikram Chatterji","startTime":892.085,"endTime":893.695,"body":"to just use AI for the sake of it."},{"speaker":"Vikram Chatterji","startTime":894.048,"endTime":897.558,"body":"to get to your point about the rock"},{"speaker":"Vikram Chatterji","startTime":894.048,"endTime":897.558,"body":"between the rock and hard place."},{"speaker":"Vikram Chatterji","startTime":897.732,"endTime":901.262,"body":"it's very important for For"},{"speaker":"Vikram Chatterji","startTime":897.732,"endTime":901.262,"body":"engineering leaders to think about"},{"speaker":"Vikram Chatterji","startTime":901.532,"endTime":902.902,"body":"what are the heuristics, right?"},{"speaker":"Vikram Chatterji","startTime":902.902,"endTime":906.472,"body":"That, that is gonna, is gonna help"},{"speaker":"Vikram Chatterji","startTime":902.902,"endTime":906.472,"body":"me figure out, you know, is this"},{"speaker":"Vikram Chatterji","startTime":906.472,"endTime":909.752,"body":"something that we should be even going"},{"speaker":"Vikram Chatterji","startTime":906.472,"endTime":909.752,"body":"ahead with, and that includes things"},{"speaker":"Vikram Chatterji","startTime":909.752,"endTime":911.492,"body":"like, what does the business need?"},{"speaker":"Vikram Chatterji","startTime":911.802,"endTime":915.492,"body":"Because what I've seen is folks"},{"speaker":"Vikram Chatterji","startTime":911.802,"endTime":915.492,"body":"just do a massive hackathon, within"},{"speaker":"Vikram Chatterji","startTime":915.492,"endTime":919.832,"body":"their org, and you're going to get"},{"speaker":"Vikram Chatterji","startTime":915.492,"endTime":919.832,"body":"a hundred ideas just given how open"},{"speaker":"Vikram Chatterji","startTime":919.832,"endTime":921.972,"body":"ended AI can be right now, right?"},{"speaker":"Vikram Chatterji","startTime":921.982,"endTime":925.262,"body":"In terms of whether it's generation"},{"speaker":"Vikram Chatterji","startTime":921.982,"endTime":925.262,"body":"of text, whether it's generation of"},{"speaker":"Vikram Chatterji","startTime":925.562,"endTime":930.422,"body":"images Completion of a task with agents,"},{"speaker":"Vikram Chatterji","startTime":925.562,"endTime":930.422,"body":"you get a hundred use cases, but it's"},{"speaker":"Vikram Chatterji","startTime":930.432,"endTime":933.872,"body":"very, very important for them to then"},{"speaker":"Vikram Chatterji","startTime":930.432,"endTime":933.872,"body":"go back to like, you know, product and"},{"speaker":"Vikram Chatterji","startTime":933.872,"endTime":937.862,"body":"business owners to figure out which"},{"speaker":"Vikram Chatterji","startTime":933.872,"endTime":937.862,"body":"one of them should they prioritize."},{"speaker":"Vikram Chatterji","startTime":938.452,"endTime":942.562,"body":"and also on the back of that,"},{"speaker":"Vikram Chatterji","startTime":938.452,"endTime":942.562,"body":"which one of them can you actually"},{"speaker":"Vikram Chatterji","startTime":942.562,"endTime":944.622,"body":"get out the door very quickly."},{"speaker":"Vikram Chatterji","startTime":944.832,"endTime":947.932,"body":"And that's kind of where the operational"},{"speaker":"Vikram Chatterji","startTime":944.832,"endTime":947.932,"body":"rigor has to kick in of like trying"},{"speaker":"Vikram Chatterji","startTime":947.942,"endTime":949.622,"body":"out X number of ideas very, very fast."},{"speaker":"Vikram Chatterji","startTime":949.632,"endTime":951.912,"body":"So you have to have"},{"speaker":"Vikram Chatterji","startTime":949.632,"endTime":951.912,"body":"that machinery in place."},{"speaker":"Vikram Chatterji","startTime":952.272,"endTime":956.632,"body":"And the ability to push back and say that,"},{"speaker":"Vikram Chatterji","startTime":952.272,"endTime":956.632,"body":"hey, maybe I don't need to use AI at all,"},{"speaker":"Vikram Chatterji","startTime":956.712,"endTime":958.922,"body":"but when I do, here's how I need to do it."},{"speaker":"Vikram Chatterji","startTime":958.922,"endTime":962.242,"body":"And at that point, we can talk about this"},{"speaker":"Vikram Chatterji","startTime":958.922,"endTime":962.242,"body":"more, but we have to think it through in"},{"speaker":"Vikram Chatterji","startTime":962.242,"endTime":966.512,"body":"terms of if I succeed, what does that mean"},{"speaker":"Vikram Chatterji","startTime":962.242,"endTime":966.512,"body":"for me in terms of the number of engineers"},{"speaker":"Vikram Chatterji","startTime":966.512,"endTime":971.352,"body":"I need for this, the evals, the cost"},{"speaker":"Vikram Chatterji","startTime":966.512,"endTime":971.352,"body":"of productionizing this thing at scale."},{"speaker":"Vikram Chatterji","startTime":971.722,"endTime":973.742,"body":"so there's a bunch of things that"},{"speaker":"Vikram Chatterji","startTime":971.722,"endTime":973.742,"body":"people have to think about and a lot"},{"speaker":"Vikram Chatterji","startTime":973.742,"endTime":975.822,"body":"of trade offs at the onset itself."},{"speaker":"Andrew Zigler","startTime":976.591,"endTime":979.171,"body":"So it sounds like in"},{"speaker":"Andrew Zigler","startTime":976.591,"endTime":979.171,"body":"balance with that proliferation of"},{"speaker":"Andrew Zigler","startTime":979.171,"endTime":984.121,"body":"ideas, you really need an evaluation"},{"speaker":"Andrew Zigler","startTime":979.171,"endTime":984.121,"body":"framework or a way to understand and"},{"speaker":"Andrew Zigler","startTime":984.121,"endTime":988.691,"body":"extract scenarios that are maybe have"},{"speaker":"Andrew Zigler","startTime":984.121,"endTime":988.691,"body":"a higher ROI or a higher impact on your"},{"speaker":"Andrew Zigler","startTime":988.691,"endTime":992.961,"body":"business and focus on those because I"},{"speaker":"Andrew Zigler","startTime":988.691,"endTime":992.961,"body":"think that's kind of part of the problem"},{"speaker":"Andrew Zigler","startTime":992.961,"endTime":996.141,"body":"too is you're drowning in possible"},{"speaker":"Andrew Zigler","startTime":992.961,"endTime":996.141,"body":"solutions and everyone like that."},{"speaker":"Andrew Zigler","startTime":996.141,"endTime":999.231,"body":"Can come up with a, maybe a way"},{"speaker":"Andrew Zigler","startTime":996.141,"endTime":999.231,"body":"to integrate it in some way, but"},{"speaker":"Andrew Zigler","startTime":999.231,"endTime":1001.031,"body":"is that the most effective way?"},{"speaker":"Andrew Zigler","startTime":1001.241,"endTime":1003.341,"body":"Is that where we should"},{"speaker":"Andrew Zigler","startTime":1001.241,"endTime":1003.341,"body":"focus our attention?"},{"speaker":"Andrew Zigler","startTime":1003.581,"endTime":1007.181,"body":"And the more attention you put"},{"speaker":"Andrew Zigler","startTime":1003.581,"endTime":1007.181,"body":"on something, it can skew then"},{"speaker":"Andrew Zigler","startTime":1007.181,"endTime":1010.421,"body":"how the rest of your organization"},{"speaker":"Andrew Zigler","startTime":1007.181,"endTime":1010.421,"body":"is using and thinking about AI."},{"speaker":"Andrew Zigler","startTime":1010.421,"endTime":1014.881,"body":"So those decisions, especially early on,"},{"speaker":"Andrew Zigler","startTime":1010.421,"endTime":1014.881,"body":"you know, they're really impactful How"},{"speaker":"Andrew Zigler","startTime":1014.881,"endTime":1020.491,"body":"would you advise, or what habits do you"},{"speaker":"Andrew Zigler","startTime":1014.881,"endTime":1020.491,"body":"think make for somebody to be able to"},{"speaker":"Andrew Zigler","startTime":1020.491,"endTime":1024.201,"body":"evaluate and de risk that experimentation?"},{"speaker":"Andrew Zigler","startTime":1024.211,"endTime":1027.161,"body":"Like, what are some, what are those tools"},{"speaker":"Andrew Zigler","startTime":1024.211,"endTime":1027.161,"body":"that those kinds of people are always"},{"speaker":"Andrew Zigler","startTime":1027.371,"endTime":1029.791,"body":"using again and again to do that well?"},{"speaker":"Vikram Chatterji","startTime":1030.207,"endTime":1031.387,"body":"Yeah,"},{"speaker":"Vikram Chatterji","startTime":1030.207,"endTime":1031.387,"body":"it's a great question."},{"speaker":"Vikram Chatterji","startTime":1031.617,"endTime":1034.947,"body":"I will say it depends a lot on the"},{"speaker":"Vikram Chatterji","startTime":1031.617,"endTime":1034.947,"body":"organization and you have to just know"},{"speaker":"Vikram Chatterji","startTime":1034.947,"endTime":1040.257,"body":"your organization well So if you're"},{"speaker":"Vikram Chatterji","startTime":1034.947,"endTime":1040.257,"body":"a big bank, right The amount of harm"},{"speaker":"Vikram Chatterji","startTime":1040.307,"endTime":1044.177,"body":"that can happen if you put something"},{"speaker":"Vikram Chatterji","startTime":1040.307,"endTime":1044.177,"body":"out there in the consumer world with"},{"speaker":"Vikram Chatterji","startTime":1044.207,"endTime":1047.167,"body":"AI and it misfires is very, very large."},{"speaker":"Vikram Chatterji","startTime":1047.397,"endTime":1049.917,"body":"It can literally derail your"},{"speaker":"Vikram Chatterji","startTime":1047.397,"endTime":1049.917,"body":"entire bank's reputation."},{"speaker":"Vikram Chatterji","startTime":1049.917,"endTime":1053.017,"body":"For a commoditized entity like"},{"speaker":"Vikram Chatterji","startTime":1049.917,"endTime":1053.017,"body":"a bank, you're going to be"},{"speaker":"Vikram Chatterji","startTime":1053.027,"endTime":1054.477,"body":"out of business very quickly."},{"speaker":"Vikram Chatterji","startTime":1054.644,"endTime":1059.194,"body":"On the other hand, if you're, let's say,"},{"speaker":"Vikram Chatterji","startTime":1054.644,"endTime":1059.194,"body":"a DoorDash or an Instacart, the bar is"},{"speaker":"Vikram Chatterji","startTime":1059.194,"endTime":1063.024,"body":"probably a bit lower because, not to"},{"speaker":"Vikram Chatterji","startTime":1059.194,"endTime":1063.024,"body":"say that they have a low bar in general,"},{"speaker":"Vikram Chatterji","startTime":1063.074,"endTime":1067.254,"body":"but if something goes wrong with their"},{"speaker":"Vikram Chatterji","startTime":1063.074,"endTime":1067.254,"body":"chatbot or something like that, it's not"},{"speaker":"Vikram Chatterji","startTime":1067.254,"endTime":1069.584,"body":"going to be the end of the world because"},{"speaker":"Vikram Chatterji","startTime":1067.254,"endTime":1069.584,"body":"they're not dealing with people's money."},{"speaker":"Vikram Chatterji","startTime":1069.624,"endTime":1072.264,"body":"They're dealing with hungry"},{"speaker":"Vikram Chatterji","startTime":1069.624,"endTime":1072.264,"body":"people, which is bad, but they're"},{"speaker":"Vikram Chatterji","startTime":1072.264,"endTime":1073.694,"body":"not dealing with people's money."},{"speaker":"Andrew Zigler","startTime":1074.107,"endTime":1076.417,"body":"I maybe didn't get"},{"speaker":"Andrew Zigler","startTime":1074.107,"endTime":1076.417,"body":"my burger, but, you know, my bank"},{"speaker":"Andrew Zigler","startTime":1076.417,"endTime":1079.657,"body":"account didn't have, like, an"},{"speaker":"Andrew Zigler","startTime":1076.417,"endTime":1079.657,"body":"unauthorized transaction or something."},{"speaker":"Andrew Zigler","startTime":1079.657,"endTime":1080.937,"body":"They're totally different stakes."},{"speaker":"Vikram Chatterji","startTime":1081.358,"endTime":1082.338,"body":"they're"},{"speaker":"Vikram Chatterji","startTime":1081.358,"endTime":1082.338,"body":"totally different stakes."},{"speaker":"Vikram Chatterji","startTime":1082.548,"endTime":1085.548,"body":"and so what I've seen as a result of"},{"speaker":"Vikram Chatterji","startTime":1082.548,"endTime":1085.548,"body":"that is when you talk to these large"},{"speaker":"Vikram Chatterji","startTime":1085.548,"endTime":1088.178,"body":"enterprises, they're very, very excited"},{"speaker":"Vikram Chatterji","startTime":1085.548,"endTime":1088.178,"body":"about generative AI, and they're very"},{"speaker":"Vikram Chatterji","startTime":1088.178,"endTime":1090.168,"body":"excited to add agents and everything else."},{"speaker":"Vikram Chatterji","startTime":1090.688,"endTime":1094.212,"body":"But they're very, they're taking a"},{"speaker":"Vikram Chatterji","startTime":1090.688,"endTime":1094.212,"body":"very, very careful crawl, walk, run"},{"speaker":"Vikram Chatterji","startTime":1094.212,"endTime":1095.635,"body":"approach to it , which I think is good."},{"speaker":"Vikram Chatterji","startTime":1096.175,"endTime":1098.975,"body":"what I'm seeing with the other,"},{"speaker":"Vikram Chatterji","startTime":1096.175,"endTime":1098.975,"body":"like, companies where they are,"},{"speaker":"Vikram Chatterji","startTime":1099.055,"endTime":1100.635,"body":"earlier stages, they're tech first."},{"speaker":"Vikram Chatterji","startTime":1100.975,"endTime":1103.885,"body":"let's say, as an example, like"},{"speaker":"Vikram Chatterji","startTime":1100.975,"endTime":1103.885,"body":"a DoorDash or like an Instacart."},{"speaker":"Vikram Chatterji","startTime":1104.51,"endTime":1108.32,"body":"Or like an Airbnb or a Twilio,"},{"speaker":"Vikram Chatterji","startTime":1104.51,"endTime":1108.32,"body":"they're taking a much more,"},{"speaker":"Vikram Chatterji","startTime":1108.36,"endTime":1110.37,"body":"experimental approach to this, right?"},{"speaker":"Vikram Chatterji","startTime":1110.37,"endTime":1111.41,"body":"Like, let's try things out."},{"speaker":"Vikram Chatterji","startTime":1111.46,"endTime":1112.49,"body":"Let's see how it goes."},{"speaker":"Vikram Chatterji","startTime":1112.75,"endTime":1113.6,"body":"Let's see how we feel."},{"speaker":"Vikram Chatterji","startTime":1113.64,"endTime":1117.75,"body":"It's very much on the lines of, you know,"},{"speaker":"Vikram Chatterji","startTime":1113.64,"endTime":1117.75,"body":"build fast, break things, learn quickly."},{"speaker":"Vikram Chatterji","startTime":1117.99,"endTime":1121.47,"body":"And that's really led to them kind"},{"speaker":"Vikram Chatterji","startTime":1117.99,"endTime":1121.47,"body":"of, Figuring things out as they go."},{"speaker":"Vikram Chatterji","startTime":1121.47,"endTime":1124.78,"body":"And it is also useful when, as this"},{"speaker":"Vikram Chatterji","startTime":1121.47,"endTime":1124.78,"body":"industry is moving super quickly."},{"speaker":"Vikram Chatterji","startTime":1125.78,"endTime":1129.87,"body":"Based on the organization, I've seen"},{"speaker":"Vikram Chatterji","startTime":1125.78,"endTime":1129.87,"body":"like the barrier to entry from a, a"},{"speaker":"Vikram Chatterji","startTime":1129.87,"endTime":1132.29,"body":"fault tolerance perspective is different."},{"speaker":"Vikram Chatterji","startTime":1133.14,"endTime":1136.8,"body":"Within that, if you're an engineering"},{"speaker":"Vikram Chatterji","startTime":1133.14,"endTime":1136.8,"body":"leader at a faster moving company,"},{"speaker":"Vikram Chatterji","startTime":1137.38,"endTime":1140.55,"body":"if you're all about experimentation"},{"speaker":"Vikram Chatterji","startTime":1137.38,"endTime":1140.55,"body":"and going fast, then the question"},{"speaker":"Vikram Chatterji","startTime":1140.55,"endTime":1143.92,"body":"becomes, how do you plan well, and"},{"speaker":"Vikram Chatterji","startTime":1140.55,"endTime":1143.92,"body":"there is a certain crawl walk around"},{"speaker":"Vikram Chatterji","startTime":1143.92,"endTime":1146.6,"body":"there as well, to be honest, Andrew,"},{"speaker":"Vikram Chatterji","startTime":1143.92,"endTime":1146.6,"body":"because what we've seen there is."},{"speaker":"Vikram Chatterji","startTime":1147.03,"endTime":1151.03,"body":"You have to think about, you know, is it"},{"speaker":"Vikram Chatterji","startTime":1147.03,"endTime":1151.03,"body":"going to be 15 use cases, 10 use cases?"},{"speaker":"Vikram Chatterji","startTime":1151.09,"endTime":1154.55,"body":"You have to have some forecasting there"},{"speaker":"Vikram Chatterji","startTime":1151.09,"endTime":1154.55,"body":"because once based on the forecasting,"},{"speaker":"Vikram Chatterji","startTime":1154.55,"endTime":1155.68,"body":"you have to do a couple of things."},{"speaker":"Vikram Chatterji","startTime":1155.85,"endTime":1159.5,"body":"You first think about the compute"},{"speaker":"Vikram Chatterji","startTime":1155.85,"endTime":1159.5,"body":"costs, and then staff yourself with,"},{"speaker":"Vikram Chatterji","startTime":1159.73,"endTime":1164.62,"body":"enough, I don't know, A100s and"},{"speaker":"Vikram Chatterji","startTime":1159.73,"endTime":1164.62,"body":"have that like available for you"},{"speaker":"Vikram Chatterji","startTime":1164.66,"endTime":1165.95,"body":"so that everyone can just build."},{"speaker":"Vikram Chatterji","startTime":1165.98,"endTime":1168.7,"body":"Otherwise everyone's going to come back to"},{"speaker":"Vikram Chatterji","startTime":1165.98,"endTime":1168.7,"body":"you and say like, Hey, where's my GPU at?"},{"speaker":"Vikram Chatterji","startTime":1169.27,"endTime":1171.45,"body":"Uh, so you have to have like"},{"speaker":"Vikram Chatterji","startTime":1169.27,"endTime":1171.45,"body":"X amount of compute and give"},{"speaker":"Vikram Chatterji","startTime":1171.46,"endTime":1172.81,"body":"that out very judiciously."},{"speaker":"Vikram Chatterji","startTime":1173.085,"endTime":1176.425,"body":"You then have to think about, like, how"},{"speaker":"Vikram Chatterji","startTime":1173.085,"endTime":1176.425,"body":"can you optimize the compute and, you"},{"speaker":"Vikram Chatterji","startTime":1176.425,"endTime":1181.505,"body":"know, start to invest in tooling at that"},{"speaker":"Vikram Chatterji","startTime":1176.425,"endTime":1181.505,"body":"layer to minimize the cost of compute."},{"speaker":"Vikram Chatterji","startTime":1182.045,"endTime":1186.045,"body":"And then comes the eval piece, which"},{"speaker":"Vikram Chatterji","startTime":1182.045,"endTime":1186.045,"body":"is kind of what Galileo does, where"},{"speaker":"Vikram Chatterji","startTime":1186.045,"endTime":1188.555,"body":"you have to think about how do you"},{"speaker":"Vikram Chatterji","startTime":1186.045,"endTime":1188.555,"body":"create the right kind of guardrails in"},{"speaker":"Vikram Chatterji","startTime":1188.555,"endTime":1191.945,"body":"place for, like, an AI CICD process."},{"speaker":"Vikram Chatterji","startTime":1192.325,"endTime":1194.805,"body":"and then basically go to the teams"},{"speaker":"Vikram Chatterji","startTime":1192.325,"endTime":1194.805,"body":"and say, awesome, all right, you"},{"speaker":"Vikram Chatterji","startTime":1194.805,"endTime":1196.845,"body":"want to, you want to launch things,"},{"speaker":"Vikram Chatterji","startTime":1194.805,"endTime":1196.845,"body":"you want to experiment with things?"},{"speaker":"Vikram Chatterji","startTime":1197.33,"endTime":1199.19,"body":"Here's the stack that you can use."},{"speaker":"Vikram Chatterji","startTime":1199.82,"endTime":1201.07,"body":"Go knock yourself out, right?"},{"speaker":"Vikram Chatterji","startTime":1201.15,"endTime":1204.08,"body":"And some people will use the entire"},{"speaker":"Vikram Chatterji","startTime":1201.15,"endTime":1204.08,"body":"stack, some won't, but you have to"},{"speaker":"Vikram Chatterji","startTime":1204.08,"endTime":1209.48,"body":"create that enablement almost within"},{"speaker":"Vikram Chatterji","startTime":1204.08,"endTime":1209.48,"body":"the team before they can like go crazy."},{"speaker":"Andrew Zigler","startTime":1210.144,"endTime":1211.044,"body":"That's a big unlock."},{"speaker":"Andrew Zigler","startTime":1211.044,"endTime":1215.034,"body":"So let me try to unpack this playbook,"},{"speaker":"Andrew Zigler","startTime":1211.044,"endTime":1215.034,"body":"because I think there's a lot of"},{"speaker":"Andrew Zigler","startTime":1215.064,"endTime":1216.544,"body":"really interesting tips in here."},{"speaker":"Andrew Zigler","startTime":1216.724,"endTime":1220.304,"body":"One of them being first and foremost,"},{"speaker":"Andrew Zigler","startTime":1216.724,"endTime":1220.304,"body":"understanding your company, your company"},{"speaker":"Andrew Zigler","startTime":1220.374,"endTime":1224.234,"body":"culture, their risk tolerance and"},{"speaker":"Andrew Zigler","startTime":1220.374,"endTime":1224.234,"body":"what they're doing, and understanding"},{"speaker":"Andrew Zigler","startTime":1224.244,"endTime":1227.534,"body":"that there's a big difference between"},{"speaker":"Andrew Zigler","startTime":1224.244,"endTime":1227.534,"body":"a traditional enterprise and a"},{"speaker":"Andrew Zigler","startTime":1227.544,"endTime":1232.484,"body":"digital native company experimenting"},{"speaker":"Andrew Zigler","startTime":1227.544,"endTime":1232.484,"body":"with AI right now, especially"},{"speaker":"Andrew Zigler","startTime":1232.484,"endTime":1233.784,"body":"with different levels of risk."},{"speaker":"Andrew Zigler","startTime":1233.959,"endTime":1234.989,"body":"within what they're working on."},{"speaker":"Andrew Zigler","startTime":1234.989,"endTime":1237.799,"body":"So it's about understanding your"},{"speaker":"Andrew Zigler","startTime":1234.989,"endTime":1237.799,"body":"own company, your own environment,"},{"speaker":"Andrew Zigler","startTime":1238.059,"endTime":1241.119,"body":"the level of risk and the"},{"speaker":"Andrew Zigler","startTime":1238.059,"endTime":1241.119,"body":"tolerance for experimentation."},{"speaker":"Andrew Zigler","startTime":1241.119,"endTime":1246.369,"body":"But then getting into that, you know,"},{"speaker":"Andrew Zigler","startTime":1241.119,"endTime":1246.369,"body":"crawl, walk, run loop that's going to get"},{"speaker":"Andrew Zigler","startTime":1246.369,"endTime":1248.769,"body":"you up and get you moving on this process."},{"speaker":"Andrew Zigler","startTime":1249.109,"endTime":1253.689,"body":"Um, it's about creating, the"},{"speaker":"Andrew Zigler","startTime":1249.109,"endTime":1253.689,"body":"actual resources to enable those"},{"speaker":"Andrew Zigler","startTime":1253.689,"endTime":1255.579,"body":"teams to make effective tools."},{"speaker":"Andrew Zigler","startTime":1255.579,"endTime":1259.319,"body":"And so if you're going to create, a way"},{"speaker":"Andrew Zigler","startTime":1255.579,"endTime":1259.319,"body":"for them to grow within your company,"},{"speaker":"Andrew Zigler","startTime":1259.319,"endTime":1261.199,"body":"you have to think about resourcing them."},{"speaker":"Andrew Zigler","startTime":1261.554,"endTime":1265.444,"body":"And prioritizing them based"},{"speaker":"Andrew Zigler","startTime":1261.554,"endTime":1265.444,"body":"upon, again, you know, maybe the"},{"speaker":"Andrew Zigler","startTime":1265.444,"endTime":1267.324,"body":"profile of your company culture."},{"speaker":"Andrew Zigler","startTime":1267.769,"endTime":1270.519,"body":"So it really does start with"},{"speaker":"Andrew Zigler","startTime":1267.769,"endTime":1270.519,"body":"understanding your culture."},{"speaker":"Vikram Chatterji","startTime":1271.165,"endTime":1271.715,"body":"exactly."},{"speaker":"Vikram Chatterji","startTime":1271.715,"endTime":1273.055,"body":"And you hit on a good point."},{"speaker":"Vikram Chatterji","startTime":1273.055,"endTime":1274.365,"body":"It's the, it's the culture."},{"speaker":"Vikram Chatterji","startTime":1274.365,"endTime":1276.255,"body":"It's the, it's the people."},{"speaker":"Vikram Chatterji","startTime":1276.896,"endTime":1280.446,"body":"again, it comes down to like also the,"},{"speaker":"Vikram Chatterji","startTime":1276.896,"endTime":1280.446,"body":"the business that you're running, whether"},{"speaker":"Vikram Chatterji","startTime":1280.446,"endTime":1283.112,"body":"you've been, you know, an Instacart or"},{"speaker":"Vikram Chatterji","startTime":1280.446,"endTime":1283.112,"body":"a DoorDash, again, going back to that"},{"speaker":"Vikram Chatterji","startTime":1283.112,"endTime":1287.433,"body":"example, they have many different ways"},{"speaker":"Vikram Chatterji","startTime":1283.112,"endTime":1287.433,"body":"of, instituting generative AI, but maybe"},{"speaker":"Vikram Chatterji","startTime":1287.433,"endTime":1291.023,"body":"there's a different company that doesn't"},{"speaker":"Vikram Chatterji","startTime":1287.433,"endTime":1291.023,"body":"have those many use cases and that's okay."},{"speaker":"Vikram Chatterji","startTime":1291.349,"endTime":1293.709,"body":"you kind of have to look at all of"},{"speaker":"Vikram Chatterji","startTime":1291.349,"endTime":1293.709,"body":"those different angles and then figure"},{"speaker":"Vikram Chatterji","startTime":1293.709,"endTime":1297.029,"body":"out, how you want to act and, and"},{"speaker":"Vikram Chatterji","startTime":1293.709,"endTime":1297.029,"body":"how, how quickly you want to act."},{"speaker":"Vikram Chatterji","startTime":1297.039,"endTime":1298.149,"body":"But it does stem with that."},{"speaker":"Vikram Chatterji","startTime":1298.159,"endTime":1301.589,"body":"As an engineering leader, I would say"},{"speaker":"Vikram Chatterji","startTime":1298.159,"endTime":1301.589,"body":"like step one is always, always just that."},{"speaker":"Andrew Zigler","startTime":1302.278,"endTime":1302.678,"body":"Right."},{"speaker":"Andrew Zigler","startTime":1303.218,"endTime":1306.768,"body":"And part of company culture, and it's"},{"speaker":"Andrew Zigler","startTime":1303.218,"endTime":1306.768,"body":"kind of what I want to focus on next"},{"speaker":"Andrew Zigler","startTime":1306.768,"endTime":1311.568,"body":"is also, you know, fears and anxieties"},{"speaker":"Andrew Zigler","startTime":1306.768,"endTime":1311.568,"body":"around, you know, Making the right or"},{"speaker":"Andrew Zigler","startTime":1311.568,"endTime":1316.768,"body":"wrong decisions or creating tools that"},{"speaker":"Andrew Zigler","startTime":1311.568,"endTime":1316.768,"body":"are doing jobs that traditionally people"},{"speaker":"Andrew Zigler","startTime":1316.768,"endTime":1320.548,"body":"did within their org and understanding"},{"speaker":"Andrew Zigler","startTime":1316.768,"endTime":1320.548,"body":"how people need to reprioritize"},{"speaker":"Andrew Zigler","startTime":1320.848,"endTime":1323.328,"body":"their time to better use these tools."},{"speaker":"Andrew Zigler","startTime":1323.358,"endTime":1326.508,"body":"And in doing all of this, I"},{"speaker":"Andrew Zigler","startTime":1323.358,"endTime":1326.508,"body":"think we're all trying to make"},{"speaker":"Andrew Zigler","startTime":1326.508,"endTime":1328.098,"body":"systems that are future proof."},{"speaker":"Andrew Zigler","startTime":1328.308,"endTime":1332.078,"body":"We don't want to rebuild these AI"},{"speaker":"Andrew Zigler","startTime":1328.308,"endTime":1332.078,"body":"workflows again and again and again."},{"speaker":"Andrew Zigler","startTime":1332.078,"endTime":1335.938,"body":"We want to make it once and"},{"speaker":"Andrew Zigler","startTime":1332.078,"endTime":1335.938,"body":"evaluate it and iterate on it."},{"speaker":"Andrew Zigler","startTime":1336.128,"endTime":1340.983,"body":"So, That's part of justifying too, the ROI"},{"speaker":"Andrew Zigler","startTime":1336.128,"endTime":1340.983,"body":"and going back to creating the resources"},{"speaker":"Andrew Zigler","startTime":1341.003,"endTime":1342.753,"body":"within your team for that to grow."},{"speaker":"Andrew Zigler","startTime":1343.063,"endTime":1346.953,"body":"So how can an engineering leader,"},{"speaker":"Andrew Zigler","startTime":1343.063,"endTime":1346.953,"body":"perhaps someone who's in a company"},{"speaker":"Andrew Zigler","startTime":1346.953,"endTime":1350.482,"body":"culture that's maybe more on the"},{"speaker":"Andrew Zigler","startTime":1346.953,"endTime":1350.482,"body":"digital native side, having an"},{"speaker":"Andrew Zigler","startTime":1350.492,"endTime":1352.632,"body":"appetite for risk and experimentation."},{"speaker":"Andrew Zigler","startTime":1352.632,"endTime":1355.062,"body":"Maybe they even have a"},{"speaker":"Andrew Zigler","startTime":1352.632,"endTime":1355.062,"body":"little bit of resources."},{"speaker":"Andrew Zigler","startTime":1355.437,"endTime":1359.857,"body":"How can they shift those conversations"},{"speaker":"Andrew Zigler","startTime":1355.437,"endTime":1359.857,"body":"with non technical leadership from"},{"speaker":"Andrew Zigler","startTime":1360.037,"endTime":1363.822,"body":"immediate gains to Creating future"},{"speaker":"Andrew Zigler","startTime":1360.037,"endTime":1363.822,"body":"proof solutions that are going"},{"speaker":"Andrew Zigler","startTime":1363.822,"endTime":1366.992,"body":"to help the company in maybe like"},{"speaker":"Andrew Zigler","startTime":1363.822,"endTime":1366.992,"body":"a year or five years from now."},{"speaker":"Vikram Chatterji","startTime":1367.81,"endTime":1370.245,"body":"it comes down"},{"speaker":"Vikram Chatterji","startTime":1367.81,"endTime":1370.245,"body":"to, you know, Building trust."},{"speaker":"Vikram Chatterji","startTime":1370.435,"endTime":1373.589,"body":"I think the first thing becomes like as"},{"speaker":"Vikram Chatterji","startTime":1370.435,"endTime":1373.589,"body":"an engineering leader, I've, talked to"},{"speaker":"Vikram Chatterji","startTime":1373.589,"endTime":1377.089,"body":"a lot of different leaders in the space"},{"speaker":"Vikram Chatterji","startTime":1373.589,"endTime":1377.089,"body":"and it all comes down to number one,"},{"speaker":"Vikram Chatterji","startTime":1377.099,"endTime":1382.109,"body":"first, having a gut instinct reaction,"},{"speaker":"Vikram Chatterji","startTime":1377.099,"endTime":1382.109,"body":"a gut instinct on their own side,"},{"speaker":"Vikram Chatterji","startTime":1382.359,"endTime":1386.109,"body":"personally, like some kind of a trust"},{"speaker":"Vikram Chatterji","startTime":1382.359,"endTime":1386.109,"body":"that, you know, these use cases are great."},{"speaker":"Vikram Chatterji","startTime":1386.489,"endTime":1389.589,"body":"And this use case is actually going"},{"speaker":"Vikram Chatterji","startTime":1386.489,"endTime":1389.589,"body":"to be a good one to start with."},{"speaker":"Vikram Chatterji","startTime":1390.264,"endTime":1392.754,"body":"The second is, staff that."},{"speaker":"Vikram Chatterji","startTime":1392.774,"endTime":1395.644,"body":"Just go very, very fast and build"},{"speaker":"Vikram Chatterji","startTime":1392.774,"endTime":1395.644,"body":"out a prototype and start to see"},{"speaker":"Vikram Chatterji","startTime":1395.644,"endTime":1396.804,"body":"does that actually add value?"},{"speaker":"Vikram Chatterji","startTime":1397.134,"endTime":1400.124,"body":"And then shop that around"},{"speaker":"Vikram Chatterji","startTime":1397.134,"endTime":1400.124,"body":"with others in your business."},{"speaker":"Vikram Chatterji","startTime":1400.124,"endTime":1403.324,"body":"They could be leaders in the business,"},{"speaker":"Vikram Chatterji","startTime":1400.124,"endTime":1403.324,"body":"depending on how large the company is."},{"speaker":"Vikram Chatterji","startTime":1403.724,"endTime":1407.114,"body":"So that, and you can now with, with"},{"speaker":"Vikram Chatterji","startTime":1403.724,"endTime":1407.114,"body":"Gen AI, the unlock is that you can"},{"speaker":"Vikram Chatterji","startTime":1407.124,"endTime":1408.464,"body":"build a prototype pretty quickly."},{"speaker":"Vikram Chatterji","startTime":1408.964,"endTime":1412.004,"body":"And so you can at least start to get a"},{"speaker":"Vikram Chatterji","startTime":1408.964,"endTime":1412.004,"body":"sense for like, what is the appetite here?"},{"speaker":"Vikram Chatterji","startTime":1412.364,"endTime":1413.774,"body":"and that's typically"},{"speaker":"Vikram Chatterji","startTime":1412.364,"endTime":1413.774,"body":"what I've seen happen."},{"speaker":"Vikram Chatterji","startTime":1414.489,"endTime":1417.189,"body":"And then from, from there you"},{"speaker":"Vikram Chatterji","startTime":1414.489,"endTime":1417.189,"body":"can start to figure out what's"},{"speaker":"Vikram Chatterji","startTime":1417.189,"endTime":1418.929,"body":"the, what the KPIs are, right?"},{"speaker":"Vikram Chatterji","startTime":1418.969,"endTime":1422.239,"body":"Because the main thing is you want to be"},{"speaker":"Vikram Chatterji","startTime":1418.969,"endTime":1422.239,"body":"able to ship something pretty quickly."},{"speaker":"Vikram Chatterji","startTime":1422.249,"endTime":1425.319,"body":"You want to go to production quickly with"},{"speaker":"Vikram Chatterji","startTime":1422.249,"endTime":1425.319,"body":"the right checks and balances in place."},{"speaker":"Vikram Chatterji","startTime":1425.329,"endTime":1429.379,"body":"So you start with at least one to"},{"speaker":"Vikram Chatterji","startTime":1425.329,"endTime":1429.379,"body":"two use cases as quickly as you"},{"speaker":"Vikram Chatterji","startTime":1429.379,"endTime":1433.039,"body":"possibly can with the right checks"},{"speaker":"Vikram Chatterji","startTime":1429.379,"endTime":1433.039,"body":"and balances and guardrails in place."},{"speaker":"Vikram Chatterji","startTime":1433.409,"endTime":1435.329,"body":"And then start to see how that looks."},{"speaker":"Vikram Chatterji","startTime":1435.909,"endTime":1438.479,"body":"Roll it out to 1%, 5%, 10%."},{"speaker":"Vikram Chatterji","startTime":1438.479,"endTime":1439.559,"body":"Start to see what you learn."},{"speaker":"Vikram Chatterji","startTime":1439.969,"endTime":1443.139,"body":"And then with that playbook, you can"},{"speaker":"Vikram Chatterji","startTime":1439.969,"endTime":1443.139,"body":"go very fast with everything else."},{"speaker":"Vikram Chatterji","startTime":1444.109,"endTime":1448.039,"body":"This is exactly what we've seen with the"},{"speaker":"Vikram Chatterji","startTime":1444.109,"endTime":1448.039,"body":"largest banks in the world, as well as"},{"speaker":"Vikram Chatterji","startTime":1448.049,"endTime":1451.979,"body":"the digital natives, it's just that the"},{"speaker":"Vikram Chatterji","startTime":1448.049,"endTime":1451.979,"body":"digital natives are just moving much,"},{"speaker":"Vikram Chatterji","startTime":1451.999,"endTime":1453.959,"body":"much faster than the largest banks."},{"speaker":"Vikram Chatterji","startTime":1454.489,"endTime":1457.29,"body":"Uh, but this playbook is kind of similar"},{"speaker":"Andrew Zigler","startTime":1457.359,"endTime":1461.309,"body":"It makes a lot of sense in"},{"speaker":"Andrew Zigler","startTime":1457.359,"endTime":1461.309,"body":"terms of how those teams can get started."},{"speaker":"Andrew Zigler","startTime":1461.309,"endTime":1464.869,"body":"It's also about evaluating, going"},{"speaker":"Andrew Zigler","startTime":1461.309,"endTime":1464.869,"body":"back to the evaluative frameworks from"},{"speaker":"Andrew Zigler","startTime":1464.889,"endTime":1467.199,"body":"earlier and having measurements in place."},{"speaker":"Andrew Zigler","startTime":1467.569,"endTime":1469.289,"body":"to look at the results week after week."},{"speaker":"Andrew Zigler","startTime":1469.289,"endTime":1472.839,"body":"And that creates within the company"},{"speaker":"Andrew Zigler","startTime":1469.289,"endTime":1472.839,"body":"culture a healthy socialization"},{"speaker":"Andrew Zigler","startTime":1472.839,"endTime":1476.779,"body":"and understanding of the tools that"},{"speaker":"Andrew Zigler","startTime":1472.839,"endTime":1476.779,"body":"people are building and we're using."},{"speaker":"Andrew Zigler","startTime":1476.779,"endTime":1479.509,"body":"And that's what cultivates the trust."},{"speaker":"Andrew Zigler","startTime":1479.719,"endTime":1484.449,"body":"And it's also something really"},{"speaker":"Andrew Zigler","startTime":1479.719,"endTime":1484.449,"body":"hard to find in an LLM based world."},{"speaker":"Andrew Zigler","startTime":1484.819,"endTime":1486.339,"body":"You know, LLMs are stochastic."},{"speaker":"Andrew Zigler","startTime":1486.339,"endTime":1487.979,"body":"They don't want to repeat themselves."},{"speaker":"Andrew Zigler","startTime":1487.979,"endTime":1490.609,"body":"So when you put them into an"},{"speaker":"Andrew Zigler","startTime":1487.979,"endTime":1490.609,"body":"environment where you want a"},{"speaker":"Andrew Zigler","startTime":1490.609,"endTime":1495.084,"body":"repeatable workflow, Where you have"},{"speaker":"Andrew Zigler","startTime":1490.609,"endTime":1495.084,"body":"an AI agent evaluating things on the"},{"speaker":"Andrew Zigler","startTime":1495.084,"endTime":1497.564,"body":"fly, there's, so much to consider."},{"speaker":"Andrew Zigler","startTime":1497.744,"endTime":1500.974,"body":"and that's kind of part of, you"},{"speaker":"Andrew Zigler","startTime":1497.744,"endTime":1500.974,"body":"know, the dauntingness of, that"},{"speaker":"Andrew Zigler","startTime":1500.974,"endTime":1502.494,"body":"I, I was alluding to earlier."},{"speaker":"Andrew Zigler","startTime":1502.834,"endTime":1506.014,"body":"in our initial conversation, you know,"},{"speaker":"Andrew Zigler","startTime":1502.834,"endTime":1506.014,"body":"there was something that you, you"},{"speaker":"Andrew Zigler","startTime":1506.014,"endTime":1509.714,"body":"mentioned to me that's really resonated"},{"speaker":"Andrew Zigler","startTime":1506.014,"endTime":1509.714,"body":"with me since, and is, you mentioned how,"},{"speaker":"Andrew Zigler","startTime":1509.714,"endTime":1514.794,"body":"you know, AI agents in the future, or"},{"speaker":"Andrew Zigler","startTime":1509.714,"endTime":1514.794,"body":"even now, they can act as smart routers or"},{"speaker":"Andrew Zigler","startTime":1514.954,"endTime":1519.964,"body":"kind of like load balancers for workflows"},{"speaker":"Andrew Zigler","startTime":1514.954,"endTime":1519.964,"body":"and help optimize them over time."},{"speaker":"Andrew Zigler","startTime":1519.964,"endTime":1521.744,"body":"And that was really fascinating to me."},{"speaker":"Andrew Zigler","startTime":1522.154,"endTime":1525.154,"body":"I'm wondering, maybe you could dive"},{"speaker":"Andrew Zigler","startTime":1522.154,"endTime":1525.154,"body":"into that a little more, just on a"},{"speaker":"Andrew Zigler","startTime":1525.154,"endTime":1528.394,"body":"technical level, now that we've talked"},{"speaker":"Andrew Zigler","startTime":1525.154,"endTime":1528.394,"body":"about introducing these products and,"},{"speaker":"Andrew Zigler","startTime":1528.584,"endTime":1530.564,"body":"or these projects within your company."},{"speaker":"Andrew Zigler","startTime":1530.844,"endTime":1536.354,"body":"If you are somebody who's revolutionizing"},{"speaker":"Andrew Zigler","startTime":1530.844,"endTime":1536.354,"body":"a workflow, Right now with AI, how"},{"speaker":"Andrew Zigler","startTime":1536.444,"endTime":1538.974,"body":"should they best kind of think about it?"},{"speaker":"Vikram Chatterji","startTime":1539.686,"endTime":1541.916,"body":"is in terms"},{"speaker":"Vikram Chatterji","startTime":1539.686,"endTime":1541.916,"body":"of agents in particular, how"},{"speaker":"Vikram Chatterji","startTime":1541.916,"endTime":1542.386,"body":"that works."},{"speaker":"Vikram Chatterji","startTime":1542.386,"endTime":1547.379,"body":"So, for context, for folks like Galileo"},{"speaker":"Vikram Chatterji","startTime":1542.386,"endTime":1547.379,"body":"is, leading provider of, Evaluation"},{"speaker":"Vikram Chatterji","startTime":1547.379,"endTime":1551.539,"body":"tooling for any AI developer out there"},{"speaker":"Vikram Chatterji","startTime":1547.379,"endTime":1551.539,"body":"that's building an AI app, right?"},{"speaker":"Vikram Chatterji","startTime":1551.999,"endTime":1555.219,"body":"and what that means is an eval"},{"speaker":"Vikram Chatterji","startTime":1551.999,"endTime":1555.219,"body":"basically includes not just the"},{"speaker":"Vikram Chatterji","startTime":1555.219,"endTime":1558.429,"body":"metrics, but also like the dataset that"},{"speaker":"Vikram Chatterji","startTime":1555.219,"endTime":1558.429,"body":"you're working with and it includes"},{"speaker":"Vikram Chatterji","startTime":1558.429,"endTime":1561.529,"body":"like a workflow so you can really"},{"speaker":"Vikram Chatterji","startTime":1558.429,"endTime":1561.529,"body":"understand what your failure modes are."},{"speaker":"Vikram Chatterji","startTime":1561.989,"endTime":1565.299,"body":"Build out evals around that and"},{"speaker":"Vikram Chatterji","startTime":1561.989,"endTime":1565.299,"body":"then use that in your CICD process."},{"speaker":"Vikram Chatterji","startTime":1565.669,"endTime":1568.599,"body":"So you're not, you know exactly if you're"},{"speaker":"Vikram Chatterji","startTime":1565.669,"endTime":1568.599,"body":"shipping a good product and once you"},{"speaker":"Vikram Chatterji","startTime":1568.599,"endTime":1569.999,"body":"ship, you can check for regression tests."},{"speaker":"Vikram Chatterji","startTime":1570.439,"endTime":1572.579,"body":"That's the net net of how evals work."},{"speaker":"Vikram Chatterji","startTime":1572.999,"endTime":1577.769,"body":"now, with agents, what's interesting"},{"speaker":"Vikram Chatterji","startTime":1572.999,"endTime":1577.769,"body":"is we've kind of moved from people"},{"speaker":"Vikram Chatterji","startTime":1577.769,"endTime":1580.9,"body":"building, like, just chatbots that"},{"speaker":"Vikram Chatterji","startTime":1577.769,"endTime":1580.9,"body":"was, like, It almost felt like that's"},{"speaker":"Vikram Chatterji","startTime":1580.9,"endTime":1584.72,"body":"just as far as the imagination could"},{"speaker":"Vikram Chatterji","startTime":1580.9,"endTime":1584.72,"body":"go in terms of use cases to now"},{"speaker":"Vikram Chatterji","startTime":1584.97,"endTime":1586.49,"body":"it's like you can complete any task."},{"speaker":"Vikram Chatterji","startTime":1586.54,"endTime":1588.43,"body":"What's the task that your product does?"},{"speaker":"Vikram Chatterji","startTime":1589.27,"endTime":1591.88,"body":"You know, we started seeing this with"},{"speaker":"Vikram Chatterji","startTime":1589.27,"endTime":1591.88,"body":"operators launch as well, which is"},{"speaker":"Vikram Chatterji","startTime":1591.88,"endTime":1595.63,"body":"a good example of an agent where all"},{"speaker":"Vikram Chatterji","startTime":1591.88,"endTime":1595.63,"body":"of a sudden I started to see booking."},{"speaker":"Vikram Chatterji","startTime":1595.63,"endTime":1599.12,"body":"com's folks talking about like,"},{"speaker":"Vikram Chatterji","startTime":1595.63,"endTime":1599.12,"body":"Hey, now you can book a hotel"},{"speaker":"Vikram Chatterji","startTime":1599.12,"endTime":1600.4,"body":"room with just natural language."},{"speaker":"Vikram Chatterji","startTime":1600.4,"endTime":1601.37,"body":"And then box."},{"speaker":"Vikram Chatterji","startTime":1601.37,"endTime":1604.89,"body":"com's CEOs started talking about how"},{"speaker":"Vikram Chatterji","startTime":1601.37,"endTime":1604.89,"body":"you can like add files and folders."},{"speaker":"Vikram Chatterji","startTime":1605.01,"endTime":1607.15,"body":"So it just unlocked all"},{"speaker":"Vikram Chatterji","startTime":1605.01,"endTime":1607.15,"body":"these use cases for people."},{"speaker":"Vikram Chatterji","startTime":1607.15,"endTime":1609.16,"body":"And that's kind of what we're"},{"speaker":"Vikram Chatterji","startTime":1607.15,"endTime":1609.16,"body":"seeing with agents coming up."},{"speaker":"Vikram Chatterji","startTime":1609.56,"endTime":1609.91,"body":"Are they appropriate?"},{"speaker":"Vikram Chatterji","startTime":1610.085,"endTime":1612.5,"body":"Are these Are these apps perfect?"},{"speaker":"Vikram Chatterji","startTime":1612.61,"endTime":1612.89,"body":"No."},{"speaker":"Vikram Chatterji","startTime":1613.37,"endTime":1614.07,"body":"And here's why."},{"speaker":"Vikram Chatterji","startTime":1614.28,"endTime":1618.13,"body":"Because what's happening with agents"},{"speaker":"Vikram Chatterji","startTime":1614.28,"endTime":1618.13,"body":"is, it's essentially a way to say like,"},{"speaker":"Vikram Chatterji","startTime":1618.13,"endTime":1622.89,"body":"Hey, LLM, instead of just generating"},{"speaker":"Vikram Chatterji","startTime":1618.13,"endTime":1622.89,"body":"something, why don't you just act"},{"speaker":"Vikram Chatterji","startTime":1622.97,"endTime":1625.56,"body":"towards choosing the right kind of tool?"},{"speaker":"Vikram Chatterji","startTime":1625.92,"endTime":1627.26,"body":"Or here's a function I've created."},{"speaker":"Vikram Chatterji","startTime":1627.505,"endTime":1628.745,"body":"Wrapped it in a certain manner."},{"speaker":"Vikram Chatterji","startTime":1629.275,"endTime":1630.575,"body":"Do X for me, right?"},{"speaker":"Vikram Chatterji","startTime":1630.755,"endTime":1632.765,"body":"So making it do specific things."},{"speaker":"Vikram Chatterji","startTime":1632.895,"endTime":1636.599,"body":"the tool piece of this especially is"},{"speaker":"Vikram Chatterji","startTime":1632.895,"endTime":1636.599,"body":"interesting because now the API can act"},{"speaker":"Vikram Chatterji","startTime":1636.797,"endTime":1641.437,"body":"as, as you mentioned, as a smart router"},{"speaker":"Vikram Chatterji","startTime":1636.797,"endTime":1641.437,"body":"to figure out, you know, which tool"},{"speaker":"Vikram Chatterji","startTime":1641.437,"endTime":1643.067,"body":"should I use for finishing this task?"},{"speaker":"Vikram Chatterji","startTime":1643.462,"endTime":1646.472,"body":"Which is, I think, the biggest unlock"},{"speaker":"Vikram Chatterji","startTime":1643.462,"endTime":1646.472,"body":"right now, because earlier you would"},{"speaker":"Vikram Chatterji","startTime":1646.472,"endTime":1649.682,"body":"just code all of that, you would like,"},{"speaker":"Vikram Chatterji","startTime":1646.472,"endTime":1649.682,"body":"literally, deterministically say,"},{"speaker":"Vikram Chatterji","startTime":1649.732,"endTime":1650.992,"body":"this is exactly what you need to do."},{"speaker":"Vikram Chatterji","startTime":1651.372,"endTime":1655.542,"body":"Versus now, it's almost like, I just"},{"speaker":"Vikram Chatterji","startTime":1651.372,"endTime":1655.542,"body":"want this done, you figure it out for me."},{"speaker":"Vikram Chatterji","startTime":1656.032,"endTime":1659.142,"body":"Which is also why there's almost like"},{"speaker":"Vikram Chatterji","startTime":1656.032,"endTime":1659.142,"body":"this, leader worker relationship,"},{"speaker":"Vikram Chatterji","startTime":1659.152,"endTime":1661.702,"body":"almost, with an agent that's happening."},{"speaker":"Vikram Chatterji","startTime":1662.12,"endTime":1663.72,"body":"and so that's been a"},{"speaker":"Vikram Chatterji","startTime":1662.12,"endTime":1663.72,"body":"very interesting thing."},{"speaker":"Vikram Chatterji","startTime":1664.05,"endTime":1667.97,"body":"Big unlock, because now these,"},{"speaker":"Vikram Chatterji","startTime":1664.05,"endTime":1667.97,"body":"these agents can just find out"},{"speaker":"Vikram Chatterji","startTime":1668.29,"endTime":1669.67,"body":"the right tool and go ahead."},{"speaker":"Vikram Chatterji","startTime":1670.07,"endTime":1673.62,"body":"However, it does lead to different"},{"speaker":"Vikram Chatterji","startTime":1670.07,"endTime":1673.62,"body":"kinds of failure modes, right?"},{"speaker":"Vikram Chatterji","startTime":1673.67,"endTime":1675.94,"body":"Like, did it choose the right tool or not?"},{"speaker":"Vikram Chatterji","startTime":1676.26,"endTime":1678.72,"body":"And did the right tool get"},{"speaker":"Vikram Chatterji","startTime":1676.26,"endTime":1678.72,"body":"called in the right way or not?"},{"speaker":"Vikram Chatterji","startTime":1679.13,"endTime":1681.78,"body":"Even if it did choose the right"},{"speaker":"Vikram Chatterji","startTime":1679.13,"endTime":1681.78,"body":"tool, what happened after that?"},{"speaker":"Vikram Chatterji","startTime":1682.08,"endTime":1685.46,"body":"Can I see the entire flow of"},{"speaker":"Vikram Chatterji","startTime":1682.08,"endTime":1685.46,"body":"how all of that worked out?"},{"speaker":"Vikram Chatterji","startTime":1685.891,"endTime":1687.901,"body":"at the end of the day,"},{"speaker":"Vikram Chatterji","startTime":1685.891,"endTime":1687.901,"body":"did it complete the task?"},{"speaker":"Vikram Chatterji","startTime":1688.375,"endTime":1689.845,"body":"What do you mean by complete the task?"},{"speaker":"Vikram Chatterji","startTime":1689.845,"endTime":1692.045,"body":"How do you measure quality"},{"speaker":"Vikram Chatterji","startTime":1689.845,"endTime":1692.045,"body":"of task completion?"},{"speaker":"Vikram Chatterji","startTime":1692.455,"endTime":1694.895,"body":"Did it plan the whole"},{"speaker":"Vikram Chatterji","startTime":1692.455,"endTime":1694.895,"body":"task properly or not?"},{"speaker":"Vikram Chatterji","startTime":1695.175,"endTime":1697.995,"body":"So there's a bunch of these"},{"speaker":"Vikram Chatterji","startTime":1695.175,"endTime":1697.995,"body":"different kinds of qualitative"},{"speaker":"Vikram Chatterji","startTime":1698.175,"endTime":1699.665,"body":"stuff along the way that you need."},{"speaker":"Andrew Zigler","startTime":1699.854,"endTime":1703.004,"body":"And they need to get"},{"speaker":"Andrew Zigler","startTime":1699.854,"endTime":1703.004,"body":"evaluated in the evaluation system"},{"speaker":"Andrew Zigler","startTime":1703.004,"endTime":1704.264,"body":"that we've been talking about."},{"speaker":"Andrew Zigler","startTime":1704.324,"endTime":1704.504,"body":"So"},{"speaker":"Andrew Zigler","startTime":1704.514,"endTime":1707.674,"body":"those, all of those things need to"},{"speaker":"Andrew Zigler","startTime":1704.514,"endTime":1707.674,"body":"get looked at because when you're,"},{"speaker":"Andrew Zigler","startTime":1707.724,"endTime":1710.874,"body":"when you're in this environment"},{"speaker":"Andrew Zigler","startTime":1707.724,"endTime":1710.874,"body":"where it's sitting as a router or"},{"speaker":"Andrew Zigler","startTime":1710.874,"endTime":1714.499,"body":"it's making smart decisions, it's no"},{"speaker":"Andrew Zigler","startTime":1710.874,"endTime":1714.499,"body":"longer like a chatbot where Chatbot,"},{"speaker":"Andrew Zigler","startTime":1714.499,"endTime":1716.469,"body":"or it's QueryResponse, QueryResponse."},{"speaker":"Andrew Zigler","startTime":1716.499,"endTime":1720.409,"body":"Instead, it's request, or you"},{"speaker":"Andrew Zigler","startTime":1716.499,"endTime":1720.409,"body":"know, demand, or like, you know,"},{"speaker":"Andrew Zigler","startTime":1720.409,"endTime":1724.379,"body":"you need it to do something, and"},{"speaker":"Andrew Zigler","startTime":1720.409,"endTime":1724.379,"body":"it's then a sequence of actions."},{"speaker":"Andrew Zigler","startTime":1725.099,"endTime":1728.189,"body":"And those actions lack visibility to you."},{"speaker":"Andrew Zigler","startTime":1728.199,"endTime":1730.949,"body":"You're not staring at it making"},{"speaker":"Andrew Zigler","startTime":1728.199,"endTime":1730.949,"body":"the decisions, or the actions,"},{"speaker":"Andrew Zigler","startTime":1730.949,"endTime":1733.949,"body":"or making the API calls, you just"},{"speaker":"Andrew Zigler","startTime":1730.949,"endTime":1733.949,"body":"see what it tells you at the end."},{"speaker":"Andrew Zigler","startTime":1733.949,"endTime":1740.139,"body":"So that's kind of where that goes"},{"speaker":"Andrew Zigler","startTime":1733.949,"endTime":1740.139,"body":"back to building trust as well,"},{"speaker":"Andrew Zigler","startTime":1740.219,"endTime":1742.829,"body":"because you need to understand"},{"speaker":"Andrew Zigler","startTime":1740.219,"endTime":1742.829,"body":"what's happening in those stages."},{"speaker":"Vikram Chatterji","startTime":1743.465,"endTime":1744.475,"body":"Yeah, exactly."},{"speaker":"Vikram Chatterji","startTime":1744.935,"endTime":1747.675,"body":"It's funny how similar this is"},{"speaker":"Vikram Chatterji","startTime":1744.935,"endTime":1747.675,"body":"to working with a human being."},{"speaker":"Andrew Zigler","startTime":1747.974,"endTime":1748.634,"body":"Yeah, exactly."},{"speaker":"Andrew Zigler","startTime":1748.634,"endTime":1750.174,"body":"That's what's come in the mind for me too."},{"speaker":"Andrew Zigler","startTime":1750.204,"endTime":1754.134,"body":"I really liked what you said about"},{"speaker":"Andrew Zigler","startTime":1750.204,"endTime":1754.134,"body":"leaders and, contributors about it"},{"speaker":"Andrew Zigler","startTime":1754.134,"endTime":1758.684,"body":"kind of being this balance of the,"},{"speaker":"Andrew Zigler","startTime":1754.134,"endTime":1758.684,"body":"you as an engineer are overseeing the"},{"speaker":"Andrew Zigler","startTime":1758.684,"endTime":1762.414,"body":"output or the work of the agent and"},{"speaker":"Andrew Zigler","startTime":1758.684,"endTime":1762.414,"body":"you're responsible for its success."},{"speaker":"Andrew Zigler","startTime":1762.424,"endTime":1766.494,"body":"Just like how a manager is responsible"},{"speaker":"Andrew Zigler","startTime":1762.424,"endTime":1766.494,"body":"for their ICs, you know, success, you"},{"speaker":"Andrew Zigler","startTime":1766.494,"endTime":1770.204,"body":"have to give it the right tools, the"},{"speaker":"Andrew Zigler","startTime":1766.494,"endTime":1770.204,"body":"right environment and the right context."},{"speaker":"Andrew Zigler","startTime":1770.204,"endTime":1772.684,"body":"So it's a, I think it's a big unlock."},{"speaker":"Andrew Zigler","startTime":1772.884,"endTime":1775.134,"body":"Are you kind of seeing that"},{"speaker":"Andrew Zigler","startTime":1772.884,"endTime":1775.134,"body":"from folks that are starting"},{"speaker":"Andrew Zigler","startTime":1775.134,"endTime":1776.404,"body":"to engage in these workflows?"},{"speaker":"Vikram Chatterji","startTime":1777.345,"endTime":1779.325,"body":"we are,"},{"speaker":"Vikram Chatterji","startTime":1777.345,"endTime":1779.325,"body":"because, that's exactly the kind"},{"speaker":"Vikram Chatterji","startTime":1779.325,"endTime":1780.495,"body":"of question that they're asking."},{"speaker":"Vikram Chatterji","startTime":1780.835,"endTime":1784.615,"body":"around like, hey, how do I make"},{"speaker":"Vikram Chatterji","startTime":1780.835,"endTime":1784.615,"body":"sure that everything worked fine?"},{"speaker":"Vikram Chatterji","startTime":1784.985,"endTime":1788.225,"body":"And also if it did work fine and"},{"speaker":"Vikram Chatterji","startTime":1784.985,"endTime":1788.225,"body":"the answer is correct, can I see"},{"speaker":"Vikram Chatterji","startTime":1788.235,"endTime":1789.915,"body":"what the route was that it took?"},{"speaker":"Vikram Chatterji","startTime":1790.34,"endTime":1793.79,"body":"Because maybe it's just making"},{"speaker":"Vikram Chatterji","startTime":1790.34,"endTime":1793.79,"body":"unnecessary API calls, that it"},{"speaker":"Vikram Chatterji","startTime":1793.8,"endTime":1795.05,"body":"doesn't have to in the first place."},{"speaker":"Vikram Chatterji","startTime":1795.05,"endTime":1796.47,"body":"Can I optimize this even further?"},{"speaker":"Vikram Chatterji","startTime":1796.47,"endTime":1800.64,"body":"So there's a big question around what are"},{"speaker":"Vikram Chatterji","startTime":1796.47,"endTime":1800.64,"body":"the failure modes, but also can I have"},{"speaker":"Vikram Chatterji","startTime":1800.64,"endTime":1805.525,"body":"a visualization into like how the, how"},{"speaker":"Vikram Chatterji","startTime":1800.64,"endTime":1805.525,"body":"the entire, AI app like actually made"},{"speaker":"Vikram Chatterji","startTime":1805.525,"endTime":1807.785,"body":"its, decisions and how it was planning."},{"speaker":"Vikram Chatterji","startTime":1807.785,"endTime":1809.555,"body":"So I can maybe tweak"},{"speaker":"Vikram Chatterji","startTime":1807.785,"endTime":1809.555,"body":"things here and there."},{"speaker":"Vikram Chatterji","startTime":1809.925,"endTime":1810.825,"body":"So the system itself."},{"speaker":"Vikram Chatterji","startTime":1811.403,"endTime":1814.843,"body":"I think the folks at Databricks,"},{"speaker":"Vikram Chatterji","startTime":1811.403,"endTime":1814.843,"body":"uh, Matej Zaharia coined the terms,"},{"speaker":"Vikram Chatterji","startTime":1815.113,"endTime":1819.19,"body":"compound systems for what we at"},{"speaker":"Vikram Chatterji","startTime":1815.113,"endTime":1819.19,"body":"Galileo basically called your AI app."},{"speaker":"Vikram Chatterji","startTime":1819.58,"endTime":1823.14,"body":"that compound system is becoming"},{"speaker":"Vikram Chatterji","startTime":1819.58,"endTime":1823.14,"body":"more complex because of these agentic"},{"speaker":"Vikram Chatterji","startTime":1823.14,"endTime":1826.075,"body":"scenarios where essentially that's,"},{"speaker":"Vikram Chatterji","startTime":1823.14,"endTime":1826.075,"body":"you know, People are adding function"},{"speaker":"Vikram Chatterji","startTime":1826.075,"endTime":1830.225,"body":"calls, so it's, it's fascinating because"},{"speaker":"Vikram Chatterji","startTime":1826.075,"endTime":1830.225,"body":"now it's even closer to classical"},{"speaker":"Vikram Chatterji","startTime":1830.225,"endTime":1832.955,"body":"software engineering and it's all"},{"speaker":"Vikram Chatterji","startTime":1830.225,"endTime":1832.955,"body":"about, like, how good are your functions"},{"speaker":"Vikram Chatterji","startTime":1832.955,"endTime":1835.615,"body":"and how well are you managing it all?"},{"speaker":"Vikram Chatterji","startTime":1835.685,"endTime":1839.925,"body":"And those engineers come to us and"},{"speaker":"Vikram Chatterji","startTime":1835.685,"endTime":1839.925,"body":"they ask us about, like, great,"},{"speaker":"Vikram Chatterji","startTime":1839.925,"endTime":1843.96,"body":"this is fun, but how do I, you know,"},{"speaker":"Vikram Chatterji","startTime":1839.925,"endTime":1843.96,"body":"What is a unit test for me now?"},{"speaker":"Vikram Chatterji","startTime":1844.01,"endTime":1845.57,"body":"And what is a regression test for me now?"},{"speaker":"Vikram Chatterji","startTime":1845.61,"endTime":1846.85,"body":"And that's where evals come in."},{"speaker":"Andrew Zigler","startTime":1847.519,"endTime":1847.939,"body":"Right."},{"speaker":"Andrew Zigler","startTime":1848.054,"endTime":1852.534,"body":"I'd like to understand a little more"},{"speaker":"Andrew Zigler","startTime":1848.054,"endTime":1852.534,"body":"about how does Galileo open that box"},{"speaker":"Andrew Zigler","startTime":1852.534,"endTime":1858.084,"body":"so that you look in to understand,"},{"speaker":"Andrew Zigler","startTime":1852.534,"endTime":1858.084,"body":"how those tools are working and, and,"},{"speaker":"Andrew Zigler","startTime":1858.084,"endTime":1861.084,"body":"and how somebody would use something"},{"speaker":"Andrew Zigler","startTime":1858.084,"endTime":1861.084,"body":"like that to build confidence."},{"speaker":"Andrew Zigler","startTime":1861.084,"endTime":1864.924,"body":"Because what you're describing is like"},{"speaker":"Andrew Zigler","startTime":1861.084,"endTime":1864.924,"body":"a whole new category to me of how we"},{"speaker":"Andrew Zigler","startTime":1864.924,"endTime":1869.624,"body":"think about tools and when you're in an"},{"speaker":"Andrew Zigler","startTime":1864.924,"endTime":1869.624,"body":"environment where you're defining a novel."},{"speaker":"Andrew Zigler","startTime":1869.669,"endTime":1870.389,"body":"Category."},{"speaker":"Andrew Zigler","startTime":1870.389,"endTime":1873.039,"body":"That's really, uh, you have to"},{"speaker":"Andrew Zigler","startTime":1870.389,"endTime":1873.039,"body":"do a lot of, like, definition"},{"speaker":"Andrew Zigler","startTime":1873.039,"endTime":1874.209,"body":"setting and understanding."},{"speaker":"Andrew Zigler","startTime":1874.209,"endTime":1877.409,"body":"So we're all for the first time"},{"speaker":"Andrew Zigler","startTime":1874.209,"endTime":1877.409,"body":"opening the box and looking"},{"speaker":"Andrew Zigler","startTime":1877.409,"endTime":1878.629,"body":"inside all the workflows."},{"speaker":"Andrew Zigler","startTime":1878.669,"endTime":1880.809,"body":"Workflows that we're building"},{"speaker":"Andrew Zigler","startTime":1878.669,"endTime":1880.809,"body":"now for the first time."},{"speaker":"Andrew Zigler","startTime":1881.169,"endTime":1882.749,"body":"What does Galileo provide?"},{"speaker":"Andrew Zigler","startTime":1882.769,"endTime":1886.029,"body":"What are the things that"},{"speaker":"Andrew Zigler","startTime":1882.769,"endTime":1886.029,"body":"people should be looking for?"},{"speaker":"Vikram Chatterji","startTime":1887.07,"endTime":1890.35,"body":"what Galileo tries"},{"speaker":"Vikram Chatterji","startTime":1887.07,"endTime":1890.35,"body":"to do is our, our end goal is to help"},{"speaker":"Vikram Chatterji","startTime":1890.35,"endTime":1893.13,"body":"you build high quality AI apps fast."},{"speaker":"Vikram Chatterji","startTime":1893.44,"endTime":1894.19,"body":"That's our end goal."},{"speaker":"Vikram Chatterji","startTime":1894.65,"endTime":1899.795,"body":"so we win if you're, Building those apps"},{"speaker":"Vikram Chatterji","startTime":1894.65,"endTime":1899.795,"body":"10x faster and those apps are 10x better."},{"speaker":"Vikram Chatterji","startTime":1899.865,"endTime":1900.445,"body":"That's our goal."},{"speaker":"Vikram Chatterji","startTime":1900.865,"endTime":1903.87,"body":"Now in order to do that, that"},{"speaker":"Vikram Chatterji","startTime":1900.865,"endTime":1903.87,"body":"includes what I think of as the"},{"speaker":"Vikram Chatterji","startTime":1903.88,"endTime":1907.02,"body":"visualization layer, meaning like you"},{"speaker":"Vikram Chatterji","startTime":1903.88,"endTime":1907.02,"body":"could just see your traces and spans."},{"speaker":"Vikram Chatterji","startTime":1907.35,"endTime":1908.79,"body":"I feel like that's the easy part."},{"speaker":"Vikram Chatterji","startTime":1908.82,"endTime":1911.6,"body":"It is, it's good for them, but"},{"speaker":"Vikram Chatterji","startTime":1908.82,"endTime":1911.6,"body":"it's highly commoditizable, right?"},{"speaker":"Vikram Chatterji","startTime":1911.6,"endTime":1912.92,"body":"Anybody can build that thing."},{"speaker":"Vikram Chatterji","startTime":1913.32,"endTime":1916.0,"body":"So we obsess about the user"},{"speaker":"Vikram Chatterji","startTime":1913.32,"endTime":1916.0,"body":"experience at that, at that layer."},{"speaker":"Vikram Chatterji","startTime":1916.5,"endTime":1921.426,"body":"But then beyond that, what we've been"},{"speaker":"Vikram Chatterji","startTime":1916.5,"endTime":1921.426,"body":"seeing is we initially gave the developers"},{"speaker":"Vikram Chatterji","startTime":1921.426,"endTime":1926.244,"body":"the ability to just build their own"},{"speaker":"Vikram Chatterji","startTime":1921.426,"endTime":1926.244,"body":"metrics as well And score their agents."},{"speaker":"Vikram Chatterji","startTime":1926.304,"endTime":1930.034,"body":"And what we saw was developers struggled"},{"speaker":"Vikram Chatterji","startTime":1926.304,"endTime":1930.034,"body":"with that because they kind of had an"},{"speaker":"Vikram Chatterji","startTime":1930.044,"endTime":1933.754,"body":"idea about like, Hey, I need to see if"},{"speaker":"Vikram Chatterji","startTime":1930.044,"endTime":1933.754,"body":"it's planned this thing out well enough."},{"speaker":"Vikram Chatterji","startTime":1934.144,"endTime":1936.474,"body":"But then in order to build"},{"speaker":"Vikram Chatterji","startTime":1934.144,"endTime":1936.474,"body":"that metric, they almost had"},{"speaker":"Vikram Chatterji","startTime":1936.474,"endTime":1938.024,"body":"to build a very complex prompt."},{"speaker":"Vikram Chatterji","startTime":1938.064,"endTime":1940.054,"body":"They had to figure out the"},{"speaker":"Vikram Chatterji","startTime":1938.064,"endTime":1940.054,"body":"instructions, they had to figure"},{"speaker":"Vikram Chatterji","startTime":1940.054,"endTime":1941.224,"body":"out how do you optimize this?"},{"speaker":"Vikram Chatterji","startTime":1941.224,"endTime":1942.994,"body":"Do I use a GPT 4 model for this?"},{"speaker":"Vikram Chatterji","startTime":1942.994,"endTime":1943.644,"body":"Something else?"},{"speaker":"Vikram Chatterji","startTime":1945.084,"endTime":1947.444,"body":"that's the layer where we basically"},{"speaker":"Vikram Chatterji","startTime":1945.084,"endTime":1947.444,"body":"realized that, wait, this is"},{"speaker":"Vikram Chatterji","startTime":1947.444,"endTime":1950.284,"body":"actually, ripe for a lot of research."},{"speaker":"Vikram Chatterji","startTime":1950.294,"endTime":1955.014,"body":"So we have a, a fairly large staff of AI"},{"speaker":"Vikram Chatterji","startTime":1950.294,"endTime":1955.014,"body":"researchers that are constantly working"},{"speaker":"Vikram Chatterji","startTime":1955.014,"endTime":1957.498,"body":"on, we call our, uh, our Luna evaluation."},{"speaker":"Vikram Chatterji","startTime":1957.498,"endTime":1961.454,"body":"Metrics, where it's not just the,"},{"speaker":"Vikram Chatterji","startTime":1957.498,"endTime":1961.454,"body":"the prompt instructions and things of"},{"speaker":"Vikram Chatterji","startTime":1961.454,"endTime":1964.584,"body":"that nature, but we also focus a lot"},{"speaker":"Vikram Chatterji","startTime":1961.454,"endTime":1964.584,"body":"on how can we optimize the cost and"},{"speaker":"Vikram Chatterji","startTime":1964.584,"endTime":1966.454,"body":"latency of these different metrics."},{"speaker":"Vikram Chatterji","startTime":1966.544,"endTime":1967.984,"body":"So what does this mean for the user?"},{"speaker":"Vikram Chatterji","startTime":1968.304,"endTime":1971.894,"body":"What this means is I've built my agent"},{"speaker":"Vikram Chatterji","startTime":1968.304,"endTime":1971.894,"body":"and, I've been, I'm building my agent."},{"speaker":"Vikram Chatterji","startTime":1971.934,"endTime":1976.104,"body":"You could just use Galileo's TypeScript"},{"speaker":"Vikram Chatterji","startTime":1971.934,"endTime":1976.104,"body":"SDK or JavaScript SDK to be able"},{"speaker":"Vikram Chatterji","startTime":1976.104,"endTime":1977.938,"body":"to start logging your application."},{"speaker":"Vikram Chatterji","startTime":1978.328,"endTime":1981.188,"body":"and then on the other side,"},{"speaker":"Vikram Chatterji","startTime":1978.328,"endTime":1981.188,"body":"without any ground truth needed."},{"speaker":"Vikram Chatterji","startTime":1981.488,"endTime":1984.258,"body":"You basically magically see not"},{"speaker":"Vikram Chatterji","startTime":1981.488,"endTime":1984.258,"body":"just the visualization layer, but"},{"speaker":"Vikram Chatterji","startTime":1984.258,"endTime":1985.678,"body":"you also start to see two things."},{"speaker":"Vikram Chatterji","startTime":1985.728,"endTime":1991.551,"body":"One, you'll see these very, very"},{"speaker":"Vikram Chatterji","startTime":1985.728,"endTime":1991.551,"body":"metrics show up with an explanation"},{"speaker":"Vikram Chatterji","startTime":1991.551,"endTime":1993.621,"body":"for exactly what went wrong and why."},{"speaker":"Vikram Chatterji","startTime":1994.141,"endTime":1997.501,"body":"The second thing that you see are"},{"speaker":"Vikram Chatterji","startTime":1994.141,"endTime":1997.501,"body":"automatic insights as well, which"},{"speaker":"Vikram Chatterji","startTime":1997.501,"endTime":2000.791,"body":"are fairly easy to understand because"},{"speaker":"Vikram Chatterji","startTime":1997.501,"endTime":2000.791,"body":"what we want to do essentially as a"},{"speaker":"Vikram Chatterji","startTime":2000.791,"endTime":2005.211,"body":"developer, what they want as well is I"},{"speaker":"Vikram Chatterji","startTime":2000.791,"endTime":2005.211,"body":"just am trying to build this agent out."},{"speaker":"Vikram Chatterji","startTime":2005.211,"endTime":2007.141,"body":"I cobbled together a few things."},{"speaker":"Vikram Chatterji","startTime":2007.221,"endTime":2008.521,"body":"I'm trying to run a quick experiment."},{"speaker":"Vikram Chatterji","startTime":2008.971,"endTime":2013.171,"body":"which part of this complex"},{"speaker":"Vikram Chatterji","startTime":2008.971,"endTime":2013.171,"body":"compound system should I focus on?"},{"speaker":"Vikram Chatterji","startTime":2013.601,"endTime":2017.171,"body":"Should I focus on this API, that"},{"speaker":"Vikram Chatterji","startTime":2013.601,"endTime":2017.171,"body":"one, the prompt or something else?"},{"speaker":"Vikram Chatterji","startTime":2017.201,"endTime":2019.681,"body":"And so we basically dumb it all"},{"speaker":"Vikram Chatterji","startTime":2017.201,"endTime":2019.681,"body":"down for them as much as we can"},{"speaker":"Vikram Chatterji","startTime":2019.681,"endTime":2022.741,"body":"and tell them like, you know,"},{"speaker":"Vikram Chatterji","startTime":2019.681,"endTime":2022.741,"body":"we've gone, we have all your logs."},{"speaker":"Vikram Chatterji","startTime":2023.066,"endTime":2025.816,"body":"We also have these metrics that"},{"speaker":"Vikram Chatterji","startTime":2023.066,"endTime":2025.816,"body":"we've built out based on all of this."},{"speaker":"Vikram Chatterji","startTime":2025.896,"endTime":2027.166,"body":"Here's what we think you should do."},{"speaker":"Vikram Chatterji","startTime":2027.446,"endTime":2031.406,"body":"So it's almost like a co pilot for"},{"speaker":"Vikram Chatterji","startTime":2027.446,"endTime":2031.406,"body":"your AI application development."},{"speaker":"Vikram Chatterji","startTime":2031.416,"endTime":2035.036,"body":"That's the journey we want to go"},{"speaker":"Vikram Chatterji","startTime":2031.416,"endTime":2035.036,"body":"with them on as they're going from"},{"speaker":"Vikram Chatterji","startTime":2035.076,"endTime":2036.146,"body":"like building towards scaling."},{"speaker":"Andrew Zigler","startTime":2036.782,"endTime":2040.502,"body":"That's very fascinating"},{"speaker":"Andrew Zigler","startTime":2036.782,"endTime":2040.502,"body":"to me, because when someone's going to"},{"speaker":"Andrew Zigler","startTime":2040.702,"endTime":2044.232,"body":"look at all of these different variables,"},{"speaker":"Andrew Zigler","startTime":2040.702,"endTime":2044.232,"body":"what Galileo is doing is it's helping you"},{"speaker":"Andrew Zigler","startTime":2044.292,"endTime":2048.072,"body":"isolate those variables and find the ones"},{"speaker":"Andrew Zigler","startTime":2044.292,"endTime":2048.072,"body":"that are gonna have the biggest impact for"},{"speaker":"Andrew Zigler","startTime":2048.072,"endTime":2051.932,"body":"you to focus on, which kind of resonates"},{"speaker":"Andrew Zigler","startTime":2048.072,"endTime":2051.932,"body":"with like the whole top line objective of"},{"speaker":"Andrew Zigler","startTime":2051.932,"endTime":2056.202,"body":"folks having to, or being in a position"},{"speaker":"Andrew Zigler","startTime":2051.932,"endTime":2056.202,"body":"where they are experimenting with AI or"},{"speaker":"Andrew Zigler","startTime":2056.202,"endTime":2060.892,"body":"building workflows as they're trying to"},{"speaker":"Andrew Zigler","startTime":2056.202,"endTime":2060.892,"body":"evaluate and justify where they should be"},{"speaker":"Andrew Zigler","startTime":2060.892,"endTime":2065.402,"body":"spending their time, because it's moving"},{"speaker":"Andrew Zigler","startTime":2060.892,"endTime":2065.402,"body":"so quickly, that Their time always needs"},{"speaker":"Andrew Zigler","startTime":2065.402,"endTime":2069.252,"body":"to be spent on the most impactful part"},{"speaker":"Andrew Zigler","startTime":2065.402,"endTime":2069.252,"body":"of the project, or anything else you're"},{"speaker":"Andrew Zigler","startTime":2069.252,"endTime":2073.052,"body":"working on is likely just, adding risk to"},{"speaker":"Andrew Zigler","startTime":2069.252,"endTime":2073.052,"body":"the project because you're doing things"},{"speaker":"Andrew Zigler","startTime":2073.052,"endTime":2074.212,"body":"that are probably going to be outdated."},{"speaker":"Andrew Zigler","startTime":2074.462,"endTime":2076.812,"body":"By the time they're really"},{"speaker":"Andrew Zigler","startTime":2074.462,"endTime":2076.812,"body":"in practice or really in use."},{"speaker":"Andrew Zigler","startTime":2077.842,"endTime":2080.942,"body":"And that requires like a whole"},{"speaker":"Andrew Zigler","startTime":2077.842,"endTime":2080.942,"body":"mental, I think, flip about how"},{"speaker":"Andrew Zigler","startTime":2080.942,"endTime":2084.132,"body":"we build and how we look at them."},{"speaker":"Andrew Zigler","startTime":2084.132,"endTime":2090.752,"body":"So it helps you isolate the variables and"},{"speaker":"Andrew Zigler","startTime":2084.132,"endTime":2090.752,"body":"make those tools a in a classical way."},{"speaker":"Andrew Zigler","startTime":2090.792,"endTime":2092.662,"body":"Where you can evaluate and understand."},{"speaker":"Vikram Chatterji","startTime":2093.423,"endTime":2095.013,"body":"Yeah,"},{"speaker":"Vikram Chatterji","startTime":2093.423,"endTime":2095.013,"body":"yeah, no, I agree with you."},{"speaker":"Vikram Chatterji","startTime":2095.013,"endTime":2095.843,"body":"That's exactly right."},{"speaker":"Andrew Zigler","startTime":2096.505,"endTime":2100.205,"body":"And when someone is, is"},{"speaker":"Andrew Zigler","startTime":2096.505,"endTime":2100.205,"body":"maybe building and managing a bunch of"},{"speaker":"Andrew Zigler","startTime":2100.215,"endTime":2104.595,"body":"these AI bots or tools or workflows,"},{"speaker":"Andrew Zigler","startTime":2100.215,"endTime":2104.595,"body":"you know, are you seeing that it's"},{"speaker":"Andrew Zigler","startTime":2104.595,"endTime":2108.595,"body":"like people are like almost like doing"},{"speaker":"Andrew Zigler","startTime":2104.595,"endTime":2108.595,"body":"performance evaluations on their agents?"},{"speaker":"Andrew Zigler","startTime":2108.605,"endTime":2112.215,"body":"Like maybe like how somebody would do for,"},{"speaker":"Andrew Zigler","startTime":2108.605,"endTime":2112.215,"body":"maybe someone's getting ramped up, right,"},{"speaker":"Andrew Zigler","startTime":2112.385,"endTime":2114.885,"body":"as a BDR or as a customer success manager."},{"speaker":"Andrew Zigler","startTime":2114.885,"endTime":2118.235,"body":"And there's, you know, basic tenants"},{"speaker":"Andrew Zigler","startTime":2114.885,"endTime":2118.235,"body":"that you want them to do, across the"},{"speaker":"Andrew Zigler","startTime":2118.235,"endTime":2121.725,"body":"board and you're evaluating their"},{"speaker":"Andrew Zigler","startTime":2118.235,"endTime":2121.725,"body":"interactions with customers or whatnot."},{"speaker":"Andrew Zigler","startTime":2122.165,"endTime":2125.175,"body":"Because you understand what is good"},{"speaker":"Andrew Zigler","startTime":2122.165,"endTime":2125.175,"body":"and what is bad in the environment"},{"speaker":"Andrew Zigler","startTime":2125.175,"endTime":2126.475,"body":"of your company's culture."},{"speaker":"Andrew Zigler","startTime":2126.705,"endTime":2128.945,"body":"are you saying that that's how"},{"speaker":"Andrew Zigler","startTime":2126.705,"endTime":2128.945,"body":"people are using and evolving,"},{"speaker":"Andrew Zigler","startTime":2129.225,"endTime":2131.275,"body":"evolving as they build these tools?"},{"speaker":"Vikram Chatterji","startTime":2131.876,"endTime":2132.846,"body":"Yeah, it's similar."},{"speaker":"Vikram Chatterji","startTime":2132.876,"endTime":2136.493,"body":"It's very similar because, um, to"},{"speaker":"Vikram Chatterji","startTime":2132.876,"endTime":2136.493,"body":"take your example for a second,"},{"speaker":"Vikram Chatterji","startTime":2136.503,"endTime":2138.823,"body":"like you, let's say you hire an SDR."},{"speaker":"Vikram Chatterji","startTime":2138.843,"endTime":2140.933,"body":"The SDR is mostly."},{"speaker":"Vikram Chatterji","startTime":2141.311,"endTime":2143.858,"body":"focused on outbounding, and then"},{"speaker":"Vikram Chatterji","startTime":2141.311,"endTime":2143.858,"body":"you got to see the quality of their"},{"speaker":"Vikram Chatterji","startTime":2143.878,"endTime":2148.758,"body":"outbound, who they outbounded to, what"},{"speaker":"Vikram Chatterji","startTime":2143.878,"endTime":2148.758,"body":"the result of that was, did they book"},{"speaker":"Vikram Chatterji","startTime":2148.758,"endTime":2150.278,"body":"a call or not, and all sorts of stuff."},{"speaker":"Vikram Chatterji","startTime":2150.278,"endTime":2151.638,"body":"There's an entire funnel there."},{"speaker":"Vikram Chatterji","startTime":2152.054,"endTime":2154.789,"body":"Imagine all that's being"},{"speaker":"Vikram Chatterji","startTime":2152.054,"endTime":2154.789,"body":"done by an AI agent."},{"speaker":"Vikram Chatterji","startTime":2155.209,"endTime":2158.079,"body":"That AI agent is basically going to have"},{"speaker":"Vikram Chatterji","startTime":2155.209,"endTime":2158.079,"body":"to call, do a bunch of different kinds of"},{"speaker":"Vikram Chatterji","startTime":2158.269,"endTime":2160.069,"body":"API calls to make sure all that happens."},{"speaker":"Vikram Chatterji","startTime":2160.509,"endTime":2163.659,"body":"And now the question becomes, great,"},{"speaker":"Vikram Chatterji","startTime":2160.509,"endTime":2163.659,"body":"if it's a, an AI agent that's not"},{"speaker":"Vikram Chatterji","startTime":2163.659,"endTime":2167.419,"body":"sleeping at night, then how do I"},{"speaker":"Vikram Chatterji","startTime":2163.659,"endTime":2167.419,"body":"make sure that there's some sense of"},{"speaker":"Vikram Chatterji","startTime":2167.879,"endTime":2169.809,"body":"potential failure modes that can happen?"},{"speaker":"Vikram Chatterji","startTime":2170.219,"endTime":2173.329,"body":"You would probably do this with"},{"speaker":"Vikram Chatterji","startTime":2170.219,"endTime":2173.329,"body":"the SDR as well, the human SDR."},{"speaker":"Vikram Chatterji","startTime":2173.634,"endTime":2176.054,"body":"So you probably want to have like"},{"speaker":"Vikram Chatterji","startTime":2173.634,"endTime":2176.054,"body":"some kind of inspection, right?"},{"speaker":"Vikram Chatterji","startTime":2176.054,"endTime":2177.274,"body":"You have an expectation setting."},{"speaker":"Vikram Chatterji","startTime":2177.554,"endTime":2180.264,"body":"I want you to book 10 calls this week."},{"speaker":"Vikram Chatterji","startTime":2180.334,"endTime":2181.094,"body":"That's my expectation."},{"speaker":"Vikram Chatterji","startTime":2181.134,"endTime":2185.004,"body":"And then with a human, you have some"},{"speaker":"Vikram Chatterji","startTime":2181.134,"endTime":2185.004,"body":"level of inspection around, great, like"},{"speaker":"Vikram Chatterji","startTime":2185.294,"endTime":2187.674,"body":"how many outbound calls did you do today?"},{"speaker":"Vikram Chatterji","startTime":2187.794,"endTime":2189.334,"body":"And then I'm going to"},{"speaker":"Vikram Chatterji","startTime":2187.794,"endTime":2189.334,"body":"start to look at a funnel."},{"speaker":"Vikram Chatterji","startTime":2189.764,"endTime":2190.294,"body":"it's similar."},{"speaker":"Vikram Chatterji","startTime":2190.404,"endTime":2194.034,"body":"You come up with that mental model of"},{"speaker":"Vikram Chatterji","startTime":2190.404,"endTime":2194.034,"body":"what those potential failure modes would"},{"speaker":"Vikram Chatterji","startTime":2194.044,"endTime":2196.111,"body":"be, as the person who's building the app."},{"speaker":"Vikram Chatterji","startTime":2196.491,"endTime":2198.551,"body":"And then you have to build out"},{"speaker":"Vikram Chatterji","startTime":2196.491,"endTime":2198.551,"body":"the guardrails accordingly."},{"speaker":"Vikram Chatterji","startTime":2198.801,"endTime":2201.941,"body":"And then what happens is as you go"},{"speaker":"Vikram Chatterji","startTime":2198.801,"endTime":2201.941,"body":"through the motion of interacting"},{"speaker":"Vikram Chatterji","startTime":2201.941,"endTime":2206.071,"body":"with that human SDR or the agent or"},{"speaker":"Vikram Chatterji","startTime":2201.941,"endTime":2206.071,"body":"the AI agent, you start to figure"},{"speaker":"Vikram Chatterji","startTime":2206.071,"endTime":2208.981,"body":"out more failure modes because you"},{"speaker":"Vikram Chatterji","startTime":2206.071,"endTime":2208.981,"body":"go deeper and deeper and deeper."},{"speaker":"Vikram Chatterji","startTime":2208.981,"endTime":2210.701,"body":"You're like, Oh shit, this can go wrong."},{"speaker":"Vikram Chatterji","startTime":2211.141,"endTime":2211.821,"body":"That can go wrong."},{"speaker":"Vikram Chatterji","startTime":2211.821,"endTime":2215.781,"body":"And then on the fly, you're going to have"},{"speaker":"Vikram Chatterji","startTime":2211.821,"endTime":2215.781,"body":"to create more evals and more metrics."},{"speaker":"Vikram Chatterji","startTime":2215.781,"endTime":2220.271,"body":"And you also start generating more data"},{"speaker":"Vikram Chatterji","startTime":2215.781,"endTime":2220.271,"body":"around like, ah, you know, like when it's."},{"speaker":"Vikram Chatterji","startTime":2220.971,"endTime":2224.551,"body":"When it's trying to reach out to this"},{"speaker":"Vikram Chatterji","startTime":2220.971,"endTime":2224.551,"body":"specific kind of person, uh, with this"},{"speaker":"Vikram Chatterji","startTime":2224.551,"endTime":2226.721,"body":"specific queue, that's when it's failing."},{"speaker":"Vikram Chatterji","startTime":2226.721,"endTime":2228.801,"body":"So maybe I should isolate"},{"speaker":"Vikram Chatterji","startTime":2226.721,"endTime":2228.801,"body":"this data a little bit."},{"speaker":"Vikram Chatterji","startTime":2228.811,"endTime":2232.201,"body":"That's when you start to create like"},{"speaker":"Vikram Chatterji","startTime":2228.811,"endTime":2232.201,"body":"this, this data set that you want"},{"speaker":"Vikram Chatterji","startTime":2232.201,"endTime":2233.481,"body":"to test against and these metrics."},{"speaker":"Vikram Chatterji","startTime":2233.481,"endTime":2237.854,"body":"So that's the flow that people have been"},{"speaker":"Vikram Chatterji","startTime":2233.481,"endTime":2237.854,"body":"going down of like, exploring and creating"},{"speaker":"Vikram Chatterji","startTime":2238.274,"endTime":2240.094,"body":"evals and making it part of their process."},{"speaker":"Andrew Zigler","startTime":2240.673,"endTime":2243.683,"body":"What really stands out to"},{"speaker":"Andrew Zigler","startTime":2240.673,"endTime":2243.683,"body":"me about that is it can get very complex."},{"speaker":"Andrew Zigler","startTime":2243.903,"endTime":2245.663,"body":"And it sounds like a whole new system."},{"speaker":"Andrew Zigler","startTime":2245.713,"endTime":2247.543,"body":"Set of skills, even."},{"speaker":"Andrew Zigler","startTime":2248.04,"endTime":2252.389,"body":"from your perspective, what do you"},{"speaker":"Andrew Zigler","startTime":2248.04,"endTime":2252.389,"body":"think is the most powerful habit or"},{"speaker":"Andrew Zigler","startTime":2252.389,"endTime":2256.939,"body":"skill that an engineer or an engineering"},{"speaker":"Andrew Zigler","startTime":2252.389,"endTime":2256.939,"body":"leader should be picking up right now"},{"speaker":"Andrew Zigler","startTime":2257.119,"endTime":2258.829,"body":"to stay ahead of this kind of curve?"},{"speaker":"Vikram Chatterji","startTime":2259.37,"endTime":2260.17,"body":"They have to build."},{"speaker":"Vikram Chatterji","startTime":2260.17,"endTime":2262.24,"body":"I think the best engineering"},{"speaker":"Vikram Chatterji","startTime":2260.17,"endTime":2262.24,"body":"leaders that I've seen, they're"},{"speaker":"Vikram Chatterji","startTime":2262.24,"endTime":2263.32,"body":"doing two things right now."},{"speaker":"Vikram Chatterji","startTime":2263.75,"endTime":2266.31,"body":"they're building, they're"},{"speaker":"Vikram Chatterji","startTime":2263.75,"endTime":2266.31,"body":"keeping themselves, up skilled."},{"speaker":"Vikram Chatterji","startTime":2266.37,"endTime":2267.55,"body":"I think everyone has to do that."},{"speaker":"Vikram Chatterji","startTime":2267.98,"endTime":2269.64,"body":"I certainly do that, all the time."},{"speaker":"Vikram Chatterji","startTime":2270.01,"endTime":2271.0,"body":"everyone has to do that."},{"speaker":"Vikram Chatterji","startTime":2271.05,"endTime":2274.11,"body":"And if you're expecting your team to"},{"speaker":"Vikram Chatterji","startTime":2271.05,"endTime":2274.11,"body":"build out these kinds of AI apps, you"},{"speaker":"Vikram Chatterji","startTime":2274.13,"endTime":2275.69,"body":"have to get very, very familiar with it."},{"speaker":"Vikram Chatterji","startTime":2276.17,"endTime":2277.66,"body":"that's one very large part of it."},{"speaker":"Vikram Chatterji","startTime":2278.267,"endTime":2281.48,"body":"And the second, the second piece is"},{"speaker":"Vikram Chatterji","startTime":2278.267,"endTime":2281.48,"body":"just, being in touch with the community,"},{"speaker":"Vikram Chatterji","startTime":2281.85,"endTime":2283.06,"body":"like learning from each other."},{"speaker":"Vikram Chatterji","startTime":2283.44,"endTime":2286.37,"body":"Because I've noticed that organizations"},{"speaker":"Vikram Chatterji","startTime":2283.44,"endTime":2286.37,"body":"that are moving really fast are the ones"},{"speaker":"Vikram Chatterji","startTime":2286.41,"endTime":2290.41,"body":"where they have eng leaders who are also"},{"speaker":"Vikram Chatterji","startTime":2286.41,"endTime":2290.41,"body":"talking to other eng leaders about what"},{"speaker":"Vikram Chatterji","startTime":2290.41,"endTime":2292.03,"body":"they have learned really, really quickly."},{"speaker":"Vikram Chatterji","startTime":2292.03,"endTime":2294.79,"body":"Because everything's moving"},{"speaker":"Vikram Chatterji","startTime":2292.03,"endTime":2294.79,"body":"so quickly, you can't wait to"},{"speaker":"Vikram Chatterji","startTime":2295.06,"endTime":2296.46,"body":"make those mistakes yourselves."},{"speaker":"Vikram Chatterji","startTime":2296.85,"endTime":2297.14,"body":"Right."},{"speaker":"Vikram Chatterji","startTime":2297.18,"endTime":2297.49,"body":"so."},{"speaker":"Vikram Chatterji","startTime":2297.825,"endTime":2301.345,"body":"I'm seeing folks where there is like a"},{"speaker":"Vikram Chatterji","startTime":2297.825,"endTime":2301.345,"body":"good cross pollination amongst leaders,"},{"speaker":"Vikram Chatterji","startTime":2301.505,"endTime":2303.095,"body":"that they're moving much, much faster."},{"speaker":"Vikram Chatterji","startTime":2303.355,"endTime":2306.185,"body":"It could be that, you know, an eng"},{"speaker":"Vikram Chatterji","startTime":2303.355,"endTime":2306.185,"body":"leader might tell somebody else that"},{"speaker":"Vikram Chatterji","startTime":2306.225,"endTime":2309.035,"body":"you should get Galileo because without"},{"speaker":"Vikram Chatterji","startTime":2306.225,"endTime":2309.035,"body":"evals, it's going to be really bad."},{"speaker":"Vikram Chatterji","startTime":2309.525,"endTime":2311.965,"body":"And the other one is going to"},{"speaker":"Vikram Chatterji","startTime":2309.525,"endTime":2311.965,"body":"probably learn that the hard way,"},{"speaker":"Vikram Chatterji","startTime":2312.165,"endTime":2314.255,"body":"which has happened with a lot"},{"speaker":"Vikram Chatterji","startTime":2312.165,"endTime":2314.255,"body":"of engineering leaders before."},{"speaker":"Vikram Chatterji","startTime":2314.675,"endTime":2316.895,"body":"So I feel like those two things"},{"speaker":"Vikram Chatterji","startTime":2314.675,"endTime":2316.895,"body":"are very, very important."},{"speaker":"Vikram Chatterji","startTime":2317.095,"endTime":2321.065,"body":"keep being, at the forefront"},{"speaker":"Vikram Chatterji","startTime":2317.095,"endTime":2321.065,"body":"of learning by doing."},{"speaker":"Vikram Chatterji","startTime":2321.215,"endTime":2323.765,"body":"Because you're an engineering leader,"},{"speaker":"Vikram Chatterji","startTime":2321.215,"endTime":2323.765,"body":"dust off those skills and you can"},{"speaker":"Vikram Chatterji","startTime":2323.765,"endTime":2325.955,"body":"actually build simple apps on the side."},{"speaker":"Vikram Chatterji","startTime":2326.419,"endTime":2328.689,"body":"And the second one is just being"},{"speaker":"Vikram Chatterji","startTime":2326.419,"endTime":2328.689,"body":"a big part of the community"},{"speaker":"Vikram Chatterji","startTime":2328.689,"endTime":2329.729,"body":"somehow, being in touch."},{"speaker":"Andrew Zigler","startTime":2330.638,"endTime":2333.548,"body":"Build,"},{"speaker":"Andrew Zigler","startTime":2330.638,"endTime":2333.548,"body":"Read, and Communicate."},{"speaker":"Andrew Zigler","startTime":2334.128,"endTime":2335.988,"body":"Three core defining traits."},{"speaker":"Andrew Zigler","startTime":2336.328,"endTime":2337.948,"body":"And that's a really powerful takeaway."},{"speaker":"Andrew Zigler","startTime":2338.508,"endTime":2344.338,"body":"For me, what really stands out about"},{"speaker":"Andrew Zigler","startTime":2338.508,"endTime":2344.338,"body":"that is that people who put in place"},{"speaker":"Andrew Zigler","startTime":2344.348,"endTime":2348.828,"body":"small incremental changes over time, you"},{"speaker":"Andrew Zigler","startTime":2344.348,"endTime":2348.828,"body":"know, they get ahead faster and faster."},{"speaker":"Andrew Zigler","startTime":2348.938,"endTime":2352.268,"body":"In an incremental world like"},{"speaker":"Andrew Zigler","startTime":2348.938,"endTime":2352.268,"body":"AI, somebody who's building that"},{"speaker":"Andrew Zigler","startTime":2352.268,"endTime":2356.758,"body":"tool yesterday is going to be"},{"speaker":"Andrew Zigler","startTime":2352.268,"endTime":2356.758,"body":"much more ahead of you tomorrow."},{"speaker":"Andrew Zigler","startTime":2356.873,"endTime":2360.803,"body":"If you didn't build it, so those"},{"speaker":"Andrew Zigler","startTime":2356.873,"endTime":2360.803,"body":"engineers that are out there building"},{"speaker":"Andrew Zigler","startTime":2360.823,"endTime":2363.423,"body":"and talking about what they're"},{"speaker":"Andrew Zigler","startTime":2360.823,"endTime":2363.423,"body":"building, and they're reading about"},{"speaker":"Andrew Zigler","startTime":2363.423,"endTime":2364.403,"body":"what other people are building."},{"speaker":"Andrew Zigler","startTime":2364.403,"endTime":2368.173,"body":"Those are the ones that are getting"},{"speaker":"Andrew Zigler","startTime":2364.403,"endTime":2368.173,"body":"ahead and are staying on the forefront."},{"speaker":"Andrew Zigler","startTime":2368.373,"endTime":2370.513,"body":"And I, that's really great takeaway."},{"speaker":"Andrew Zigler","startTime":2370.883,"endTime":2373.413,"body":"And Vikram, this has been"},{"speaker":"Andrew Zigler","startTime":2370.883,"endTime":2373.413,"body":"an incredible conversation."},{"speaker":"Andrew Zigler","startTime":2373.413,"endTime":2375.973,"body":"There's so much insight packed"},{"speaker":"Andrew Zigler","startTime":2373.413,"endTime":2375.973,"body":"into what we talked about today."},{"speaker":"Andrew Zigler","startTime":2376.313,"endTime":2380.683,"body":"I want to thank you for sharing and giving"},{"speaker":"Andrew Zigler","startTime":2376.313,"endTime":2380.683,"body":"our listeners some actionable takeaways."},{"speaker":"Andrew Zigler","startTime":2380.683,"endTime":2384.603,"body":"before we wrap up, where can our"},{"speaker":"Andrew Zigler","startTime":2380.683,"endTime":2384.603,"body":"audience go to learn more about"},{"speaker":"Andrew Zigler","startTime":2384.623,"endTime":2386.873,"body":"Galileo and the work that you're doing?"},{"speaker":"Vikram Chatterji","startTime":2387.464,"endTime":2388.134,"body":"Yeah, for sure."},{"speaker":"Vikram Chatterji","startTime":2388.134,"endTime":2390.184,"body":"So we are at Galileo."},{"speaker":"Vikram Chatterji","startTime":2390.185,"endTime":2390.744,"body":"ai."},{"speaker":"Vikram Chatterji","startTime":2390.924,"endTime":2393.914,"body":"You can also check us out"},{"speaker":"Vikram Chatterji","startTime":2390.924,"endTime":2393.914,"body":"on LinkedIn and on Twitter."},{"speaker":"Vikram Chatterji","startTime":2394.926,"endTime":2398.106,"body":"post a lot of content on our website, we"},{"speaker":"Vikram Chatterji","startTime":2394.926,"endTime":2398.106,"body":"put that in our blogs and the website."},{"speaker":"Vikram Chatterji","startTime":2398.248,"endTime":2399.628,"body":"there's an entire research section."},{"speaker":"Vikram Chatterji","startTime":2399.638,"endTime":2403.118,"body":"We publish our papers and everything"},{"speaker":"Vikram Chatterji","startTime":2399.638,"endTime":2403.118,"body":"else that we've worked on."},{"speaker":"Vikram Chatterji","startTime":2403.538,"endTime":2407.318,"body":"For AI evals, we've been around for"},{"speaker":"Vikram Chatterji","startTime":2403.538,"endTime":2407.318,"body":"four years, so there's a rich history"},{"speaker":"Vikram Chatterji","startTime":2407.318,"endTime":2410.524,"body":"of, a large body of work there,"},{"speaker":"Vikram Chatterji","startTime":2407.318,"endTime":2410.524,"body":"so they can check it out there."},{"speaker":"Vikram Chatterji","startTime":2410.554,"endTime":2413.374,"body":"We're also hiring right now for"},{"speaker":"Vikram Chatterji","startTime":2410.554,"endTime":2413.374,"body":"a lot of, uh, engineers who have"},{"speaker":"Vikram Chatterji","startTime":2413.384,"endTime":2414.674,"body":"built out their own AI apps."},{"speaker":"Vikram Chatterji","startTime":2414.994,"endTime":2418.004,"body":"excited to chat with anybody who's"},{"speaker":"Vikram Chatterji","startTime":2414.994,"endTime":2418.004,"body":"interested in being at the forefront"},{"speaker":"Vikram Chatterji","startTime":2418.004,"endTime":2419.652,"body":"of, uh, helping builders build"},{"speaker":"Andrew Zigler","startTime":2420.543,"endTime":2423.803,"body":"That's fantastic to hear,"},{"speaker":"Andrew Zigler","startTime":2420.543,"endTime":2423.803,"body":"and we'll definitely, I'll include those"},{"speaker":"Andrew Zigler","startTime":2423.803,"endTime":2426.713,"body":"notes in the show, notes on Substack."},{"speaker":"Andrew Zigler","startTime":2426.713,"endTime":2429.163,"body":"So if you listen to this and you're"},{"speaker":"Andrew Zigler","startTime":2426.713,"endTime":2429.163,"body":"interested in getting involved with"},{"speaker":"Andrew Zigler","startTime":2429.163,"endTime":2433.233,"body":"Galileo, or following up on anything"},{"speaker":"Andrew Zigler","startTime":2429.163,"endTime":2433.233,"body":"that we talked about today, you know,"},{"speaker":"Andrew Zigler","startTime":2433.373,"endTime":2437.813,"body":"please be sure to subscribe, share our"},{"speaker":"Andrew Zigler","startTime":2433.373,"endTime":2437.813,"body":"episode, you've made it this far, and if"},{"speaker":"Andrew Zigler","startTime":2437.813,"endTime":2441.753,"body":"you check out our Substack, there's even"},{"speaker":"Andrew Zigler","startTime":2437.813,"endTime":2441.753,"body":"more insights from today's discussion."},{"speaker":"Andrew Zigler","startTime":2442.083,"endTime":2444.513,"body":"And we'd also love to"},{"speaker":"Andrew Zigler","startTime":2442.083,"endTime":2444.513,"body":"hear from you on socials."},{"speaker":"Andrew Zigler","startTime":2444.533,"endTime":2445.623,"body":"So don't be a stranger."},{"speaker":"Andrew Zigler","startTime":2445.933,"endTime":2448.983,"body":"And that's it for this"},{"speaker":"Andrew Zigler","startTime":2445.933,"endTime":2448.983,"body":"week's Dev Interrupted."},{"speaker":"Andrew Zigler","startTime":2449.073,"endTime":2450.183,"body":"We'll see you next time."}]}