{"version":"1.0.0","segments":[{"speaker":"Evan Morikawa","startTime":0.0,"endTime":1.559,"body":"it took a"},{"speaker":"Evan Morikawa","startTime":0.0,"endTime":1.559,"body":"lot of tweaking to kind"},{"speaker":"Evan Morikawa","startTime":1.559,"endTime":5.0,"body":"of find what the right"},{"speaker":"Evan Morikawa","startTime":1.559,"endTime":5.0,"body":"utilization metrics were and"},{"speaker":"Evan Morikawa","startTime":5.005,"endTime":6.41,"body":"then how to optimize those."},{"speaker":"Evan Morikawa","startTime":6.41,"endTime":8.42,"body":"And all of those were"},{"speaker":"Evan Morikawa","startTime":6.41,"endTime":8.42,"body":"really critical to"},{"speaker":"Evan Morikawa","startTime":8.42,"endTime":9.92,"body":"getting more out of it."},{"speaker":"Evan Morikawa","startTime":10.19,"endTime":13.04,"body":"But for us, everything"},{"speaker":"Evan Morikawa","startTime":10.19,"endTime":13.04,"body":"was framed in terms of."},{"speaker":"Evan Morikawa","startTime":14.235,"endTime":17.305,"body":"Every improvement that we"},{"speaker":"Evan Morikawa","startTime":14.235,"endTime":17.305,"body":"have represents more users we"},{"speaker":"Evan Morikawa","startTime":17.305,"endTime":18.625,"body":"could let onto the platform."},{"speaker":"Evan Morikawa","startTime":19.105,"endTime":21.815,"body":"As opposed to say like, oh it's"},{"speaker":"Evan Morikawa","startTime":19.105,"endTime":21.815,"body":"driving down our margins or"},{"speaker":"Evan Morikawa","startTime":21.815,"endTime":27.9,"body":"making it faster, we kept Cost"},{"speaker":"Evan Morikawa","startTime":21.815,"endTime":27.9,"body":"and latency as relatively fixed."},{"speaker":"Evan Morikawa","startTime":27.9,"endTime":30.03,"body":"And the thing that could get"},{"speaker":"Evan Morikawa","startTime":27.9,"endTime":30.03,"body":"to move was we got to get"},{"speaker":"Evan Morikawa","startTime":30.03,"endTime":31.35,"body":"more users onto the system."},{"speaker":"Evan Morikawa","startTime":31.92,"endTime":34.8,"body":"What's the impact of"},{"speaker":"Evan Morikawa","startTime":31.92,"endTime":34.8,"body":"AI generated code."},{"speaker":"Evan Morikawa","startTime":35.28,"endTime":36.3,"body":"How can you measure it?"},{"speaker":"Evan Morikawa","startTime":37.05,"endTime":39.63,"body":"Join linear B and"},{"speaker":"Evan Morikawa","startTime":37.05,"endTime":39.63,"body":"ThoughtWorks global head"},{"speaker":"Evan Morikawa","startTime":39.63,"endTime":40.86,"body":"of AI software delivery."},{"speaker":"Evan Morikawa","startTime":41.07,"endTime":44.97,"body":"As we explore the metrics that"},{"speaker":"Evan Morikawa","startTime":41.07,"endTime":44.97,"body":"measure the impact gen AI has on"},{"speaker":"Evan Morikawa","startTime":44.97,"endTime":48.24,"body":"the software delivery process in"},{"speaker":"Evan Morikawa","startTime":44.97,"endTime":48.24,"body":"is first of its kind workshop."},{"speaker":"Evan Morikawa","startTime":49.02,"endTime":52.62,"body":"We'll share new data insights"},{"speaker":"Evan Morikawa","startTime":49.02,"endTime":52.62,"body":"from our gen AI impact report."},{"speaker":"Evan Morikawa","startTime":53.07,"endTime":55.47,"body":"Case studies into how teams"},{"speaker":"Evan Morikawa","startTime":53.07,"endTime":55.47,"body":"are successfully leveraging"},{"speaker":"Evan Morikawa","startTime":55.47,"endTime":58.89,"body":"gen AI impact measurements,"},{"speaker":"Evan Morikawa","startTime":55.47,"endTime":58.89,"body":"including adoption"},{"speaker":"Evan Morikawa","startTime":58.95,"endTime":60.51,"body":"benefits and risk metrics."},{"speaker":"Evan Morikawa","startTime":60.93,"endTime":63.99,"body":"Plus a live demo of how you"},{"speaker":"Evan Morikawa","startTime":60.93,"endTime":63.99,"body":"can measure the impact of"},{"speaker":"Evan Morikawa","startTime":63.99,"endTime":65.58,"body":"your gen AI initiative today."},{"speaker":"Evan Morikawa","startTime":66.148,"endTime":68.518,"body":"Register at linear bead"},{"speaker":"Evan Morikawa","startTime":66.148,"endTime":68.518,"body":"at IO slash events."},{"speaker":"Evan Morikawa","startTime":68.908,"endTime":72.688,"body":"And join us January 25th,"},{"speaker":"Evan Morikawa","startTime":68.908,"endTime":72.688,"body":"January 30th, or on demand"},{"speaker":"Evan Morikawa","startTime":72.748,"endTime":75.988,"body":"to start measuring the ROI of"},{"speaker":"Evan Morikawa","startTime":72.748,"endTime":75.988,"body":"your gen AI initiative today."},{"speaker":"Ben Lloyd Pearson","startTime":76.776,"endTime":77.376,"body":"Hey everyone."},{"speaker":"Ben Lloyd Pearson","startTime":77.406,"endTime":79.086,"body":"Welcome back to Dev Interrupted."},{"speaker":"Ben Lloyd Pearson","startTime":79.146,"endTime":81.306,"body":"I'm Ben Lloyd Pearson,"},{"speaker":"Ben Lloyd Pearson","startTime":79.146,"endTime":81.306,"body":"director of Developer"},{"speaker":"Ben Lloyd Pearson","startTime":81.306,"endTime":82.776,"body":"Relations here at Linear B."},{"speaker":"Ben Lloyd Pearson","startTime":82.866,"endTime":86.256,"body":"I'm pleased to have Evan"},{"speaker":"Ben Lloyd Pearson","startTime":82.866,"endTime":86.256,"body":"Morikawa joining us today."},{"speaker":"Ben Lloyd Pearson","startTime":86.355,"endTime":87.465,"body":"welcome to the show, Evan."},{"speaker":"Evan Morikawa","startTime":87.525,"endTime":87.825,"body":"Awesome."},{"speaker":"Evan Morikawa","startTime":87.825,"endTime":88.365,"body":"Thank you."},{"speaker":"Ben Lloyd Pearson","startTime":88.575,"endTime":90.315,"body":"Yeah,"},{"speaker":"Ben Lloyd Pearson","startTime":88.575,"endTime":90.315,"body":"so full disclosure, Evan"},{"speaker":"Ben Lloyd Pearson","startTime":90.315,"endTime":93.435,"body":"and I worked together in a"},{"speaker":"Ben Lloyd Pearson","startTime":90.315,"endTime":93.435,"body":"past life at an API company."},{"speaker":"Ben Lloyd Pearson","startTime":93.495,"endTime":95.505,"body":"Uh, you may not remember"},{"speaker":"Ben Lloyd Pearson","startTime":93.495,"endTime":95.505,"body":"this, but you were actually"},{"speaker":"Ben Lloyd Pearson","startTime":95.505,"endTime":97.965,"body":"a big part of why I decided"},{"speaker":"Ben Lloyd Pearson","startTime":95.505,"endTime":97.965,"body":"to join that company."},{"speaker":"Ben Lloyd Pearson","startTime":98.025,"endTime":98.115,"body":"What?"},{"speaker":"Ben Lloyd Pearson","startTime":98.115,"endTime":98.955,"body":"I did not know that."},{"speaker":"Ben Lloyd Pearson","startTime":98.955,"endTime":99.105,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":100.125,"endTime":100.665,"body":"Wow."},{"speaker":"Ben Lloyd Pearson","startTime":100.695,"endTime":100.755,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":100.755,"endTime":103.35,"body":"You had saw your video on"},{"speaker":"Ben Lloyd Pearson","startTime":100.755,"endTime":103.35,"body":"YouTube and, uh Oh, wow."},{"speaker":"Ben Lloyd Pearson","startTime":103.46,"endTime":103.75,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":104.02,"endTime":104.31,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":104.92,"endTime":105.34,"body":"So cool."},{"speaker":"Ben Lloyd Pearson","startTime":105.34,"endTime":106.93,"body":"Yeah, we've got a little bit"},{"speaker":"Ben Lloyd Pearson","startTime":105.34,"endTime":106.93,"body":"of a history, so it's really"},{"speaker":"Ben Lloyd Pearson","startTime":106.93,"endTime":109.12,"body":"great to to just have an"},{"speaker":"Ben Lloyd Pearson","startTime":106.93,"endTime":109.12,"body":"opportunity to catch up with"},{"speaker":"Ben Lloyd Pearson","startTime":109.12,"endTime":111.67,"body":"you, talk about some of new"},{"speaker":"Ben Lloyd Pearson","startTime":109.12,"endTime":111.67,"body":"stuff that you're working on."},{"speaker":"Ben Lloyd Pearson","startTime":111.95,"endTime":114.05,"body":"I know per, you know,"},{"speaker":"Ben Lloyd Pearson","startTime":111.95,"endTime":114.05,"body":"personally I was a little"},{"speaker":"Ben Lloyd Pearson","startTime":114.05,"endTime":116.51,"body":"bit envious when I saw that"},{"speaker":"Ben Lloyd Pearson","startTime":114.05,"endTime":116.51,"body":"you were going to open api."},{"speaker":"Ben Lloyd Pearson","startTime":116.51,"endTime":119.18,"body":"I, 'cause I was like,"},{"speaker":"Ben Lloyd Pearson","startTime":116.51,"endTime":119.18,"body":"that sounds so cool, . And"},{"speaker":"Ben Lloyd Pearson","startTime":119.185,"endTime":121.97,"body":"here we are talking about"},{"speaker":"Ben Lloyd Pearson","startTime":119.185,"endTime":121.97,"body":"how cool your work is."},{"speaker":"Ben Lloyd Pearson","startTime":121.97,"endTime":122.36,"body":"So."},{"speaker":"Ben Lloyd Pearson","startTime":122.422,"endTime":123.412,"body":"let's kick it off."},{"speaker":"Ben Lloyd Pearson","startTime":123.412,"endTime":125.842,"body":"Yeah, I mean, I mean,"},{"speaker":"Ben Lloyd Pearson","startTime":123.412,"endTime":125.842,"body":"opening eye, I feel like"},{"speaker":"Ben Lloyd Pearson","startTime":125.842,"endTime":126.862,"body":"needs no introduction."},{"speaker":"Ben Lloyd Pearson","startTime":126.862,"endTime":129.142,"body":"I mean, I feel like everyone"},{"speaker":"Ben Lloyd Pearson","startTime":126.862,"endTime":129.142,"body":"is talking about it."},{"speaker":"Ben Lloyd Pearson","startTime":129.192,"endTime":131.832,"body":"it's gone viral, at the"},{"speaker":"Ben Lloyd Pearson","startTime":129.192,"endTime":131.832,"body":"center of the conversation"},{"speaker":"Ben Lloyd Pearson","startTime":131.832,"endTime":133.812,"body":"around AI and LLMs."},{"speaker":"Ben Lloyd Pearson","startTime":133.812,"endTime":137.962,"body":"the release of ChatGPT has"},{"speaker":"Ben Lloyd Pearson","startTime":133.812,"endTime":137.962,"body":"kicked off a global phenomenon."},{"speaker":"Ben Lloyd Pearson","startTime":138.282,"endTime":140.542,"body":"Uh, and I, I want to walk"},{"speaker":"Ben Lloyd Pearson","startTime":138.282,"endTime":140.542,"body":"through that story, uh, in"},{"speaker":"Ben Lloyd Pearson","startTime":140.542,"endTime":143.392,"body":"particular, you know, what it"},{"speaker":"Ben Lloyd Pearson","startTime":140.542,"endTime":143.392,"body":"took for you to scale ChatGPT"},{"speaker":"Ben Lloyd Pearson","startTime":143.412,"endTime":145.042,"body":"when you had that viral moment."},{"speaker":"Ben Lloyd Pearson","startTime":145.442,"endTime":147.892,"body":"and especially, I, I, I know"},{"speaker":"Ben Lloyd Pearson","startTime":145.442,"endTime":147.892,"body":"I've heard a little bit that"},{"speaker":"Ben Lloyd Pearson","startTime":147.922,"endTime":149.962,"body":"there was a shortage of GPUs"},{"speaker":"Ben Lloyd Pearson","startTime":147.922,"endTime":149.962,"body":"that also affected this."},{"speaker":"Ben Lloyd Pearson","startTime":150.522,"endTime":152.967,"body":"I, I do want to dig into that"},{"speaker":"Ben Lloyd Pearson","startTime":150.522,"endTime":152.967,"body":"and you know, I just wanna see,"},{"speaker":"Ben Lloyd Pearson","startTime":152.967,"endTime":155.397,"body":"learn a little bit about how"},{"speaker":"Ben Lloyd Pearson","startTime":152.967,"endTime":155.397,"body":"you've been flexible and nimble"},{"speaker":"Ben Lloyd Pearson","startTime":155.397,"endTime":158.937,"body":"while this industry has just"},{"speaker":"Ben Lloyd Pearson","startTime":155.397,"endTime":158.937,"body":"like rapidly shifted around you."},{"speaker":"Ben Lloyd Pearson","startTime":159.057,"endTime":159.297,"body":"Yep."},{"speaker":"Ben Lloyd Pearson","startTime":159.417,"endTime":160.887,"body":"So let's just kick it off."},{"speaker":"Ben Lloyd Pearson","startTime":160.892,"endTime":162.657,"body":"Where, where does this"},{"speaker":"Ben Lloyd Pearson","startTime":160.892,"endTime":162.657,"body":"story all start for you?"},{"speaker":"Ben Lloyd Pearson","startTime":162.657,"endTime":164.727,"body":"Like how did you get to"},{"speaker":"Ben Lloyd Pearson","startTime":162.657,"endTime":164.727,"body":"where you are right now?"},{"speaker":"Evan Morikawa","startTime":164.817,"endTime":168.247,"body":"Yeah, so I mean,"},{"speaker":"Evan Morikawa","startTime":164.817,"endTime":168.247,"body":"as you mentioned, I was working"},{"speaker":"Evan Morikawa","startTime":168.247,"endTime":169.867,"body":"at an API company before."},{"speaker":"Evan Morikawa","startTime":170.367,"endTime":173.577,"body":"And at the time,"},{"speaker":"Evan Morikawa","startTime":170.367,"endTime":173.577,"body":"that's all OpenAI had."},{"speaker":"Evan Morikawa","startTime":173.577,"endTime":176.277,"body":"In fact, that was the light"},{"speaker":"Evan Morikawa","startTime":173.577,"endTime":176.277,"body":"bulb moment, was learning"},{"speaker":"Evan Morikawa","startTime":176.277,"endTime":178.747,"body":"that three years ago or so."},{"speaker":"Evan Morikawa","startTime":178.747,"endTime":181.087,"body":"Now OpenAI was starting"},{"speaker":"Evan Morikawa","startTime":178.747,"endTime":181.087,"body":"this apply to team."},{"speaker":"Evan Morikawa","startTime":182.132,"endTime":184.242,"body":"And Applied was all"},{"speaker":"Evan Morikawa","startTime":182.132,"endTime":184.242,"body":"about bringing this crazy"},{"speaker":"Evan Morikawa","startTime":184.482,"endTime":186.332,"body":"technology safely to market."},{"speaker":"Evan Morikawa","startTime":186.332,"endTime":189.222,"body":"And at the time, that was"},{"speaker":"Evan Morikawa","startTime":186.332,"endTime":189.222,"body":"just a single developer"},{"speaker":"Evan Morikawa","startTime":189.222,"endTime":192.632,"body":"facing API around the"},{"speaker":"Evan Morikawa","startTime":189.222,"endTime":192.632,"body":"very first GPT 3 models."},{"speaker":"Evan Morikawa","startTime":193.162,"endTime":194.862,"body":"which is good, because up"},{"speaker":"Evan Morikawa","startTime":193.162,"endTime":194.862,"body":"until that, like, it's still"},{"speaker":"Evan Morikawa","startTime":194.862,"endTime":197.282,"body":"the case that I do not have a"},{"speaker":"Evan Morikawa","startTime":194.862,"endTime":197.282,"body":"machine learning background."},{"speaker":"Evan Morikawa","startTime":197.652,"endTime":200.142,"body":"OpenAI was much more"},{"speaker":"Evan Morikawa","startTime":197.652,"endTime":200.142,"body":"research lab focused."},{"speaker":"Evan Morikawa","startTime":200.462,"endTime":203.642,"body":"But then here now, it's starting"},{"speaker":"Evan Morikawa","startTime":200.462,"endTime":203.642,"body":"up this brand new small team"},{"speaker":"Evan Morikawa","startTime":203.642,"endTime":205.482,"body":"doing APIs and products."},{"speaker":"Evan Morikawa","startTime":205.482,"endTime":207.672,"body":"And I was like, ah, I've"},{"speaker":"Evan Morikawa","startTime":205.482,"endTime":207.672,"body":"been doing APIs and products."},{"speaker":"Evan Morikawa","startTime":208.062,"endTime":209.672,"body":"Maybe I can help"},{"speaker":"Evan Morikawa","startTime":208.062,"endTime":209.672,"body":"contribute here."},{"speaker":"Evan Morikawa","startTime":210.192,"endTime":213.672,"body":"I think there was also that"},{"speaker":"Evan Morikawa","startTime":210.192,"endTime":213.672,"body":"feeling, too, that, you know,"},{"speaker":"Evan Morikawa","startTime":213.672,"endTime":216.492,"body":"I feel like I had a reasonable"},{"speaker":"Evan Morikawa","startTime":213.672,"endTime":216.492,"body":"sense of how computers"},{"speaker":"Evan Morikawa","startTime":216.492,"endTime":219.742,"body":"worked by that point in my"},{"speaker":"Evan Morikawa","startTime":216.492,"endTime":219.742,"body":"career, except for this."},{"speaker":"Evan Morikawa","startTime":219.812,"endTime":222.831,"body":"This still is this kind"},{"speaker":"Evan Morikawa","startTime":219.812,"endTime":222.831,"body":"of magic, certainly at the"},{"speaker":"Evan Morikawa","startTime":222.831,"endTime":224.891,"body":"beginning was this magic"},{"speaker":"Evan Morikawa","startTime":222.831,"endTime":224.891,"body":"black box, like, okay, I'm"},{"speaker":"Evan Morikawa","startTime":224.891,"endTime":225.821,"body":"gonna see what that's..."},{"speaker":"Evan Morikawa","startTime":225.886,"endTime":227.206,"body":"All about as well."},{"speaker":"Evan Morikawa","startTime":227.386,"endTime":230.866,"body":"I also knew of like some of"},{"speaker":"Evan Morikawa","startTime":227.386,"endTime":230.866,"body":"the, the, the founders through"},{"speaker":"Evan Morikawa","startTime":230.866,"endTime":235.216,"body":"just the, the broader network"},{"speaker":"Evan Morikawa","startTime":230.866,"endTime":235.216,"body":"and yeah, reached out and that"},{"speaker":"Evan Morikawa","startTime":235.216,"endTime":236.446,"body":"was the start of it there."},{"speaker":"Evan Morikawa","startTime":236.506,"endTime":238.876,"body":"Uh, when I joined,"},{"speaker":"Evan Morikawa","startTime":236.506,"endTime":238.876,"body":"applied was very small."},{"speaker":"Evan Morikawa","startTime":238.876,"endTime":241.996,"body":"There were only basically"},{"speaker":"Evan Morikawa","startTime":238.876,"endTime":241.996,"body":"like half a dozen engineers"},{"speaker":"Evan Morikawa","startTime":241.996,"endTime":243.526,"body":"working on all of the."},{"speaker":"Evan Morikawa","startTime":243.976,"endTime":245.736,"body":"APIs and systems for that."},{"speaker":"Evan Morikawa","startTime":246.196,"endTime":249.966,"body":"and that steadily grew as we"},{"speaker":"Evan Morikawa","startTime":246.196,"endTime":249.966,"body":"were trying to work on iterating"},{"speaker":"Evan Morikawa","startTime":249.966,"endTime":251.546,"body":"on these language models."},{"speaker":"Evan Morikawa","startTime":251.676,"endTime":256.116,"body":"I think the next big, the"},{"speaker":"Evan Morikawa","startTime":251.676,"endTime":256.116,"body":"big really first release or"},{"speaker":"Evan Morikawa","startTime":256.116,"endTime":259.786,"body":"push of any kind was when"},{"speaker":"Evan Morikawa","startTime":256.116,"endTime":259.786,"body":"we tweaked these to write"},{"speaker":"Evan Morikawa","startTime":259.786,"endTime":263.796,"body":"code and worked with GitHub"},{"speaker":"Evan Morikawa","startTime":259.786,"endTime":263.796,"body":"to launch GitHub Copilot."},{"speaker":"Evan Morikawa","startTime":264.256,"endTime":266.946,"body":"So GitHub Copilot actually"},{"speaker":"Evan Morikawa","startTime":264.256,"endTime":266.946,"body":"initially ran through our"},{"speaker":"Evan Morikawa","startTime":266.946,"endTime":270.166,"body":"servers at launch, because"},{"speaker":"Evan Morikawa","startTime":266.946,"endTime":270.166,"body":"it was very difficult"},{"speaker":"Evan Morikawa","startTime":270.166,"endTime":271.446,"body":"to run or deploy these."},{"speaker":"Evan Morikawa","startTime":271.816,"endTime":273.226,"body":"That was definitely the"},{"speaker":"Evan Morikawa","startTime":271.816,"endTime":273.226,"body":"first time we had any"},{"speaker":"Evan Morikawa","startTime":273.226,"endTime":275.876,"body":"experience running this"},{"speaker":"Evan Morikawa","startTime":273.226,"endTime":275.876,"body":"at any sort of scale."},{"speaker":"Evan Morikawa","startTime":276.856,"endTime":280.156,"body":"But still all the way through,"},{"speaker":"Evan Morikawa","startTime":276.856,"endTime":280.156,"body":"basically up until ChatGPT."},{"speaker":"Evan Morikawa","startTime":280.511,"endTime":283.871,"body":"And still to this day, have"},{"speaker":"Evan Morikawa","startTime":280.511,"endTime":283.871,"body":"this very core of an API"},{"speaker":"Evan Morikawa","startTime":283.891,"endTime":287.641,"body":"business that powers all these"},{"speaker":"Evan Morikawa","startTime":283.891,"endTime":287.641,"body":"other AI powered applications"},{"speaker":"Evan Morikawa","startTime":287.641,"endTime":290.861,"body":"that a lot of people are"},{"speaker":"Evan Morikawa","startTime":287.641,"endTime":290.861,"body":"trying to build on now."},{"speaker":"Evan Morikawa","startTime":291.302,"endTime":293.342,"body":"and then along came ChatGPT."},{"speaker":"Evan Morikawa","startTime":294.658,"endTime":296.638,"body":"It's actually kind of an"},{"speaker":"Evan Morikawa","startTime":294.658,"endTime":296.638,"body":"interesting story, because when"},{"speaker":"Evan Morikawa","startTime":298.348,"endTime":302.638,"body":"ChatGPT launched, it wasn't"},{"speaker":"Evan Morikawa","startTime":298.348,"endTime":302.638,"body":"necessarily sure whether or"},{"speaker":"Evan Morikawa","startTime":302.638,"endTime":306.148,"body":"not it'd be a scary thing"},{"speaker":"Evan Morikawa","startTime":302.638,"endTime":306.148,"body":"or a totally normal thing."},{"speaker":"Evan Morikawa","startTime":306.158,"endTime":308.618,"body":"You know, at one, on one hand..."},{"speaker":"Evan Morikawa","startTime":309.123,"endTime":310.623,"body":"The model that was"},{"speaker":"Evan Morikawa","startTime":309.123,"endTime":310.623,"body":"powering it, GPT 3."},{"speaker":"Evan Morikawa","startTime":310.623,"endTime":311.823,"body":"5."},{"speaker":"Evan Morikawa","startTime":311.863,"endTime":315.233,"body":"That had been out for"},{"speaker":"Evan Morikawa","startTime":311.863,"endTime":315.233,"body":"several years at that point"},{"speaker":"Evan Morikawa","startTime":315.233,"endTime":316.393,"body":"in various iterations."},{"speaker":"Evan Morikawa","startTime":316.913,"endTime":319.443,"body":"people could already sign"},{"speaker":"Evan Morikawa","startTime":316.913,"endTime":319.443,"body":"up for free through the"},{"speaker":"Evan Morikawa","startTime":319.443,"endTime":320.583,"body":"developer playground."},{"speaker":"Evan Morikawa","startTime":320.613,"endTime":323.393,"body":"And in fact, really noticed a"},{"speaker":"Evan Morikawa","startTime":320.613,"endTime":323.393,"body":"lot of people like playing with"},{"speaker":"Evan Morikawa","startTime":323.393,"endTime":326.073,"body":"the models in that kind of way."},{"speaker":"Evan Morikawa","startTime":326.753,"endTime":329.033,"body":"So, in some senses,"},{"speaker":"Evan Morikawa","startTime":326.753,"endTime":329.033,"body":"there wasn't that much."},{"speaker":"Evan Morikawa","startTime":329.423,"endTime":331.863,"body":"Different about it, you"},{"speaker":"Evan Morikawa","startTime":329.423,"endTime":331.863,"body":"know, maybe a new UI."},{"speaker":"Evan Morikawa","startTime":331.873,"endTime":336.263,"body":"Also the model had been improved"},{"speaker":"Evan Morikawa","startTime":331.873,"endTime":336.263,"body":"dramatically to make up things"},{"speaker":"Evan Morikawa","startTime":336.263,"endTime":338.663,"body":"less and be more conversational."},{"speaker":"Evan Morikawa","startTime":339.243,"endTime":340.903,"body":"But you know, on the flip"},{"speaker":"Evan Morikawa","startTime":339.243,"endTime":340.903,"body":"side, this is also the first"},{"speaker":"Evan Morikawa","startTime":340.903,"endTime":344.953,"body":"time we would ever be offering"},{"speaker":"Evan Morikawa","startTime":340.903,"endTime":344.953,"body":"anything without a wait list."},{"speaker":"Evan Morikawa","startTime":345.003,"endTime":348.693,"body":"This would be a free to"},{"speaker":"Evan Morikawa","startTime":345.003,"endTime":348.693,"body":"use application out there."},{"speaker":"Evan Morikawa","startTime":348.693,"endTime":351.043,"body":"And yeah, that definitely"},{"speaker":"Evan Morikawa","startTime":348.693,"endTime":351.043,"body":"changes things as well."},{"speaker":"Evan Morikawa","startTime":351.043,"endTime":354.543,"body":"Um, you know, actually"},{"speaker":"Evan Morikawa","startTime":351.043,"endTime":354.543,"body":"on launch day, I think it"},{"speaker":"Evan Morikawa","startTime":354.543,"endTime":355.943,"body":"launched on a Wednesday."},{"speaker":"Evan Morikawa","startTime":355.943,"endTime":357.693,"body":"It was like November 30th."},{"speaker":"Evan Morikawa","startTime":358.533,"endTime":363.799,"body":"And we, kind of by design,"},{"speaker":"Evan Morikawa","startTime":358.533,"endTime":363.799,"body":"sort of thought this would be"},{"speaker":"Evan Morikawa","startTime":363.869,"endTime":365.559,"body":"a low key research preview."},{"speaker":"Evan Morikawa","startTime":366.459,"endTime":369.839,"body":"Just a blog post, a"},{"speaker":"Evan Morikawa","startTime":366.459,"endTime":369.839,"body":"tweet, and nothing else."},{"speaker":"Evan Morikawa","startTime":370.389,"endTime":374.569,"body":"And actually on launch day,"},{"speaker":"Evan Morikawa","startTime":370.389,"endTime":374.569,"body":"nothing crazy happened."},{"speaker":"Evan Morikawa","startTime":374.899,"endTime":377.629,"body":"Like some people came and"},{"speaker":"Evan Morikawa","startTime":374.899,"endTime":377.629,"body":"used it and never passed"},{"speaker":"Evan Morikawa","startTime":377.929,"endTime":379.499,"body":"number five on Hacker News."},{"speaker":"Evan Morikawa","startTime":379.979,"endTime":383.409,"body":"We had all the capacity we"},{"speaker":"Evan Morikawa","startTime":379.979,"endTime":383.409,"body":"needed, traffic tapered off."},{"speaker":"Evan Morikawa","startTime":383.469,"endTime":386.559,"body":"We're like, great, quick"},{"speaker":"Evan Morikawa","startTime":383.469,"endTime":386.559,"body":"little launch, like move on."},{"speaker":"Evan Morikawa","startTime":386.894,"endTime":390.054,"body":"You know, actually, at the time,"},{"speaker":"Evan Morikawa","startTime":386.894,"endTime":390.054,"body":"we were preparing internally"},{"speaker":"Evan Morikawa","startTime":390.054,"endTime":393.754,"body":"for the launch of GPT 4, which"},{"speaker":"Evan Morikawa","startTime":390.054,"endTime":393.754,"body":"was coming up next, so this"},{"speaker":"Evan Morikawa","startTime":393.754,"endTime":398.114,"body":"was, this was actually a way"},{"speaker":"Evan Morikawa","startTime":393.754,"endTime":398.114,"body":"for us to experiment with a"},{"speaker":"Evan Morikawa","startTime":398.114,"endTime":401.044,"body":"lot of the recent fine tuning"},{"speaker":"Evan Morikawa","startTime":398.114,"endTime":401.044,"body":"that had gone into the older"},{"speaker":"Evan Morikawa","startTime":401.044,"endTime":404.044,"body":"models to, like, make them"},{"speaker":"Evan Morikawa","startTime":401.044,"endTime":404.044,"body":"safer and more conversational."},{"speaker":"Evan Morikawa","startTime":404.044,"endTime":405.764,"body":"It was the next"},{"speaker":"Evan Morikawa","startTime":404.044,"endTime":405.764,"body":"day, or rather 4 a."},{"speaker":"Evan Morikawa","startTime":405.764,"endTime":405.974,"body":"m."},{"speaker":"Evan Morikawa","startTime":405.974,"endTime":409.674,"body":"the next morning, when"},{"speaker":"Evan Morikawa","startTime":405.974,"endTime":409.674,"body":"our on call starts to get"},{"speaker":"Evan Morikawa","startTime":409.674,"endTime":413.484,"body":"paged because traffic is"},{"speaker":"Evan Morikawa","startTime":409.674,"endTime":413.484,"body":"starting to really rise."},{"speaker":"Evan Morikawa","startTime":414.134,"endTime":416.644,"body":"There was this graph, we were"},{"speaker":"Evan Morikawa","startTime":414.134,"endTime":416.644,"body":"trying to figure out what was"},{"speaker":"Evan Morikawa","startTime":416.644,"endTime":420.214,"body":"going on, and all the traffic"},{"speaker":"Evan Morikawa","startTime":416.644,"endTime":420.214,"body":"was only coming from Japan."},{"speaker":"Evan Morikawa","startTime":420.344,"endTime":422.184,"body":"And we were very confused,"},{"speaker":"Evan Morikawa","startTime":420.344,"endTime":422.184,"body":"we actually thought we were"},{"speaker":"Evan Morikawa","startTime":422.184,"endTime":424.654,"body":"getting like DDoS'd or like"},{"speaker":"Evan Morikawa","startTime":422.184,"endTime":424.654,"body":"attacked or something like that."},{"speaker":"Evan Morikawa","startTime":425.064,"endTime":427.674,"body":"but no, they had just woken"},{"speaker":"Evan Morikawa","startTime":425.064,"endTime":427.674,"body":"up first, it was starting to"},{"speaker":"Evan Morikawa","startTime":427.834,"endTime":430.798,"body":"virally spread through Twitter,"},{"speaker":"Evan Morikawa","startTime":427.834,"endTime":430.798,"body":"and then yeah, and then by"},{"speaker":"Evan Morikawa","startTime":430.798,"endTime":434.468,"body":"the time the morning of the"},{"speaker":"Evan Morikawa","startTime":430.798,"endTime":434.468,"body":"East and the West Coast picked"},{"speaker":"Evan Morikawa","startTime":434.468,"endTime":436.268,"body":"up, it was like very clear."},{"speaker":"Evan Morikawa","startTime":436.643,"endTime":438.233,"body":"That we were getting"},{"speaker":"Evan Morikawa","startTime":436.643,"endTime":438.233,"body":"hammered here."},{"speaker":"Evan Morikawa","startTime":438.643,"endTime":442.563,"body":"unfortunately there, you know,"},{"speaker":"Evan Morikawa","startTime":438.643,"endTime":442.563,"body":"normally you can just like"},{"speaker":"Evan Morikawa","startTime":443.143,"endTime":445.783,"body":"throw more servers at the"},{"speaker":"Evan Morikawa","startTime":443.143,"endTime":445.783,"body":"problem, but yes, there is"},{"speaker":"Evan Morikawa","startTime":445.783,"endTime":450.263,"body":"like a very finite supply of"},{"speaker":"Evan Morikawa","startTime":445.783,"endTime":450.263,"body":"GPUs here, so there's really"},{"speaker":"Evan Morikawa","startTime":450.263,"endTime":451.253,"body":"not much we can do about it."},{"speaker":"Evan Morikawa","startTime":451.283,"endTime":455.213,"body":"The, we did have some"},{"speaker":"Evan Morikawa","startTime":451.283,"endTime":455.213,"body":"contingency in place for this."},{"speaker":"Evan Morikawa","startTime":455.243,"endTime":457.403,"body":"The idea being that"},{"speaker":"Evan Morikawa","startTime":455.243,"endTime":457.403,"body":"we could throw up a."},{"speaker":"Evan Morikawa","startTime":457.968,"endTime":461.838,"body":"Like, WeAreAtCapacityPage,"},{"speaker":"Evan Morikawa","startTime":457.968,"endTime":461.838,"body":"this kind of like, bouncer"},{"speaker":"Evan Morikawa","startTime":461.838,"endTime":463.458,"body":"model, if you will, right?"},{"speaker":"Evan Morikawa","startTime":463.458,"endTime":466.098,"body":"Like, oh, the club is full,"},{"speaker":"Evan Morikawa","startTime":463.458,"endTime":466.098,"body":"when some people leave,"},{"speaker":"Evan Morikawa","startTime":466.098,"endTime":467.198,"body":"we can let more people in."},{"speaker":"Evan Morikawa","startTime":467.718,"endTime":469.838,"body":"Uh, that was actually explicitly"},{"speaker":"Evan Morikawa","startTime":467.718,"endTime":469.838,"body":"done because we also did"},{"speaker":"Evan Morikawa","startTime":469.838,"endTime":472.208,"body":"not want another waitlist."},{"speaker":"Evan Morikawa","startTime":472.428,"endTime":475.608,"body":"like, no one likes a waitlist,"},{"speaker":"Evan Morikawa","startTime":472.428,"endTime":475.608,"body":"so we're like, oh, we can..."},{"speaker":"Evan Morikawa","startTime":475.913,"endTime":478.183,"body":"Try it this way, but"},{"speaker":"Evan Morikawa","startTime":475.913,"endTime":478.183,"body":"unfortunately that"},{"speaker":"Evan Morikawa","startTime":478.183,"endTime":483.173,"body":"WeAreAtCapacity page was up"},{"speaker":"Evan Morikawa","startTime":478.183,"endTime":483.173,"body":"a lot for the first while"},{"speaker":"Evan Morikawa","startTime":483.173,"endTime":485.293,"body":"while we were trying to"},{"speaker":"Evan Morikawa","startTime":483.173,"endTime":485.293,"body":"like scramble to get, to"},{"speaker":"Evan Morikawa","startTime":485.293,"endTime":486.533,"body":"get more capacity online."},{"speaker":"Evan Morikawa","startTime":486.533,"endTime":489.853,"body":"And it also just like fixed the"},{"speaker":"Evan Morikawa","startTime":486.533,"endTime":489.853,"body":"long tail of other stuff too."},{"speaker":"Evan Morikawa","startTime":489.853,"endTime":493.683,"body":"GPU capacity was definitely"},{"speaker":"Evan Morikawa","startTime":489.853,"endTime":493.683,"body":"a dominant concern,"},{"speaker":"Evan Morikawa","startTime":493.683,"endTime":495.673,"body":"but we also had all the"},{"speaker":"Evan Morikawa","startTime":493.683,"endTime":495.673,"body":"other scaling problems."},{"speaker":"Evan Morikawa","startTime":496.318,"endTime":499.138,"body":"Kind of other everybody"},{"speaker":"Evan Morikawa","startTime":496.318,"endTime":499.138,"body":"else in engineering has too."},{"speaker":"Ben Lloyd Pearson","startTime":499.888,"endTime":500.158,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":500.218,"endTime":500.518,"body":"Wow."},{"speaker":"Ben Lloyd Pearson","startTime":500.518,"endTime":502.018,"body":"That's a fascinating story."},{"speaker":"Ben Lloyd Pearson","startTime":502.018,"endTime":504.388,"body":"And, and trust me, I"},{"speaker":"Ben Lloyd Pearson","startTime":502.018,"endTime":504.388,"body":"remember those, uh,"},{"speaker":"Ben Lloyd Pearson","startTime":504.388,"endTime":506.398,"body":"capacity pages quite well."},{"speaker":"Ben Lloyd Pearson","startTime":507.598,"endTime":508.138,"body":", so yeah."},{"speaker":"Ben Lloyd Pearson","startTime":508.138,"endTime":510.748,"body":"So we, I wanna talk about a,"},{"speaker":"Ben Lloyd Pearson","startTime":508.138,"endTime":510.748,"body":"a little bit before we get"},{"speaker":"Ben Lloyd Pearson","startTime":510.748,"endTime":513.418,"body":"into the GPU stuff and some of"},{"speaker":"Ben Lloyd Pearson","startTime":510.748,"endTime":513.418,"body":"the scaling issues about this"},{"speaker":"Ben Lloyd Pearson","startTime":513.418,"endTime":517.198,"body":"black box that is ai because"},{"speaker":"Ben Lloyd Pearson","startTime":513.418,"endTime":517.198,"body":"it is really how it feels to"},{"speaker":"Ben Lloyd Pearson","startTime":517.198,"endTime":519.938,"body":"a lot of people, and many of"},{"speaker":"Ben Lloyd Pearson","startTime":517.198,"endTime":519.938,"body":"the productized versions of it."},{"speaker":"Ben Lloyd Pearson","startTime":519.943,"endTime":522.488,"body":"That's, that's really kind of"},{"speaker":"Ben Lloyd Pearson","startTime":519.943,"endTime":522.488,"body":"how they, they perform, right?"},{"speaker":"Ben Lloyd Pearson","startTime":522.488,"endTime":522.548,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":522.888,"endTime":524.978,"body":"And, you know, there's"},{"speaker":"Ben Lloyd Pearson","startTime":522.888,"endTime":524.978,"body":"no shortage of content on"},{"speaker":"Ben Lloyd Pearson","startTime":524.978,"endTime":528.478,"body":"the web that describes how"},{"speaker":"Ben Lloyd Pearson","startTime":524.978,"endTime":528.478,"body":"generative AI and LLMs are"},{"speaker":"Ben Lloyd Pearson","startTime":528.478,"endTime":531.828,"body":"going to do both wonderful and"},{"speaker":"Ben Lloyd Pearson","startTime":528.478,"endTime":531.828,"body":"horrible things to all of us."},{"speaker":"Ben Lloyd Pearson","startTime":532.548,"endTime":534.978,"body":"And, you know, as this"},{"speaker":"Ben Lloyd Pearson","startTime":532.548,"endTime":534.978,"body":"initial hype wave wears"},{"speaker":"Ben Lloyd Pearson","startTime":534.978,"endTime":537.608,"body":"off, I think we're really"},{"speaker":"Ben Lloyd Pearson","startTime":534.978,"endTime":537.608,"body":"starting to see like concrete"},{"speaker":"Ben Lloyd Pearson","startTime":537.618,"endTime":540.678,"body":"use cases that truly bring"},{"speaker":"Ben Lloyd Pearson","startTime":537.618,"endTime":540.678,"body":"a lot of value to people."},{"speaker":"Ben Lloyd Pearson","startTime":540.928,"endTime":542.418,"body":"And, you know, I'm thinking,"},{"speaker":"Ben Lloyd Pearson","startTime":540.928,"endTime":542.418,"body":"you know, just from my"},{"speaker":"Ben Lloyd Pearson","startTime":542.418,"endTime":545.938,"body":"personal experience, like"},{"speaker":"Ben Lloyd Pearson","startTime":542.418,"endTime":545.938,"body":"my grammar checker software"},{"speaker":"Ben Lloyd Pearson","startTime":545.968,"endTime":548.398,"body":"giving me more intelligent"},{"speaker":"Ben Lloyd Pearson","startTime":545.968,"endTime":548.398,"body":"advice about my writing."},{"speaker":"Ben Lloyd Pearson","startTime":555.648,"endTime":557.678,"body":"And it can't do everything, but"},{"speaker":"Ben Lloyd Pearson","startTime":555.648,"endTime":557.678,"body":"there's still some things that"},{"speaker":"Ben Lloyd Pearson","startTime":557.698,"endTime":559.318,"body":"it just saves me so much time."},{"speaker":"Ben Lloyd Pearson","startTime":559.775,"endTime":561.275,"body":"so, you know, I think it"},{"speaker":"Ben Lloyd Pearson","startTime":559.775,"endTime":561.275,"body":"would be really interesting to"},{"speaker":"Ben Lloyd Pearson","startTime":561.275,"endTime":564.035,"body":"hear from an engineer that's"},{"speaker":"Ben Lloyd Pearson","startTime":561.275,"endTime":564.035,"body":"actually building this stuff."},{"speaker":"Ben Lloyd Pearson","startTime":564.065,"endTime":566.649,"body":"Like, about the biggest"},{"speaker":"Ben Lloyd Pearson","startTime":564.065,"endTime":566.649,"body":"misconceptions, the biggest"},{"speaker":"Ben Lloyd Pearson","startTime":566.695,"endTime":569.845,"body":"misunderstandings that you've"},{"speaker":"Ben Lloyd Pearson","startTime":566.695,"endTime":569.845,"body":"seen, related to all of this."},{"speaker":"Ben Lloyd Pearson","startTime":569.845,"endTime":572.155,"body":"Uh, and, and specifically,"},{"speaker":"Ben Lloyd Pearson","startTime":569.845,"endTime":572.155,"body":"if there's like one or"},{"speaker":"Ben Lloyd Pearson","startTime":572.155,"endTime":574.405,"body":"two things that you can"},{"speaker":"Ben Lloyd Pearson","startTime":572.155,"endTime":574.405,"body":"clarify for the world."},{"speaker":"Ben Lloyd Pearson","startTime":574.935,"endTime":577.175,"body":"About AI, like what"},{"speaker":"Ben Lloyd Pearson","startTime":574.935,"endTime":577.175,"body":"would those things be?"},{"speaker":"Evan Morikawa","startTime":577.889,"endTime":578.789,"body":"that's"},{"speaker":"Evan Morikawa","startTime":577.889,"endTime":578.789,"body":"a good question."},{"speaker":"Evan Morikawa","startTime":578.799,"endTime":581.372,"body":"The, certainly one"},{"speaker":"Evan Morikawa","startTime":578.799,"endTime":581.372,"body":"misconception is that it"},{"speaker":"Evan Morikawa","startTime":581.372,"endTime":584.212,"body":"is definitely, certainly"},{"speaker":"Evan Morikawa","startTime":581.372,"endTime":584.212,"body":"not as it exists today."},{"speaker":"Evan Morikawa","startTime":584.252,"endTime":587.022,"body":"This completely omnipotent"},{"speaker":"Evan Morikawa","startTime":584.252,"endTime":587.022,"body":"system here, right?"},{"speaker":"Evan Morikawa","startTime":587.032,"endTime":589.872,"body":"It is, there are quirks"},{"speaker":"Evan Morikawa","startTime":587.032,"endTime":589.872,"body":"about how this thing works."},{"speaker":"Evan Morikawa","startTime":589.902,"endTime":592.802,"body":"It's actually quite helpful"},{"speaker":"Evan Morikawa","startTime":589.902,"endTime":592.802,"body":"to remember somewhat how"},{"speaker":"Evan Morikawa","startTime":592.802,"endTime":594.042,"body":"these things are trained."},{"speaker":"Evan Morikawa","startTime":594.532,"endTime":598.062,"body":"They're trained by predicting"},{"speaker":"Evan Morikawa","startTime":594.532,"endTime":598.062,"body":"the next word, for all words"},{"speaker":"Evan Morikawa","startTime":598.062,"endTime":599.292,"body":"and phrases on the internet."},{"speaker":"Evan Morikawa","startTime":599.832,"endTime":603.812,"body":"And that, though, is in itself"},{"speaker":"Evan Morikawa","startTime":599.832,"endTime":603.812,"body":"deceptive, because this is"},{"speaker":"Evan Morikawa","startTime":603.812,"endTime":607.322,"body":"much more than a Mad Lib system"},{"speaker":"Evan Morikawa","startTime":603.812,"endTime":607.322,"body":"or an autocomplete engine."},{"speaker":"Evan Morikawa","startTime":607.842,"endTime":610.702,"body":"Because it has turned out that"},{"speaker":"Evan Morikawa","startTime":607.842,"endTime":610.702,"body":"in order to be able to predict"},{"speaker":"Evan Morikawa","startTime":610.702,"endTime":613.492,"body":"the next word, you kind of"},{"speaker":"Evan Morikawa","startTime":610.702,"endTime":613.492,"body":"need to know a huge amount"},{"speaker":"Evan Morikawa","startTime":613.492,"endTime":617.152,"body":"about society and structure"},{"speaker":"Evan Morikawa","startTime":613.492,"endTime":617.152,"body":"and context and culture and"},{"speaker":"Evan Morikawa","startTime":617.162,"endTime":618.322,"body":"things like that as well, too."},{"speaker":"Evan Morikawa","startTime":619.117,"endTime":622.177,"body":"But at the same time,"},{"speaker":"Evan Morikawa","startTime":619.117,"endTime":622.177,"body":"they're also very steerable"},{"speaker":"Evan Morikawa","startTime":622.177,"endTime":624.847,"body":"based on the context that"},{"speaker":"Evan Morikawa","startTime":622.177,"endTime":624.847,"body":"you give it ahead of time."},{"speaker":"Evan Morikawa","startTime":625.327,"endTime":628.227,"body":"You know, when GPT 3 first"},{"speaker":"Evan Morikawa","startTime":625.327,"endTime":628.227,"body":"came out, this was, in fact,"},{"speaker":"Evan Morikawa","startTime":628.227,"endTime":630.887,"body":"the title of the paper is"},{"speaker":"Evan Morikawa","startTime":628.227,"endTime":630.887,"body":"about few shot prompting."},{"speaker":"Evan Morikawa","startTime":631.427,"endTime":634.147,"body":"Few shot here basically refers"},{"speaker":"Evan Morikawa","startTime":631.427,"endTime":634.147,"body":"to the idea that you just give"},{"speaker":"Evan Morikawa","startTime":634.157,"endTime":637.517,"body":"the model a handful, like three"},{"speaker":"Evan Morikawa","startTime":634.157,"endTime":637.517,"body":"or four examples of what you"},{"speaker":"Evan Morikawa","startTime":637.517,"endTime":639.847,"body":"want it to do, and that kind"},{"speaker":"Evan Morikawa","startTime":637.517,"endTime":639.847,"body":"of eggs it into the right..."},{"speaker":"Evan Morikawa","startTime":640.362,"endTime":640.992,"body":"Direction."},{"speaker":"Evan Morikawa","startTime":641.372,"endTime":643.672,"body":"in some ways this is"},{"speaker":"Evan Morikawa","startTime":641.372,"endTime":643.672,"body":"not too unfamiliar."},{"speaker":"Evan Morikawa","startTime":643.682,"endTime":646.502,"body":"I think, you know, if you"},{"speaker":"Evan Morikawa","startTime":643.682,"endTime":646.502,"body":"go on Google, old Google at"},{"speaker":"Evan Morikawa","startTime":646.502,"endTime":649.032,"body":"least, if you type some, if you"},{"speaker":"Evan Morikawa","startTime":646.502,"endTime":649.032,"body":"type a question one way, you"},{"speaker":"Evan Morikawa","startTime":649.032,"endTime":651.062,"body":"get Yahoo answers, but if you"},{"speaker":"Evan Morikawa","startTime":649.032,"endTime":651.062,"body":"type it a different way, you"},{"speaker":"Evan Morikawa","startTime":651.062,"endTime":652.352,"body":"get like a scientific paper."},{"speaker":"Evan Morikawa","startTime":652.792,"endTime":654.882,"body":"so kind of in the same"},{"speaker":"Evan Morikawa","startTime":652.792,"endTime":654.882,"body":"way, you can steer it in"},{"speaker":"Evan Morikawa","startTime":654.882,"endTime":655.832,"body":"the direction of things."},{"speaker":"Evan Morikawa","startTime":656.267,"endTime":658.197,"body":"the one thing that's a"},{"speaker":"Evan Morikawa","startTime":656.267,"endTime":658.197,"body":"little weird about this,"},{"speaker":"Evan Morikawa","startTime":658.197,"endTime":662.527,"body":"though, is this has left"},{"speaker":"Evan Morikawa","startTime":658.197,"endTime":662.527,"body":"some very seemingly black box"},{"speaker":"Evan Morikawa","startTime":662.527,"endTime":666.817,"body":"types of prompt engineering"},{"speaker":"Evan Morikawa","startTime":662.527,"endTime":666.817,"body":"into, into this right now."},{"speaker":"Evan Morikawa","startTime":667.147,"endTime":669.217,"body":"Uh, there were some papers"},{"speaker":"Evan Morikawa","startTime":667.147,"endTime":669.217,"body":"recently that have been getting"},{"speaker":"Evan Morikawa","startTime":669.217,"endTime":671.707,"body":"a lot of press that simply..."},{"speaker":"Evan Morikawa","startTime":671.907,"endTime":676.417,"body":"Say, if you ask in the prompt,"},{"speaker":"Evan Morikawa","startTime":671.907,"endTime":676.417,"body":"literally take a deep breath"},{"speaker":"Evan Morikawa","startTime":676.477,"endTime":680.557,"body":"and think step by step, it"},{"speaker":"Evan Morikawa","startTime":676.477,"endTime":680.557,"body":"does much better on a large"},{"speaker":"Evan Morikawa","startTime":681.037,"endTime":686.187,"body":"like category of tasks, that's"},{"speaker":"Evan Morikawa","startTime":681.037,"endTime":686.187,"body":"something there feels . Wow."},{"speaker":"Evan Morikawa","startTime":686.337,"endTime":688.047,"body":"Like kind of wrong on one hand."},{"speaker":"Ben Lloyd Pearson","startTime":688.427,"endTime":690.167,"body":"It feels very"},{"speaker":"Ben Lloyd Pearson","startTime":688.427,"endTime":690.167,"body":"humanistic though, doesn't it?"},{"speaker":"Evan Morikawa","startTime":690.172,"endTime":692.027,"body":"Well, that's a,"},{"speaker":"Evan Morikawa","startTime":690.172,"endTime":692.027,"body":"that's actually the great point."},{"speaker":"Evan Morikawa","startTime":692.032,"endTime":694.997,"body":"On the flip side though,"},{"speaker":"Evan Morikawa","startTime":692.032,"endTime":694.997,"body":"if you kind of assume that"},{"speaker":"Evan Morikawa","startTime":694.997,"endTime":698.057,"body":"the models are going to"},{"speaker":"Evan Morikawa","startTime":694.997,"endTime":698.057,"body":"approach kind of a human"},{"speaker":"Evan Morikawa","startTime":698.057,"endTime":699.407,"body":"type level of intelligence."},{"speaker":"Evan Morikawa","startTime":699.942,"endTime":702.492,"body":"It's worth to ask yourself"},{"speaker":"Evan Morikawa","startTime":699.942,"endTime":702.492,"body":"if you throw, if you threw a"},{"speaker":"Evan Morikawa","startTime":702.492,"endTime":706.562,"body":"relatively competent human in"},{"speaker":"Evan Morikawa","startTime":702.492,"endTime":706.562,"body":"what you're asking it to do"},{"speaker":"Evan Morikawa","startTime":706.592,"endTime":709.592,"body":"with as much context as you"},{"speaker":"Evan Morikawa","startTime":706.592,"endTime":709.592,"body":"gave it, how would they perform?"},{"speaker":"Evan Morikawa","startTime":709.682,"endTime":709.772,"body":"Mm-Hmm."},{"speaker":"Evan Morikawa","startTime":711.102,"endTime":714.282,"body":"and it's not unreasonable to"},{"speaker":"Evan Morikawa","startTime":711.102,"endTime":714.282,"body":"think that these models kind"},{"speaker":"Evan Morikawa","startTime":714.282,"endTime":716.937,"body":"of mimic that because they"},{"speaker":"Evan Morikawa","startTime":714.282,"endTime":716.937,"body":"are mimicking human speech."},{"speaker":"Evan Morikawa","startTime":717.862,"endTime":719.692,"body":"We've seen it on the internet."},{"speaker":"Evan Morikawa","startTime":720.282,"endTime":723.132,"body":"so yeah, in, in fact, if you"},{"speaker":"Evan Morikawa","startTime":720.282,"endTime":723.132,"body":"are, I actually think some"},{"speaker":"Evan Morikawa","startTime":723.132,"endTime":725.022,"body":"of the people who are the,"},{"speaker":"Evan Morikawa","startTime":723.132,"endTime":725.022,"body":"could be the best at prompt"},{"speaker":"Evan Morikawa","startTime":725.022,"endTime":729.222,"body":"engineering are like teachers,"},{"speaker":"Evan Morikawa","startTime":725.022,"endTime":729.222,"body":"engineering managers, tutors,"},{"speaker":"Evan Morikawa","startTime":729.222,"endTime":732.912,"body":"people who are used to like"},{"speaker":"Evan Morikawa","startTime":729.222,"endTime":732.912,"body":"asking the right questions and"},{"speaker":"Evan Morikawa","startTime":732.912,"endTime":736.332,"body":"setting up the right context"},{"speaker":"Evan Morikawa","startTime":732.912,"endTime":736.332,"body":"for people to like, help them"},{"speaker":"Evan Morikawa","startTime":736.332,"endTime":738.072,"body":"arrive at the right conclusions."},{"speaker":"Evan Morikawa","startTime":738.482,"endTime":740.882,"body":"Uh, and if you kind of think"},{"speaker":"Evan Morikawa","startTime":738.482,"endTime":740.882,"body":"about it that way, you get"},{"speaker":"Evan Morikawa","startTime":740.882,"endTime":741.967,"body":"like weirdly better results."},{"speaker":"Evan Morikawa","startTime":742.57,"endTime":743.41,"body":"across there."},{"speaker":"Ben Lloyd Pearson","startTime":744.36,"endTime":744.54,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":744.54,"endTime":746.04,"body":"And I think you're,"},{"speaker":"Ben Lloyd Pearson","startTime":744.54,"endTime":746.04,"body":"you're actually partially"},{"speaker":"Ben Lloyd Pearson","startTime":746.04,"endTime":747.39,"body":"answering my next question."},{"speaker":"Ben Lloyd Pearson","startTime":747.39,"endTime":749.79,"body":"So, you know, in your, in your"},{"speaker":"Ben Lloyd Pearson","startTime":747.39,"endTime":749.79,"body":"opinion, like what are the"},{"speaker":"Ben Lloyd Pearson","startTime":749.79,"endTime":753.27,"body":"situations that are ideal for"},{"speaker":"Ben Lloyd Pearson","startTime":749.79,"endTime":753.27,"body":"generative, generative ai?"},{"speaker":"Ben Lloyd Pearson","startTime":753.54,"endTime":753.66,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":753.66,"endTime":755.7,"body":"And alternatively, like"},{"speaker":"Ben Lloyd Pearson","startTime":753.66,"endTime":755.7,"body":"when would you steer someone"},{"speaker":"Ben Lloyd Pearson","startTime":755.7,"endTime":756.99,"body":"away from it as a solution?"},{"speaker":"Evan Morikawa","startTime":756.99,"endTime":757.285,"body":"Yeah, yeah, yeah."},{"speaker":"Evan Morikawa","startTime":757.5,"endTime":760.05,"body":"some places I think has"},{"speaker":"Evan Morikawa","startTime":757.5,"endTime":760.05,"body":"absolutely been helpful."},{"speaker":"Evan Morikawa","startTime":760.08,"endTime":762.42,"body":"I really just started to"},{"speaker":"Evan Morikawa","startTime":760.08,"endTime":762.42,"body":"tap at, right now certainly"},{"speaker":"Evan Morikawa","startTime":762.42,"endTime":765.33,"body":"software engineering, coding,"},{"speaker":"Evan Morikawa","startTime":762.42,"endTime":765.33,"body":"boilerplate type things."},{"speaker":"Evan Morikawa","startTime":765.33,"endTime":767.64,"body":"That was the first place"},{"speaker":"Evan Morikawa","startTime":765.33,"endTime":767.64,"body":"that we internally really"},{"speaker":"Evan Morikawa","startTime":767.85,"endTime":770.844,"body":"dogfooded any of this was when"},{"speaker":"Evan Morikawa","startTime":767.85,"endTime":770.844,"body":"we ourselves Started to use"},{"speaker":"Evan Morikawa","startTime":770.844,"endTime":773.325,"body":"Copilot as an educational tool."},{"speaker":"Evan Morikawa","startTime":773.765,"endTime":776.895,"body":"I think that is still, despite"},{"speaker":"Evan Morikawa","startTime":773.765,"endTime":776.895,"body":"how much has been talked"},{"speaker":"Evan Morikawa","startTime":776.895,"endTime":780.605,"body":"about, still an underrated,"},{"speaker":"Evan Morikawa","startTime":776.895,"endTime":780.605,"body":"undertapped ability here."},{"speaker":"Evan Morikawa","startTime":780.605,"endTime":786.295,"body":"The idea of a personalized"},{"speaker":"Evan Morikawa","startTime":780.605,"endTime":786.295,"body":"tutor everywhere you go."},{"speaker":"Evan Morikawa","startTime":786.295,"endTime":790.395,"body":"Like, TAs, think about"},{"speaker":"Evan Morikawa","startTime":786.295,"endTime":790.395,"body":"university, the professor can"},{"speaker":"Evan Morikawa","startTime":790.395,"endTime":792.695,"body":"talk at you all day, but it"},{"speaker":"Evan Morikawa","startTime":790.395,"endTime":792.695,"body":"was the TAs where I really"},{"speaker":"Evan Morikawa","startTime":792.695,"endTime":795.665,"body":"learned things, and those"},{"speaker":"Evan Morikawa","startTime":792.695,"endTime":795.665,"body":"follow up sessions, because you"},{"speaker":"Evan Morikawa","startTime":795.665,"endTime":797.285,"body":"could ask follow up questions."},{"speaker":"Evan Morikawa","startTime":797.635,"endTime":800.195,"body":"And you can frame it in a"},{"speaker":"Evan Morikawa","startTime":797.635,"endTime":800.195,"body":"way that makes sense to you."},{"speaker":"Evan Morikawa","startTime":800.675,"endTime":804.255,"body":"that kind of iterative"},{"speaker":"Evan Morikawa","startTime":800.675,"endTime":804.255,"body":"learning, I mean this is where"},{"speaker":"Evan Morikawa","startTime":804.255,"endTime":805.945,"body":"I personally use it the most."},{"speaker":"Evan Morikawa","startTime":805.955,"endTime":809.185,"body":"Like the thought of reading"},{"speaker":"Evan Morikawa","startTime":805.955,"endTime":809.185,"body":"any paper without this thing"},{"speaker":"Evan Morikawa","startTime":809.385,"endTime":812.255,"body":"on the side, or without"},{"speaker":"Evan Morikawa","startTime":809.385,"endTime":812.255,"body":"being able to like, just"},{"speaker":"Evan Morikawa","startTime":812.275,"endTime":814.975,"body":"being able to like ask it for"},{"speaker":"Evan Morikawa","startTime":812.275,"endTime":814.975,"body":"concrete examples of things."},{"speaker":"Evan Morikawa","startTime":814.975,"endTime":817.485,"body":"Being able to rephrase"},{"speaker":"Evan Morikawa","startTime":814.975,"endTime":817.485,"body":"and reword and take"},{"speaker":"Evan Morikawa","startTime":817.485,"endTime":818.495,"body":"follow up questions."},{"speaker":"Evan Morikawa","startTime":818.595,"endTime":819.935,"body":"That I think is going"},{"speaker":"Evan Morikawa","startTime":818.595,"endTime":819.935,"body":"to be a huge deal."},{"speaker":"Evan Morikawa","startTime":820.575,"endTime":823.645,"body":"there are some places, on the"},{"speaker":"Evan Morikawa","startTime":820.575,"endTime":823.645,"body":"flip side of this, This, though,"},{"speaker":"Evan Morikawa","startTime":823.745,"endTime":827.475,"body":"uh, yeah, like it still has,"},{"speaker":"Evan Morikawa","startTime":823.745,"endTime":827.475,"body":"we have not in any way, shape,"},{"speaker":"Evan Morikawa","startTime":827.475,"endTime":831.465,"body":"or form solved this, like,"},{"speaker":"Evan Morikawa","startTime":827.475,"endTime":831.465,"body":"perfectly verifiable problem."},{"speaker":"Evan Morikawa","startTime":831.515,"endTime":834.875,"body":"This should not be your"},{"speaker":"Evan Morikawa","startTime":831.515,"endTime":834.875,"body":"end state for medical"},{"speaker":"Evan Morikawa","startTime":834.875,"endTime":837.595,"body":"advice on a huge slew"},{"speaker":"Evan Morikawa","startTime":834.875,"endTime":837.595,"body":"of the topics right now."},{"speaker":"Evan Morikawa","startTime":837.895,"endTime":840.635,"body":"It should not be the thing"},{"speaker":"Evan Morikawa","startTime":837.895,"endTime":840.635,"body":"that is, you are trying to"},{"speaker":"Evan Morikawa","startTime":840.645,"endTime":843.735,"body":"use to cite case law for"},{"speaker":"Evan Morikawa","startTime":840.645,"endTime":843.735,"body":"your own trial, for example."},{"speaker":"Ben Lloyd Pearson","startTime":843.735,"endTime":844.345,"body":"I"},{"speaker":"Ben Lloyd Pearson","startTime":843.735,"endTime":844.345,"body":"think a lot of people"},{"speaker":"Ben Lloyd Pearson","startTime":844.345,"endTime":845.415,"body":"call this hallucination."},{"speaker":"Evan Morikawa","startTime":845.435,"endTime":845.935,"body":"That's right."},{"speaker":"Evan Morikawa","startTime":846.409,"endTime":849.129,"body":"Now, on the flip side,"},{"speaker":"Evan Morikawa","startTime":846.409,"endTime":849.129,"body":"there's actually law."},{"speaker":"Evan Morikawa","startTime":849.139,"endTime":851.749,"body":"I think it's a really"},{"speaker":"Evan Morikawa","startTime":849.139,"endTime":851.749,"body":"interesting area as well, too."},{"speaker":"Evan Morikawa","startTime":852.099,"endTime":854.699,"body":"Especially some of the"},{"speaker":"Evan Morikawa","startTime":852.099,"endTime":854.699,"body":"side effects of this, like"},{"speaker":"Evan Morikawa","startTime":854.729,"endTime":857.719,"body":"the power of the embedding"},{"speaker":"Evan Morikawa","startTime":854.729,"endTime":857.719,"body":"models that we have."},{"speaker":"Evan Morikawa","startTime":858.369,"endTime":861.249,"body":"you know, it is very good"},{"speaker":"Evan Morikawa","startTime":858.369,"endTime":861.249,"body":"at saying what patents"},{"speaker":"Evan Morikawa","startTime":861.249,"endTime":862.339,"body":"are similar to this one."},{"speaker":"Evan Morikawa","startTime":862.399,"endTime":864.509,"body":"What cases are"},{"speaker":"Evan Morikawa","startTime":862.399,"endTime":864.509,"body":"similar to this one."},{"speaker":"Evan Morikawa","startTime":864.809,"endTime":866.909,"body":"And in ways that are much"},{"speaker":"Evan Morikawa","startTime":864.809,"endTime":866.909,"body":"more than just do they"},{"speaker":"Evan Morikawa","startTime":866.919,"endTime":868.109,"body":"have similar keywords."},{"speaker":"Evan Morikawa","startTime":868.644,"endTime":871.894,"body":"But the fact that these models"},{"speaker":"Evan Morikawa","startTime":868.644,"endTime":871.894,"body":"deeply semantically understand"},{"speaker":"Evan Morikawa","startTime":872.244,"endTime":875.734,"body":"what's going on, it can help"},{"speaker":"Evan Morikawa","startTime":872.244,"endTime":875.734,"body":"you find and search like that."},{"speaker":"Evan Morikawa","startTime":875.784,"endTime":878.764,"body":"Like that's, yeah, those types"},{"speaker":"Evan Morikawa","startTime":875.784,"endTime":878.764,"body":"of search abilities will get"},{"speaker":"Evan Morikawa","startTime":878.784,"endTime":880.124,"body":"dramatically more powerful."},{"speaker":"Ben Lloyd Pearson","startTime":880.474,"endTime":882.574,"body":"Yeah, and"},{"speaker":"Ben Lloyd Pearson","startTime":880.474,"endTime":882.574,"body":"I know personally, one area"},{"speaker":"Ben Lloyd Pearson","startTime":882.574,"endTime":885.684,"body":"that I've really found a lot"},{"speaker":"Ben Lloyd Pearson","startTime":882.574,"endTime":885.684,"body":"of benefit is I have to very"},{"speaker":"Ben Lloyd Pearson","startTime":885.684,"endTime":888.464,"body":"quickly understand a lot of"},{"speaker":"Ben Lloyd Pearson","startTime":885.684,"endTime":888.464,"body":"new technologies as a part of"},{"speaker":"Ben Lloyd Pearson","startTime":888.464,"endTime":891.224,"body":"my day and work with things"},{"speaker":"Ben Lloyd Pearson","startTime":888.464,"endTime":891.224,"body":"like, you know, I don't, I"},{"speaker":"Ben Lloyd Pearson","startTime":891.224,"endTime":894.224,"body":"don't, I haven't done a lot of"},{"speaker":"Ben Lloyd Pearson","startTime":891.224,"endTime":894.224,"body":"Regex in the past, but I find"},{"speaker":"Ben Lloyd Pearson","startTime":894.224,"endTime":895.964,"body":"myself doing a lot of it today."},{"speaker":"Ben Lloyd Pearson","startTime":896.059,"endTime":898.879,"body":"And just getting that like"},{"speaker":"Ben Lloyd Pearson","startTime":896.059,"endTime":898.879,"body":"intermediate understanding,"},{"speaker":"Ben Lloyd Pearson","startTime":898.879,"endTime":901.609,"body":"like immediately without having"},{"speaker":"Ben Lloyd Pearson","startTime":898.879,"endTime":901.609,"body":"to like parse through a bunch"},{"speaker":"Ben Lloyd Pearson","startTime":901.609,"endTime":903.109,"body":"of resources across the web."},{"speaker":"Ben Lloyd Pearson","startTime":903.109,"endTime":903.439,"body":"Like Yep."},{"speaker":"Ben Lloyd Pearson","startTime":903.649,"endTime":905.629,"body":"I, I mean, I can't even, I"},{"speaker":"Ben Lloyd Pearson","startTime":903.649,"endTime":905.629,"body":"can't even add up the number"},{"speaker":"Ben Lloyd Pearson","startTime":905.629,"endTime":907.069,"body":"of hours that that saved me."},{"speaker":"Ben Lloyd Pearson","startTime":907.129,"endTime":907.249,"body":"Yep."},{"speaker":"Ben Lloyd Pearson","startTime":907.249,"endTime":908.479,"body":"You know, so Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":908.479,"endTime":909.799,"body":"It's really great to do that are"},{"speaker":"Evan Morikawa","startTime":909.799,"endTime":911.419,"body":"probably"},{"speaker":"Evan Morikawa","startTime":909.799,"endTime":911.419,"body":"the best example of that"},{"speaker":"Evan Morikawa","startTime":911.544,"endTime":911.624,"body":"Yeah."},{"speaker":"Evan Morikawa","startTime":911.814,"endTime":912.104,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":912.829,"endTime":913.129,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":913.219,"endTime":913.429,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":913.434,"endTime":914.089,"body":"That's been wild."},{"speaker":"Ben Lloyd Pearson","startTime":914.089,"endTime":916.489,"body":"It is actually kind of blown"},{"speaker":"Ben Lloyd Pearson","startTime":914.089,"endTime":916.489,"body":"my minds at how quickly, you"},{"speaker":"Ben Lloyd Pearson","startTime":916.489,"endTime":918.649,"body":"know, 'cause I mean, reg Xs is"},{"speaker":"Ben Lloyd Pearson","startTime":916.489,"endTime":918.649,"body":"like, is not complicated, but"},{"speaker":"Ben Lloyd Pearson","startTime":918.649,"endTime":921.049,"body":"it can be time consuming if you"},{"speaker":"Ben Lloyd Pearson","startTime":918.649,"endTime":921.049,"body":"don't work in it frequently."},{"speaker":"Ben Lloyd Pearson","startTime":921.079,"endTime":921.259,"body":"Yep."},{"speaker":"Ben Lloyd Pearson","startTime":921.259,"endTime":921.649,"body":"You know, so."},{"speaker":"Ben Lloyd Pearson","startTime":921.649,"endTime":921.949,"body":"Yep, yep."},{"speaker":"Ben Lloyd Pearson","startTime":921.949,"endTime":922.189,"body":"Yep."},{"speaker":"Ben Lloyd Pearson","startTime":922.399,"endTime":924.649,"body":"So then as you know, as an"},{"speaker":"Ben Lloyd Pearson","startTime":922.399,"endTime":924.649,"body":"engineering manager, like,"},{"speaker":"Ben Lloyd Pearson","startTime":924.859,"endTime":927.319,"body":"like what expectations"},{"speaker":"Ben Lloyd Pearson","startTime":924.859,"endTime":927.319,"body":"are you setting with your"},{"speaker":"Ben Lloyd Pearson","startTime":927.319,"endTime":929.809,"body":"team in regards to the"},{"speaker":"Ben Lloyd Pearson","startTime":927.319,"endTime":929.809,"body":"use of generative ai?"},{"speaker":"Ben Lloyd Pearson","startTime":929.809,"endTime":931.369,"body":"So, you know, it sounds"},{"speaker":"Ben Lloyd Pearson","startTime":929.809,"endTime":931.369,"body":"like you're an early"},{"speaker":"Ben Lloyd Pearson","startTime":931.374,"endTime":933.349,"body":"dog fooder of copilot."},{"speaker":"Ben Lloyd Pearson","startTime":933.599,"endTime":937.499,"body":"you know, is, is, is that, are"},{"speaker":"Ben Lloyd Pearson","startTime":933.599,"endTime":937.499,"body":"you are as an organization or"},{"speaker":"Ben Lloyd Pearson","startTime":937.504,"endTime":940.709,"body":"are you like, systematically"},{"speaker":"Ben Lloyd Pearson","startTime":937.504,"endTime":940.709,"body":"adopting tools like that and"},{"speaker":"Ben Lloyd Pearson","startTime":940.709,"endTime":942.689,"body":"you know, what are the, like"},{"speaker":"Ben Lloyd Pearson","startTime":940.709,"endTime":942.689,"body":"changes that you've seen?"},{"speaker":"Evan Morikawa","startTime":942.809,"endTime":943.199,"body":"yeah."},{"speaker":"Evan Morikawa","startTime":943.199,"endTime":944.699,"body":"So getting this really."},{"speaker":"Evan Morikawa","startTime":945.604,"endTime":948.834,"body":"We definitely want to"},{"speaker":"Evan Morikawa","startTime":945.604,"endTime":948.834,"body":"more and more have this"},{"speaker":"Evan Morikawa","startTime":948.834,"endTime":950.274,"body":"help us be productive."},{"speaker":"Evan Morikawa","startTime":950.364,"endTime":953.704,"body":"I mean the, we actually have"},{"speaker":"Evan Morikawa","startTime":950.364,"endTime":953.704,"body":"a research team called the AI"},{"speaker":"Evan Morikawa","startTime":953.714,"endTime":957.024,"body":"scientist team, which is very"},{"speaker":"Evan Morikawa","startTime":953.714,"endTime":957.024,"body":"much long term about being"},{"speaker":"Evan Morikawa","startTime":957.024,"endTime":958.624,"body":"able to make this, this work."},{"speaker":"Evan Morikawa","startTime":959.064,"endTime":963.326,"body":"at the same time though, the"},{"speaker":"Evan Morikawa","startTime":959.064,"endTime":963.326,"body":"There's a pretty wide gap"},{"speaker":"Evan Morikawa","startTime":963.346,"endTime":966.086,"body":"between what works just straight"},{"speaker":"Evan Morikawa","startTime":963.346,"endTime":966.086,"body":"off the bat from the prompt"},{"speaker":"Evan Morikawa","startTime":966.086,"endTime":970.216,"body":"in ChatGPT and like an actual"},{"speaker":"Evan Morikawa","startTime":966.086,"endTime":970.216,"body":"tool you'll use day to day."},{"speaker":"Evan Morikawa","startTime":970.556,"endTime":974.836,"body":"Like, yeah, some people did"},{"speaker":"Evan Morikawa","startTime":970.556,"endTime":974.836,"body":"a quick plug in to like hack"},{"speaker":"Evan Morikawa","startTime":974.836,"endTime":978.766,"body":"in before VS Code, but I"},{"speaker":"Evan Morikawa","startTime":974.836,"endTime":978.766,"body":"mean it still takes a lot"},{"speaker":"Evan Morikawa","startTime":978.766,"endTime":982.426,"body":"of product work to put the"},{"speaker":"Evan Morikawa","startTime":978.766,"endTime":982.426,"body":"whole experience together and"},{"speaker":"Evan Morikawa","startTime":982.426,"endTime":983.746,"body":"make it work really nicely."},{"speaker":"Evan Morikawa","startTime":984.366,"endTime":987.956,"body":"I think this kind of immediate"},{"speaker":"Evan Morikawa","startTime":984.366,"endTime":987.956,"body":"generation of like developer"},{"speaker":"Evan Morikawa","startTime":987.956,"endTime":991.496,"body":"productivity tools, it will take"},{"speaker":"Evan Morikawa","startTime":987.956,"endTime":991.496,"body":"a fairly large investment to..."},{"speaker":"Evan Morikawa","startTime":992.236,"endTime":995.186,"body":"Like, really put it"},{"speaker":"Evan Morikawa","startTime":992.236,"endTime":995.186,"body":"naturally into a workflow."},{"speaker":"Evan Morikawa","startTime":995.476,"endTime":997.736,"body":"That's actually why I'm"},{"speaker":"Evan Morikawa","startTime":995.476,"endTime":997.736,"body":"very optimistic about the"},{"speaker":"Evan Morikawa","startTime":997.766,"endTime":1002.456,"body":"coexistence of both a tool like"},{"speaker":"Evan Morikawa","startTime":997.766,"endTime":1002.456,"body":"ChatGPT and the API ecosystem."},{"speaker":"Evan Morikawa","startTime":1002.506,"endTime":1005.54,"body":"like, ChatGPT is about, we"},{"speaker":"Evan Morikawa","startTime":1002.506,"endTime":1005.54,"body":"think it can be useful in"},{"speaker":"Evan Morikawa","startTime":1005.55,"endTime":1006.87,"body":"lots of different places."},{"speaker":"Evan Morikawa","startTime":1007.27,"endTime":1009.85,"body":"In a much more kind of"},{"speaker":"Evan Morikawa","startTime":1007.27,"endTime":1009.85,"body":"generic sense, but there"},{"speaker":"Evan Morikawa","startTime":1009.85,"endTime":1013.06,"body":"are also a huge number of"},{"speaker":"Evan Morikawa","startTime":1009.85,"endTime":1013.06,"body":"industries where like being"},{"speaker":"Evan Morikawa","startTime":1013.06,"endTime":1014.95,"body":"in flow matters a huge amount."},{"speaker":"Evan Morikawa","startTime":1014.95,"endTime":1016.63,"body":"Developer tools, law, right?"},{"speaker":"Evan Morikawa","startTime":1016.72,"endTime":1020.05,"body":"Medical systems, like all these"},{"speaker":"Evan Morikawa","startTime":1016.72,"endTime":1020.05,"body":"other places, you'd also need"},{"speaker":"Evan Morikawa","startTime":1020.05,"endTime":1021.1,"body":"and want a lot of these like."},{"speaker":"Evan Morikawa","startTime":1021.1,"endTime":1022.84,"body":"integrated applications"},{"speaker":"Evan Morikawa","startTime":1021.1,"endTime":1022.84,"body":"as well too."},{"speaker":"Evan Morikawa","startTime":1023.39,"endTime":1027.04,"body":"but yeah, like we absolutely"},{"speaker":"Evan Morikawa","startTime":1023.39,"endTime":1027.04,"body":"believe this will make us and"},{"speaker":"Evan Morikawa","startTime":1027.04,"endTime":1031.022,"body":"everybody else substantially"},{"speaker":"Evan Morikawa","startTime":1027.04,"endTime":1031.022,"body":"more productive um,, over time."},{"speaker":"Evan Morikawa","startTime":1031.027,"endTime":1031.682,"body":"For sure."},{"speaker":"Ben Lloyd Pearson","startTime":1032.252,"endTime":1032.522,"body":"cool."},{"speaker":"Ben Lloyd Pearson","startTime":1032.527,"endTime":1035.252,"body":"So let's pivot a little bit"},{"speaker":"Ben Lloyd Pearson","startTime":1032.527,"endTime":1035.252,"body":"to talk about what I think"},{"speaker":"Ben Lloyd Pearson","startTime":1035.252,"endTime":1038.372,"body":"is gonna be the most unique"},{"speaker":"Ben Lloyd Pearson","startTime":1035.252,"endTime":1038.372,"body":"aspect of this discussion."},{"speaker":"Ben Lloyd Pearson","startTime":1038.372,"endTime":1040.502,"body":"So, you know, in recent"},{"speaker":"Ben Lloyd Pearson","startTime":1038.372,"endTime":1040.502,"body":"years, there's been something"},{"speaker":"Ben Lloyd Pearson","startTime":1040.502,"endTime":1042.062,"body":"that has brought gamers."},{"speaker":"Ben Lloyd Pearson","startTime":1042.122,"endTime":1045.512,"body":"Crypto enthusiasts and"},{"speaker":"Ben Lloyd Pearson","startTime":1042.122,"endTime":1045.512,"body":"LLM experts together."},{"speaker":"Ben Lloyd Pearson","startTime":1045.632,"endTime":1048.932,"body":"And that is the frustration"},{"speaker":"Ben Lloyd Pearson","startTime":1045.632,"endTime":1048.932,"body":"over this shortage of GPUs."},{"speaker":"Ben Lloyd Pearson","startTime":1049.082,"endTime":1049.292,"body":"Right?"},{"speaker":"Ben Lloyd Pearson","startTime":1049.682,"endTime":1053.282,"body":", I tried to upgrade my P PC"},{"speaker":"Ben Lloyd Pearson","startTime":1049.682,"endTime":1053.282,"body":"a couple years ago, . So,"},{"speaker":"Ben Lloyd Pearson","startTime":1053.332,"endTime":1055.492,"body":"you know, before we get into"},{"speaker":"Ben Lloyd Pearson","startTime":1053.332,"endTime":1055.492,"body":"that shortage, I, I think"},{"speaker":"Ben Lloyd Pearson","startTime":1055.492,"endTime":1057.832,"body":"it, it would be beneficial to"},{"speaker":"Ben Lloyd Pearson","startTime":1055.492,"endTime":1057.832,"body":"just step back for a moment."},{"speaker":"Ben Lloyd Pearson","startTime":1057.837,"endTime":1057.892,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":1057.892,"endTime":1061.042,"body":"So can you describe like"},{"speaker":"Ben Lloyd Pearson","startTime":1057.892,"endTime":1061.042,"body":"the technical role that GPUs"},{"speaker":"Ben Lloyd Pearson","startTime":1061.042,"endTime":1064.172,"body":"play, in the open AI tech"},{"speaker":"Ben Lloyd Pearson","startTime":1061.042,"endTime":1064.172,"body":"stack, uh, and you know,"},{"speaker":"Ben Lloyd Pearson","startTime":1064.172,"endTime":1066.782,"body":"why are they in so important"},{"speaker":"Ben Lloyd Pearson","startTime":1064.172,"endTime":1066.782,"body":"and how do you all use them?"},{"speaker":"Evan Morikawa","startTime":1066.782,"endTime":1067.682,"body":"Yeah, absolutely."},{"speaker":"Evan Morikawa","startTime":1068.077,"endTime":1071.234,"body":"So at the end of the day, when"},{"speaker":"Evan Morikawa","startTime":1068.077,"endTime":1071.234,"body":"you ask Chachi PT a question,"},{"speaker":"Evan Morikawa","startTime":1071.284,"endTime":1075.744,"body":"it's taking your text and it's"},{"speaker":"Evan Morikawa","startTime":1071.284,"endTime":1075.744,"body":"doing a huge amount of matrix"},{"speaker":"Evan Morikawa","startTime":1075.744,"endTime":1078.564,"body":"multiplication to kind of"},{"speaker":"Evan Morikawa","startTime":1075.744,"endTime":1078.564,"body":"predict the next word, right?"},{"speaker":"Evan Morikawa","startTime":1078.564,"endTime":1080.184,"body":"That's what all these"},{"speaker":"Evan Morikawa","startTime":1078.564,"endTime":1080.184,"body":"hundreds of billions of"},{"speaker":"Evan Morikawa","startTime":1080.184,"endTime":1081.324,"body":"model weights are for."},{"speaker":"Evan Morikawa","startTime":1081.864,"endTime":1085.724,"body":"And at the end of the day,"},{"speaker":"Evan Morikawa","startTime":1081.864,"endTime":1085.724,"body":"we're basically doing one math"},{"speaker":"Evan Morikawa","startTime":1085.724,"endTime":1089.154,"body":"operation, multiplying and"},{"speaker":"Evan Morikawa","startTime":1085.724,"endTime":1089.154,"body":"adding a lot, which, like we're"},{"speaker":"Evan Morikawa","startTime":1089.154,"endTime":1091.884,"body":"talking like quadrillions and"},{"speaker":"Evan Morikawa","startTime":1089.154,"endTime":1091.884,"body":"quadrillions of operations"},{"speaker":"Evan Morikawa","startTime":1091.889,"endTime":1093.047,"body":"a second here to do this."},{"speaker":"Evan Morikawa","startTime":1093.697,"endTime":1098.952,"body":"So, the ability for a GPU to"},{"speaker":"Evan Morikawa","startTime":1093.697,"endTime":1098.952,"body":"be able to GPUs are just many"},{"speaker":"Evan Morikawa","startTime":1098.952,"endTime":1101.032,"body":"orders of magnitude faster here."},{"speaker":"Evan Morikawa","startTime":1101.042,"endTime":1106.192,"body":"For a sense of scale, the latest"},{"speaker":"Evan Morikawa","startTime":1101.042,"endTime":1106.192,"body":"GPU that will be running this"},{"speaker":"Evan Morikawa","startTime":1106.222,"endTime":1111.042,"body":"NVIDIA H100 that everyone's been"},{"speaker":"Evan Morikawa","startTime":1106.222,"endTime":1111.042,"body":"trying to get, uh, that can do"},{"speaker":"Evan Morikawa","startTime":1111.062,"endTime":1114.942,"body":"about 2 quadrillion floating"},{"speaker":"Evan Morikawa","startTime":1111.062,"endTime":1114.942,"body":"point operations per second."},{"speaker":"Evan Morikawa","startTime":1115.552,"endTime":1119.272,"body":"your laptop CPU probably"},{"speaker":"Evan Morikawa","startTime":1115.552,"endTime":1119.272,"body":"can do on the order of a"},{"speaker":"Evan Morikawa","startTime":1119.272,"endTime":1120.482,"body":"couple hundred billion."},{"speaker":"Evan Morikawa","startTime":1120.897,"endTime":1122.687,"body":"So we're talking like"},{"speaker":"Evan Morikawa","startTime":1120.897,"endTime":1122.687,"body":"thousands of times"},{"speaker":"Evan Morikawa","startTime":1122.687,"endTime":1124.327,"body":"difference in speed here."},{"speaker":"Evan Morikawa","startTime":1124.777,"endTime":1129.257,"body":"So yes, you can run these"},{"speaker":"Evan Morikawa","startTime":1124.777,"endTime":1129.257,"body":"things on CPUs, but the"},{"speaker":"Evan Morikawa","startTime":1129.257,"endTime":1133.487,"body":"performance difference"},{"speaker":"Evan Morikawa","startTime":1129.257,"endTime":1133.487,"body":"we're talking about is 100x."},{"speaker":"Evan Morikawa","startTime":1133.947,"endTime":1137.437,"body":"So it's, especially for"},{"speaker":"Evan Morikawa","startTime":1133.947,"endTime":1137.437,"body":"models this size, it's like"},{"speaker":"Evan Morikawa","startTime":1137.437,"endTime":1138.947,"body":"really important to do that."},{"speaker":"Evan Morikawa","startTime":1139.297,"endTime":1141.747,"body":"The other thing that's"},{"speaker":"Evan Morikawa","startTime":1139.297,"endTime":1141.747,"body":"significant is that the"},{"speaker":"Evan Morikawa","startTime":1141.747,"endTime":1146.157,"body":"models are so large, they"},{"speaker":"Evan Morikawa","startTime":1141.747,"endTime":1146.157,"body":"don't fit on just one GPU."},{"speaker":"Evan Morikawa","startTime":1146.367,"endTime":1148.767,"body":"We need to put them on"},{"speaker":"Evan Morikawa","startTime":1146.367,"endTime":1148.767,"body":"multiple different GPUs."},{"speaker":"Evan Morikawa","startTime":1149.287,"endTime":1152.317,"body":"That's actually where things"},{"speaker":"Evan Morikawa","startTime":1149.287,"endTime":1152.317,"body":"jump dramatically in complexity."},{"speaker":"Evan Morikawa","startTime":1152.347,"endTime":1155.767,"body":"Much like the rest of this world"},{"speaker":"Evan Morikawa","startTime":1152.347,"endTime":1155.767,"body":"knows that yeah, your simple"},{"speaker":"Evan Morikawa","startTime":1155.767,"endTime":1159.087,"body":"single-threaded application"},{"speaker":"Evan Morikawa","startTime":1155.767,"endTime":1159.087,"body":"makes sense, but once you run"},{"speaker":"Evan Morikawa","startTime":1159.087,"endTime":1162.207,"body":"it massively concurrently on"},{"speaker":"Evan Morikawa","startTime":1159.087,"endTime":1162.207,"body":"a, on a globally distributed"},{"speaker":"Evan Morikawa","startTime":1162.207,"endTime":1164.157,"body":"system, that's where the"},{"speaker":"Evan Morikawa","startTime":1162.207,"endTime":1164.157,"body":"hard problems come from."},{"speaker":"Evan Morikawa","startTime":1164.662,"endTime":1169.262,"body":"Similarly here, once you run"},{"speaker":"Evan Morikawa","startTime":1164.662,"endTime":1169.262,"body":"these on multiple GPUs, things"},{"speaker":"Evan Morikawa","startTime":1169.262,"endTime":1170.152,"body":"get a lot more difficult."},{"speaker":"Evan Morikawa","startTime":1170.732,"endTime":1174.352,"body":"now you really care about how"},{"speaker":"Evan Morikawa","startTime":1170.732,"endTime":1174.352,"body":"fast your memory bandwidth is."},{"speaker":"Evan Morikawa","startTime":1174.392,"endTime":1176.812,"body":"You really care about how"},{"speaker":"Evan Morikawa","startTime":1174.392,"endTime":1176.812,"body":"fast your interconnect, your"},{"speaker":"Evan Morikawa","startTime":1176.812,"endTime":1180.532,"body":"like network bandwidth is"},{"speaker":"Evan Morikawa","startTime":1176.812,"endTime":1180.532,"body":"between GPUs, between boxes."},{"speaker":"Evan Morikawa","startTime":1181.032,"endTime":1183.232,"body":"and it's gotten to the"},{"speaker":"Evan Morikawa","startTime":1181.032,"endTime":1183.232,"body":"point now where every"},{"speaker":"Evan Morikawa","startTime":1183.232,"endTime":1184.462,"body":"single one of those..."},{"speaker":"Evan Morikawa","startTime":1184.682,"endTime":1189.222,"body":"These metrics can become a"},{"speaker":"Evan Morikawa","startTime":1184.682,"endTime":1189.222,"body":"bottleneck at various points"},{"speaker":"Evan Morikawa","startTime":1189.222,"endTime":1190.252,"body":"in the development cycle."},{"speaker":"Evan Morikawa","startTime":1190.262,"endTime":1192.132,"body":"So we really care about"},{"speaker":"Evan Morikawa","startTime":1190.262,"endTime":1192.132,"body":"all of them, and we"},{"speaker":"Evan Morikawa","startTime":1192.132,"endTime":1193.122,"body":"really maximize them."},{"speaker":"Evan Morikawa","startTime":1193.122,"endTime":1195.942,"body":"And anytime there's a new"},{"speaker":"Evan Morikawa","startTime":1193.122,"endTime":1195.942,"body":"or faster interconnect,"},{"speaker":"Evan Morikawa","startTime":1196.542,"endTime":1199.172,"body":"that usually almost"},{"speaker":"Evan Morikawa","startTime":1196.542,"endTime":1199.172,"body":"directly translates to"},{"speaker":"Evan Morikawa","startTime":1199.242,"endTime":1201.692,"body":"improved speed for chat GPT."},{"speaker":"Evan Morikawa","startTime":1202.022,"endTime":1205.446,"body":"That directly translates,"},{"speaker":"Evan Morikawa","startTime":1202.022,"endTime":1205.446,"body":"if you can make it run twice"},{"speaker":"Evan Morikawa","startTime":1205.446,"endTime":1208.816,"body":"as fast, that's twice as"},{"speaker":"Evan Morikawa","startTime":1205.446,"endTime":1208.816,"body":"many users as can access"},{"speaker":"Evan Morikawa","startTime":1208.816,"endTime":1210.326,"body":"it on the same hardware."},{"speaker":"Evan Morikawa","startTime":1210.606,"endTime":1213.166,"body":"Or that's twice as large"},{"speaker":"Evan Morikawa","startTime":1210.606,"endTime":1213.166,"body":"of a model as you can run."},{"speaker":"Evan Morikawa","startTime":1213.581,"endTime":1215.021,"body":"on the same hardware today."},{"speaker":"Evan Morikawa","startTime":1215.166,"endTime":1217.956,"body":"so these really make a, make"},{"speaker":"Evan Morikawa","startTime":1215.166,"endTime":1217.956,"body":"a huge difference in, in what"},{"speaker":"Evan Morikawa","startTime":1217.956,"endTime":1219.186,"body":"it's capable of going forward."},{"speaker":"Ben Lloyd Pearson","startTime":1219.651,"endTime":1221.311,"body":"Nice,"},{"speaker":"Ben Lloyd Pearson","startTime":1219.651,"endTime":1221.311,"body":"so I think that describes"},{"speaker":"Ben Lloyd Pearson","startTime":1221.311,"endTime":1224.651,"body":"pretty well the impact that,"},{"speaker":"Ben Lloyd Pearson","startTime":1221.311,"endTime":1224.651,"body":"you know, a shortage of GPUs"},{"speaker":"Ben Lloyd Pearson","startTime":1224.691,"endTime":1226.341,"body":"would have on your company."},{"speaker":"Ben Lloyd Pearson","startTime":1226.381,"endTime":1229.821,"body":"So, beyond a page that says,"},{"speaker":"Ben Lloyd Pearson","startTime":1226.381,"endTime":1229.821,"body":"hey, we're really busy right"},{"speaker":"Ben Lloyd Pearson","startTime":1229.821,"endTime":1233.171,"body":"now, come back later, what"},{"speaker":"Ben Lloyd Pearson","startTime":1229.821,"endTime":1233.171,"body":"other strategies did you all"},{"speaker":"Ben Lloyd Pearson","startTime":1233.211,"endTime":1237.881,"body":"take to adapt to the sudden"},{"speaker":"Ben Lloyd Pearson","startTime":1233.211,"endTime":1237.881,"body":"influx and the lack of this"},{"speaker":"Ben Lloyd Pearson","startTime":1237.881,"endTime":1239.171,"body":"hardware resource that you need."},{"speaker":"Evan Morikawa","startTime":1239.689,"endTime":1240.939,"body":"Have to do"},{"speaker":"Evan Morikawa","startTime":1239.689,"endTime":1240.939,"body":"a little bit of everything."},{"speaker":"Evan Morikawa","startTime":1246.064,"endTime":1249.124,"body":"We were working very closely"},{"speaker":"Evan Morikawa","startTime":1246.064,"endTime":1249.124,"body":"with Microsoft, who subsequently"},{"speaker":"Evan Morikawa","startTime":1249.134,"endTime":1251.994,"body":"was working closely with"},{"speaker":"Evan Morikawa","startTime":1249.134,"endTime":1251.994,"body":"NVIDIA on this to build that"},{"speaker":"Evan Morikawa","startTime":1251.994,"endTime":1255.334,"body":"capacity here, but also it"},{"speaker":"Evan Morikawa","startTime":1251.994,"endTime":1255.334,"body":"was about making the most of"},{"speaker":"Evan Morikawa","startTime":1255.334,"endTime":1256.824,"body":"the resources that we had."},{"speaker":"Evan Morikawa","startTime":1257.154,"endTime":1260.894,"body":"So optimizations were"},{"speaker":"Evan Morikawa","startTime":1257.154,"endTime":1260.894,"body":"hugely important here."},{"speaker":"Evan Morikawa","startTime":1261.094,"endTime":1264.181,"body":"And this is the long tail of"},{"speaker":"Evan Morikawa","startTime":1261.094,"endTime":1264.181,"body":"sort of your classic put it"},{"speaker":"Evan Morikawa","startTime":1264.181,"endTime":1267.838,"body":"through a  out the parts that"},{"speaker":"Evan Morikawa","startTime":1264.181,"endTime":1267.838,"body":"are slow, optimize those."},{"speaker":"Evan Morikawa","startTime":1268.343,"endTime":1271.553,"body":"but for us, those optimizations"},{"speaker":"Evan Morikawa","startTime":1268.343,"endTime":1271.553,"body":"are all across the stack."},{"speaker":"Evan Morikawa","startTime":1271.553,"endTime":1276.763,"body":"It's from low level cuda kernel"},{"speaker":"Evan Morikawa","startTime":1271.553,"endTime":1276.763,"body":"compiler optimizations to"},{"speaker":"Evan Morikawa","startTime":1276.763,"endTime":1279.193,"body":"sort of more business logic."},{"speaker":"Evan Morikawa","startTime":1279.193,"endTime":1281.053,"body":"How are we batching"},{"speaker":"Evan Morikawa","startTime":1279.193,"endTime":1281.053,"body":"requests together?"},{"speaker":"Evan Morikawa","startTime":1281.053,"endTime":1284.083,"body":"How are we maximally utilizing"},{"speaker":"Evan Morikawa","startTime":1281.053,"endTime":1284.083,"body":"these, uh, these things?"},{"speaker":"Evan Morikawa","startTime":1284.623,"endTime":1288.273,"body":"it took a lot of exploration,"},{"speaker":"Evan Morikawa","startTime":1284.623,"endTime":1288.273,"body":"like we discovered that we, we"},{"speaker":"Evan Morikawa","startTime":1288.273,"endTime":1292.168,"body":"started with a very basic GPU"},{"speaker":"Evan Morikawa","startTime":1288.273,"endTime":1292.168,"body":"utilization metric, you know?"},{"speaker":"Evan Morikawa","startTime":1293.138,"endTime":1296.558,"body":"From, from whatever the Nvidia"},{"speaker":"Evan Morikawa","startTime":1293.138,"endTime":1296.558,"body":"box bits out, we found that"},{"speaker":"Evan Morikawa","startTime":1296.558,"endTime":1300.668,"body":"it was actually misleading"},{"speaker":"Evan Morikawa","startTime":1296.558,"endTime":1300.668,"body":"because we were not like,"},{"speaker":"Evan Morikawa","startTime":1300.668,"endTime":1303.098,"body":"it could be doing more math"},{"speaker":"Evan Morikawa","startTime":1300.668,"endTime":1303.098,"body":"when the same time it was"},{"speaker":"Evan Morikawa","startTime":1303.098,"endTime":1305.738,"body":"on, or we were actually"},{"speaker":"Evan Morikawa","startTime":1303.098,"endTime":1305.738,"body":"running outta memory instead."},{"speaker":"Evan Morikawa","startTime":1305.743,"endTime":1309.388,"body":"So it took a lot of tweaking"},{"speaker":"Evan Morikawa","startTime":1305.743,"endTime":1309.388,"body":"to kind of find what the right"},{"speaker":"Evan Morikawa","startTime":1309.388,"endTime":1312.268,"body":"utilization metrics were and"},{"speaker":"Evan Morikawa","startTime":1309.388,"endTime":1312.268,"body":"then how to optimize those."},{"speaker":"Evan Morikawa","startTime":1312.268,"endTime":1314.278,"body":"And all of those were"},{"speaker":"Evan Morikawa","startTime":1312.268,"endTime":1314.278,"body":"really critical to"},{"speaker":"Evan Morikawa","startTime":1314.278,"endTime":1315.778,"body":"getting more out of it."},{"speaker":"Evan Morikawa","startTime":1316.048,"endTime":1318.898,"body":"But for us, everything"},{"speaker":"Evan Morikawa","startTime":1316.048,"endTime":1318.898,"body":"was framed in terms of."},{"speaker":"Evan Morikawa","startTime":1320.093,"endTime":1323.163,"body":"Every improvement that we"},{"speaker":"Evan Morikawa","startTime":1320.093,"endTime":1323.163,"body":"have represents more users we"},{"speaker":"Evan Morikawa","startTime":1323.163,"endTime":1324.483,"body":"could let onto the platform."},{"speaker":"Evan Morikawa","startTime":1324.963,"endTime":1327.673,"body":"As opposed to say like, oh it's"},{"speaker":"Evan Morikawa","startTime":1324.963,"endTime":1327.673,"body":"driving down our margins or"},{"speaker":"Evan Morikawa","startTime":1327.673,"endTime":1333.758,"body":"making it faster, we kept Cost"},{"speaker":"Evan Morikawa","startTime":1327.673,"endTime":1333.758,"body":"and latency as relatively fixed."},{"speaker":"Evan Morikawa","startTime":1333.758,"endTime":1335.888,"body":"And the thing that could get"},{"speaker":"Evan Morikawa","startTime":1333.758,"endTime":1335.888,"body":"to move was we got to get"},{"speaker":"Evan Morikawa","startTime":1335.888,"endTime":1337.208,"body":"more users onto the system."},{"speaker":"Ben Lloyd Pearson","startTime":1337.778,"endTime":1338.798,"body":"Gotcha, gotcha."},{"speaker":"Ben Lloyd Pearson","startTime":1338.798,"endTime":1342.211,"body":"And you know, I've always"},{"speaker":"Ben Lloyd Pearson","startTime":1338.798,"endTime":1342.211,"body":"kind of felt that that, GPUs"},{"speaker":"Ben Lloyd Pearson","startTime":1342.216,"endTime":1345.031,"body":"were chosen for LLMs, mostly"},{"speaker":"Ben Lloyd Pearson","startTime":1342.216,"endTime":1345.031,"body":"outta convenience because"},{"speaker":"Ben Lloyd Pearson","startTime":1345.031,"endTime":1347.191,"body":"it was the hardware that was"},{"speaker":"Ben Lloyd Pearson","startTime":1345.031,"endTime":1347.191,"body":"available at the time that"},{"speaker":"Ben Lloyd Pearson","startTime":1347.196,"endTime":1350.086,"body":"was most closely adapted to"},{"speaker":"Ben Lloyd Pearson","startTime":1347.196,"endTime":1350.086,"body":"the, the needs of that Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":1350.086,"endTime":1350.366,"body":"Community."},{"speaker":"Ben Lloyd Pearson","startTime":1350.946,"endTime":1354.366,"body":"So are you looking at other"},{"speaker":"Ben Lloyd Pearson","startTime":1350.946,"endTime":1354.366,"body":"hardware options out there?"},{"speaker":"Ben Lloyd Pearson","startTime":1354.366,"endTime":1356.246,"body":"Like are there things"},{"speaker":"Ben Lloyd Pearson","startTime":1354.366,"endTime":1356.246,"body":"that are coming up in the"},{"speaker":"Ben Lloyd Pearson","startTime":1356.246,"endTime":1359.276,"body":"market that you think have"},{"speaker":"Ben Lloyd Pearson","startTime":1356.246,"endTime":1359.276,"body":"potential to replace GPUs,"},{"speaker":"Ben Lloyd Pearson","startTime":1359.286,"endTime":1361.016,"body":"specifically for generative AI?"},{"speaker":"Evan Morikawa","startTime":1361.016,"endTime":1363.406,"body":"Yeah, well,"},{"speaker":"Evan Morikawa","startTime":1361.016,"endTime":1363.406,"body":"so even though we all call"},{"speaker":"Evan Morikawa","startTime":1363.406,"endTime":1368.056,"body":"them GPUs, like these are not"},{"speaker":"Evan Morikawa","startTime":1363.406,"endTime":1368.056,"body":"the graphics processing units"},{"speaker":"Evan Morikawa","startTime":1368.126,"endTime":1370.136,"body":"of your desktop PCs anymore."},{"speaker":"Evan Morikawa","startTime":1378.821,"endTime":1383.961,"body":"Uh, machines, and especially"},{"speaker":"Evan Morikawa","startTime":1378.821,"endTime":1383.961,"body":"with Google calling theirs TPUs,"},{"speaker":"Evan Morikawa","startTime":1384.011,"endTime":1387.231,"body":"the kind of, this is why it's"},{"speaker":"Evan Morikawa","startTime":1384.011,"endTime":1387.231,"body":"sort of like AI accelerators,"},{"speaker":"Evan Morikawa","startTime":1387.231,"endTime":1390.051,"body":"more the generic trend now,"},{"speaker":"Evan Morikawa","startTime":1387.231,"endTime":1390.051,"body":"but I still call them GPUs."},{"speaker":"Evan Morikawa","startTime":1391.101,"endTime":1394.941,"body":"I mean at this point they"},{"speaker":"Evan Morikawa","startTime":1391.101,"endTime":1394.941,"body":"are hyper specialized to"},{"speaker":"Evan Morikawa","startTime":1394.941,"endTime":1399.041,"body":"do this exact one type"},{"speaker":"Evan Morikawa","startTime":1394.941,"endTime":1399.041,"body":"of matrix operation."},{"speaker":"Evan Morikawa","startTime":1399.814,"endTime":1402.344,"body":"Um, another, another concrete"},{"speaker":"Evan Morikawa","startTime":1399.814,"endTime":1402.344,"body":"example here that they've been"},{"speaker":"Evan Morikawa","startTime":1402.344,"endTime":1406.914,"body":"optimized only for AI things"},{"speaker":"Evan Morikawa","startTime":1402.344,"endTime":1406.914,"body":"is doing lower precision math."},{"speaker":"Evan Morikawa","startTime":1407.464,"endTime":1409.964,"body":"Most people, when they have"},{"speaker":"Evan Morikawa","startTime":1407.464,"endTime":1409.964,"body":"a floating point number, you"},{"speaker":"Evan Morikawa","startTime":1409.964,"endTime":1412.254,"body":"get 32 bits to preserve it."},{"speaker":"Evan Morikawa","startTime":1412.364,"endTime":1416.324,"body":"We do math with, you can do math"},{"speaker":"Evan Morikawa","startTime":1412.364,"endTime":1416.324,"body":"with 16 bits, so with 8 bits."},{"speaker":"Evan Morikawa","startTime":1416.774,"endTime":1419.004,"body":"which means you can just like,"},{"speaker":"Evan Morikawa","startTime":1416.774,"endTime":1419.004,"body":"do more at the same time."},{"speaker":"Evan Morikawa","startTime":1419.004,"endTime":1421.094,"body":"And now they have"},{"speaker":"Evan Morikawa","startTime":1419.004,"endTime":1421.094,"body":"dedicated circuits to"},{"speaker":"Evan Morikawa","startTime":1421.094,"endTime":1424.424,"body":"do that on the upcoming"},{"speaker":"Evan Morikawa","startTime":1421.094,"endTime":1424.424,"body":"GPUs that are coming out."},{"speaker":"Evan Morikawa","startTime":1424.454,"endTime":1426.154,"body":"So in a lot of ways they"},{"speaker":"Evan Morikawa","startTime":1424.454,"endTime":1426.154,"body":"are super specialized."},{"speaker":"Evan Morikawa","startTime":1426.464,"endTime":1428.594,"body":"the other thing though"},{"speaker":"Evan Morikawa","startTime":1426.464,"endTime":1428.594,"body":"is the software stack."},{"speaker":"Evan Morikawa","startTime":1430.249,"endTime":1434.836,"body":"NVIDIA has, their CUDA"},{"speaker":"Evan Morikawa","startTime":1430.249,"endTime":1434.836,"body":"stack, their software stack,"},{"speaker":"Evan Morikawa","startTime":1434.846,"endTime":1437.286,"body":"the kind of compiler layer"},{"speaker":"Evan Morikawa","startTime":1434.846,"endTime":1437.286,"body":"on top of there has been"},{"speaker":"Evan Morikawa","startTime":1437.466,"endTime":1439.536,"body":"hugely specialized to that."},{"speaker":"Evan Morikawa","startTime":1439.776,"endTime":1442.376,"body":"It's currently very"},{"speaker":"Evan Morikawa","startTime":1439.776,"endTime":1442.376,"body":"difficult for people to use"},{"speaker":"Evan Morikawa","startTime":1442.426,"endTime":1445.836,"body":"AMD or Intel or the other"},{"speaker":"Evan Morikawa","startTime":1442.426,"endTime":1445.836,"body":"manufacturers out there."},{"speaker":"Evan Morikawa","startTime":1446.076,"endTime":1448.966,"body":"Actually, there is a product,"},{"speaker":"Evan Morikawa","startTime":1446.076,"endTime":1448.966,"body":"OpenAI Triton, which is"},{"speaker":"Evan Morikawa","startTime":1448.976,"endTime":1451.766,"body":"explicitly designed to try"},{"speaker":"Evan Morikawa","startTime":1448.976,"endTime":1451.766,"body":"and better abstract that."},{"speaker":"Evan Morikawa","startTime":1452.311,"endTime":1455.221,"body":"and that's potentially a"},{"speaker":"Evan Morikawa","startTime":1452.311,"endTime":1455.221,"body":"huge deal because the ability"},{"speaker":"Evan Morikawa","startTime":1455.571,"endTime":1459.321,"body":"to use other hardware much"},{"speaker":"Evan Morikawa","startTime":1455.571,"endTime":1459.321,"body":"more easily is definitely"},{"speaker":"Evan Morikawa","startTime":1459.321,"endTime":1462.551,"body":"a big thing, uh, will be a"},{"speaker":"Evan Morikawa","startTime":1459.321,"endTime":1462.551,"body":"big thing for this market."},{"speaker":"Evan Morikawa","startTime":1462.581,"endTime":1467.181,"body":"But right now, yeah, Nvidia"},{"speaker":"Evan Morikawa","startTime":1462.581,"endTime":1467.181,"body":"has an incredible hold on this."},{"speaker":"Evan Morikawa","startTime":1467.211,"endTime":1469.971,"body":"It's reflected in their"},{"speaker":"Evan Morikawa","startTime":1467.211,"endTime":1469.971,"body":"share price right now."},{"speaker":"Evan Morikawa","startTime":1470.421,"endTime":1470.511,"body":"Yeah."},{"speaker":"Evan Morikawa","startTime":1470.571,"endTime":1473.061,"body":"Um, a lot of it is because"},{"speaker":"Evan Morikawa","startTime":1470.571,"endTime":1473.061,"body":"they own the hardware,"},{"speaker":"Evan Morikawa","startTime":1473.091,"endTime":1475.701,"body":"the software stack, and a"},{"speaker":"Evan Morikawa","startTime":1473.091,"endTime":1475.701,"body":"lot of the interconnects."},{"speaker":"Evan Morikawa","startTime":1475.751,"endTime":1478.591,"body":"for example, we, we"},{"speaker":"Evan Morikawa","startTime":1475.751,"endTime":1478.591,"body":"use InfiniBand, which"},{"speaker":"Evan Morikawa","startTime":1478.591,"endTime":1479.941,"body":"is a like ultra high."},{"speaker":"Evan Morikawa","startTime":1480.681,"endTime":1481.971,"body":"Bandwidth interconnect."},{"speaker":"Evan Morikawa","startTime":1482.361,"endTime":1485.871,"body":"That company that developed at"},{"speaker":"Evan Morikawa","startTime":1482.361,"endTime":1485.871,"body":"Melanox is also owned by Nvidia."},{"speaker":"Evan Morikawa","startTime":1485.871,"endTime":1489.321,"body":"So they like really have the"},{"speaker":"Evan Morikawa","startTime":1485.871,"endTime":1489.321,"body":"stack top to bottom here."},{"speaker":"Ben Lloyd Pearson","startTime":1489.501,"endTime":1491.031,"body":"Yeah, they,"},{"speaker":"Ben Lloyd Pearson","startTime":1489.501,"endTime":1491.031,"body":"they definitely got in early."},{"speaker":"Ben Lloyd Pearson","startTime":1491.061,"endTime":1493.851,"body":"'cause I, I remember playing"},{"speaker":"Ben Lloyd Pearson","startTime":1491.061,"endTime":1493.851,"body":"around with LLM tools, uh,"},{"speaker":"Ben Lloyd Pearson","startTime":1493.851,"endTime":1497.901,"body":"years ago and like Nvidia was"},{"speaker":"Ben Lloyd Pearson","startTime":1493.851,"endTime":1497.901,"body":"the only option in the market."},{"speaker":"Ben Lloyd Pearson","startTime":1497.901,"endTime":1500.091,"body":"There really wasn't anything"},{"speaker":"Ben Lloyd Pearson","startTime":1497.901,"endTime":1500.091,"body":"else to, to look at."},{"speaker":"Ben Lloyd Pearson","startTime":1500.091,"endTime":1500.092,"body":"Yeah."},{"speaker":"Ben Lloyd Pearson","startTime":1500.097,"endTime":1502.131,"body":"And I remember Cuda in"},{"speaker":"Ben Lloyd Pearson","startTime":1500.097,"endTime":1502.131,"body":"particular, that was around"},{"speaker":"Ben Lloyd Pearson","startTime":1502.131,"endTime":1504.381,"body":"the time that it was really"},{"speaker":"Ben Lloyd Pearson","startTime":1502.131,"endTime":1504.381,"body":"taken off and, yep, yep."},{"speaker":"Evan Morikawa","startTime":1504.441,"endTime":1504.771,"body":"Yeah."},{"speaker":"Evan Morikawa","startTime":1504.981,"endTime":1506.126,"body":"I actually should note"},{"speaker":"Evan Morikawa","startTime":1504.981,"endTime":1506.126,"body":"though, these are."},{"speaker":"Evan Morikawa","startTime":1506.981,"endTime":1509.791,"body":"It's been very, very"},{"speaker":"Evan Morikawa","startTime":1506.981,"endTime":1509.791,"body":"difficult for the chip"},{"speaker":"Evan Morikawa","startTime":1509.791,"endTime":1513.111,"body":"manufacturers to even, like,"},{"speaker":"Evan Morikawa","startTime":1509.791,"endTime":1513.111,"body":"get the right chips, though."},{"speaker":"Evan Morikawa","startTime":1513.481,"endTime":1516.161,"body":"another example, despite"},{"speaker":"Evan Morikawa","startTime":1513.481,"endTime":1516.161,"body":"NVIDIA's dominance here, their"},{"speaker":"Evan Morikawa","startTime":1516.191,"endTime":1520.951,"body":"upcoming chip, this H100, is"},{"speaker":"Evan Morikawa","startTime":1516.191,"endTime":1520.951,"body":"kind of widely known, well,"},{"speaker":"Evan Morikawa","startTime":1520.971,"endTime":1525.211,"body":"within this specific subset of"},{"speaker":"Evan Morikawa","startTime":1520.971,"endTime":1525.211,"body":"the industry here, that it has,"},{"speaker":"Evan Morikawa","startTime":1526.041,"endTime":1530.551,"body":"it doesn't have enough memory"},{"speaker":"Evan Morikawa","startTime":1526.041,"endTime":1530.551,"body":"bandwidth relative to how much"},{"speaker":"Evan Morikawa","startTime":1530.761,"endTime":1532.251,"body":"compute they added into it."},{"speaker":"Evan Morikawa","startTime":1532.706,"endTime":1535.206,"body":"So it's very, it's"},{"speaker":"Evan Morikawa","startTime":1532.706,"endTime":1535.206,"body":"getting increasingly"},{"speaker":"Evan Morikawa","startTime":1535.206,"endTime":1536.666,"body":"difficult to utilize this."},{"speaker":"Evan Morikawa","startTime":1537.166,"endTime":1538.696,"body":"But the reason that happened"},{"speaker":"Evan Morikawa","startTime":1537.166,"endTime":1538.696,"body":"is because they didn't"},{"speaker":"Evan Morikawa","startTime":1538.706,"endTime":1541.676,"body":"know about how large the"},{"speaker":"Evan Morikawa","startTime":1538.706,"endTime":1541.676,"body":"models were going to get."},{"speaker":"Evan Morikawa","startTime":1541.706,"endTime":1544.476,"body":"They didn't know, it was"},{"speaker":"Evan Morikawa","startTime":1541.706,"endTime":1544.476,"body":"very difficult to predict"},{"speaker":"Evan Morikawa","startTime":1544.476,"endTime":1547.186,"body":"this on the scale of like"},{"speaker":"Evan Morikawa","startTime":1544.476,"endTime":1547.186,"body":"chip development cycles."},{"speaker":"Evan Morikawa","startTime":1547.206,"endTime":1549.726,"body":"So, right, ChatGPT is"},{"speaker":"Evan Morikawa","startTime":1547.206,"endTime":1549.726,"body":"less than a year old."},{"speaker":"Evan Morikawa","startTime":1550.156,"endTime":1552.796,"body":"And one year in"},{"speaker":"Evan Morikawa","startTime":1550.156,"endTime":1552.796,"body":"semiconductor manufacturing"},{"speaker":"Evan Morikawa","startTime":1552.796,"endTime":1554.746,"body":"design is nothing, so."},{"speaker":"Evan Morikawa","startTime":1555.256,"endTime":1556.996,"body":"Yeah, it's very"},{"speaker":"Evan Morikawa","startTime":1555.256,"endTime":1556.996,"body":"difficult for anybody to"},{"speaker":"Evan Morikawa","startTime":1556.996,"endTime":1558.436,"body":"predict, uh, to do this."},{"speaker":"Ben Lloyd Pearson","startTime":1558.623,"endTime":1560.303,"body":"so I wanna,"},{"speaker":"Ben Lloyd Pearson","startTime":1558.623,"endTime":1560.303,"body":"I wanna transition a little"},{"speaker":"Ben Lloyd Pearson","startTime":1560.303,"endTime":1563.843,"body":"bit into talking about how"},{"speaker":"Ben Lloyd Pearson","startTime":1560.303,"endTime":1563.843,"body":"you approach scaling, uh, the"},{"speaker":"Ben Lloyd Pearson","startTime":1563.843,"endTime":1566.483,"body":"engineering function at Open"},{"speaker":"Ben Lloyd Pearson","startTime":1563.843,"endTime":1566.483,"body":"the Eye as this was going on."},{"speaker":"Ben Lloyd Pearson","startTime":1566.803,"endTime":1570.193,"body":"So, you know, the rapid success,"},{"speaker":"Ben Lloyd Pearson","startTime":1566.803,"endTime":1570.193,"body":"no secret at this point."},{"speaker":"Ben Lloyd Pearson","startTime":1570.283,"endTime":1573.043,"body":"your, your leadership has"},{"speaker":"Ben Lloyd Pearson","startTime":1570.283,"endTime":1573.043,"body":"been very open about sharing"},{"speaker":"Ben Lloyd Pearson","startTime":1573.043,"endTime":1575.953,"body":"the challenges of this,"},{"speaker":"Ben Lloyd Pearson","startTime":1573.043,"endTime":1575.953,"body":"this rapid sudden virality."},{"speaker":"Ben Lloyd Pearson","startTime":1576.613,"endTime":1578.923,"body":"And, you know, it's one thing"},{"speaker":"Ben Lloyd Pearson","startTime":1576.613,"endTime":1578.923,"body":"to deal with sustained growth"},{"speaker":"Ben Lloyd Pearson","startTime":1578.923,"endTime":1581.983,"body":"over a long period, but when"},{"speaker":"Ben Lloyd Pearson","startTime":1578.923,"endTime":1581.983,"body":"you, when you deal with, like,"},{"speaker":"Ben Lloyd Pearson","startTime":1582.023,"endTime":1585.063,"body":"this sudden surge, it's, it's"},{"speaker":"Ben Lloyd Pearson","startTime":1582.023,"endTime":1585.063,"body":"an entirely different beast."},{"speaker":"Ben Lloyd Pearson","startTime":1585.063,"endTime":1587.523,"body":"Cause, I mean, not only do you"},{"speaker":"Ben Lloyd Pearson","startTime":1585.063,"endTime":1587.523,"body":"have to deal with potentially"},{"speaker":"Ben Lloyd Pearson","startTime":1587.533,"endTime":1590.153,"body":"much higher peaks, but you"},{"speaker":"Ben Lloyd Pearson","startTime":1587.533,"endTime":1590.153,"body":"don't know how much of that"},{"speaker":"Ben Lloyd Pearson","startTime":1590.153,"endTime":1592.393,"body":"is going to stick around"},{"speaker":"Ben Lloyd Pearson","startTime":1590.153,"endTime":1592.393,"body":"for the long term, right?"},{"speaker":"Ben Lloyd Pearson","startTime":1593.053,"endTime":1596.583,"body":"So, you know, walk me through"},{"speaker":"Ben Lloyd Pearson","startTime":1593.053,"endTime":1596.583,"body":"how that played out for your"},{"speaker":"Ben Lloyd Pearson","startTime":1596.583,"endTime":1598.093,"body":"engineering organization."},{"speaker":"Evan Morikawa","startTime":1598.293,"endTime":1598.753,"body":"Yeah."},{"speaker":"Evan Morikawa","startTime":1598.853,"endTime":1603.629,"body":"so for instance, staying nimble"},{"speaker":"Evan Morikawa","startTime":1598.853,"endTime":1603.629,"body":"has been a huge piece of this."},{"speaker":"Evan Morikawa","startTime":1603.639,"endTime":1604.519,"body":"One thing that is like."},{"speaker":"Evan Morikawa","startTime":1604.889,"endTime":1608.249,"body":"very much helped in by design"},{"speaker":"Evan Morikawa","startTime":1604.889,"endTime":1608.249,"body":"is doing everything we can to"},{"speaker":"Evan Morikawa","startTime":1608.249,"endTime":1611.869,"body":"try and treat everything like"},{"speaker":"Evan Morikawa","startTime":1608.249,"endTime":1611.869,"body":"a tiny early stage startup."},{"speaker":"Evan Morikawa","startTime":1612.369,"endTime":1614.759,"body":"this originally was"},{"speaker":"Evan Morikawa","startTime":1612.369,"endTime":1614.759,"body":"true in terms of raw,"},{"speaker":"Evan Morikawa","startTime":1615.009,"endTime":1616.039,"body":"like, head count here."},{"speaker":"Evan Morikawa","startTime":1616.229,"endTime":1618.969,"body":"Yes, there are a huge number"},{"speaker":"Evan Morikawa","startTime":1616.229,"endTime":1618.969,"body":"of people that contributed"},{"speaker":"Evan Morikawa","startTime":1618.969,"endTime":1621.679,"body":"to the research and the model"},{"speaker":"Evan Morikawa","startTime":1618.969,"endTime":1621.679,"body":"training, but at the end of"},{"speaker":"Evan Morikawa","startTime":1621.679,"endTime":1624.789,"body":"the day, the kind of product"},{"speaker":"Evan Morikawa","startTime":1621.679,"endTime":1624.789,"body":"engineering, design, like,"},{"speaker":"Evan Morikawa","startTime":1624.809,"endTime":1627.489,"body":"parts that is applied was"},{"speaker":"Evan Morikawa","startTime":1624.809,"endTime":1627.489,"body":"only several dozen people"},{"speaker":"Evan Morikawa","startTime":1627.499,"endTime":1629.339,"body":"when the chat GPT launched."},{"speaker":"Evan Morikawa","startTime":1629.349,"endTime":1631.649,"body":"So it was still like"},{"speaker":"Evan Morikawa","startTime":1629.349,"endTime":1631.649,"body":"a much smaller group."},{"speaker":"Evan Morikawa","startTime":1632.259,"endTime":1637.839,"body":"Even still, we intentionally set"},{"speaker":"Evan Morikawa","startTime":1632.259,"endTime":1637.839,"body":"up ChatGPT as a more vertically"},{"speaker":"Evan Morikawa","startTime":1637.839,"endTime":1642.579,"body":"integrated sort of separate"},{"speaker":"Evan Morikawa","startTime":1637.839,"endTime":1642.579,"body":"product team within Applied."},{"speaker":"Evan Morikawa","startTime":1643.319,"endTime":1645.879,"body":"If you think of Applied"},{"speaker":"Evan Morikawa","startTime":1643.319,"endTime":1645.879,"body":"and the API as this three"},{"speaker":"Evan Morikawa","startTime":1645.879,"endTime":1649.519,"body":"year old startup, ChatGPT"},{"speaker":"Evan Morikawa","startTime":1645.879,"endTime":1649.519,"body":"looks, feels and acts like"},{"speaker":"Evan Morikawa","startTime":1649.519,"endTime":1651.099,"body":"a ten month old startup."},{"speaker":"Evan Morikawa","startTime":1651.446,"endTime":1655.646,"body":"And concretely that was in the"},{"speaker":"Evan Morikawa","startTime":1651.446,"endTime":1655.646,"body":"form of we intentionally started"},{"speaker":"Evan Morikawa","startTime":1655.646,"endTime":1658.946,"body":"on a separate repo, separate"},{"speaker":"Evan Morikawa","startTime":1655.646,"endTime":1658.946,"body":"clusters, different controls,"},{"speaker":"Evan Morikawa","startTime":1659.046,"endTime":1660.566,"body":"taking on a little bit of that."},{"speaker":"Evan Morikawa","startTime":1661.101,"endTime":1663.131,"body":"kind of tech debt and"},{"speaker":"Evan Morikawa","startTime":1661.101,"endTime":1663.131,"body":"duplication at the start to"},{"speaker":"Evan Morikawa","startTime":1663.131,"endTime":1666.901,"body":"like really like optimize"},{"speaker":"Evan Morikawa","startTime":1663.131,"endTime":1666.901,"body":"for iteration there."},{"speaker":"Evan Morikawa","startTime":1667.161,"endTime":1669.911,"body":"Whereas gradually the API"},{"speaker":"Evan Morikawa","startTime":1667.161,"endTime":1669.911,"body":"started to optimize a little"},{"speaker":"Evan Morikawa","startTime":1669.911,"endTime":1673.391,"body":"bit more for like stability and"},{"speaker":"Evan Morikawa","startTime":1669.911,"endTime":1673.391,"body":"SLAs and stuff like that too."},{"speaker":"Evan Morikawa","startTime":1673.841,"endTime":1677.401,"body":"Now this is of course changing,"},{"speaker":"Evan Morikawa","startTime":1673.841,"endTime":1677.401,"body":"like ChatGPT also has like huge"},{"speaker":"Evan Morikawa","startTime":1677.401,"endTime":1679.101,"body":"stability and SLA concerns."},{"speaker":"Evan Morikawa","startTime":1679.101,"endTime":1682.311,"body":"We are kind of working"},{"speaker":"Evan Morikawa","startTime":1679.101,"endTime":1682.311,"body":"to build out more broad"},{"speaker":"Evan Morikawa","startTime":1682.311,"endTime":1684.501,"body":"platform teams as well."},{"speaker":"Evan Morikawa","startTime":1689.586,"endTime":1691.076,"body":"Fast iterating was important."},{"speaker":"Evan Morikawa","startTime":1691.356,"endTime":1692.936,"body":"The other kind of key"},{"speaker":"Evan Morikawa","startTime":1691.356,"endTime":1692.936,"body":"piece about this was"},{"speaker":"Evan Morikawa","startTime":1692.936,"endTime":1696.096,"body":"having the research teams"},{"speaker":"Evan Morikawa","startTime":1692.936,"endTime":1696.096,"body":"deeply embedded here."},{"speaker":"Evan Morikawa","startTime":1696.466,"endTime":1699.196,"body":"So, while I talk about Applied,"},{"speaker":"Evan Morikawa","startTime":1696.466,"endTime":1699.196,"body":"because that's the group"},{"speaker":"Evan Morikawa","startTime":1699.196,"endTime":1703.366,"body":"I'm in, in reality, ChatGPT"},{"speaker":"Evan Morikawa","startTime":1699.196,"endTime":1703.366,"body":"effort heavily involved a"},{"speaker":"Evan Morikawa","startTime":1703.366,"endTime":1706.556,"body":"huge chunk of researchers"},{"speaker":"Evan Morikawa","startTime":1703.366,"endTime":1706.556,"body":"from various research teams."},{"speaker":"Evan Morikawa","startTime":1706.976,"endTime":1709.516,"body":"They were the ones who were"},{"speaker":"Evan Morikawa","startTime":1706.976,"endTime":1709.516,"body":"actually constantly tweaking"},{"speaker":"Evan Morikawa","startTime":1709.516,"endTime":1711.086,"body":"and updating and fine tuning."},{"speaker":"Evan Morikawa","startTime":1711.431,"endTime":1715.361,"body":"The models based on end user"},{"speaker":"Evan Morikawa","startTime":1711.431,"endTime":1715.361,"body":"feedback as well here too."},{"speaker":"Evan Morikawa","startTime":1715.531,"endTime":1719.351,"body":"So keeping these as very"},{"speaker":"Evan Morikawa","startTime":1715.531,"endTime":1719.351,"body":"vertically integrated, like"},{"speaker":"Evan Morikawa","startTime":1719.511,"endTime":1722.671,"body":"teams with both product"},{"speaker":"Evan Morikawa","startTime":1719.511,"endTime":1722.671,"body":"engineering design and research"},{"speaker":"Evan Morikawa","startTime":1722.681,"endTime":1724.241,"body":"was also super important."},{"speaker":"Ben Lloyd Pearson","startTime":1724.631,"endTime":1726.101,"body":"Yeah, so"},{"speaker":"Ben Lloyd Pearson","startTime":1724.631,"endTime":1726.101,"body":"what would you say is like"},{"speaker":"Ben Lloyd Pearson","startTime":1726.101,"endTime":1728.961,"body":"the biggest improvement that"},{"speaker":"Ben Lloyd Pearson","startTime":1726.101,"endTime":1728.961,"body":"has come out of this from"},{"speaker":"Ben Lloyd Pearson","startTime":1729.131,"endTime":1731.441,"body":"an engineering perspective,"},{"speaker":"Ben Lloyd Pearson","startTime":1729.131,"endTime":1731.441,"body":"from you know, just dealing"},{"speaker":"Ben Lloyd Pearson","startTime":1731.441,"endTime":1732.941,"body":"with all of this scale?"},{"speaker":"Evan Morikawa","startTime":1733.581,"endTime":1735.001,"body":"the biggest"},{"speaker":"Evan Morikawa","startTime":1733.581,"endTime":1735.001,"body":"improvement that's come"},{"speaker":"Evan Morikawa","startTime":1735.001,"endTime":1738.208,"body":"out, actually our ability"},{"speaker":"Evan Morikawa","startTime":1735.001,"endTime":1738.208,"body":"to work together as like"},{"speaker":"Evan Morikawa","startTime":1738.208,"endTime":1739.317,"body":"a single research team."},{"speaker":"Evan Morikawa","startTime":1739.318,"endTime":1745.558,"body":"Product group, there was an"},{"speaker":"Evan Morikawa","startTime":1739.318,"endTime":1745.558,"body":"early fear that it would be"},{"speaker":"Evan Morikawa","startTime":1745.588,"endTime":1748.188,"body":"like the worst case scenario"},{"speaker":"Evan Morikawa","startTime":1745.588,"endTime":1748.188,"body":"for us would be the type of"},{"speaker":"Evan Morikawa","startTime":1748.188,"endTime":1751.378,"body":"place where research trains"},{"speaker":"Evan Morikawa","startTime":1748.188,"endTime":1751.378,"body":"a model, throws it over"},{"speaker":"Evan Morikawa","startTime":1751.378,"endTime":1753.568,"body":"the wall, go productize it."},{"speaker":"Evan Morikawa","startTime":1753.858,"endTime":1755.748,"body":"And it was like"},{"speaker":"Evan Morikawa","startTime":1753.858,"endTime":1755.748,"body":"this one way street."},{"speaker":"Evan Morikawa","startTime":1756.088,"endTime":1759.358,"body":"And we spent a lot of effort"},{"speaker":"Evan Morikawa","startTime":1756.088,"endTime":1759.358,"body":"making sure that that was not"},{"speaker":"Evan Morikawa","startTime":1759.558,"endTime":1761.458,"body":"how we developed anything."},{"speaker":"Evan Morikawa","startTime":1761.818,"endTime":1764.458,"body":"But, you know, that was"},{"speaker":"Evan Morikawa","startTime":1761.818,"endTime":1764.458,"body":"all like abstract until"},{"speaker":"Evan Morikawa","startTime":1764.468,"endTime":1766.818,"body":"you like are actually in"},{"speaker":"Evan Morikawa","startTime":1764.468,"endTime":1766.818,"body":"the trenches developing."},{"speaker":"Evan Morikawa","startTime":1767.153,"endTime":1769.103,"body":"And like really working"},{"speaker":"Evan Morikawa","startTime":1767.153,"endTime":1769.103,"body":"on a product here."},{"speaker":"Evan Morikawa","startTime":1769.233,"endTime":1772.253,"body":"and of course the reality"},{"speaker":"Evan Morikawa","startTime":1769.233,"endTime":1772.253,"body":"is about like, just like it,"},{"speaker":"Evan Morikawa","startTime":1772.253,"endTime":1773.543,"body":"it's very messy to begin with."},{"speaker":"Evan Morikawa","startTime":1773.543,"endTime":1775.673,"body":"You just kind of like have"},{"speaker":"Evan Morikawa","startTime":1773.543,"endTime":1775.673,"body":"to tweak it as it goes along."},{"speaker":"Evan Morikawa","startTime":1775.673,"endTime":1778.823,"body":"But now that there's a much"},{"speaker":"Evan Morikawa","startTime":1775.673,"endTime":1778.823,"body":"stronger focus around sort of"},{"speaker":"Evan Morikawa","startTime":1778.823,"endTime":1781.993,"body":"these clear products we have"},{"speaker":"Evan Morikawa","startTime":1778.823,"endTime":1781.993,"body":"with this clear API product"},{"speaker":"Evan Morikawa","startTime":1781.998,"endTime":1784.333,"body":"that we need to build, we"},{"speaker":"Evan Morikawa","startTime":1781.998,"endTime":1784.333,"body":"have this clear consumer"},{"speaker":"Evan Morikawa","startTime":1784.743,"endTime":1786.093,"body":"app that we're focusing for."},{"speaker":"Evan Morikawa","startTime":1786.093,"endTime":1788.073,"body":"I think like that"},{"speaker":"Evan Morikawa","startTime":1786.093,"endTime":1788.073,"body":"has helped a lot."},{"speaker":"Evan Morikawa","startTime":1788.628,"endTime":1792.718,"body":"Really integrate the research"},{"speaker":"Evan Morikawa","startTime":1788.628,"endTime":1792.718,"body":"and product and engineering and"},{"speaker":"Evan Morikawa","startTime":1792.718,"endTime":1794.276,"body":"design and kind of this one uh,"},{"speaker":"Ben Lloyd Pearson","startTime":1794.456,"endTime":1794.966,"body":"push."},{"speaker":"Ben Lloyd Pearson","startTime":1795.296,"endTime":1796.496,"body":"And, and I imagine that"},{"speaker":"Ben Lloyd Pearson","startTime":1795.296,"endTime":1796.496,"body":"probably gives your"},{"speaker":"Ben Lloyd Pearson","startTime":1796.496,"endTime":1798.746,"body":"engineers an opportunity to"},{"speaker":"Ben Lloyd Pearson","startTime":1796.496,"endTime":1798.746,"body":"learn more about how this"},{"speaker":"Ben Lloyd Pearson","startTime":1798.746,"endTime":1800.336,"body":"stuff is created, right?"},{"speaker":"Evan Morikawa","startTime":1800.426,"endTime":1801.086,"body":"Yes."},{"speaker":"Evan Morikawa","startTime":1801.186,"endTime":1802.926,"body":"it has definitely"},{"speaker":"Evan Morikawa","startTime":1801.186,"endTime":1802.926,"body":"been necessary."},{"speaker":"Evan Morikawa","startTime":1803.136,"endTime":1805.536,"body":"So it has been the case"},{"speaker":"Evan Morikawa","startTime":1803.136,"endTime":1805.536,"body":"actually that everybody in"},{"speaker":"Evan Morikawa","startTime":1805.536,"endTime":1808.476,"body":"applied and engineering did"},{"speaker":"Evan Morikawa","startTime":1805.536,"endTime":1808.476,"body":"not need to have a machine"},{"speaker":"Evan Morikawa","startTime":1808.476,"endTime":1809.916,"body":"learning background to do this."},{"speaker":"Evan Morikawa","startTime":1809.921,"endTime":1812.466,"body":"I do not have a PhD"},{"speaker":"Evan Morikawa","startTime":1809.921,"endTime":1812.466,"body":"in machine learning."},{"speaker":"Evan Morikawa","startTime":1812.966,"endTime":1814.676,"body":"but that's, that's fine for now."},{"speaker":"Evan Morikawa","startTime":1814.826,"endTime":1818.516,"body":"Uh, certainly the, the interest"},{"speaker":"Evan Morikawa","startTime":1814.826,"endTime":1818.516,"body":"and the, to pick up and"},{"speaker":"Evan Morikawa","startTime":1818.521,"endTime":1820.886,"body":"learn a lot of things along"},{"speaker":"Evan Morikawa","startTime":1818.521,"endTime":1820.886,"body":"the way is important, but."},{"speaker":"Evan Morikawa","startTime":1821.966,"endTime":1824.756,"body":"Yeah, a lot of our, most of our"},{"speaker":"Evan Morikawa","startTime":1821.966,"endTime":1824.756,"body":"problems are product problems."},{"speaker":"Evan Morikawa","startTime":1824.756,"endTime":1825.656,"body":"They're engineering problems."},{"speaker":"Evan Morikawa","startTime":1825.656,"endTime":1827.226,"body":"They're distributed"},{"speaker":"Evan Morikawa","startTime":1825.656,"endTime":1827.226,"body":"systems problems."},{"speaker":"Evan Morikawa","startTime":1827.226,"endTime":1831.586,"body":"They're kind of classic like"},{"speaker":"Evan Morikawa","startTime":1827.226,"endTime":1831.586,"body":"that, but at the same time, it"},{"speaker":"Evan Morikawa","startTime":1831.586,"endTime":1834.206,"body":"has been really important for"},{"speaker":"Evan Morikawa","startTime":1831.586,"endTime":1834.206,"body":"everybody to at least get a"},{"speaker":"Evan Morikawa","startTime":1834.406,"endTime":1838.726,"body":"reasonable understanding of how"},{"speaker":"Evan Morikawa","startTime":1834.406,"endTime":1838.726,"body":"all these models fit together."},{"speaker":"Evan Morikawa","startTime":1839.173,"endTime":1841.493,"body":"Because a lot of the"},{"speaker":"Evan Morikawa","startTime":1839.173,"endTime":1841.493,"body":"engineering considerations"},{"speaker":"Evan Morikawa","startTime":1841.493,"endTime":1845.333,"body":"are deeply tied with the way"},{"speaker":"Evan Morikawa","startTime":1841.493,"endTime":1845.333,"body":"these are structured, the"},{"speaker":"Evan Morikawa","startTime":1845.333,"endTime":1847.773,"body":"hardware that we're using,"},{"speaker":"Evan Morikawa","startTime":1845.333,"endTime":1847.773,"body":"the way things are deployed."},{"speaker":"Evan Morikawa","startTime":1848.138,"endTime":1849.558,"body":"Those all definitely matter."},{"speaker":"Ben Lloyd Pearson","startTime":1849.758,"endTime":1851.508,"body":"Yeah, this"},{"speaker":"Ben Lloyd Pearson","startTime":1849.758,"endTime":1851.508,"body":"might be a tough question to"},{"speaker":"Ben Lloyd Pearson","startTime":1851.508,"endTime":1855.908,"body":"answer, but if you could go"},{"speaker":"Ben Lloyd Pearson","startTime":1851.508,"endTime":1855.908,"body":"back in time to Evan a year ago,"},{"speaker":"Ben Lloyd Pearson","startTime":1856.038,"endTime":1859.508,"body":"and tell him, hey, your product"},{"speaker":"Ben Lloyd Pearson","startTime":1856.038,"endTime":1859.508,"body":"is going to go viral someday,"},{"speaker":"Ben Lloyd Pearson","startTime":1860.178,"endTime":1863.048,"body":"are there any changes that you"},{"speaker":"Ben Lloyd Pearson","startTime":1860.178,"endTime":1863.048,"body":"would have made in your approach"},{"speaker":"Ben Lloyd Pearson","startTime":1863.048,"endTime":1865.538,"body":"to respond, or to anticipate"},{"speaker":"Evan Morikawa","startTime":1865.538,"endTime":1865.758,"body":"that?"},{"speaker":"Evan Morikawa","startTime":1865.818,"endTime":1871.748,"body":"you know, maybe not,"},{"speaker":"Evan Morikawa","startTime":1865.818,"endTime":1871.748,"body":"because I, I'm inherently"},{"speaker":"Evan Morikawa","startTime":1871.748,"endTime":1874.088,"body":"a bit of a skeptic when it"},{"speaker":"Evan Morikawa","startTime":1871.748,"endTime":1874.088,"body":"comes to things, like, I"},{"speaker":"Evan Morikawa","startTime":1874.088,"endTime":1875.338,"body":"would not have believed..."},{"speaker":"Evan Morikawa","startTime":1876.163,"endTime":1877.403,"body":"My thing would go viral."},{"speaker":"Evan Morikawa","startTime":1878.643,"endTime":1882.723,"body":"Also because I believe"},{"speaker":"Evan Morikawa","startTime":1878.643,"endTime":1882.723,"body":"in not prematurely"},{"speaker":"Evan Morikawa","startTime":1882.783,"endTime":1884.473,"body":"optimizing the system."},{"speaker":"Evan Morikawa","startTime":1885.364,"endTime":1891.023,"body":"We have this deeply iterative"},{"speaker":"Evan Morikawa","startTime":1885.364,"endTime":1891.023,"body":"model baked into here."},{"speaker":"Evan Morikawa","startTime":1891.363,"endTime":1895.153,"body":"So I actually would have been"},{"speaker":"Evan Morikawa","startTime":1891.363,"endTime":1895.153,"body":"afraid that we would have spent"},{"speaker":"Evan Morikawa","startTime":1895.153,"endTime":1898.943,"body":"this huge amount of time making"},{"speaker":"Evan Morikawa","startTime":1895.153,"endTime":1898.943,"body":"sure the infrastructure was"},{"speaker":"Evan Morikawa","startTime":1898.943,"endTime":1901.893,"body":"load tested up the wazoo, making"},{"speaker":"Evan Morikawa","startTime":1898.943,"endTime":1901.893,"body":"sure the product was perfect"},{"speaker":"Evan Morikawa","startTime":1901.913,"endTime":1902.887,"body":"before it even got there."},{"speaker":"Ben Lloyd Pearson","startTime":1904.488,"endTime":1905.568,"body":"So you're"},{"speaker":"Ben Lloyd Pearson","startTime":1904.488,"endTime":1905.568,"body":"saying just swing for the"},{"speaker":"Ben Lloyd Pearson","startTime":1905.568,"endTime":1909.498,"body":"fences and deal with what"},{"speaker":"Ben Lloyd Pearson","startTime":1905.568,"endTime":1909.498,"body":"happens after the fact."},{"speaker":"Evan Morikawa","startTime":1909.498,"endTime":1916.687,"body":"Now, I should"},{"speaker":"Evan Morikawa","startTime":1909.498,"endTime":1916.687,"body":"note, though, that iterating"},{"speaker":"Evan Morikawa","startTime":1917.348,"endTime":1922.508,"body":"a product quickly does not,"},{"speaker":"Evan Morikawa","startTime":1917.348,"endTime":1922.508,"body":"especially here, it's very"},{"speaker":"Evan Morikawa","startTime":1922.508,"endTime":1924.928,"body":"important that that does"},{"speaker":"Evan Morikawa","startTime":1922.508,"endTime":1924.928,"body":"not compromise the kind"},{"speaker":"Evan Morikawa","startTime":1924.928,"endTime":1927.438,"body":"of safety mission that we"},{"speaker":"Evan Morikawa","startTime":1924.928,"endTime":1927.438,"body":"have to begin with as well."},{"speaker":"Evan Morikawa","startTime":1930.753,"endTime":1934.803,"body":"Like safety, our safe, not"},{"speaker":"Evan Morikawa","startTime":1930.753,"endTime":1934.803,"body":"being happy enough with our"},{"speaker":"Evan Morikawa","startTime":1934.803,"endTime":1937.293,"body":"safety systems, with the red"},{"speaker":"Evan Morikawa","startTime":1934.803,"endTime":1937.293,"body":"teaming results that we're"},{"speaker":"Evan Morikawa","startTime":1937.293,"endTime":1941.583,"body":"getting, that is the primary"},{"speaker":"Evan Morikawa","startTime":1937.293,"endTime":1941.583,"body":"thing that will delay launches."},{"speaker":"Evan Morikawa","startTime":1941.583,"endTime":1944.613,"body":"That is the non-negotiable"},{"speaker":"Evan Morikawa","startTime":1941.583,"endTime":1944.613,"body":"before we can ship something."},{"speaker":"Evan Morikawa","startTime":1944.793,"endTime":1945.063,"body":"Yeah."},{"speaker":"Evan Morikawa","startTime":1945.213,"endTime":1948.243,"body":"so that, that is the place"},{"speaker":"Evan Morikawa","startTime":1945.213,"endTime":1948.243,"body":"that we can, should, and"},{"speaker":"Evan Morikawa","startTime":1948.248,"endTime":1951.363,"body":"would like spend even"},{"speaker":"Evan Morikawa","startTime":1948.248,"endTime":1951.363,"body":"more time iterating on."},{"speaker":"Evan Morikawa","startTime":1951.603,"endTime":1954.843,"body":"But here again, a lot of the"},{"speaker":"Evan Morikawa","startTime":1951.603,"endTime":1954.843,"body":"philosophy around this is"},{"speaker":"Evan Morikawa","startTime":1954.843,"endTime":1956.573,"body":"that, we kind of see the."},{"speaker":"Evan Morikawa","startTime":1957.358,"endTime":1960.618,"body":"Safety systems are the red"},{"speaker":"Evan Morikawa","startTime":1957.358,"endTime":1960.618,"body":"teaming layers coming in layers."},{"speaker":"Evan Morikawa","startTime":1960.618,"endTime":1964.648,"body":"Like we have a very active,"},{"speaker":"Evan Morikawa","startTime":1960.618,"endTime":1964.648,"body":"network of experienced"},{"speaker":"Evan Morikawa","startTime":1964.648,"endTime":1966.698,"body":"red teamers who will"},{"speaker":"Evan Morikawa","startTime":1964.648,"endTime":1966.698,"body":"go in and test stuff."},{"speaker":"Evan Morikawa","startTime":1966.988,"endTime":1970.268,"body":"We have very controlled rollouts"},{"speaker":"Evan Morikawa","startTime":1966.988,"endTime":1970.268,"body":"through various stages to try"},{"speaker":"Evan Morikawa","startTime":1970.268,"endTime":1971.358,"body":"and like catch everything."},{"speaker":"Evan Morikawa","startTime":1971.908,"endTime":1975.418,"body":"But it's still the case that"},{"speaker":"Evan Morikawa","startTime":1971.908,"endTime":1975.418,"body":"there's no way to catch all"},{"speaker":"Evan Morikawa","startTime":1975.418,"endTime":1977.788,"body":"of it until you actually"},{"speaker":"Evan Morikawa","startTime":1975.418,"endTime":1977.788,"body":"get it out into the world."},{"speaker":"Ben Lloyd Pearson","startTime":1977.788,"endTime":1981.807,"body":"Absolutely"},{"speaker":"Evan Morikawa","startTime":1981.808,"endTime":1982.728,"body":"not, yeah."},{"speaker":"Evan Morikawa","startTime":1984.338,"endTime":1991.048,"body":"Having a lot of people thinking"},{"speaker":"Evan Morikawa","startTime":1984.338,"endTime":1991.048,"body":"and working and doing this is"},{"speaker":"Evan Morikawa","startTime":1991.048,"endTime":1994.293,"body":"a big part of what it takes to"},{"speaker":"Evan Morikawa","startTime":1991.048,"endTime":1994.293,"body":"make these things move forward."},{"speaker":"Ben Lloyd Pearson","startTime":1994.293,"endTime":1997.06,"body":"So, I got"},{"speaker":"Ben Lloyd Pearson","startTime":1994.293,"endTime":1997.06,"body":"one I want to talk about and"},{"speaker":"Ben Lloyd Pearson","startTime":1997.06,"endTime":2000.197,"body":"it kind of brings us back full"},{"speaker":"Ben Lloyd Pearson","startTime":1997.06,"endTime":2000.197,"body":"circle to how this conversation"},{"speaker":"Ben Lloyd Pearson","startTime":2000.197,"endTime":2001.243,"body":"started and that's APIs."},{"speaker":"Ben Lloyd Pearson","startTime":2002.043,"endTime":2004.393,"body":"So, you know, we mentioned"},{"speaker":"Ben Lloyd Pearson","startTime":2002.043,"endTime":2004.393,"body":"that, the part of generative"},{"speaker":"Ben Lloyd Pearson","startTime":2004.433,"endTime":2008.493,"body":"API that really excites me is"},{"speaker":"Ben Lloyd Pearson","startTime":2004.433,"endTime":2008.493,"body":"seeing it pop up in all the"},{"speaker":"Ben Lloyd Pearson","startTime":2008.493,"endTime":2010.193,"body":"tools that I use everyday."},{"speaker":"Ben Lloyd Pearson","startTime":2010.553,"endTime":2012.763,"body":"you know, in, at LinearB,"},{"speaker":"Ben Lloyd Pearson","startTime":2010.553,"endTime":2012.763,"body":"you know, we've trained"},{"speaker":"Ben Lloyd Pearson","startTime":2012.763,"endTime":2015.743,"body":"ChatGPT to use multiple"},{"speaker":"Ben Lloyd Pearson","startTime":2012.763,"endTime":2015.743,"body":"aspects of our platform."},{"speaker":"Ben Lloyd Pearson","startTime":2015.743,"endTime":2018.443,"body":"So we can do things like ask"},{"speaker":"Ben Lloyd Pearson","startTime":2015.743,"endTime":2018.443,"body":"it questions, it can write"},{"speaker":"Ben Lloyd Pearson","startTime":2018.443,"endTime":2021.503,"body":"configuration files for us"},{"speaker":"Ben Lloyd Pearson","startTime":2018.443,"endTime":2021.503,"body":"that are highly specialized."},{"speaker":"Ben Lloyd Pearson","startTime":2021.923,"endTime":2024.533,"body":"And, uh, you know, that's, like,"},{"speaker":"Ben Lloyd Pearson","startTime":2021.923,"endTime":2024.533,"body":"to me, that's, like, the real,"},{"speaker":"Ben Lloyd Pearson","startTime":2024.543,"endTime":2026.213,"body":"like, fascinating use cases."},{"speaker":"Ben Lloyd Pearson","startTime":2026.623,"endTime":2028.323,"body":"but, you know, obviously for"},{"speaker":"Ben Lloyd Pearson","startTime":2026.623,"endTime":2028.323,"body":"that to happen, there's got"},{"speaker":"Ben Lloyd Pearson","startTime":2028.323,"endTime":2031.363,"body":"to be a really strong API"},{"speaker":"Ben Lloyd Pearson","startTime":2028.323,"endTime":2031.363,"body":"for developers to, to build"},{"speaker":"Ben Lloyd Pearson","startTime":2031.363,"endTime":2032.683,"body":"functionality on top of it."},{"speaker":"Ben Lloyd Pearson","startTime":2033.073,"endTime":2035.493,"body":"And, you know, I think since we"},{"speaker":"Ben Lloyd Pearson","startTime":2033.073,"endTime":2035.493,"body":"both worked at API companies,"},{"speaker":"Ben Lloyd Pearson","startTime":2035.513,"endTime":2038.163,"body":"we understand that, you know,"},{"speaker":"Ben Lloyd Pearson","startTime":2035.513,"endTime":2038.163,"body":"it really comes down to making"},{"speaker":"Ben Lloyd Pearson","startTime":2038.163,"endTime":2040.523,"body":"sure the API is performant,"},{"speaker":"Ben Lloyd Pearson","startTime":2038.163,"endTime":2040.523,"body":"and that it's built for"},{"speaker":"Ben Lloyd Pearson","startTime":2040.523,"endTime":2044.143,"body":"real world use cases rather"},{"speaker":"Ben Lloyd Pearson","startTime":2040.523,"endTime":2044.143,"body":"than theoretical situations."},{"speaker":"Ben Lloyd Pearson","startTime":2044.929,"endTime":2048.049,"body":"what role generally speaking,"},{"speaker":"Ben Lloyd Pearson","startTime":2044.929,"endTime":2048.049,"body":"do you think APIs or, or the"},{"speaker":"Ben Lloyd Pearson","startTime":2048.049,"endTime":2051.109,"body":"APIs for your products are"},{"speaker":"Ben Lloyd Pearson","startTime":2048.049,"endTime":2051.109,"body":"gonna play in the success"},{"speaker":"Ben Lloyd Pearson","startTime":2051.199,"endTime":2052.789,"body":"of Open AI in ChatGPT?"},{"speaker":"Evan Morikawa","startTime":2052.849,"endTime":2053.299,"body":"Yeah."},{"speaker":"Evan Morikawa","startTime":2053.359,"endTime":2056.929,"body":"The, there is absolutely no"},{"speaker":"Evan Morikawa","startTime":2053.359,"endTime":2056.929,"body":"way, despite how I think we"},{"speaker":"Evan Morikawa","startTime":2056.929,"endTime":2058.189,"body":"have a very talented team."},{"speaker":"Evan Morikawa","startTime":2058.249,"endTime":2061.729,"body":"There is no way our one team"},{"speaker":"Evan Morikawa","startTime":2058.249,"endTime":2061.729,"body":"can out-innovate the vast,"},{"speaker":"Evan Morikawa","startTime":2061.729,"endTime":2065.989,"body":"some of all of the creativity"},{"speaker":"Evan Morikawa","startTime":2061.729,"endTime":2065.989,"body":"and startup energy and like"},{"speaker":"Evan Morikawa","startTime":2066.369,"endTime":2070.119,"body":"company focuses that it has on"},{"speaker":"Evan Morikawa","startTime":2066.369,"endTime":2070.119,"body":"making this work right now, as"},{"speaker":"Evan Morikawa","startTime":2070.119,"endTime":2071.439,"body":"I mentioned earlier, like the."},{"speaker":"Evan Morikawa","startTime":2071.964,"endTime":2076.494,"body":"Ability to have this, to"},{"speaker":"Evan Morikawa","startTime":2071.964,"endTime":2076.494,"body":"have AI deeply integrated"},{"speaker":"Evan Morikawa","startTime":2076.494,"endTime":2080.069,"body":"with everything you're doing"},{"speaker":"Evan Morikawa","startTime":2076.494,"endTime":2080.069,"body":"somewhat seamlessly and"},{"speaker":"Evan Morikawa","startTime":2080.069,"endTime":2082.654,"body":"transparently, that's where"},{"speaker":"Evan Morikawa","startTime":2080.069,"endTime":2082.654,"body":"a lot of real power is."},{"speaker":"Evan Morikawa","startTime":2090.199,"endTime":2095.179,"body":"One of the best places for us to"},{"speaker":"Evan Morikawa","startTime":2090.199,"endTime":2095.179,"body":"sort of try out new ideas first."},{"speaker":"Ben Lloyd Pearson","startTime":2095.229,"endTime":2096.379,"body":"That's"},{"speaker":"Ben Lloyd Pearson","startTime":2095.229,"endTime":2096.379,"body":"actually a great point, because"},{"speaker":"Ben Lloyd Pearson","startTime":2096.379,"endTime":2098.559,"body":"I wanted to ask, like what kind"},{"speaker":"Ben Lloyd Pearson","startTime":2096.379,"endTime":2098.559,"body":"of feedback are you getting"},{"speaker":"Ben Lloyd Pearson","startTime":2098.559,"endTime":2099.889,"body":"from developers in the field?"},{"speaker":"Evan Morikawa","startTime":2099.929,"endTime":2101.969,"body":"That's one of the"},{"speaker":"Evan Morikawa","startTime":2099.929,"endTime":2101.969,"body":"most important pieces about it."},{"speaker":"Evan Morikawa","startTime":2102.616,"endTime":2105.136,"body":"Consumer apps, it's very"},{"speaker":"Evan Morikawa","startTime":2102.616,"endTime":2105.136,"body":"difficult to get feedback"},{"speaker":"Evan Morikawa","startTime":2105.146,"endTime":2107.986,"body":"on unless you start doing"},{"speaker":"Evan Morikawa","startTime":2105.146,"endTime":2107.986,"body":"really aggregate stuff."},{"speaker":"Evan Morikawa","startTime":2108.373,"endTime":2110.093,"body":"But yeah, if you sit down"},{"speaker":"Evan Morikawa","startTime":2108.373,"endTime":2110.093,"body":"and you like talk with the"},{"speaker":"Evan Morikawa","startTime":2110.093,"endTime":2114.463,"body":"developers building on the API,"},{"speaker":"Evan Morikawa","startTime":2110.093,"endTime":2114.463,"body":"we really learn where the actual"},{"speaker":"Evan Morikawa","startTime":2114.463,"endTime":2116.063,"body":"problems are in the system."},{"speaker":"Evan Morikawa","startTime":2117.753,"endTime":2120.573,"body":"The gap between some a"},{"speaker":"Evan Morikawa","startTime":2117.753,"endTime":2120.573,"body":"cool demo and something"},{"speaker":"Evan Morikawa","startTime":2120.573,"endTime":2122.433,"body":"that's useful is massive."},{"speaker":"Evan Morikawa","startTime":2122.853,"endTime":2123.723,"body":"And that is true here."},{"speaker":"Evan Morikawa","startTime":2123.728,"endTime":2125.043,"body":"That is true in every industry."},{"speaker":"Evan Morikawa","startTime":2125.103,"endTime":2127.593,"body":"It is especially true here,"},{"speaker":"Evan Morikawa","startTime":2125.103,"endTime":2127.593,"body":"esp, like the more hype that"},{"speaker":"Evan Morikawa","startTime":2127.593,"endTime":2130.203,"body":"there is, the further that gap"},{"speaker":"Evan Morikawa","startTime":2127.593,"endTime":2130.203,"body":"becomes and that only, that"},{"speaker":"Evan Morikawa","startTime":2130.203,"endTime":2133.323,"body":"gap can only be closed when"},{"speaker":"Evan Morikawa","startTime":2130.203,"endTime":2133.323,"body":"you're like really working with"},{"speaker":"Evan Morikawa","startTime":2133.323,"endTime":2136.713,"body":"companies and developers like"},{"speaker":"Evan Morikawa","startTime":2133.323,"endTime":2136.713,"body":"figuring out the hard way and"},{"speaker":"Evan Morikawa","startTime":2136.713,"endTime":2137.733,"body":"learning all the quirks of it."},{"speaker":"Evan Morikawa","startTime":2138.063,"endTime":2140.403,"body":"It is definitely the case"},{"speaker":"Evan Morikawa","startTime":2138.063,"endTime":2140.403,"body":"that the API ecosystem has"},{"speaker":"Evan Morikawa","startTime":2140.403,"endTime":2143.853,"body":"discovered far more quirks"},{"speaker":"Evan Morikawa","startTime":2140.403,"endTime":2143.853,"body":"of the models than our"},{"speaker":"Evan Morikawa","startTime":2143.853,"endTime":2145.083,"body":"own research teams have."},{"speaker":"Ben Lloyd Pearson","startTime":2145.493,"endTime":2146.783,"body":"so what,"},{"speaker":"Ben Lloyd Pearson","startTime":2145.493,"endTime":2146.783,"body":"what are some of the engineering"},{"speaker":"Ben Lloyd Pearson","startTime":2146.783,"endTime":2149.513,"body":"challenges that have been"},{"speaker":"Ben Lloyd Pearson","startTime":2146.783,"endTime":2149.513,"body":"unique to the API versus the,"},{"speaker":"Ben Lloyd Pearson","startTime":2149.543,"endTime":2151.133,"body":"the more general purpose tools?"},{"speaker":"Evan Morikawa","startTime":2151.792,"endTime":2155.052,"body":"we have lots"},{"speaker":"Evan Morikawa","startTime":2151.792,"endTime":2155.052,"body":"of similar challenges to other"},{"speaker":"Evan Morikawa","startTime":2155.062,"endTime":2158.272,"body":"API companies, you know, like"},{"speaker":"Evan Morikawa","startTime":2155.062,"endTime":2158.272,"body":"database connection limits,"},{"speaker":"Evan Morikawa","startTime":2159.012,"endTime":2160.522,"body":"networking shenanigans."},{"speaker":"Evan Morikawa","startTime":2161.039,"endTime":2164.189,"body":"the GPU constraints though,"},{"speaker":"Evan Morikawa","startTime":2161.039,"endTime":2164.189,"body":"I would say those are,"},{"speaker":"Evan Morikawa","startTime":2164.199,"endTime":2165.689,"body":"those are quite different."},{"speaker":"Evan Morikawa","startTime":2165.819,"endTime":2168.649,"body":"We've needed to jump"},{"speaker":"Evan Morikawa","startTime":2165.819,"endTime":2168.649,"body":"immediately to tons of"},{"speaker":"Evan Morikawa","startTime":2168.649,"endTime":2170.249,"body":"clusters all over the world."},{"speaker":"Evan Morikawa","startTime":2170.689,"endTime":2172.669,"body":"That was mostly done"},{"speaker":"Evan Morikawa","startTime":2170.689,"endTime":2172.669,"body":"because we were chasing"},{"speaker":"Evan Morikawa","startTime":2172.669,"endTime":2174.169,"body":"GPUs wherever we could have."},{"speaker":"Evan Morikawa","startTime":2174.189,"endTime":2178.019,"body":"So we are like a, we"},{"speaker":"Evan Morikawa","startTime":2174.189,"endTime":2178.019,"body":"suddenly found ourselves"},{"speaker":"Evan Morikawa","startTime":2178.019,"endTime":2179.599,"body":"multi cluster, multi region."},{"speaker":"Evan Morikawa","startTime":2180.169,"endTime":2183.059,"body":"On the flip side though, we've"},{"speaker":"Evan Morikawa","startTime":2180.169,"endTime":2183.059,"body":"also spent a fairly large"},{"speaker":"Evan Morikawa","startTime":2183.059,"endTime":2186.269,"body":"amount of effort making it"},{"speaker":"Evan Morikawa","startTime":2183.059,"endTime":2186.269,"body":"such that there really aren't"},{"speaker":"Evan Morikawa","startTime":2186.489,"endTime":2190.654,"body":"that many special unique things"},{"speaker":"Evan Morikawa","startTime":2186.489,"endTime":2190.654,"body":"about Our deployment stack."},{"speaker":"Evan Morikawa","startTime":2191.161,"endTime":2194.271,"body":"we've been using stock"},{"speaker":"Evan Morikawa","startTime":2191.161,"endTime":2194.271,"body":"Azure Kubernetes service,"},{"speaker":"Evan Morikawa","startTime":2194.301,"endTime":2196.991,"body":"we use a lot, we use like"},{"speaker":"Evan Morikawa","startTime":2194.301,"endTime":2196.991,"body":"Datadog, we use a lot of just"},{"speaker":"Evan Morikawa","startTime":2196.991,"endTime":2198.551,"body":"like off the shelf tools."},{"speaker":"Evan Morikawa","startTime":2198.911,"endTime":2200.361,"body":"I think that's actually been"},{"speaker":"Evan Morikawa","startTime":2198.911,"endTime":2200.361,"body":"really important, that's"},{"speaker":"Evan Morikawa","startTime":2200.361,"endTime":2203.961,"body":"really helped our development"},{"speaker":"Evan Morikawa","startTime":2200.361,"endTime":2203.961,"body":"team stay small, it's meant"},{"speaker":"Evan Morikawa","startTime":2203.971,"endTime":2206.551,"body":"that new hires come in, kind"},{"speaker":"Evan Morikawa","startTime":2203.971,"endTime":2206.551,"body":"of know what they're doing."},{"speaker":"Evan Morikawa","startTime":2207.131,"endTime":2211.556,"body":"the more we can do to try"},{"speaker":"Evan Morikawa","startTime":2207.131,"endTime":2211.556,"body":"and treat things as a Just"},{"speaker":"Evan Morikawa","startTime":2211.556,"endTime":2213.836,"body":"another service that takes"},{"speaker":"Evan Morikawa","startTime":2211.556,"endTime":2213.836,"body":"text and spits it out the"},{"speaker":"Evan Morikawa","startTime":2213.836,"endTime":2215.866,"body":"other side, you know, a cube"},{"speaker":"Evan Morikawa","startTime":2213.836,"endTime":2215.866,"body":"service is a cube service."},{"speaker":"Evan Morikawa","startTime":2215.866,"endTime":2217.296,"body":"People know how"},{"speaker":"Evan Morikawa","startTime":2215.866,"endTime":2217.296,"body":"to deal with that."},{"speaker":"Evan Morikawa","startTime":2217.716,"endTime":2219.756,"body":"It's a, at least"},{"speaker":"Evan Morikawa","startTime":2217.716,"endTime":2219.756,"body":"it's a known unknown."},{"speaker":"Evan Morikawa","startTime":2220.176,"endTime":2222.836,"body":"I'm trying to, like, minimize"},{"speaker":"Evan Morikawa","startTime":2220.176,"endTime":2222.836,"body":"the unknown unknowns here."},{"speaker":"Evan Morikawa","startTime":2223.356,"endTime":2226.376,"body":"but, yeah, at the same time,"},{"speaker":"Evan Morikawa","startTime":2223.356,"endTime":2226.376,"body":"the scaling characteristics"},{"speaker":"Evan Morikawa","startTime":2226.376,"endTime":2228.286,"body":"of this are very strange."},{"speaker":"Evan Morikawa","startTime":2228.316,"endTime":2231.096,"body":"I was mentioning that when,"},{"speaker":"Evan Morikawa","startTime":2228.316,"endTime":2231.096,"body":"yeah, talking about these, like,"},{"speaker":"Evan Morikawa","startTime":2231.106,"endTime":2235.336,"body":"utilization metrics were hard to"},{"speaker":"Evan Morikawa","startTime":2231.106,"endTime":2235.336,"body":"figure out, initially there too."},{"speaker":"Evan Morikawa","startTime":2235.336,"endTime":2237.336,"body":"And the scaling challenges of"},{"speaker":"Evan Morikawa","startTime":2235.336,"endTime":2237.336,"body":"this, I think, are also going"},{"speaker":"Evan Morikawa","startTime":2237.336,"endTime":2238.986,"body":"to get pretty, pretty nuts."},{"speaker":"Evan Morikawa","startTime":2239.006,"endTime":2239.746,"body":"Like, we..."},{"speaker":"Evan Morikawa","startTime":2240.176,"endTime":2246.186,"body":"the, the, the ambitions for"},{"speaker":"Evan Morikawa","startTime":2240.176,"endTime":2246.186,"body":"the scale of capacity that we"},{"speaker":"Evan Morikawa","startTime":2246.186,"endTime":2250.136,"body":"need to ramp up to, it's both"},{"speaker":"Evan Morikawa","startTime":2246.186,"endTime":2250.136,"body":"usage going up, the models"},{"speaker":"Evan Morikawa","startTime":2250.136,"endTime":2253.056,"body":"getting bigger, the models"},{"speaker":"Evan Morikawa","startTime":2250.136,"endTime":2253.056,"body":"having all these different"},{"speaker":"Evan Morikawa","startTime":2253.066,"endTime":2256.526,"body":"types of modalities for"},{"speaker":"Evan Morikawa","startTime":2253.066,"endTime":2256.526,"body":"them, yeah, that's all just"},{"speaker":"Evan Morikawa","startTime":2256.656,"endTime":2258.146,"body":"getting started right now."},{"speaker":"Evan Morikawa","startTime":2258.642,"endTime":2262.102,"body":"yeah, like we found ourselves"},{"speaker":"Evan Morikawa","startTime":2258.642,"endTime":2262.102,"body":"suddenly When Dali came out,"},{"speaker":"Evan Morikawa","startTime":2262.122,"endTime":2264.162,"body":"we're like, oh, we're suddenly"},{"speaker":"Evan Morikawa","startTime":2262.122,"endTime":2264.162,"body":"not dealing with text anymore."},{"speaker":"Evan Morikawa","startTime":2264.162,"endTime":2267.162,"body":"Now it's all, everything"},{"speaker":"Evan Morikawa","startTime":2264.162,"endTime":2267.162,"body":"that is image processing and"},{"speaker":"Evan Morikawa","startTime":2267.162,"endTime":2270.262,"body":"image generation, now we're"},{"speaker":"Evan Morikawa","startTime":2267.162,"endTime":2270.262,"body":"in audio as well with all"},{"speaker":"Evan Morikawa","startTime":2270.262,"endTime":2271.662,"body":"speech in and speech out."},{"speaker":"Evan Morikawa","startTime":2272.232,"endTime":2276.252,"body":"Um, so, so yeah, the, the,"},{"speaker":"Evan Morikawa","startTime":2272.232,"endTime":2276.252,"body":"it's become a lot more complex."},{"speaker":"Ben Lloyd Pearson","startTime":2277.342,"endTime":2277.762,"body":"Awesome."},{"speaker":"Ben Lloyd Pearson","startTime":2277.772,"endTime":2279.022,"body":"They're all very fascinating."},{"speaker":"Ben Lloyd Pearson","startTime":2279.022,"endTime":2281.812,"body":"I'm really happy we got to,"},{"speaker":"Ben Lloyd Pearson","startTime":2279.022,"endTime":2281.812,"body":"to learn about all this from,"},{"speaker":"Ben Lloyd Pearson","startTime":2281.832,"endTime":2283.712,"body":"from inside the organization."},{"speaker":"Ben Lloyd Pearson","startTime":2284.332,"endTime":2286.202,"body":"And I have a couple of just"},{"speaker":"Ben Lloyd Pearson","startTime":2284.332,"endTime":2286.202,"body":"quick questions, because"},{"speaker":"Ben Lloyd Pearson","startTime":2286.202,"endTime":2288.552,"body":"they're things that I think"},{"speaker":"Ben Lloyd Pearson","startTime":2286.202,"endTime":2288.552,"body":"our audience is really"},{"speaker":"Ben Lloyd Pearson","startTime":2288.552,"endTime":2290.452,"body":"going to be interested in."},{"speaker":"Ben Lloyd Pearson","startTime":2291.102,"endTime":2294.332,"body":"The first is, what is the"},{"speaker":"Ben Lloyd Pearson","startTime":2291.102,"endTime":2294.332,"body":"most interesting or useful"},{"speaker":"Ben Lloyd Pearson","startTime":2294.362,"endTime":2297.142,"body":"adaptation you've seen so"},{"speaker":"Ben Lloyd Pearson","startTime":2294.362,"endTime":2297.142,"body":"far of one of your products?"},{"speaker":"Evan Morikawa","startTime":2297.572,"endTime":2301.142,"body":"So, I think this"},{"speaker":"Evan Morikawa","startTime":2297.572,"endTime":2301.142,"body":"is still emerging yet, but like,"},{"speaker":"Evan Morikawa","startTime":2301.382,"endTime":2304.212,"body":"there's a lot of work of people"},{"speaker":"Evan Morikawa","startTime":2301.382,"endTime":2304.212,"body":"trying to build longer form"},{"speaker":"Evan Morikawa","startTime":2304.222,"endTime":2305.952,"body":"agents right now on the system."},{"speaker":"Evan Morikawa","startTime":2305.952,"endTime":2308.562,"body":"That's been a huge focus of"},{"speaker":"Evan Morikawa","startTime":2305.952,"endTime":2308.562,"body":"a lot of startup activity."},{"speaker":"Evan Morikawa","startTime":2311.717,"endTime":2315.007,"body":"As I mentioned, a lot of the"},{"speaker":"Evan Morikawa","startTime":2311.717,"endTime":2315.007,"body":"like, law applications I think"},{"speaker":"Evan Morikawa","startTime":2315.007,"endTime":2318.857,"body":"are very, very, very, very"},{"speaker":"Evan Morikawa","startTime":2315.007,"endTime":2318.857,"body":"like, have a huge potential"},{"speaker":"Evan Morikawa","startTime":2318.857,"endTime":2320.947,"body":"to feel like that industry"},{"speaker":"Evan Morikawa","startTime":2318.857,"endTime":2320.947,"body":"should feel very different."},{"speaker":"Ben Lloyd Pearson","startTime":2321.237,"endTime":2322.267,"body":"I would"},{"speaker":"Ben Lloyd Pearson","startTime":2321.237,"endTime":2322.267,"body":"love to have a lawyer in"},{"speaker":"Ben Lloyd Pearson","startTime":2322.277,"endTime":2323.647,"body":"my pocket, just saying."},{"speaker":"Evan Morikawa","startTime":2323.647,"endTime":2327.577,"body":"The other"},{"speaker":"Evan Morikawa","startTime":2323.647,"endTime":2327.577,"body":"one of course is like the"},{"speaker":"Evan Morikawa","startTime":2327.577,"endTime":2329.007,"body":"education side of things."},{"speaker":"Evan Morikawa","startTime":2330.217,"endTime":2334.707,"body":"We have a real effort to figure"},{"speaker":"Evan Morikawa","startTime":2330.217,"endTime":2334.707,"body":"out how to use these tools."},{"speaker":"Evan Morikawa","startTime":2334.707,"endTime":2337.787,"body":"To me it is kind of analogous"},{"speaker":"Evan Morikawa","startTime":2334.707,"endTime":2337.787,"body":"to, I don't know, math"},{"speaker":"Evan Morikawa","startTime":2337.787,"endTime":2340.217,"body":"classes had to do something"},{"speaker":"Evan Morikawa","startTime":2337.787,"endTime":2340.217,"body":"different when my calculator"},{"speaker":"Evan Morikawa","startTime":2340.217,"endTime":2341.127,"body":"showed up everywhere."},{"speaker":"Evan Morikawa","startTime":2341.617,"endTime":2347.577,"body":"but we will, figuring that out"},{"speaker":"Evan Morikawa","startTime":2341.617,"endTime":2347.577,"body":"and sort of like, getting in a"},{"speaker":"Evan Morikawa","startTime":2347.577,"endTime":2350.847,"body":"world where people start to use"},{"speaker":"Evan Morikawa","startTime":2347.577,"endTime":2350.847,"body":"these as tools and can like,"},{"speaker":"Evan Morikawa","startTime":2351.327,"endTime":2353.907,"body":"help them be just dramatically"},{"speaker":"Evan Morikawa","startTime":2351.327,"endTime":2353.907,"body":"more impactful at things,"},{"speaker":"Evan Morikawa","startTime":2353.907,"endTime":2355.037,"body":"I think that's a huge deal."},{"speaker":"Evan Morikawa","startTime":2355.417,"endTime":2358.677,"body":"I actually really liked"},{"speaker":"Evan Morikawa","startTime":2355.417,"endTime":2358.677,"body":"Spotify's feature they"},{"speaker":"Evan Morikawa","startTime":2358.677,"endTime":2361.907,"body":"launched recently, which"},{"speaker":"Evan Morikawa","startTime":2358.677,"endTime":2361.907,"body":"was using, text to speech."},{"speaker":"Evan Morikawa","startTime":2362.187,"endTime":2363.997,"body":"So they release these podcasts."},{"speaker":"Evan Morikawa","startTime":2364.842,"endTime":2369.162,"body":"In other languages,"},{"speaker":"Evan Morikawa","startTime":2364.842,"endTime":2369.162,"body":"with the voice of the"},{"speaker":"Evan Morikawa","startTime":2369.322,"endTime":2371.172,"body":"original podcasters."},{"speaker":"Evan Morikawa","startTime":2371.182,"endTime":2373.872,"body":"So yes, you can like listen"},{"speaker":"Evan Morikawa","startTime":2371.182,"endTime":2373.872,"body":"to Lex Freeman in Spanish,"},{"speaker":"Evan Morikawa","startTime":2373.872,"endTime":2376.282,"body":"but it is clearly his voice."},{"speaker":"Evan Morikawa","startTime":2376.522,"endTime":2379.702,"body":"And if you think about how"},{"speaker":"Evan Morikawa","startTime":2376.522,"endTime":2379.702,"body":"much work or money it used"},{"speaker":"Evan Morikawa","startTime":2379.702,"endTime":2382.312,"body":"to take to dub something,"},{"speaker":"Evan Morikawa","startTime":2379.702,"endTime":2382.312,"body":"or to hire somebody to do"},{"speaker":"Evan Morikawa","startTime":2382.312,"endTime":2384.492,"body":"that, like, that's insane."},{"speaker":"Evan Morikawa","startTime":2384.492,"endTime":2387.722,"body":"the other partnership I think"},{"speaker":"Evan Morikawa","startTime":2384.492,"endTime":2387.722,"body":"is really cool is Be My Eyes."},{"speaker":"Evan Morikawa","startTime":2387.822,"endTime":2391.322,"body":"They've been using the GPT 4"},{"speaker":"Evan Morikawa","startTime":2387.822,"endTime":2391.322,"body":"with vision capabilities to"},{"speaker":"Evan Morikawa","startTime":2391.322,"endTime":2392.612,"body":"help visually impaired people."},{"speaker":"Evan Morikawa","startTime":2393.182,"endTime":2394.372,"body":"Take a photo of your closet."},{"speaker":"Evan Morikawa","startTime":2394.952,"endTime":2397.892,"body":"Like, what, what should"},{"speaker":"Evan Morikawa","startTime":2394.952,"endTime":2397.892,"body":"I be wearing today?"},{"speaker":"Evan Morikawa","startTime":2398.462,"endTime":2399.422,"body":"yeah, that's, that's nuts."},{"speaker":"Evan Morikawa","startTime":2399.422,"endTime":2402.842,"body":"You can, you gonna have a device"},{"speaker":"Evan Morikawa","startTime":2399.422,"endTime":2402.842,"body":"in your pocket that can deeply"},{"speaker":"Evan Morikawa","startTime":2402.842,"endTime":2405.932,"body":"meaningfully and semantically"},{"speaker":"Evan Morikawa","startTime":2402.842,"endTime":2405.932,"body":"describe what you're looking at."},{"speaker":"Evan Morikawa","startTime":2405.937,"endTime":2408.152,"body":"And if you're a visually"},{"speaker":"Evan Morikawa","startTime":2405.937,"endTime":2408.152,"body":"impaired person, that's,"},{"speaker":"Evan Morikawa","startTime":2408.542,"endTime":2410.882,"body":"yeah, that's obviously,"},{"speaker":"Evan Morikawa","startTime":2408.542,"endTime":2410.882,"body":"that's massives a huge deal."},{"speaker":"Ben Lloyd Pearson","startTime":2413.372,"endTime":2413.942,"body":"awesome."},{"speaker":"Ben Lloyd Pearson","startTime":2414.032,"endTime":2417.392,"body":"Uh, well it is been really"},{"speaker":"Ben Lloyd Pearson","startTime":2414.032,"endTime":2417.392,"body":"great learning, uh, the"},{"speaker":"Ben Lloyd Pearson","startTime":2417.392,"endTime":2420.272,"body":"inside perspective from"},{"speaker":"Ben Lloyd Pearson","startTime":2417.392,"endTime":2420.272,"body":"you on OpenAI and, and all"},{"speaker":"Ben Lloyd Pearson","startTime":2420.272,"endTime":2421.982,"body":"the engineering work that"},{"speaker":"Ben Lloyd Pearson","startTime":2420.272,"endTime":2421.982,"body":"you're doing over there."},{"speaker":"Ben Lloyd Pearson","startTime":2422.372,"endTime":2424.322,"body":"Uh, if people wanna learn"},{"speaker":"Ben Lloyd Pearson","startTime":2422.372,"endTime":2424.322,"body":"more about you or the work"},{"speaker":"Ben Lloyd Pearson","startTime":2424.322,"endTime":2426.182,"body":"that you're doing, where's"},{"speaker":"Ben Lloyd Pearson","startTime":2424.322,"endTime":2426.182,"body":"the best place to send them?"},{"speaker":"Evan Morikawa","startTime":2426.352,"endTime":2430.822,"body":"yeah, so I'm,"},{"speaker":"Evan Morikawa","startTime":2426.352,"endTime":2430.822,"body":"I am E zero M on Twitter, and"},{"speaker":"Evan Morikawa","startTime":2431.172,"endTime":2434.202,"body":"actually, I, I actually point"},{"speaker":"Evan Morikawa","startTime":2431.172,"endTime":2434.202,"body":"including a lot of our new hires"},{"speaker":"Evan Morikawa","startTime":2434.202,"endTime":2438.462,"body":"to open AI's blog, which both"},{"speaker":"Evan Morikawa","startTime":2434.202,"endTime":2438.462,"body":"is a mix of like the product"},{"speaker":"Evan Morikawa","startTime":2438.467,"endTime":2441.612,"body":"and research releases too, but"},{"speaker":"Evan Morikawa","startTime":2438.467,"endTime":2441.612,"body":"kind of that whole arc gives"},{"speaker":"Evan Morikawa","startTime":2441.617,"endTime":2444.182,"body":"a pretty good view of the."},{"speaker":"Evan Morikawa","startTime":2444.577,"endTime":2446.197,"body":"Kind of the state of what"},{"speaker":"Evan Morikawa","startTime":2444.577,"endTime":2446.197,"body":"we're doing, but also state of"},{"speaker":"Evan Morikawa","startTime":2446.197,"endTime":2447.267,"body":"the kind of the industry too."},{"speaker":"Ben Lloyd Pearson","startTime":2447.517,"endTime":2447.817,"body":"Awesome."},{"speaker":"Ben Lloyd Pearson","startTime":2447.817,"endTime":2449.757,"body":"And I know personally, I've"},{"speaker":"Ben Lloyd Pearson","startTime":2447.817,"endTime":2449.757,"body":"been silently watching your"},{"speaker":"Ben Lloyd Pearson","startTime":2449.757,"endTime":2452.017,"body":"LinkedIn too and seeing a lot"},{"speaker":"Ben Lloyd Pearson","startTime":2449.757,"endTime":2452.017,"body":"of your posts about all the"},{"speaker":"Ben Lloyd Pearson","startTime":2452.017,"endTime":2453.127,"body":"cool stuff that's happening."},{"speaker":"Ben Lloyd Pearson","startTime":2453.327,"endTime":2454.417,"body":"So, so yeah."},{"speaker":"Ben Lloyd Pearson","startTime":2454.687,"endTime":2456.887,"body":"Well, it was really great"},{"speaker":"Ben Lloyd Pearson","startTime":2454.687,"endTime":2456.887,"body":"having you here today."},{"speaker":"Ben Lloyd Pearson","startTime":2456.947,"endTime":2459.347,"body":"I'm glad we got a chance"},{"speaker":"Ben Lloyd Pearson","startTime":2456.947,"endTime":2459.347,"body":"to catch up and talk"},{"speaker":"Ben Lloyd Pearson","startTime":2459.547,"endTime":2460.387,"body":"about what you're doing."},{"speaker":"Ben Lloyd Pearson","startTime":2460.387,"endTime":2461.637,"body":"So thanks for showing up."},{"speaker":"Evan Morikawa","startTime":2461.697,"endTime":2462.157,"body":"Likewise."},{"speaker":"Evan Morikawa","startTime":2462.167,"endTime":2462.737,"body":"Thank you."}]}