We published a paper by way of example with one of our big customers, Merck, that illustrated how by making judicious design decisions at the very beginning of the clinical trial, you can save upwards of a hundred million dollars in the downstream cost of the trial.
SPEAKER_01What is the the budget typically expected pre-AI for this clinical trial phase of new development?
SPEAKER_00The overall cost of developing the drug and running the trials and doing everything necessary to bring a drug from conception to market is of the order of one to two billion dollars. Wow. And these trials oftentimes run 10 years or more. How far opposing generative AI? And so we're using AI to greatly accelerate that process from a matter of weeks or even months down to potentially 20 minutes to get the first drop.
SPEAKER_01Patrick Lung is a pioneering force at the intersection of AI and healthcare. From Google Duplex to hedge fund innovation, he's now driving Faro Health's mission to slash drug trial costs and bring life-saving treatments to patients faster. Welcome to Using AI at work. I'm your host, Chris Dagle. Each week we'll be learning how today's business owners, entrepreneurs, and ambitious professionals are getting more done with smart use of tomorrow's tech. Let's get started. Right now, every business leader is asking the same question. What are we going to do about AI? If this is you, ChiefAIOfficer.com has the answer. We give you a simple path forward where we provide executive and team training so your people know exactly how to safely use generative AI in their day-to-day. We also manage the deployment and implementation to make sure tools actually get adopted and deliver results. And we'll also guide company-wide transformation so AI becomes part of your operating system, not just another shiny object. The companies that act now will increase productivity, cut costs, and grow faster than their competitors. Those that wait will get left behind. So if you want to make AI work in your business, visit chiefaiofficer.com and see how we're helping companies of all sizes finally get results from AI. Hey everybody, welcome to another episode of Using AI at Work with an amazing executive who is using AI not only in his day-to-day, but their business is based on it. I'd like to introduce Patrew Lung from Faro Health. Patrick and I had the opportunity to connect before the episode to kind of discuss what they're doing, but also how he's using generative AI in his day-to-day. So, Patrick, before we get started, if you don't mind, just maybe introduce yourself to the community and uh let us know um kind of what brought you here today.
SPEAKER_00Sure. Um thanks for having me on the show, Chris. It's great to be here. Um so I'm a tech entrepreneur and executive. I've been um in this industry since the.com boom way back in the 90s. And um it's uh I I worked at Google um for about 11 years. And as part of that, I was part of this team called the Google Duplex, which produced this very, very super realistic conversational AI system that could call up businesses on the user's behalf and book restaurant reservations. Yeah, and at the time it really felt like, wow, we're now in this world of the movie Hair. And this was before large language models came along, but it just sounded so real um that people were really freaking out about it. And so to me, that was in some ways the dawn of the AI kind of revolution that we're in right now. Yeah. Um, that of course was greatly accelerated by the introduction of live language models, which Google played a big part in with the invention of the Transformer architecture. And so since Google, I've been basically applying AI in different ways for a couple of years at a hedge fund in New York to Sigma Investments, and then I started my own company using AI to predict tropical rainforest ecosystem growth. Um, and now I'm at Faro Health, where we're applying large language models and other forms of AI to make clinical trials better and bring drugs to market um quicker and hopefully at lower cost to the user and so on and so forth.
SPEAKER_01Nice. So um tell me a little bit about the impact that Faro Health is having by introducing AI into this process for uh pharmaceutical companies.
SPEAKER_00Yeah, well, we're sort of tackling this problem um right at the beginning, which is when um clinical scientists first start conceiving clinical trials and designing them. And so it's sort of a general engineering principle that the earlier on you're able to make changes and corrections, um the higher impact, the lower cost those changes are. And so we published a paper by way of example with one of our big customers, Merck, uh, last year that illustrated how by making judicious design decisions at the very beginning of the clinical trial, you can save upwards of $100 million in the downstream cost of the trial. And so it just shows you like you know a few clicks and you can make these changes to the design, and it has these profound implications over the next potentially up to 10 years during the course of the trial.
SPEAKER_01So, what what is the budget typically expected pre-AI for this clinical trial phase of new development?
SPEAKER_00Yeah, well, it's really kind of interesting. I mean, the cost of clinical trial of bringing drugs to market, the overall cost of developing the drug and running the trials and doing everything necessary to bring a drug from conception to market is of the order of one to two billion dollars. And these trials oftentimes run 10 years or more. And this has actually been getting worse over the last 50 years. There's this kind of um kind of ironically named E-Room's law, which is more's law spelled backwards, which basically says that every nine years the cost and time required to bring drugs to market doubles. And and so um, you know, there's a lot of um concern around this. Obviously, it's it's caused delays in terms of really groundbreaking new treatments making their way into patients' hands. And so um we are looking to use AI to bend Moore's law, to bend ERUM's law, I should say, um, downward, and really start to counteract this trend of much more expensive, uh ever spiraling costs. Um and so we're doing that in a couple of different ways. Um one of them, one obvious way is by automating clinical writing. So there's this whole plethora of documentation required um by regulatory authorities, um, starting with the protocol document, uh, which is a very complex document that involves a lot of sort of structured data as well as um lots of verbiage around describing certain safety aspects and the goals and rationale of the trial. And there's many, many different sections that need to be drawn up by a team of clinical writers. And so we're using AI to greatly accelerate that process from a matter of weeks or even months down to potentially you know 20 minutes to get the first draft. So so um that's just one obvious example. I mean, there's other things we're doing that I think are actually in many ways sort of more interesting in terms of using AI to optimize the very design of the trial and suggest what uh activities and and how they should be used in the in the trial itself, um, as well as performing automatic research. But it gives you an idea. I mean, there's there's just this whole green field of opportunity where we can apply these language models to really greatly improve the process by which clinical trials are created.
SPEAKER_01You know, so I for the listeners, I I think this is a good point. Now, this is something that's very complicated, sophisticated, technical, and if they're able to leverage AI to be able to compress that cycle from weeks to days, let's say, um, just think about what it could be doing in your business if you're not doing something as technical or precise as clinical trial, uh pretrial documentation and things like that. So I think that's a fantastic opportunity. So, Patrick, before we jump in, I I didn't know about your work with uh Google Duplex, and I'd like to ask a few questions um specifically about Google uh about AI voice. And I don't know if you are paying much attention to uh what's happening in the space, but uh currently, right now, if I wanted to, I could spend twenty to fifty dollars per month to be able to get access to a voice engine that would be able to take uh human uh voice input and reply with synthetic voice output pretty quickly. Have you been paying uh much attention to the developments in that space in particular?
SPEAKER_00Yeah, I mean, um this technology has been around for a while actually, like it predates large language models. And at the time we were building duplex, uh we were kind of transitioning from using voice recordings to voice synthesis. At the time, um the quality wasn't quite there. Like we really, really wanted that system to be extremely lifelike, and so we were using kind of recordings that we would sort of um stitch together in a very s intelligent way. Um but but nowadays, and for some time now, actually, um AI-based systems have been able to synthesize voices really convincingly. And you can I remember was this kind of use case of, I don't know, I can't remember who the celebrity was. Was it Matt Damon or something where they were kind of making Matt Damon say anything and then they can make him sort of speak French, which I'm to the best of my knowledge, he he maybe can't. Um so it's sort of like it's pretty um both awe-inspiring as well as slightly scary, right? Like we're in this world now where and and you know, now there's videos as well where you can sort of paste someone's face onto a video and make them sort of do or say anything. Dance, yeah. So so you know, we're into this world where it's certainly going to be very disruptive and feels like entertainment and advertising and and maybe even, yeah, I mean the field of acting, who knows, and modeling. We're already seeing cases where there's digital AI-based models. Um so so you know, this is just one little slice of disruption that all this AI technology is creating.
SPEAKER_01Yeah, that's an area that I'm I'm paying a lot of attention to because in my usage of it, AI voice isn't quite there. The latency they advertise is supposed to be less than human latency when it comes to a response. However, in practice, I'm not seeing that. So I'm always interested. That's not the topic of our conversation, of course, but I'm always interested in um I want to hear somebody say, yes, it's ready. Because I've got a lot of applications backed up for it waiting for that. So um so Patrick, let's talk about not specifically um what Faro Health is doing right now, but how you guys internally are using, let's let's just stick with generative AI in the conversation. Obviously, you you're you're demonstrating to clients that we can compress uh a process from weeks to days. Uh how are you guys seeing that impact internally with just operating and scaling the business?
SPEAKER_00Yeah, I mean, we've been really aggressively kind of adopting AI all over the place uh in our organization. I mean, there's so many different things it can do for an organization to kind of become more productive. I mean, starting with engineering, right? I mean, the obvious use case with all this large language model kind of revolution is coding. And so we've certainly used AI to produce prototypes, to vibe code prototypes and evaluate them. Like it's a great way of quickly building um some kind of you know feature concept and getting in front of customers and getting feedback. Um, we're also using it to you know write unit tests and do other forms of development, like you know, when we start encountering a new API or a new sort of system we haven't that no one's really um used before, we can use AI to very quickly build a proof of concept sort of integration and access that system really easily without having to learn all the necessary API calls. Um but even beyond engineering, I mean, certainly QA and DevOps, which are adjacent areas to software engineering, um, we're using AI to accelerate and automate the production of tests and code and IAC and infrastructure as code and things like that. Um and even beyond technology, you know, we have um UX designers and we have salespeople using AI to perform research and to um you know put together, maybe even to help craft slides or emails and things like that. And so the possibilities are really kind of endless. I mean, there's there's just so many different ways in which this technology can be used. And the fact that it's also multimodal, where it can take images as well as text as input, and it can produce images as output. Like it makes this technology so flexible and so usable by non-technical people that I'm seeing it's it's just different from any other technology revolution I've been involved in. It's really different from the Dark Arm boom or the crypto boom or many other kind of like um even the image uh recognition boom, there was a sort of a smaller AI revolution ten years ago, right? Where people were saying that where there were all these new capabilities coming out as far as recognizing people's faces and generating images and things like that. Even that's been completely eclipsed by this technology because it's so accessible.
SPEAKER_01So you've been around this for a while. Um what are your thoughts as far as the acceleration of it? Do you think that we're that that the acceleration is increasing? Do you think that we've probably seen um kind of maximum velocity when it comes to uh improvement of the models and uh accuracy of their output and things like that? What are your thoughts on that?
SPEAKER_00Yeah, I mean, I am not in the camp of people that think that large language models before too long will be you know AGI, artificial general intelligence, and we'll be, you know, we'll be having these kind of philosophical discourses and they'll be creating masterworks. Like I don't believe that. Um and there's a few I think there's a few reasons why. The first is that um there's only so much, there's only so far you can get by analyzing web pages and and even sort of books out there. Like I think we've already got to the point where a lot of the useful information has been processed. And so um it's not like we can sort of you know have future generations of these systems ingest ten times more data in order to become smarter. So we're we we are starting to hit the limits of I think how much different useful data we can feed these things. Um in terms of really rich textual input, not in terms of quantitative information, which there's you know vast amounts of every single day. I think another is is just more inherent to the design of these systems themselves. Like, I don't know if you you or your your viewers have heard of ARC AGI, but it's this kind of test for quote artificial general intelligence. And if you go look at the ARC AGI site and look at the kind of tests that they're giving these systems, you would probably, on the face of it, think, wow, that's it. That's all it takes. That's how we're defining AGI, like moving blocks around on a screen. Like it's kind of stuff in many cases that kind of a smart kid could do, because all you're doing is sort of doing pattern matching and this kind of thing. But what it illustrates is that even though these systems are capable of generating these convincing looking text um uh paragraphs or these convincing looking images, when it comes down to it, their ability to creatively think, like to solve problems that they haven't seen before, is very, very limited. And um, Apple also um published a paper earlier on this year. I was just sharing this around the office actually yesterday, where they talk about how even though these systems are so smart, they can't even play something like Towers of Hanoi, which is a relatively simple kind of logic game, past about nine or ten different stages, it just totally falls down. They, you know, they said it totally collapses, even if you give them the algorithm itself for solving this problem. And so I think, you know, if you if you and I know this is not entirely fair, but if you characterize these systems as being sort of super well-informed, glorified, autocomplete, where they're taking a string of words and predicting the next most likely um word, there's only so far that that approach can take you, in my opinion. And so I think while, yes, there's gonna be disruption and disruption in the job market the same way there is with any technology evolution, the chances that humanity is gonna be completely usurped by large language model-based systems in the next three to five years, which many people out there in my field actually believe. Yeah. I don't think it's gonna happen. I do think there are safety issues for sure that we need to be aware of, but that's just I guess that's my position on the matter.
SPEAKER_01You know, you mentioned that three to five year timeline. I think that when GPT-5 came out in the in November of 2022, people were talking about a three to five year timeline then.
SPEAKER_00Yeah, well, you know, the i there's this kind of joke in the AI world that got blown away by large language models where for the last 50 years everyone's saying that, you know, general intelligence is maybe 20 years away and that's still held over the last 50 years. So now, you know, I guess Silicon Valley came along and it's like, well, now it's three to five years because we're faster. But it's still the same kind of effect, right? Where it seems around the corner for a long, long time.
SPEAKER_01And it's interesting because when when that came out, people were, you know, amazed at what was possible with 3.5, which when you compare that to what you can do with some of the uh capabilities of Gemini or Claude with the coding or just GPT with general capabilities, it's a lot better for sure, but it's not it's not what one would expect on a three to five year timeline, like, oh, we're only three years out from major. I mean, that that's my opinion, and I guess it might be a little biased or you know, because I'm immersed in this. I use it and have been active power user for oh, you know, since I got my hands on it, really. Um so I I don't know, just just an interesting observation on my part. So I'd like to talk about um what you guys have been able to do because this is uh an AI native company, right? Like you guys the thesis I would imagine when originally launching was leveraging AI to compress these cycles of clinical trials. Um were you able to find talent that was I guess because you're in San Francisco, right? We're actually in San Diego. San Diego, oh great. Um better weather. Um were you were you struggling to find talent that was uh AI fluent enough to be able to really contribute to what was happening in the company?
SPEAKER_00Yeah, I mean I think first of all it's worth pointing out that we were not an AI startup from inception. Okay. So we were we were more of a kind of a traditional SaaS company software as service. Um and so in the last couple of years, we've been pivoting and in some ways actually cannibalizing our own kind of product in and team in order to really heavily pivot towards AI, to the point where most of our new um product engineering efforts are centered around AI. Uh and so that necessitated both retraining existing um engineers as well as hiring specialists, hiring people who had actually implemented AI features in the past and had a data science background. And to answer the second part of your question, yeah, it was hard to find people actually, despite the fact that everybody sort of has been learning about this and is trying to become an AI expert. So what I what we observed is that there was this kind of effect where people would really dress up their resumes to look like they were AI experts. Yep. Um and and in fact had done enough reading to sound kind of convincing. Yeah. But when it with but we we sort of, you know, I I was lucky enough to hire someone really good in the first year that I was at the company. And so together we spent about four months trying to find really good people to form the nucleus of his team. And the the clincher was just having an actual coding test where we were asking questions, we were asking the candidates to do things that you had to have real experience actually implementing these AI systems in order to do reasonably well on. And it was just surprising and somewhat shocking like the number of candidates that kind of fell down. Um and I didn't even really consider the coding test to be that hard. But it was it was kind of like you know, like you you couldn't fake it. And so it took us months and months to find people, despite all these promising-looking candidates with really good resumes. So I I don't know what what your users can, what your kind of listeners can take away from this story, but I think a lot of it is look, if you're gonna become an AI, if you want to pivot into this, um, into this field, really gain some experience. It doesn't have to be professional experience. You can do your own projects, but sure just know what you're doing. So when people ask you a technical question, you don't fall down.
SPEAKER_01You know, I I think one of the points that I take away from this is that companies are you know, anecdotally and in the media and everything, you're seeing that companies are looking for talent that is AI enabled, right? However, if you're not already uh an AI-enabled individual, it might be hard for you to vet candidates, just like you said. I mean, I can get on there and uh here's the job interview or here's the company, here's the job description, you know, write up my resume kind of thing. And I'm sure I I know as a matter of fact that that's happening. So this idea of of identifying a few tests, essentially, the equivalent of a coding test, but for HR, for sales, whatever it is, when somebody comes in and says that they're an AI capable user of this and that role, I think that's a fantastic idea. How would you um I I guess with with a tech background, it might be hard for you to answer that, but how would you recommend people go about, let's say we've got an expert on here or a professional on here listening? They they like that idea, they want to introduce that into their their vetting and hiring process. How but they don't know, like they're not actually users themselves. What did you guys do that they might be able to borrow for their own process?
SPEAKER_00Well, I I guess overall FIRO's strategy was to sort of hire me, and then I I had the wherewithal to actually be able to pick kind of like a senior leader to to lead our data science and AI initiatives. And then he had the the skill set to vet you know individual contributors on this team. And so one strategy is to say like bring in someone senior who has a proven track record and kind of can come with synties and references. So even if you yourself are unable to fully technically vet them, um I mean, you know, another strategy is to bring in people you know who can as an advisor or as a favor or whatever, right? Yeah. And so sometimes our our like sometimes some of our investors do that with me and say, hey, would you mind interviewing this candidate for another portfolio company of mine? Well, of course, you guys are investors. I'd love to help you out. Yeah. You know, and so that's another strategy, is if you know someone, or if you know someone who knows someone who's really good and could do that favor, you can bootstrap that way and bring in someone good, and then they can help hire people from then on.
SPEAKER_01You know, I hadn't considered that, but that's a whole nother industry right there. Like the AI expert who supports other companies and vetting candidates for senior positions that require or would expect an AI literacy level. That's a great idea.
SPEAKER_00Yeah, definitely worth the investment. I mean, hiring is the most important thing you do, really, when you're building a company. So it's worth the extra effort and investment and calling in some favors, maybe even to do that. And if you do have if you are a tech company that's VC you know funded, then a decent investor will will definitely help you with that too.
SPEAKER_01And they'll probably know some people. Yeah. For sure. Absolutely. So as far as training of the the the staff, what are you guys doing to make sure? that because as you know, like and and a reference to Moore's Law, if somebody um maybe they went and got a certification, right? But they got it two years ago. Unless they've been actively on the front lines of paying attention to what's happening with developments and capabilities of the tools, uh they're not going to be current. So how are you guys making an effort to make sure that your teams that are expected to be using this stuff are like knowing that this tool's not ready yet, but oh try it in a month.
SPEAKER_00Yeah, I mean I think the biggest thing you can do to keep up is to hire people who are really curious and motivated to keep up. You know so so for instance like we didn't have anyone do all these big courses or certifications or anything. Yeah. We we did, as I mentioned, we did hire um a small team that is specializes in this. But for the most part for the for the existing engineers who are not data scientists, we basically said, look, we have these really challenging problems. Like for instance one of them recently was um building a system to take an existing PDF, like it's a super complicated multi-hundred page protocol document, and parse it and figure out like what is the structured digitized representation of this trial. So it's almost like the opposite of taking the digitized representation and generating you know like a like like a big protocol document out of it. Taking an existing protocol document, parsing it. Really challenging problem. And you know people basically said wow this is really interesting. Let's go do some research let's go read up let's just learn by doing. And so that's what we did and luckily we have a team that has a caliber that's able to actually do that and figure things out on the job and take whatever guidance they need to take or collaborate as as needed with the data scientists but essentially kind of figure it out. Because at the end of the day like as that doesn't using a large language model to solve a problem doesn't require a degree in AI. It just requires you know like experience applying an API of some third party external system and understanding the nuances of prompting and you know evaluating results and things like that. And then you could build a system without even really understanding how the AI system actually works.
SPEAKER_01I'm going to chase a little bit of a shiny object here and and address that PDF. Because I I've seen that to me it was like oh it's a document would be easy to ingest that and understand what's happening. But why PDFs in particular is it because it's it's more like a looked at like an image that happens to have text or what's the challenge with the parsing of the PDFs?
SPEAKER_00So I'm not talking about OCR which is a well-solved problem. Sure. Whether it's a PDF or a word doc doesn't really matter. Okay. What I'm talking about is making sense of what you're reading. Okay so so you know we're we're getting all these all these text blobs in you know from paragraphs, we're getting these tables, you know, even sort of images, references, all this kind of thing. How do you reconstruct the design and the internal structure of a clinical trial or some other you know highly structured kind of um form of data or or representation of something from all the text like that's the hard part. And that involves things like you know for instance at the center of the clinical trial is this thing called a schedule of activities. It consists of all these different activities that are performed on the patient during the course of the clinical trial. What are those activities? Well you know we have a library of those activities but they're not necessarily worded the same way as the as things in the in the clinical trial right and so we have to match them up. This is a process called entity resolution. It's a very you know sort of like a well-worn kind of problem in data science is to kind of normalize things. And then we can do all sorts of cool things like we have these data sets we've ingested that have the cost and complexity and you know blood draw and all these different useful metrics that we can then go and analyze the trial and provide all these metrics and insights and so on. But I mean that's just one example right so it's not just about reading the text of the PDF, it's about making sense of it all. Okay.
SPEAKER_01That helps. Patrick do you do do you do a lot of like speaking at conferences or uh industry events and things like that?
SPEAKER_00Yeah I do. I speak at two or three conferences a year and you know it's uh we're small companies that we have to sort of still be pretty um careful about how which conferences we choose to invest that time and money investment in.
SPEAKER_01But I spoke at the DIA conference um uh earlier on DC and also um scope in in Orlando so um I enjoy it I enjoy getting the word out like it's funny are have you seen a uh an I don't know an evolution of the sophistication of the questions that you get after you come off the stage from people?
SPEAKER_00Yeah I mean definitely over the last last couple of years I've observed kind of like people come in being more versatile in just how do large language models work, what are the ins and outs of applying them I see you know other speakers and other vendors kind of with some of this starting to use some of the same concepts and terminology like the need the critical need of evaluating the results of the large language model in order to kind of prevent hallucinations and omissions and things like that. And so definitely things are as you would expect with all the attention and all the capital being poured into this space um people are becoming more mature in terms of their uh questions that they ask and and so on and so forth.
SPEAKER_01But that said, you know, these live language models are complex and there are lots of nuances to getting them to work properly and so it it's sort of it's always a bit of an interesting design kind of um challenge to figure out the right level of how to pitch things like how do I communicate things in my presentations so that they they pick it up they can actually understand yeah so one of the things obviously we we uh primarily focus on a non-technical use of AI like you've already got operations that are occurring how do we introduce AI to accelerate or improve the the quality of the of the uh the user when it comes to their deliverable that they're responsible for for their role and one of the things that we noticed we do a lot of training and one of the things that we noticed is is before the training we would always ask people to self-identify on a scale of one to five where are you with your understanding of generative AI and how to use it the business. And in 2024 it was a lot of ones and twos. Starting this year we started seeing a lot more threes and fours and I I thought that was a sign that okay great people are using it but upon uh investigation like oh tell me how you're using it they're they were using it for basic things such as writing emails or summarizing documents things that happen a lot in the you know in knowledge work let's say so it it I I I realized that yes they're using it more but they don't quite understand what's possible but because they're using it they're self-identifying as threes or fours right higher up on the which which I would still say that falls into the the two category let's say um are with with the conversations that you're having with people when you when you're presenting or when uh you're you know you're at a cocktail party or whatever and people talking about it that say that they know the stuff do you feel that you're like okay yeah they're they're on parity with some of my people or do you think that this self-identification is like that same experience is happening on the technical side? Does that question make sense?
SPEAKER_00Yeah I mean I think um it's interesting you know like there was this study that that we've that we found recently where someone had actually examined programmers using large language models to automate their work and so on. And the results of the study were that actually using AI to code is kind of a detriment um for the most part until you hit about 50 hours like once you've been interested you know once you've really kind of taken your lumps and tried a bunch of things out and you know failed and learned a bunch of lessons the hard way then there was this there were a few people maybe one person really who was significantly more productive and had 50 hours under his belt and had a lot of interesting observations about the ins and outs of using AI during the course of the study. And so I think that kind of generally holds like you know if you're a smart person and you're trying a bunch of things out then once you hit a certain critical threshold maybe it's actually saving you time or maybe the results you're seeing are actually really worth it. But it it takes time like it like any new tool like this is sophisticated. You can't naively go in there. I mean I think I think there are certain things that it's really you can get some immediate quick wins with like summarizing text or generating a convincing sounding email that you then go and kind of handcraft, especially for people who don't particularly like writing like that's that that can be a kind of a quick win but as far as actually performing really valuable work there's there's a lot of research going on around this like sort of you know the complexity of tasks that AI is able to actually reasonably kind of automate and add value to is still relatively low. And so I think I think there was another study kind of talking about like what is the length of tasks that AI is able to effectively kind of um automate. And it was of the order of maybe I don't know a couple of days to maybe a week. And so when you're talking about a longer term project that requires a lot of strategic thinking and so on, then AI still doesn't really help with that. And it's also sort of dovetails with what I said before about once you had a certain complexity of like the Taraf Hanoi problem where after 10 turns like the the LLM starts collapsing. And so there is this frontier of complexity that the AI is able to actually sort of comprehend. And probably a lot of people out there in the audience have tried this right like where you try to pass something increasingly sophisticated like a whole you know book or like these days the LLMs can supposedly handle a lot of tokens right and so you try throwing on a whole you know even like a complicated McKinsey report or something and it's like does it really understand what's going on? You know yes and no right it doesn't always get the nuances. And so I think this is what we're talking about. And so what we found was that in many cases you have to really divide and conquer. You've got to take the problem break it down into much simpler subforms and then you've got to take the results and then kind of synthesize them and build these kind of agentic architectures to do this. And then before you know it like it's actually taking quite a lot of human you know effort to actually put these agentic systems together and you've got to be something of an expert. It takes hours to many many many dozens of hours to kind of become good at that. Doesn't require a degree in computer science doesn't require you to do super complicated courses on AI and Andrew Eng, Stanford understand what's going on behind at all. It requires real world experience of just trying stuff out and being smart about the results that you that you find.
SPEAKER_01That's fantastic advice so for any of you that are listening that um are experiencing some frustration on your learning curve, keep practicing. Keep experimenting that's how it's done. So you know I was going to ask that Patrick so if you've got this complex uh operation that you want to agentify or AIFI in your business you know my my question was going to be would it be better to look at that process in chunks and then address it chunk at a time so that output from one day or week long process where it started to kind of cap out on its capability, that output would then feed into another chunk of that process.
SPEAKER_00Yeah I mean what you're describing is the very rationale for agentic systems in the first place is that, you know, in in in sort of if we could wave a magic one you just describe at a high level the problem and then the then and then the the allowed language model sort of come out with the results, right? But for a number of different reasons that just doesn't work, including the AI's limitations as far as the complexity of problems it can handle at once, but also including kind of like understandability. Like you kind of want to know what's going on inside the inside the system if so you can debug it if there's you know bad results coming out and maybe you want to divide and conquer and say hey you take this piece, you take this piece with your team you can't do that if it's one big monolithic prompt, right? And so for a number of different reasons this agentic architecture is becoming super super popular out there with good reason. You know and so yeah in general in life breaking down your problems into manageable chunks. Like with software engineering you know the best practices if you have a task that you think is going to take more than a week break it break it down. Ideally into tasks that take you know a couple of days um certainly anything over a week you've got to break it down. This is a standard agile kind of guidance. And I think the same thing applies you know like in the abstract to many different problems that are out there like divide and conquer. And we've we've been doing that with very great results with what we're doing at Firo Health with you know clinical child design.
SPEAKER_01So I I know that San Diego is uh a high density um life sciences uh pharmaceutical uh all that type of thing so in in your ecosystem of San Diego as you're interacting with you know at at industry events that might be local or talking to somebody a friend who works at another company that's quasi-competitive or whatever um are you seeing certain roles within the leadership that tend to be more AI enabled and some that are kind of lagging? Obviously you're a CTO so that you know by default I would hope that other CTOs were as fluent in the understanding of AI as application as you are but have you noticed any kind of trends on on which other outside of the technology space which other executive roles tend to be really grasping the concept and like the theory and strategy of AI?
SPEAKER_00Yeah I mean a couple of obvious examples are chief security officer type of roles right where now what we're observing is that a there's this kind of growing awareness that there's safety issues and privacy issues with with these large language models, right? And so we've really been spending a lot of time putting together an AI governance model. And there's various regulatory standards arising around that as well but just what happens like how do you avoid people how do you avoid the large language model from sort of mixing data from different customers or how do you how do you ensure that you know rogue employees or rogue users won't come along and start doing this prompt injection attack to make the to make the system do bad things, right? So this is becoming a big thing and every C CISO, chief security officer or chief information security officer out there needs to start thinking about this. Another obvious example is product like if you're a chief product officer you need to be putting together your AI product strategy. So even if it's not if it doesn't involve coding or whatever, just figuring out like how is your product going to incorporate AI. I would say people involved in strategy, of course this is a classic technology disruption scenario of the kind that Clayton Christensen wrote about in the innovator's dilemma, right? Where the laggards you know who don't adopt this technology are going to get left way behind. And I think it's sort of more intense than the usual disruption because it's not like you know hard drives where it's sort of a faster horse. It's kind of like this is dramatically changing a lot of different you know industries at once and a lot of different roles at once. And so it's going to have more far-reaching consequences than some of the other um technology sort of disruptive innovations that he talks about. So so anyway like it gives you a sampling I think at this point every executive should be thinking about how they can be using AI in order to whether it's you know further the business, cut costs, you know, improve productivity, whatever the case may be because if you won't then your competitors will be you know so you you mentioned that you guys are paying a lot of attention to governance.
SPEAKER_01What does that look like for you guys? Like what are the main concerns and how are you going about I identify like even thinking about what the issues could be and addressing those?
SPEAKER_00Yeah I mean some of the ones I mentioned before you know like everybody we talk to every customer potential prospect prospective customer we talk to is asking what happens with my data? Is it going to be mixed with our competitors data? Is it going to be safe? Is it going to be subject to you know the usual sort of um safety um regulations and process and so on. How do you develop your software? How do you ensure that the models you're you're developing, the systems you're you're you're launching are of high quality you know like it's different from normal software. You can't just compare the result with an expected result and say, well you know is this what I expected or not? Because generative AI is non-deterministic. Like you can't expect the same results with the same inputs. And so there's this whole you know thing around process around evaluating results and scoring the results and ensuring you don't regress and things like that. And then increasingly you know the same way that people can craft really um kind of malevolent sort of queries or enter malevolent input into a web form and potentially do a lot of damage on the back end of the system same thing holds here right we're giving AI all this power now. Like we have these agents out there that can access file systems and do all this stuff like there are now prompt injection attacks where people can craft prompts that are designed to make the AI do bad things and access data it shouldn't and say things it shouldn't. So how do you protect against that? So this gives you a sampling right there's there's much more to it like we're we have a whole you know senior person on our team that does nothing but think about this. Wow so I think it's just really important for companies to get ahead of this because there will be disasters. There will be things that happen where oh here's an example of some bank that you know made all these bad transactions or something that happened as a result of having an AI chatbot on its server on its system or something like that. There's going to be some kind of series of events like that that really greatly heighten the um the the perception and the and the risk around this and so best to best to get ahead of it, especially if you are a company like us that's selling to large enterprises where the stakes tend to be really high.
SPEAKER_01You know one of the questions that we get a lot from again we're we're focused on non-technical businesses typically is the models using data to train and how do I how do I know ChatGPT isn't using my stuff or how do I know that my information will stay secure? Do you get that question when you're talking to non-technical business peers? And if you do, how do you explain your comfort level with the security of using my IP to in a in a generative AI model to improve it or to evaluate it or to that sort of thing?
SPEAKER_00Yeah I mean we've already seen cases where cleverly engineered prompts have been able to reveal some of the underlying data that there's been used to train the models and of course the large language model vendors themselves OpenAI and Anthropic and companies like this, Google with Gemini, they're all it's in very much in their interest to prevent this from happening, right? Because they're they're all for the most part kind of cloud providers and they understand the stakes here. And and I think to answer your question like yeah every single customer we talk to um asks this like aren't you going to be handing your data over to OpenAI and then therefore anyone using ChatGPT will be able to access it? And so there's a there's a number of different levels you can answer that question. I mean one of them being well wherever possible we use sort of private instances of the large language model that are only accessible to us. And so it is therefore impossible for the public version of you know ChatGPT to have access to any of your data. And then we have sort of various policies and assurances from the large language model vendor itself that they've made public and you know basically need to need to abide by. And you can argue by analogy like hey sort of this is the the same level of safety as using say Google Docs or Amazon AWS and so on. And so there's just a number of different I think assurances and arguments and evidence that you can provide to people to and to to assure them that this is not going to happen. Because obviously for many companies including sort of especially those whose um kind of like competitive differentiation is their IP, you know, whether it's a joint company or whether it's a finance company or whatever this is the crown jewels, right, that we're talking about. And so if your system handles those crown jewels, then you have to be really sure that they're gonna be safe.
SPEAKER_01Yeah you know I I had that question we were sponsoring an event for Vistage which is a kind of like a CEO peer group network in Chicago yesterday. The question came up from every single person that I talked to yeah and the way that I would kind of position it is that this is a very the frontier model environment is extremely competitive a lot of money going into it there's a lot at risk everybody wants to be the recognized kind of champion of that like this LLM is the the Kleenex or the Coca-Cola of you know using LLMs. And everybody goes oh well they say they you know they kind of roll their eyes and go, they say they won't use our data but how do we know? How I look at it is that our incentives are aligned. I don't want you using it. And if one case came out where I had the settings configured correctly we followed the best practice yet my results still showed up in somebody else's output if it was if that happened with OpenAI, Google would exploit that anthropic would exploit that they would say see look look what's happening over you can't trust them you can trust us. So I kind of see our incentives are aligned. I don't want you to use it you don't want to get caught using it. So therefore you're gonna do everything you can to protect my data and I think between what what your position is and kind of how I've been looking at it that's going to be a great response for me to use when this comes up because companies you know I saw a meme the other day or like a little comic strip and it was like uh who are we and then the other side was CEOs. What do we want? AI when do we want it now? What do we want it for? We don't know right like like there's just yeah there's this FOMO kind of in the marketplace and I obviously you and I understand that you can compete again as an AI enabled company can compete with companies that have 10 times more resources, bigger brains, all that stuff if they're not using this. It's like like you said, it's not a faster horse. It's a different completely different paradigm.
SPEAKER_00Yeah I actually have something to say in response to the early part of what you just said actually Chris which is that I think it's incumbent upon like if your company is really using AI in a way that um it's going to be handling private information of another customer or or or person or whatever. It's incumbent upon you to really research like look at the providers look at AWS look at you know Azure look at OpenAI look at you know um Anthropic. These are all the main vendors right there are others of course um Google and so on but just look at what their policies are look at what they commit to and make your own decisions. You know it's all a matter of public record. Like you can go to Anthropic site for instance and see this is our enterprise offering and here are all the privacy sort of assurances that we can offer you. Or look at this is how Azure like gives you this kind of private instance of GPT and and so that's why this is the best solution. And just make your own decisions like There's there's offerings out there, but I think it's incumbent upon companies to really sort of do the research and can and compare these different vendors and figure out what's best for them.
SPEAKER_01Yeah. And for those of you those of you listening say, I wouldn't even know how to evaluate that, take those policies and review them in a large language model and ask, here's my concerns based on this policy. You know, um are are we in a a risk-averse environment or a safe environment for us to be using these models for our stuff? I think that's great advice.
SPEAKER_00If you do that, best to use someone else's large language model to evaluate a given vendor. Yep.
SPEAKER_01Uh eliminate that bias. So out outside of um in in your department, but also across the company, I'm sure that people, other executives in your company look to you for uh perspective or counsel when it comes to how they should be approaching AI and their areas of responsibility within the business. What are you telling your own peers within Faro Health when they come to you for that kind of advice?
SPEAKER_00Yeah, I mean, I think by and large, it's been really just be very, very curious and try things out. Because many times our assumptions about the way things work and the best way to do things are challenged and potentially disrupted or disruptible by this new technology. And so rethinking potentially the way that we do QA or DevOps or, you know, um early stage prototyping and so on. And I think that the most important thing is you learn is you are kind of quote quote sort of failing fast, meaning just trying things out and seeing what works and what doesn't work, and basically learning uh what the best way for your organization to really use this technology is. Like I don't think there's any hard and fast rules about which model to use. Like we're we're using a bunch of different models. They're good at different things, they have different strengths, you know. Um, and there's trade-offs because as we know, these models can get expensive. If you're talking about the state-of-the-art, you know, open AI or anthropic models, like they they cost 20 times plus more than the the sort of run-of-the-mill standard models. And so you have to be really judicious, especially if you're a high volume business, like like a sort of a consumer-facing service. But if you're using it for internal use, maybe that's less of an issue. But just keeping tabs on what the best model is to use for each task is really important. Um, and I think just evaluating, like not blindly trusting the output of the model. Um, that's really important too. Like, like if you're using it for vibe coding, which is so tempting, especially if you're not a if you're not a technical company that has software engineers and you're kind of using this technology to really sort of like um build software maybe for the first time, just be a little careful. Um maybe try to get someone who knows what they're doing, like a software engineer, to take a look at the output of the code, because sometimes it can be amazing and sometimes it can be a real mess, you know, and like a liability, basically. And so the difference between having a really well-designed system and one that's like sort of spaghetti code, um it can be the difference between totally worth it versus forget it, rewrite it from scratch.
SPEAKER_01And I know that the answer to this question I'm about to ask you could change next next month with um as fast as these things develop. But have you identified one model in particular that you feel is really doing a good job at supporting the development of code or the vibe coding?
SPEAKER_00I mean, it's interesting because in many ways, um, vibe coding slash you know, using AI to code is the ultimate use case for AI. Yeah. Because it's sort of consumer-facing, like there's millions and millions of programmers all around the world who can benefit from this. You know when the output is right, because you run the program and you can see did it work or did it not work. So you get this instant feedback that results in rapid evolution, and it's high value, really high value. I mean, software engineers are highly paid. And so making them more productive is super, super valuable. And so that's where I think a lot of the innovation is going because of these three factors I mentioned, there's probably more than not even thinking of right now. Um and so yeah, I I just think that um that's an area where it's really things are evolving really fast. And there's no there's no one model I point to. Like, you know, for a while it was sort of like, oh, Opus 4, you know, this is the way, this is the way to go. But then I was playing with Gemini, and I thought, like, actually, for what I was doing, Gemini was better. And now, and now there's kind of like what is the latest OpenAI? There's some GPT code thing, codex or something that as well as people say, oh, it blows away all the other ones. And so it seems like every single month there's a new kind of horse to beat in this race. And I think it's just like, just try all of them, see which one works for you. They've got pros and cons, cost, time, capabilities, and just figure out what works for you. They're all good. They're all gonna help you. You know, I've tried I've tried all three of them. Gemini, and obviously there's more than just just three, but Gemini, you know, GPT, and um Anthrobic Claude, and they all they all can do a great job. Just I've just found for that one project that I was using it for that Gemini was the better. Yeah, because I'm an ex I'm an ex-googler or anything, but it just happened to be there. So one little tip, right? If you are into vibe coding, definitely make your system write a decent spec. So I did this thing where I I asked Anthropic Claude to do that because I was having trouble getting this system working to basically build build what I wanted to build. It wrote this really nice spec. I fed the spec right into Gemini and it reconstructed the app very easily. So you can very easily kind of like kind of cover your bases and try out different models by having a spec that essentially serves as the ingredients for the instructions to actually build the system from scratch. It's good practice anyway to have a good spec, right? Because it helps us understand things, especially as your system gets more complex, it helps people understand things.
SPEAKER_01You know, and I think that that would extend outside of just the technology as well, like any any type of strategic work that you're gonna be doing with it, like have an understanding of what you would categorize as ideal output.
SPEAKER_00Absolutely. I mean, you know, we're doing all this really complex work with respect to analyzing protocols and figuring out how do we optimize the design. And this has nothing to do with coding, really. This has to do with like quantitative analysis and medical sort of expertise and things like that. So, you know, uh, we have, you know, in some cases our CEO is actually he's a clinical expert and he's not a coder, and he's building this incredible, he's helping our system, have our team build this incredible system that does this detailed clinical analysis, right? Um, how cool is it to have a CEO that can do that? Yeah, right. You know, but it's not like he did a computer science degree to do this. And so I think, yeah, for sure, um the things that I'm talking about that apply to um coding apply to a lot of other problem domains where you're trying to really use the large language model to solve these complicated domain-specific problems.
SPEAKER_01So, Patrick, one of the the I guess things that has come up throughout this conversation today has been your um emphasis on curiosity, right, when it comes to using these things. So for anybody that's out there that isn't a CTO at a cutting-edge, you know, biosciences company, this curiosity extends to what the domain that you do know. So if it is uh executive level leadership, whatever area that you're responsible for, take that existing experience and discretion that you have on that topic and use that to guide your usage of the models as you're getting your 50 hours or your 500 or your 10,000 hours. So, Patrick, do you share a lot of your perspective on this anywhere in particular? Are you a LinkedIn guy or what?
SPEAKER_00Yeah, I I I do. I I don't actually post enough about this, but what you're saying is actually a very deep question, like the nature of curiosity. Um I I have given talks about um career development, and at least what I've seen over the last you know 20, 30 years of doing this, um meaning being sort of like an executive in the technology world, is that curiosity is one of the key traits that determines how far you go in your career. So it's not just about large language models, it's about life. You know, and and in fact, even beyond your career, you know, to make progress as a human being in the world, yeah. Um, and to and to become everything you can be in the world, curiosity is really essential to that. So uh, you know, there's various exercises and practices and all sorts of things you can do to kind of bring this out if you don't consider yourself to be a naturally curious person, like just wondering about the world. I think it is a basic human trait. Um, but just even even knowing that, knowing how valuable it is to be curious about things and just asking questions, super, super valuable.
SPEAKER_01I love it. Patrick, um, this has been a uh fascinating conversation. It was definitely outside of my uh area of experience when it comes to industries. Um, but the information that I've gotten from you today is is I've been able to translate some of these points for sure into my conversations with the non-technical business leaders. So I want to thank you for being here. I really enjoyed the conversation and I look forward to sending you more business.
SPEAKER_00Yeah, thanks, Chris. I really enjoyed the conversation. Thanks for having me on the show.
SPEAKER_01Absolutely. Thanks, everybody. So we'll see you on the next episode. Uh please check the show notes for uh further contact information for Patrick and to follow uh what they're doing at Faro Health. See you on the next episode. Thanks for tuning in to Using AI at Work. Don't forget to subscribe for more conversations about how to use AI at work. And a special thank you to our sponsor, Chief AI Officer, for empowering businesses with AI education and training. Visit their website for a free AI readiness assessment and AI strategy guide to help you get started using AI at work. That's www.chiefaiofficer.com. Follow us on Twitter at the handle UsingAI at work and visit www.usingai at work.com for free resources to help you harness AI in your role.