EP: 09 The "Hallucination" Myth: Vikram Chatterji (CEO & Co-Founder, Galileo) | What the futr

Welcome to Episode-9 of What the futr.

In this episode of What the futr Sandesh Patel and Chris Brandt are joined by CEO & Co-Founder of Galileo, Vikram Chatterji for a candid conversation about the rapidly evolving world of AI agents and what it takes to deploy them safely in the enterprise.

Whether it's rethinking how we define AI hallucinations or building a real-time control layer for autonomous agents, this conversation gets into the heart of how enterprises can govern, secure, and scale AI.

A must watch for anyone  interested in AI agents, enterprise data strategy, and the future of intelligent systems.

Subscribe for regular episodes.

👉 Explore the platform: https://futrconnect.io

👉 Business inquiries: sandesh@futrconnect.io

Transcript

00:00:03:02 - 00:00:09:03
Sandesh
You got to change the game.

00:00:09:05 - 00:00:13:18
Sandesh
Up.

00:00:13:20 - 00:00:25:11
Sandesh
One of the founders that Chris and I have, that I know pretty well, is, Do you know Pugin Kumar from, clue. Meow. Here's the paradox data guy.

00:00:25:13 - 00:00:31:09
Vikram Chatterji
Yeah. The name rings a bell. I don't, I do, I know, I don't know him personally, but the name.

00:00:31:11 - 00:00:34:22
Sandesh
Ryan worked with him for a stint. Yeah, it was another.

00:00:35:00 - 00:00:45:22
Vikram Chatterji
Yes. That's right, because I did a background check on Brian with Butch, and he was a, a Battery Ventures portfolio company. Two or something like that, for sure. But yeah.

00:00:46:00 - 00:01:04:21
Sandesh
Sutter, he was a mike Spicer, an investor and all that kind of stuff. But, and that's how we got to know hugeness through Spicer. Gotcha, gotcha. But what was funny, and I want to get your take on this because I asked him, you know, what is the day of the life of, like, a founder, like a knight?

00:01:04:21 - 00:01:26:14
Sandesh
You know, you're going after a big market, just like you're going after a big market. You know, you're you're you're still trying to get out to get deals done, right at, like, what is it like? And his comment was, well, every day about 9 to 10 things go wrong. And then there's 1 or 2 that go, well, and you just have to get used to that.

00:01:26:14 - 00:01:33:08
Sandesh
You're going to have more challenges than you're going to on successes, and then you just got to get back. Do you agree with that?

00:01:33:09 - 00:01:53:11
Vikram Chatterji
Yeah. Yeah, I hundred percent I'm actually the the winds that we get are so few that that's, that's what kind of carries fuels, gives the energy. But for the most part, I feel like, least I personally, I've just developed so much. I thought I had thick skin before we started the. This is this, like, Teflon on Teflon right now?

00:01:53:11 - 00:02:12:14
Vikram Chatterji
It feels like, you know, it's such a sweet summer child. Five years ago, you know, at this is, it's it's very different. But I do think, like, the team really helps, you know, having a Brian, having a great CMO, a really good person in finance, very good person, HR. So they kind of feel things a little bit and then.

00:02:12:14 - 00:02:23:11
Vikram Chatterji
But if they bring me the grand, bring me last week on Monday morning and it's like, okay, I didn't see this one coming, but here this thing happened. And I think, okay, we can get a call. So a good team on my hands.

00:02:23:11 - 00:02:36:17
Sandesh
So Vikram, a lot of buzz in the market about open claw. So give me a little view of what's your thoughts about it? Why is it so relevant? You know, what excites you or concerns you about it?

00:02:36:19 - 00:02:53:09
Vikram Chatterji
First of all, thank you for having me. And, on open clock, it's it feels like one of those pieces of software that just had to happen, as soon as you hear about agents, the first thing that comes to people's minds is, can I. Well, apart from the fact that everyone starts worrying about what's what does it mean for them and their jobs?

00:02:53:09 - 00:03:15:19
Vikram Chatterji
The second thing that comes to mind is, can I have my own agents? Can I make them do my own stuff? Can I have them order, actually, do my shopping for me and all sorts of other stuff. So I feel like it was a really good, leap in that direction. It got a lot of buzz in the market because it, I think it was done in a really interesting way where it was it's actually really simple.

00:03:15:19 - 00:03:39:15
Vikram Chatterji
The architecture of open doors, very simple. But the fact that it's open source and it's pretty powerful, you can download it locally on your machine, on your own hardware and actually start using it. Was very, was, was was really good from a vanity perspective. But, but so I feel like it was step one of a thousand in terms of personal automation.

00:03:39:20 - 00:04:01:03
Vikram Chatterji
Now it's it's good, but it's clunky and it's, it's hard to kind of get going. I but I do think it's going to move in that direction where I keep telling very telling anyone who's going to listen that when when you think about agents in the future, it really starts getting interesting when, physically AI becomes more of a thing and you're going to have your the dishwasher doing the dishes for you.

00:04:01:03 - 00:04:20:21
Vikram Chatterji
And, you know, those are all that automation is going to absolutely come and there's going to be some kind of a centralized recipe creator for your household. And that's going to be like the future of Open Flower or something of that nature. Right? Where anybody in your household, whether it's your grandma, granddad or anyone could be like, I want to do this, this, this and this and get it done.

00:04:20:21 - 00:04:36:02
Vikram Chatterji
You know, it's definitely going to move in that direction. But it's also scary because if it can do anything for you, then where do you stop and how do you control it? How do you trust it? That definitely this is becoming, the next question that people start to have as soon as they see how powerful it is.

00:04:36:04 - 00:04:39:17
Chris
Yeah. Anything it can do for you, it can do to you.

00:04:39:19 - 00:04:44:08
Vikram Chatterji
That's right. We've all seen the movies, right? It's in the movies where everything becomes red in color and.

00:04:44:08 - 00:04:45:06
Chris
The the.

00:04:45:08 - 00:04:46:18
Vikram Chatterji
Machine takes over.

00:04:46:20 - 00:05:07:03
Chris
I find it, I find a little. Yeah, I'm a little worried about, like, how many people are jumping on the bandwagon at this early stage because I just don't think it's quite, quite ready to just, you know, turn everything over to yet, although, you know, hey, I, I'd like I would be happy to have an home automation thing that could get turning my lights on.

00:05:07:03 - 00:05:11:18
Chris
Correct. Yeah. So I'm. Yeah. Yeah, that's a big leap. So Chris.

00:05:11:18 - 00:05:24:17
Sandesh
Let me just tick on that point. And I agree with you that we're at this very elementary stage where still do we do we want to give it access. So all of our data, like my bank information and my emails and my texts and WhatsApp, I.

00:05:24:17 - 00:05:28:14
Chris
Think it's a little early for that personally. But, you know, for their own I guess, you know.

00:05:28:15 - 00:05:40:06
Sandesh
Yeah. But like, here's the other piece of it and we're going to get into this, right? Is like, when does the confidence come in with AI. Yeah. And we'll you know Galileo plays there right.

00:05:40:08 - 00:06:00:23
Chris
Yeah. And I think the bigger thing is it's like it's so hard to make a judgment on anything because in a week it's going to be radically different at this point. Yeah. You know. Yeah. So it's like it's moving so fast. It's like you kind of have to like even if it's not ready for prime time, you kind of have to keep an eye on it because you know, you don't know what it's going to be in another month or two months.

00:06:00:23 - 00:06:01:15
Chris
You know.

00:06:01:17 - 00:06:30:23
Vikram Chatterji
It's also one of those things where, when something captures imagination of so many people, if, like, the train's already running, you know, like you. Yeah, it's you either join the train or you just get really left behind. And so the same thing happened with, frankly, with ChatGPT also when it came about, I remember when it launched, I was at a dinner with my wife and I was hearing about ChatGPT on the way to the dinner, sat down there, and then there was somebody talking about behind me, and I was like, I tried this thing, but I didn't know how to ask it.

00:06:31:03 - 00:06:56:08
Vikram Chatterji
It was kind of dumb. It was okay, but it but, you know, I heard of a couple of prompts from the person behind me. They were like, I asked you to do blah, blah, blah. And it gave me this perfect response and I came back and tried it out. And so I think people started realizing over the course of a little bit of time about how you can harness some technology and that it gets better and better and better, but, in terms of giving access and stuff, I feel like humans are is all this, this curve where there are lots of people are very privacy sensitive, as they should be, and then there are

00:06:56:08 - 00:07:14:11
Vikram Chatterji
lots of people who are like, here's all my data, do some cool stuff for me. And I do think a lot of the people where we're, we're adopting open cloud radically right now. I'm just giving it unfettered access to there's slack there, WhatsApp, their messaging app and everything else. And just in like, let's see how far we can go with this.

00:07:14:13 - 00:07:35:12
Vikram Chatterji
But but yeah, I do like the fact that coming do the Galileo piece that you mentioned, you know, I always feel like consumer, is, is, is a leading indicator of what enterprises are going to adopt, you know, whether it's search consumers adoption, search big time enterprise sort of guide to that at, at an enterprise scale.

00:07:35:14 - 00:07:52:10
Vikram Chatterji
ChatGPT folks. So like I love chatting with this thing enterprises thought like, can I create a context graph for my entire enterprise and do an internal ChatGPT? What does that mean? And I think the same thing is going to be happening with with these these agent ecosystems to where can every employee be given their own? Yeah. And recipe.

00:07:52:10 - 00:08:10:06
Vikram Chatterji
And that that world of having an agent swarm across your is 100% coming. Well, again, I don't I don't think it's going to replace jobs. All jobs it's going to be everyone's going to be able to augment their own job big time. But you'll have to know how to do it. It's just a new skill that you have to learn.

00:08:10:08 - 00:08:30:03
Chris
Yeah. For sure. Why? And I feel like, you know, people, you know, the conversation that goes on about people use AI like the UK's Google and it's that's not the right way to do it. And I think, you know, as I've got more into it, like I'm realizing that it's like prompting AI is really kind of just as complicated as programing.

00:08:30:03 - 00:08:59:14
Chris
Yes. You know, and having that systems based approach to doing things and then like, you know, refactoring and going back and analyzing, there's a lot of work in that that I don't think people understand really. Yeah. That that kind of keeps them from really, achieving everything that I can, can do for you. And I think the people who understand that, like a really excited about AI and then you have this, these naysayers who just, you know, use it like Google and, you know, it's they're like, yeah it's nothing.

00:08:59:18 - 00:09:01:02
Chris
Yeah. Nothing. Sandwich. Right.

00:09:01:02 - 00:09:25:07
Sandesh
Yep. Yeah. So but before we dive into Galileo because Chris asked that I have a question, what skillset would do you think is most important today, March 9th? Okay. 2026. Yeah. This is going to evolve. Yeah. But I know everyone's been talking about prompt engineering. Prompt engineering. Do you feel like that like for somebody that really wants to understand how this all works, is it better to start on that side?

00:09:25:07 - 00:09:40:10
Sandesh
Is it better to start on like the like. How do you like quantify value with AI? How do you look at the governance and security side of AI? Like what's skills do you think are the most important and valued with AI?

00:09:40:12 - 00:10:00:20
Vikram Chatterji
I mean, I think it depends a lot. First of all, I don't think prompt engineering necessarily is, a skill that I would necessarily start out with, with. If I'm getting into the world of AI and teaching myself, that's very circa 2023 at this point it's important. But the models have gotten much better. The tools on top of the models, the rappers have gotten much better.

00:10:00:20 - 00:10:24:19
Vikram Chatterji
Are the frameworks that you can use in orchestration. Systems are so, so good and easy to use. It's ridiculous how easy it is now. So I feel like at this moment the big thing is and this goes back to how we hire as well and inside of our company, how we think about things every single rule, every single function, every single thing that, a person is obsessed about, whether it's marketing or running a company or whether it's finance.

00:10:24:21 - 00:10:48:02
Vikram Chatterji
The big question that everyone should be asking themselves is, what can I do to augment my day to day, right, with with AI? And so, for instance, in our team, we had our CMO, and our zero and who, you know, worry about, and, and a few other folks, who are not super technical, really, they're building their own tools to make their job easier.

00:10:48:03 - 00:11:11:22
Vikram Chatterji
Right? They're connecting into HubSpot because HubSpot has an MC server. It's those are things like that. Should not be scary to anybody. Like when you hear MCP servers and tools and stuff like that, it's the more you just jump in and you start realizing that you can literally, natural language query your way towards building out, the UI and the front end for something.

00:11:11:22 - 00:11:40:07
Vikram Chatterji
And lovable, you can build like it comes with the few API hooks here and there. Super easy. It's it's it's it's it's ridiculously easy to connect it with HubSpot and all these other tools. And you can create something magical on the other side, you know, so the more people dive into this and just embrace that, this is what I want to build, the ability to build stuff and actually ship stuff and have everybody else use it has reduced so dramatically because of AI.

00:11:40:09 - 00:11:55:11
Vikram Chatterji
So, you know, that's something which I feel like everyone should just internalize and not get. The further, the more time they they spend in thinking about it versus actually diving in, the more the market's gonna start moving and it's going to become more of a daunting task for them to learn all these things.

00:11:55:13 - 00:12:02:10
Chris
Yeah, you definitely have to get into it because it's it's not what it's not what you think it's going to be at the end of the day.

00:12:02:10 - 00:12:19:08
Vikram Chatterji
It's not it's not a black terminal typing in code. It's it's not that. It's in fact, even if it's that you can use Coursera and you can literally like write type in natural language now. So it's, it's, it's it's really, really easy. I feel like, you know, everyone in our company is in sales, but also everyone's a developer now.

00:12:19:11 - 00:12:42:15
Vikram Chatterji
So that's something that everyone should just internalize that the cost of building stuff is it's like the, the, the level of abstraction is now so, so high that it's at the cognitive level of almost every human being. You don't there is deep knowledge needed that it's not like developers and engineers are going away at all, but it everyone can build stuff, which is a really powerful thing.

00:12:42:17 - 00:12:59:21
Chris
Yeah. And I find that like it seems to me that the biggest problem most people have is like, what is the problem they're trying to solve? Yes. Yeah. And like defining that, you know, because I think people have this like loose concept of what they want. But when they start thinking about like, well, what does that mean? Yeah.

00:12:59:23 - 00:13:03:14
Chris
You know, that's where it all kind of comes apart for a lot of people, I.

00:13:03:14 - 00:13:12:21
Vikram Chatterji
Think, which is great. Right? Because now the only blocker is your imagination. And that's where it should be because that's where humans excel. So that's a that's a good place to be.

00:13:12:23 - 00:13:34:16
Sandesh
Yeah. Yeah. I, I have a little side hustle going on with, a co-founder of mine, and he, he keeps telling me he's like, stop asking the how, like, don't worry about the how. Just dream big and tell me what it is that you want to do and let me go figure it out. Which is is nothing.

00:13:34:17 - 00:13:52:02
Sandesh
It's something that I have never. It's hard to do, right? It's intimidating to even dream that big. But so fun. God, that it's just, it's so exciting. It's like, can you really do this? And then when he comes back and says, yep, I can. It's like, prove it. Prove it to me. You know.

00:13:52:02 - 00:13:54:21
Chris
Give me 30 minutes. Yeah, yeah.

00:13:54:23 - 00:14:26:03
Sandesh
So let's get into Galileo today. We have, Vikram Chatterjee, CEO and founder of Galileo. AI. These guys are really making a big splash in their market. They're going after a big problem, with, what the average person would call hallucinations. But it's really just making our AI more reliable and and correct. And secure. So with that being said, Vikram, maybe you can give us a little bit of how did you get started?

00:14:26:03 - 00:14:31:10
Sandesh
Why did you get started? What was the problem that you wanted? To solve? And, we'll dive in.

00:14:31:12 - 00:15:01:04
Vikram Chatterji
Sure thing. So Galileo is an agent observability platform that's big for enterprises. It's being used by some of the largest banks, telecommunication businesses, and a bunch of others across across the US and parts of Europe, as well as a lot of, high growth startups. The problem of agent observability, as is increasingly becoming bigger and bigger, mostly because, the systems are getting more complex.

00:15:01:04 - 00:15:18:08
Vikram Chatterji
The number of agents in the enterprises we're talking before are just exponentially increasing right now, folks, for us folks. Right. Like maybe one, maybe five, maybe ten. But if you talk to them about the plan for the year, it's always like 50, 150. And the, the ocean moment for them is like, how do I know what it's doing?

00:15:18:10 - 00:15:40:09
Vikram Chatterji
But if you the one level deeper, beast from there, which we're really excited about is, and it's kind of dovetails into the history of the company, we've always been excited about what we call probabilistic software. So anything that's built with language models, any software that's built to language models is hyper probabilistic, which the software used to be super, deterministic.

00:15:40:11 - 00:15:59:23
Vikram Chatterji
The whole stack that you need for probabilistic software across testing and monitoring and, you know, runtime stopping and runtime steering is completely different. Like you can't use data for that out of the box. You can't use any kind of, testing tool you used before beforehand for this. So that's the whole market that, that that is yet to be captured.

00:15:59:23 - 00:16:33:01
Vikram Chatterji
And it kind of also dovetails into security to some extent. But, how do you trust these agent swarms in your enterprise? How do you control them? How do you create a single control plane to be able to manage all of that? It's a super hard problem to solve. And, so we built the, the software layer, the AI, data science layer, as well as a very strong platform and infrastructure layer for solving that, that we started five years ago, though, before that I was heading product at Google, I where, Google had come out with, the world's first large language model.

00:16:33:01 - 00:16:49:08
Vikram Chatterji
But back then it was, the largest, but now it's super small. It's called Bert. And it was open source. A lot of startups started using it to build out applications on top of it, but it was small. So you need to train it with a lot of data. It really fine tune it and tell it about the world.

00:16:49:10 - 00:17:11:23
Vikram Chatterji
It was fine. It was trained on Wikipedia data. And so people from enterprises had to give it like hundreds and thousands of rows of data to even hope that it gives the right response and would almost never get the right response. And so to kind of keep fixing it. So my team was one of the first few teams that built a very large scale application for financial services, using that, and was a team of software engineers back then.

00:17:11:23 - 00:17:38:07
Vikram Chatterji
This is 2017, 2020 timeframe. And, the engineers were just perplexed because the responses from this software application that they did was just giving a different answer every single time. Sometimes correct, sometimes wrong. But, you know, our question was, how do you actually test this thing? How do you know, how do I ship this and make sure that our customer Babel and Deutsche Bank and JP and see on the other side are actually not going to say that, look, this this thing really sucks.

00:17:38:07 - 00:18:05:23
Vikram Chatterji
Guys like you. I thought you were Google. And so I was surprised by how they were just no tools. And Google is very good at building internal tools and they're just no tools for this back then. And so that's kind of where the, my co-founders and I, we figured that, look, there's, massive, market risk because, there weren't very few companies that are actually using language models, for building anything back that in 2021, this is February 21st.

00:18:06:01 - 00:18:26:16
Vikram Chatterji
But these are like the the product risk was low in the sense that we knew exactly what we needed, what we wanted. And so we just went out and built it. And I feel like two, three years later, the market kind of met met our met our vision and and you know, now it's I would argue it's either going to be one of the biggest markets in the history of technology because agents are going to be everywhere.

00:18:26:18 - 00:18:48:13
Vikram Chatterji
I think agent based software applications are going to take over traditional applications. It's going to be more in volume. It just has to be because who wants to be deterministic when they can build hyperactive agents? So, that's kind of the main thesis of the problem we're gonna solve, like, as this future unfolds. You're going to be the trust and control layer for all agents across the enterprise.

00:18:48:15 - 00:19:10:06
Chris
Yeah, I think I think you hit on a really important piece there. It's sort of the non-deterministic nature of AI is it makes it like that whole development stack is really I mean, like, I think when you're you're saying it's got to be very different. I don't think people really, you know, may understand how different that has to be.

00:19:10:08 - 00:19:31:10
Chris
I mean, because like the models of testing have to be, you know, completely, completely different. Yeah. You know, and I and I also I was going to also say you know, that's one thing that I keep hearing, they kind of got into the popular culture is the idea of AI hallucinations. Yeah. And you know, as you mentioned before, like these models have gotten so much better.

00:19:31:12 - 00:19:51:22
Chris
And I do find that like the models don't hallucinate much anymore. What they do have a big problem with is alignment issues. You know, and that's a tricky one to really figure out. It's like a tool like yours that provides visibility. You can give you, you know, like, is this a hallucination or is this just, you know, Misalign.

00:19:52:00 - 00:19:53:14
Chris
What what's going on here? Right.

00:19:53:18 - 00:20:10:18
Vikram Chatterji
That's right, that's right. We. Yeah. The way I think of this is it's less and less about the models and it's more about the entire system that's coming together to share an AI application now, because way back, it used to be the model in the problem, way back in the sense like, what was it, two years.

00:20:10:18 - 00:20:16:21
Chris
In a couple of years ago, in ancient times, ancient.

00:20:16:23 - 00:20:36:11
Vikram Chatterji
But that's in prompt engineering became a really big thing because that's all you need to do, right? There's here's the model, here's a prompt, shipped the product, and then they ship the product and said, we now know this is there is no context here at all. It's completely like hallucinating because it's supposed to be about, I don't know, it's a financial analyst job for JPMorgan, but it's talking about Deutsche Bank.

00:20:36:12 - 00:20:54:13
Vikram Chatterji
What the hell? And so then drag became a thing, and people started adding context and knowledge inside of it and decided turning the the the model on the product. Listen to the context, don't guard a syllabus. This is it. This is all you supposed to do to stay within. And that became the new definition of Malice Nation. But even that wasn't performing really well.

00:20:54:13 - 00:21:14:07
Vikram Chatterji
And was a lot of chat bots came out of that. But now with agents there's this concept of tool and memory and a bunch of other stuff. And now the the agent, the agent handoffs, where one agent does one very specific thing and hands it off to another agent. And because of these complex systems, there are many different failure modes where something can go wrong.

00:21:14:09 - 00:21:46:08
Vikram Chatterji
And so that's why I like from an agent observability perspective, we can think of this as you just we just need to diagnose and tell you where things are potentially going wrong, as your agents are trying to, grapple with many different kinds of tasks and queries that are coming its way, right? If you take a very simple example of a financial analyst, a bunch of people or an agent, which is trying to do perform the task of a financial analyst, most people will probably ask it questions like generate a report which does this, this and this, but then all of a sudden, let's say the world changes a little bit more,

00:21:46:08 - 00:22:06:11
Vikram Chatterji
and then it'll say something like, you know, there's a war going on in the Middle East right now. How is that going to affect oil prices? And now the agent is kind of completely thrown off guard and say, oh, this did this. And now it has to learn what to do, right, to figure that out. And so while doing that, it might make the wrong tool call or make that might be the wrong mistake or might just say like, I don't know what to do, but because I'm an agent, I have to do something.

00:22:06:13 - 00:22:26:08
Vikram Chatterji
I'm going to give you an answer. And that's kind of those are the things that you want to you want to pick up on and be able to make sure that it's steered in the right direction. So that's kind of how we think of this. It's not the hallucination problem per se. It's like, how do you make sure that the system is very, very rigorous and you've put the right controls in place all through, and when something's going wrong, can you stop it proactively?

00:22:26:14 - 00:22:31:22
Vikram Chatterji
But if not, then can you kind of actually kind of have extreme observability on the entire system.

00:22:32:00 - 00:23:04:03
Chris
And I think the example you gave there is really interesting one, because like you've got an agent that focuses on, you know, like reporting on financial markets, and then you say, what about this war? And then, you know, that's like from a context perspective that's outside of the context of that, you know, financial kind of piece. And now and one of the things I find that I has a very hard time doing is like switching contexts or operating within multiple contexts, you know, in the process of solving a problem.

00:23:04:03 - 00:23:16:05
Chris
So like, you know, tell me about like, how does like if I was using Galileo, like, how would I, like start to identify those things and like peel them apart so that I could yeah, fix that problem.

00:23:16:05 - 00:23:33:01
Vikram Chatterji
Yeah. There are a couple of ways of using Galileo. So the way just for context, the way Galileo works is, as you're building out these agents, these developers of building these agents, they are a few lines of code inside of their, application. And then we can just do the rest. We can track everything that's happening there.

00:23:33:03 - 00:23:58:08
Vikram Chatterji
As always with these enterprises, Galileo is inside of the VPC of the enterprise, so the data never leaves, for instance. Right. Now what that what that allows us to do is two things. One is, because Galileo becomes a system of record for all each and software workloads, every single log across this particular agent in this in this use case of the financial analyst is going to be going through Galileo.

00:23:58:08 - 00:24:17:16
Vikram Chatterji
So now we have the inputs and the outputs, which tools we use which what what context is. What about chunks. We provided everything. We have all that information. So one of the features that we launch a couple of months ago after a lot of R&D was is called signals. So what that does is it's looking at all these logs and really understanding what, what is this agent trying to do.

00:24:17:18 - 00:24:36:12
Vikram Chatterji
What does good look like? What does that look like? What's the most efficient bus that's been taken? And then based on all of that, trying to figure out what those unknown unknowns are and which is it's basically think of it as an insights engine. Right? And it's been super magical for us to be using it, even internally and from our customers as part of what we've been hearing.

00:24:36:12 - 00:24:37:17
Chris
Is eating your own dogfood.

00:24:37:18 - 00:25:01:11
Vikram Chatterji
Yeah, absolutely. But we've been hearing stuff from them about things like they had no idea that, for instance, there was some PII that was being thrown out by the agents in certain places. There was some tool calls which were redundant. There were three tools that were trying to do exactly the same thing. Let's say there was a in the case of this, what you were talking about, that's probably some kind of a knowledge or search tool, right?

00:25:01:11 - 00:25:15:19
Vikram Chatterji
Which is kind of asking which is being asked by the agent like, hey, tell me about what what context do you have about a war going on right now? And then I do a search call. Maybe there are three tools that are doing exactly the same thing. So this signals feature would just tell them, like you have three different tools.

00:25:15:19 - 00:25:36:15
Vikram Chatterji
One of them's not doing really well. Kind of throws it off, but the other one is doing a good job. But you have some redundancy in here. Here's how you can fix it. Just, you know, this is what you need to do next. So that's a very proactive way to help them figure out the unknown unknowns. Because at the end of the day, with AI and machine learning, it's all about figuring out the unknown unknowns and looking into the data.

00:25:36:17 - 00:25:54:01
Vikram Chatterji
And so we proactively built this basically the signals AI agent to do that for you at scale. So that's one way to do it. The other way to do it is we made it super easy to build your own metrics for any subject matter expert to build their own metrics. We also have a lot of metrics out of the box.

00:25:54:01 - 00:26:13:21
Vikram Chatterji
We are unique in the sense that we have an entire AI research team that builds those metrics, and all of our competitors and all this kind of outsource this to other open source, third party vendors. But we've owned this ourselves. We call this the AI measurement problem, where you can build these metrics. These metrics are also part of BI language models.

00:26:13:23 - 00:26:32:15
Vikram Chatterji
So the case again of the financial analyst, you might ask the question that I want to create a metric called overall quality of task completion. Right. But you might define that in a very specific way that you that you know, as the perfect task completion, if it's powered by a model, it's going to be like 70%, 80% accurate.

00:26:32:17 - 00:26:51:11
Vikram Chatterji
And we built a technique by which you can make that go from 80% to 95% accuracy. So you can actually trust your measurement. That's what we really excel in. So there's many different ways in which you can proactively know what's going wrong. You can also reactively know what's going wrong because of these because these metrics you can get constant alerts about them.

00:26:51:13 - 00:27:08:16
Vikram Chatterji
But then the last thing is, you can take these metrics and actually use them to just stop an agent from performing that, performing that task, if it's going down the right track in the wrong direction. So you can block and steer the agent, because we built, one of the only companies in the world that's done this.

00:27:08:18 - 00:27:30:15
Vikram Chatterji
If you build, the ability to have an ultra low latency, ultra low cost version of those metrics, which can be used for literally blocking and steering, which which I think is going to have to it's going to become more and more important in the enterprise is, you know, the publicly traded company is they want to see their, stocks go down and later look at it as a blip in their dashboard and be like, shucks, there we go.

00:27:30:17 - 00:27:41:18
Vikram Chatterji
I knew, I knew this could happen. I wish I could just have a way to stop it. You know, that's that's, super important. So those are three big ways in which we've been helping, enterprises.

00:27:41:20 - 00:28:02:07
Chris
You know, I love the idea of, like, having, you know, those kind of customizable metrics because I think you touched on another really big area, which is sort of like having what's the level of confidence in your result, right? That's just can be a very hard thing to quantify, like you say. And, and you know, the nature of AI is to be very optimistic about its answer.

00:28:02:10 - 00:28:02:13
Vikram Chatterji
Right.

00:28:02:13 - 00:28:23:09
Chris
So like every answer seems like, yeah, 100% certainty. Yes. And it may only be like 25%, but it's still going to be like, you know, 100%. So yep. So like so what? Like what do you what is it you do that like you mentioned going for 7080 to like 90%. You know like confidence like how do it how do you how do you do that.

00:28:23:11 - 00:28:40:21
Vikram Chatterji
It's it's actually I mean we so this the measurement problem of AI is can be divided into a couple of different things, but we, we call we call that piece the going from 75 to 95, the last mile of measurement. And what we realized is in your in the use case that you just mentioned, right.

00:28:40:21 - 00:29:02:22
Vikram Chatterji
It's, it's kind of like, what's the, what's the certainty that this is the right answer? Let's say you asked. I don't know, Gemini asked a question and it says, absolutely. The capital of France is Washington, DC. And, you know, that one is easy to know, but if it's if it's an obscure reference for something, some tax document or something, then it might be harder for you to know whether it's right or wrong.

00:29:03:00 - 00:29:22:04
Vikram Chatterji
And so you might want to build a metric which says, you know, what's the, level of confidence of, of, of the response here or what's the certainty of this response? Out of the box? It's probably going to be around 70% accurate. But that last mile is based on the context that you have and the data that you have.

00:29:22:06 - 00:29:46:20
Vikram Chatterji
So in the case of, let's say that, that your let's say this is Intuit building an agent for, for TurboTax. Right. The subject matter expert into it, the, the people who are actually responsible for, for taxation. They're the experts at this stuff, right? They can see that this is a wrong answer. And it's most likely because the, the the AI application in the models is missing some context.

00:29:46:22 - 00:30:05:02
Vikram Chatterji
So in Galileo, there are those SMEs can come in. And when they see that the metrics are wrong, or partially correct, they can provide natural language feedback directly into the metric. And they don't have to do it a lot. They maybe like 5 to 10 times. They whenever they see something wrong. And you see just by doing that we just auto improve the metric.

00:30:05:02 - 00:30:21:23
Vikram Chatterji
We abstract all the, the, the the the architecture, the algorithms and everything else in the infrastructure away from them. All they have to do is they like they look at the answer that that's not right, and here's why. And only I know it. And they enter that feedback and then we just automatically improve it. Auto version control that metric.

00:30:22:01 - 00:30:45:10
Vikram Chatterji
And it's been surprising to us how quickly that becomes not just more accurate. It's also very, unique to your business now and, and just for you, and that, frankly, becomes the IP for your company and becomes a competitive advantage for your company against whoever into its competitor is. I don't know if they have any, but whoever the competitor is who's also in do exactly the same thing.

00:30:45:12 - 00:30:52:05
Vikram Chatterji
Right? So because now they can ship their agents, but you can because you can trust that if something goes wrong, you know.

00:30:52:07 - 00:31:12:03
Chris
And when you're dealing with so much output and it's, it's hard to really identify those things that are sort of maybe screwing up the end results. Because, you know, a lot of times things run and run and run and run and then it comes back and you're like, what went wrong? Where? Yeah, you know, like I the ability to like, kind of quickly identify like places.

00:31:12:03 - 00:31:23:16
Chris
Maybe that's not what caused the problem, or maybe it is what caused a problem. But you know, like you would have looked somewhere else because you didn't have that confidence rating to go like, oh, that's the little linchpin that made everything go haywire.

00:31:23:16 - 00:31:25:08
Vikram Chatterji
That's correct. That's exactly right. Yeah.

00:31:25:08 - 00:31:27:10
Chris
And it's that's got to be enormously time saving.

00:31:27:10 - 00:31:44:14
Vikram Chatterji
It's it is, it is. And that's that's where that's. And without that you're just stuck with a bunch of logs. Right. And no one knows what to do with it. And it's got super hard to figure that out. And it's not even logs which are as easy or simple as deterministic software logs, which is like very you know, there are a bunch of if statements.

00:31:44:14 - 00:31:50:12
Vikram Chatterji
You just know how to look through it. This is yeah, dramatically different. The nature of the very yeah.

00:31:50:14 - 00:31:54:18
Chris
Very sorry. So this I've been kind of monopolizing.

00:31:54:18 - 00:32:23:03
Sandesh
So no, this has been awesome to me. What I've been thinking is how many customers, Vikram, have this problem that you can solve it almost. I think the there's a varying maturity level in the market right now where you have the ones that can get access to GPUs, you know, are able to just do more. It seems like compute is like the new fuel these days in this economy.

00:32:23:03 - 00:32:33:10
Sandesh
But yeah. Can you give us, like, a sense of, like, who are you going after? What is that ideal customer profile for you?

00:32:33:12 - 00:33:01:23
Vikram Chatterji
Yeah. So, we've taken a very strong stance on what? Who we think our our ideal customer profile is. But then there's also the, you know, the broader market of, just agent observability and evaluations. What, what we have, we have a free product as well. If you just go to our website, I there's a, there's a button there which we have given unfettered access to people's free, free apps.

00:33:02:02 - 00:33:25:09
Vikram Chatterji
Right. And we see a lot of startups coming through that. And they're using Galileo without anybody else in the loop. Just complete product led. They're signing in. They're using it. The only time I know that they're, they're having any issues is when they reach out to us for support. Right. So, there's definitely all those folks who are startups.

00:33:25:09 - 00:33:51:16
Vikram Chatterji
They want evolves, they want metrics. They're just trying to get get to shipping something. So there's not a whole side of the market. We we think of it as they're they're the fastest moving folks. And that's where we can learn about what's coming. And so we just give as many startups as possible, just free access, especially if they're very early stage, because that those are the ones who are at the cutting edge of adopting stuff.

00:33:51:18 - 00:34:26:14
Vikram Chatterji
And so we learn from them. We constantly learn from them, but from a sales perspective, what we've seen is because of our uniqueness organically, we just run into this uniqueness of being, having, having one platform which provides agent observability across not just offline testing. And not just the, runtime monitoring of the, of the, of the product of the application, but also, runtime, blocking and steering of the, of the, of their application and then back again to offline.

00:34:26:14 - 00:35:00:08
Vikram Chatterji
So the whole trifecta, in one place because of that, the, the general, you know, likelihood of us getting adopted by an enterprise, especially if it's in a regulated market, for instance, like a bank or a telco or insurance or healthcare is massive. So, you know, we've been winning deals like no one else's business when it comes to that specific domain, because we believe in our bones that that agent governance and trust in control is extremely important.

00:35:00:08 - 00:35:28:02
Vikram Chatterji
And when we talk to them about this problem that very passionate about, it's a very direct synergy that, yes, absolutely, we get it. We can't launch anything without this. And so on that front days, I feel like the market is kind of like the cloud market in 2010, maybe, where if you talk to the global 2000 and you poll every single one of them, maybe like, 99% of them will tell you that, yes, of course, we want to do something with AI.

00:35:28:04 - 00:35:48:20
Vikram Chatterji
But if you ask them, like do you have do you have everything do already ship stuff right now? Maybe 2% of them will be like, yes, but what's going to change is in six months, it's got to it's going to become eight and then 15 and then 20 and 25. So that's the deal. When we're kind of back into in the enterprise, because that 2% is extremely aggressive about needing to ship something tomorrow.

00:35:48:22 - 00:35:58:19
Vikram Chatterji
And Galileo is kind of becoming this like mandatory checkbox and, that they absolutely need across the enterprise. And we like to partner with those kinds of customers.

00:35:58:21 - 00:36:12:08
Sandesh
Yeah. Yeah, I, I like to use the term ship the work. Like at the end of the day, we can all work on something, but we have to have a tangible outcome of some kind. Right? Like, it's like it's fun to be in the sandbox, but you got to get it out. So from a sales.

00:36:12:08 - 00:36:29:22
Chris
Person and also building in public, I think is also another really valuable aspect of that too, is I think that like that, like constant feedback loop that happens when you do that. I love those freemium sort of yeah, options there, because I think that really helps a product develop better quicker.

00:36:29:22 - 00:36:31:11
Sandesh
Yeah. Such a great point.

00:36:31:13 - 00:36:32:09
Vikram Chatterji
Yeah.

00:36:32:11 - 00:37:02:10
Sandesh
Such a great point. So from a business value perspective, it seems to me, and I don't want to oversimplify this, so please keep me honest here, but it seems like in some ways it's a no brainer. If you're going to have you're going to be using AI on your data sets. That runs your business. One of the most important things that I want, if not the most important, is people are worried about what bad things can happen.

00:37:02:10 - 00:37:15:21
Sandesh
Yeah, right. So it's almost when I when I think about that, it's hard to almost put a financial metric or a financial value to it. Yeah. But how do you guys look at that from a sales perspective?

00:37:15:23 - 00:37:35:09
Vikram Chatterji
Well, we think of it a lot in terms of what's the opportunity cost on the other side. Right. You're missing out on your agent creating tons of tons of economic value. And we need to work with them on figuring out what that means for a dollar perspective. And let's now it's March. And we tell people like, you know, this is exactly what's kept you from launching since November.

00:37:35:09 - 00:37:58:20
Vikram Chatterji
You've already lost four and a half months. You've got your competitors already out there. You know, they're they're they're beating you right now. You could have launched this by now if you just had Galileo like five months ago. So what's the opportunity cost from that perspective. So that's number one. Number two is even if they launch something, let's say with regular metrics in place, most folks are using large language models for creating these metrics.

00:37:58:20 - 00:38:28:06
Vikram Chatterji
And that's very expensive at scale, very expensive. And they can only monitor like 10% of their traffic using these large language models because it's ridiculously expensive, like one of our fortune 50 customers was spending, 23 million a year on just the metrics and the calculation, the metrics, because they're all about relevance. And so we've again, uniquely built out these small language model powered, metrics, which can reduce the cost by 99% while having 97% higher accuracy.

00:38:28:08 - 00:38:46:13
Vikram Chatterji
So, you know, we've dramatically reduced the price for those guys. So there's also the cost of just in control. That's that's meaty, but also the opportunity cost that you're leaving on the table. But then there's also that other business that you talked about, which is the, the fear factor of, you know, when something goes wrong, what does that mean?

00:38:46:13 - 00:39:06:20
Vikram Chatterji
And how do you that one's a hard, hard one to put a number on. But, and most security products I've noticed, like they go with the log with that, like it's nothing is something that's going to happen. Maybe once a year. But when that happens, you got to be ready. I think the difference here is that that something bad that can happen can actually happen ten times a day in the case of probabilistic software.

00:39:06:20 - 00:39:10:19
Vikram Chatterji
So it's just, people get that much faster.

00:39:10:21 - 00:39:33:13
Chris
Yeah. I want to mention, you know, like the transparency that you're offering here is really interesting because that's always been a big challenge. And like, I think the, the, the central locus of that challenge, you know, really has been, the European Union and, you know, like some of the regulation, privacy regulations, explainability and all that that's really hit.

00:39:33:17 - 00:39:39:05
Chris
Do you find that like that's a really great market option opportunity for you.

00:39:39:07 - 00:40:03:23
Vikram Chatterji
You know, not yes and no. I would say like, I thought that it, that regulations and stuff would be and I think it will be, but those regulations are so, generic and hard to codify. So when we do work with, companies in the EU, actually, it's been yes, it's been useful, but it's about 10% of the reason why they would need a Galileo in place.

00:40:04:01 - 00:40:39:06
Vikram Chatterji
But when they do, we actually do, because of the nature of Galileo. And it's very easy to convert the metrics into any version of what you think is important. We have a way by which they can just convert the EU, AI act for their context into actual, tangible metrics. So yes, it's been useful, but it's not been like, like it's a tailwind, but it's nothing compared to the actual tailwind, which is if people really want to learn stuff and, you know, they just want to get this out the door and there's, there are like 30 other things apart from the EU that they really need to make sure is.

00:40:39:10 - 00:40:46:20
Chris
Yeah, that's probably more of a mature market, you know, focus rather than sort of where we are right now in this sort of growing market. Yeah.

00:40:46:22 - 00:40:48:04
Vikram Chatterji
Yeah. That's right.

00:40:48:06 - 00:41:01:11
Sandesh
That's right. Yeah. And to that point, what's the future then for you guys from where you are today? Maybe you can just tell us a little about where are you and and where are you going, where are you focused?

00:41:01:13 - 00:41:20:05
Vikram Chatterji
I feel like the, the story of Galileo is always, followed the arc of how language models are being used in the world. Right. Like, when we started out, there was no market. So first two years or so, you know, revenue was dry. But the interest from customers is really high because there are a few folks who are really excited.

00:41:20:05 - 00:41:46:01
Vikram Chatterji
And this is awesome. And now it's like the last year we, you know, 300% increase in revenue on a very, very large, very large denominator. So, and this year we're seeing that being going even higher. And, you know, the year after this, even higher. But if you look at why that's happening, it's because, the adoption of AI and the adoption of agents in the enterprise is, is is very real.

00:41:46:01 - 00:42:06:18
Vikram Chatterji
Now, we think of agents as the first time that there's a really interesting use case for AI that everyone can just galvanize around. And it wasn't we didn't feel the same way about chatbots, you know, we we just felt like that's not doesn't feel like $1 trillion industry in the making. That's how many chat bots can you have.

00:42:06:18 - 00:42:25:05
Vikram Chatterji
And everyone was to be like, maybe I can have a chat bot for customer experience and then I can't think of anything else. And so, you know, it was always like that. And so it didn't feel quite as big, which is right now it's like, oh, we can automate these 100 things and they actually can happen. And then Opencore comes and says, like you actually get it's very easy.

00:42:25:07 - 00:42:45:13
Vikram Chatterji
So, we, we think that this entire market's going to keep, keep growing over the course of the next season exceeding five years exponentially. So right now, frankly, our whole our head is very much in the, in the, in the zone that, you know, our product is really awesome. It's it's being leveraged across all these very demanding customers.

00:42:45:15 - 00:43:09:15
Vikram Chatterji
It's really showing up. Well, it's all about how do we have more conversations with folks that are dealing with this problem of how do I actually just control this stuff and telling them that, look, there is an actual viable, well-proven solution that's available for that. So that's kind of where our head is at right now. Like, how do you actually get the word out that people there is a remedy for this and there is a way to work through this?

00:43:09:17 - 00:43:28:17
Vikram Chatterji
So I think it's just going to become a bigger and bigger market. The thing which I'm very excited about is there's really two big shifts that I think are going to happen between this year and next year. One is multi-agent systems are going to become more of a thing. Yeah. People are not going to have one massive monolithic agent, which is your financial analyst.

00:43:28:17 - 00:43:46:21
Vikram Chatterji
It's going to be many, like ten smaller ones just talking to each other. And that's going to open up, a lot of efficiencies, but a lot of failure modes as well. The second thing I think that is going to happen at some point in the next 1 or 2 years, hopefully, is we'll start seeing physical AI becoming more and more of a thing.

00:43:46:23 - 00:44:18:22
Vikram Chatterji
You know, you're going to be able to infuse AI on the edge, and be able to do more with all this stuff. And I know that Nvidia and others are pumping a lot of, cash into getting that done. Do. So all of that's going to move in the direction where like five years from now, when you talk about this, it's going to be probably a very different version of this conversation where agent observability is what we're calling the market now, but just in control of these, these bots that are just automating a bunch of stuff is going to be it's going to be everywhere.

00:44:19:00 - 00:44:51:23
Chris
Yeah. I think you're you're potential markets just, exponentially growing. And I think, I think the thing that's really going to drive things like this forward too, is that, you know, you talk about multi-agent kind of interactions and like all the it it's like agents within your organizations, agents outside of your organization. So yeah, the thing is that at some point, the complexity of all that gets too much for anybody to actually and we probably already there right now, it's too much for anybody to actually monitor.

00:44:51:23 - 00:45:09:14
Chris
And so like having a tool that, you know, understands non-deterministic systems is non-deterministic in many ways of its own. Yeah. You know, is is really going to be the only way to really, you know, summarize a scientist distill this into something that humans can, you know, make a value judgment about.

00:45:09:15 - 00:45:33:15
Vikram Chatterji
That's right, that's right. Yeah. No, that's exactly right. We you, you hit the nail on the head. I think the need for a platform that can actually, help these help these enterprises out from a trusting, controlled perspective is becoming the it's becoming the biggest bottleneck right now for unlocking this technology in the enterprise. It's a big thing for people that can I trust it can control it.

00:45:33:15 - 00:45:48:22
Vikram Chatterji
If you look at the board, if you if you if you, if you peer into the boardrooms of these kind of these companies, that's the number one thing they should we invest in this. Yes. But is it worth the risk? Can we take the risk? How can we take the risk. And that's kind of where, where this where where we come in.

00:45:49:00 - 00:46:12:17
Sandesh
So you solve a big problem in the market. You guys are having a ton of momentum. You guys got great leadership. I, it's amazing when I start to see at a very early stage in your a company like yours landing the companies, the customers that you're landing. That that to me, is the most telling.

00:46:12:19 - 00:46:17:05
Sandesh
How do you want people to, to reach you if they're interested in learning more about Galileo?

00:46:17:07 - 00:46:39:20
Vikram Chatterji
Oh, you can reach me directly at the Chrome at Galileo. I, we're, you know, we hiring big time across engineering. The AI research, sales, marketing, across the board. This is a really, really massive market in the making. We're right up there and our leading platform in this space, but also time.

00:46:39:20 - 00:46:40:15
Chris
To get in early.

00:46:40:16 - 00:46:58:08
Vikram Chatterji
Time to get in early. Absolutely. And, you know, apart from that, if, if folks are actually building agents as well, you know, whether it's for our personal hobby project, they can go to Galileo or I, I would highly encourage them to use the product and, and see if it can help and if and just give us some feedback.

00:46:58:10 - 00:47:12:10
Vikram Chatterji
But also if it's an enterprise, we're happy to help. And they should know that there's actually a solution out there. And some of the leading fortune 50 companies are using Galileo to ship agents much, much faster than they otherwise could. So just reach out directly to me.

00:47:12:11 - 00:47:29:14
Chris
Yeah, I got it. I got this integrate, this into my workflow because I'll tell you one thing, and I do that like I'm always. And I know we're over time, I apologize. I like, but, you know, like, managing context windows is just, like, miserable. And if I had real stats because, you know, Claude just guesses. Yeah. Half the time.

00:47:29:14 - 00:47:40:21
Chris
Anyways. So, you know, like, if there was, like, real stats that I could use. Wow. That sounds amazing. Yeah. Ability. Wow, I love it. I gotta, I gotta, I mean, to sign up, right? I've already got the window open.

00:47:40:23 - 00:47:56:03
Sandesh
So is you. Yeah. Yeah. Well, obviously, as you can tell, Vikram, we could talk to you for for for hours. And now, guys, this is this was just awesome. We're rooting for you, man. Thank you so much. Thank you. Yeah.

00:47:56:03 - 00:48:00:18
Chris
This is a tool that's desperately needed in the marketplace. It's needed in the market because you.

00:48:00:20 - 00:48:16:19
Sandesh
Market right outside of wanting you to be successful in your organization, in your people. It's important that we solve this problem in the market for AI. Yeah. It's a big one. It's, I think it's the biggest one out there, so I can't thank you enough.

00:48:16:21 - 00:48:17:18
Vikram Chatterji
Oh, thank you for giving.

00:48:17:18 - 00:48:20:23
Sandesh
Us your time. And, we wish you the best.

00:48:21:00 - 00:48:22:15
Chris
And keep us posted.

00:48:22:17 - 00:48:26:14
Vikram Chatterji
Of course. Absolutely. Thank you so much, guys. Thanks. And they really appreciate this.