Code with Jason
Code with Jason
270 - AI with Daniel Nastase
In this episode, I discuss AI with Daniel Nastase, covering Daniel's journey from building neural networks from scratch to understanding embeddings and vector databases. We explore the limitations of current AI learning models versus explanation-based reasoning, and discuss practical AI applications including agents and voice interfaces for programming.
Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Email newsletters are a really nice way to keep on top of what's going on in the programming world. Except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more. That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right, I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like? You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming. The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary. I'm sure that's what you were going for, so just wanted to let you know that you succeeded. Looking forward to more. Drew Bragg from Philadelphia says Nonsense Monthly is the only newsletter I deliberately set aside time to read. I read a lot of great newsletters, but there's just something about receiving a piece of mail, physically opening it, and sitting down to read it on paper that is just so awesome. Feels like a lost luxury. Chris Sonnier from Dickinson, Texas says just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it. Something about holding a physical piece of paper that just feels good. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsensemonthly.com. That's nonsensemonthly.com. I'll say it one more time. Nonsensemonthly.com. And now without further ado, here is today's episode. Hey, today I'm here with Daniel Nastase. Daniel, welcome.
SPEAKER_02:Jason, welcome. Very happy to be here.
SPEAKER_00:Happy to have you here. Um, and it's interesting the way that we met. You know, quite often I'll meet people first on the internet and then we'll meet in person. Um, but you and I met in person um when I was visiting Romania, and um and now we're here talking online. But can you share from your perspective, like how did that come about? I think I sent out an email to my email list saying, like, hey, I'm in Romania, something like that.
SPEAKER_02:Exactly. Uh actually the correct uh full story is that indeed initially we met online because I started to follow you and uh your online adventures and online endeavors. Uh and I actually remembered uh while preparing for the podcast how I discovered your blog. Uh and um I've seen I was subscribed uh to your newsletter, and I have seen that uh you are in Bucharest. Uh and you are the first one, the first time I actually reached out to someone that I wanted to meet in person. So uh for me was always not not to sound weird or something like that, was this um locage, let's say. And I said, okay, whatever, I'll do it. Uh Jason seems uh a cool guy, so uh yeah, most likely it will turn out to be okay. Yeah, and uh I replied with the email and uh we met uh in Bucharest, and uh yeah, it's uh it's a nice talk, and uh after that uh we we continued uh to to to uh be in contact.
SPEAKER_00:Yeah, yeah, I'm really glad we we got to meet. Um and what was I gonna say? Um oh yeah, going to um Bucharest, that was one of my favorite places I've ever gone. Um it was a really nice, really nice trip. Um, really nice city. Of course, I went there for um the Friendly RB conference. This was in the fall of 2023. Um and if anybody gets a chance to go to Friendly RB, I highly recommend it, both for the conference itself, which is great, um, but also for Romania. Um I I I didn't go in expecting this, but Bucharest had excellent food. Um that's that's something that I really value. Um I especially really like the pastries, and so we found honestly, we went to this one place first, um, and it was like supposed to be kind of a French place, and I was like pumped about that because we had just gone to Paris like a few months before and we got some like raspberry tartlets and stuff like that and croissants, and it all kind of sucked. And I was like, shit, maybe this place sucks. Um, but then I found this other bakery, and it was just like it was the best, it was every bit as good as anything that I had in France. Um, and so that was that was really nice. And then we went up um after the conference, we went up to the mountains um and we saw some bears and um what else did we do up there? Well, we took like this tour, there was like some Dracula themed stuff that we went and did. Um, just a really nice time. Um, maybe it doesn't seem that special to you because you live there and it's just like what you're surrounded by every day, but from my perspective, it was a really special trip.
SPEAKER_02:Yeah, uh speaking about uh bears, actually, we have a small issue uh with bears, if I'm not mistaken. Uh I think we have the uh most bears in Europe by uh country, and uh they can even go directly in the city, so they were uh there were cases when actually you could like go out and uh see a bear in the middle of the city, so which is not uh not the best uh thing to see. But yeah, uh there are many nice things to see and uh to do here.
SPEAKER_00:Yeah, that happens with us sometimes. Um I think our bears are getting more and more populous here in Michigan. Um and they're they're coming further south, like Michigan's a peninsula, and so it's not on the way to anything, and especially the upper part of well, Michigan's two peninsulas actually, but the upper part of the lower peninsula is like pretty isolated, and so there's there's not a lot up there, and it's getting like even more depopulated, and the bear population is growing, and sometimes they make their way down further south, and so in Grand Rapids, where near where I live, which is like the second biggest metro area in Michigan, um, some years ago we had a bear just just in the city, it just wandered into the city.
SPEAKER_02:Yeah, here I think it's uh much more often often that uh once per year. I mean, uh uh usually during the winter sinaa, it's uh most likely you you passed by uh through Sinaia when you went uh uh to the mountainside, and um in there uh during the winter they can have, I don't know, uh each each week to see uh to to have a bear encounter, so it can be quite interesting. Yeah, yeah, yeah. Because they are attracted um uh by the food, by the smell, by the yeah, so uh they they usually come to eat, and uh at one point there were also people who are feeding them, but uh yeah, things uh are getting better now.
SPEAKER_00:Okay. Um by the way, uh just a couple like animal sighting stories. Um one time, uh you'll never believe this, but I saw a fucking deer in downtown Chicago.
SPEAKER_02:Chicago, okay. So not the smallest city. Okay, yeah. I don't know how a deer got there. It's insane. Escape from the zoo or something like that.
SPEAKER_00:Well, I mean, we don't have deer in in zoos here. Um so it just got there somehow. Uh it's crazy. Okay, it's like, bro, you're really far out of here. Yeah.
SPEAKER_02:I have a friend who was in Barcelona um when was it uh close to four months ago, and he saw a boar, like a wild boar.
SPEAKER_00:A wild boar in Barcelona.
SPEAKER_02:Yeah, exactly. Walking on the streets. Uh and this friend of mine also had a couple of beers that evening, and and he was telling me, I was not I was not so sure that I was seeing right. And I and I stopped and asked someone on the screen, do you also see this? Yeah, it's a boar in the middle of Barcelona. So yeah, it maybe it happens all all over the place.
SPEAKER_00:That's crazy. Um, and a couple weeks ago, for the first time in my life, I saw a wild fox.
SPEAKER_02:Oh, okay.
SPEAKER_00:Yeah, that was pretty cool. And it was like just across the street from my house. So that was yeah, that was a really cool place to see that. And sometimes here it it happens like a couple times a year, maybe. We'll see a bald eagle. Like just at our not at our house, but like right by our house.
SPEAKER_02:Eagles are uh are fantastic. I mean, uh it's the first time we don't have uh eagles in Romania, but uh we were chatting before starting the podcast. Um we stayed uh for four months now in Andalusia in the south of Spain. In there they have eagles, and was the first time when I have seen how big they are, like when their wings are expanded and uh they're circling something, I don't know, and I was like, whoa, this is these birds are really, really nice.
SPEAKER_00:Yeah, it's crazy when a bird like lands on the ground and you can see it kind of close, or even just like in a branch that's nearby, you like you're like, whoa, this thing is like way bigger than I thought it was. Like I saw a woodpecker on the ground one time, and I'm like, whoa, what this woodpecker is huge. Um, and the opposite, like we have hummingbreed, hummingbird feeders on our porch, and we'll see them all the time. But sometimes they'll see a hummingbird and it'll like fly away and land in a branch, and it's insane how tiny those things are.
SPEAKER_02:Yep, yeah, and by the way, did you know that uh beside birds, uh humans uh have the best sight in the world from all of the like living things on Earth after birds? Uh we are the second uh in line to how clear can we see?
SPEAKER_00:Interesting. I did not know that.
SPEAKER_02:Yeah, we are very visual uh animals.
SPEAKER_00:Well, I will um respond to your fun fact with a fun fact of my own. I learned recently that um the eye has evolved in the history of the world, it has evolved something like 40 to 60 separate times. Yeah, uh that's crazy. Like flight has evolved four separate times. And I thought that was crazy. You know, it's like um birds, pterosaurs, um, bats, and insects, but the eye evolved like 40 to 60. I would never guess that it evolved independently that many times.
SPEAKER_02:Indeed, it is sounds, yeah.
SPEAKER_00:Yeah, um, well, we sure got real off topic really quickly. Um coming back around to to more related things. Um AI. So I've been thinking about AI a lot since it came out. As everyone. As everyone. Yeah, some people are sick of hearing about it. And to those people, I say, get ready for a really long rest of your life because you're not gonna stop hearing about it, it's not gonna go away.
SPEAKER_02:Like the internet, I think it will have almost the same impact, and maybe even more.
SPEAKER_00:Yeah, yeah, exactly. And it's really interesting to be here, like kind of at the beginning of this incredibly impactful revolution. Um, so I'm curious, like, how have you been using AI? What have you been thinking about? What's on your mind, all that stuff?
SPEAKER_02:Uh I when I first saw it, you know, like I I have uh uh I remember uh very clearly what was uh I was doing when uh my daughter was born, uh or uh when uh the World Trade Center uh attacks happened very, very clearly, and exactly the same thing. Uh I had a very clear memory when I first interacted with ChatGPT. I was uh with uh friend, uh we were watching a football match actually, and he showed me this. And look, this is amazing, and I was yeah, yeah, whatever. Another chat would seem then uh before. But the more after that, I know that I came back home and uh spoke with ChatGPT for close to two hours, and I was also excited and afraid in the same time and worry like this is not real. I mean, uh, for a couple of days I was sure that they will discover that actually there is like a human uh operator who answers, and it was just a big uh I don't know, scam or something like that. So um after discovering that this thing is actually real and so on, uh my first reaction was okay, how this thing was built, given the fact that okay, I be I was doing web development for close to I don't know, 18 years then, and uh uh I didn't know how to build uh anything that uh resembled with uh ChatGPT or any of this. Uh it wasn't uh traditionally then as uh programming um manner. And um I started uh I actually took uh I was between two jobs then, and I took close to two or three months and I went as deep as I could on this rabbit hole. And uh yeah, it was a painful uh trip, let's say also interesting, but also intense and painful because after three months I I was able to have like a neural network written in JavaScript, your JavaScript from scratch, that uh could uh recognize the digits. Uh you could draw on uh HTML canvas some digits and it will tell you is this uh five or ten or something like that. Yeah, I actually still have it on my GitHub.
SPEAKER_00:Uh how long did it take you to build that?
SPEAKER_02:Uh actually initially I started with TensorFlow.js. Yeah, I my my first uh interaction, my first instinct was to reach for some JavaScript stuff because JavaScript is what I've done almost all of my career. And uh uh I uh for the first thing that I discovered was TensorFlow.js, and I didn't understand nothing. I mean uh it was okay and not in the same time I it was uh and uh I realized that I had to go go deeper and uh actually see at the core uh what a neural network is, uh how it how it actually knows to do stuff, because again, it wasn't about the if then else, uh it was something else there, and uh after that, uh yeah, I started to learn I was basically pulling information from multiple uh sources, YouTube videos, books, because it wasn't like just one single thing. Okay, this is how you build this from scratch, or at least I didn't find it at that uh moment. And it took close to yeah, four months, four months to to build that and uh to understand uh like the math behind it and uh so on. But I was almost burned out at the end of those four months. Uh I said that I cannot continue is way too intense, way too uh and went back to web development because it was clear that nobody will hire me just to build a neural network that knows how to recognize digits. Because I mean, these things were uh done before, and I said, okay, if this took me four months, then yeah, most likely I will become proficient in this stuff in a couple of years in a good uh yeah, and I I went back to uh traditional web uh web development. That was actually when we met, was very close to uh the moment we uh met in Bucharest. But during the same time, I discovered I don't know if you know the latent space podcast uh run by Swix. Uh it's maybe the biggest ah, you know, Swix. Then he has a uh podcast, Latent Space, which is actually the biggest AI engineering um uh podcast.
SPEAKER_00:Oh and uh I was thinking of a different guy, but I know of Swix. Yeah, there's a different guy with a similar name who I know personally, but this guy I at least know of.
SPEAKER_02:I see. I see. Yeah, and I was listening to this uh podcast, and uh at one point uh he mentioned Langchain. And uh uh he mentioned Langchain like something uh lang chain solves the streaming uh problem in LLMs. And I said, okay, streaming sounds familiar. We I I I was at one point involved in a project uh very streaming based. And like, okay, what's the deal with streaming, AI, and so on? And after that, i I discovered like the whole um uh uh the whole uh AI integration part.
SPEAKER_00:So basically you don't let me let me pause you a second because I have some like questions and comments on what you've covered so far. Um so it's really interesting to me that you like went to the like the the the most fundamental fundamentals of AI and you like implemented your own neural network from scratch. Um I've been like studying those fundamental building blocks myself a bit. I haven't put in nearly as much time and effort as it sounds like you did, but I was I I at least got some AI books. A lot of them I've found just like pretty inscrutable. And so like I I've read parts of them, but it's like hmm, I don't understand this because I don't think I have the requisite background knowledge to be able to even understand this. And so it's like I'll read part of one book and then I'll go get a different book that has like more basic stuff in it, and then even another book that has more basic stuff than that. Um but this is just kind of like haphazardly and casually reading this stuff. I haven't done any like really serious um AI projects. I did build my own neural network um that could what could it do? It was like very basic, like inputs and outputs, like zeros and ones. Um and it it learned I don't remember what it learned. It learned like logical or or something like that.
SPEAKER_01:Okay, okay, yeah, okay.
SPEAKER_00:Yeah. Um and I think the way I built it actually was I just like asked Chat GPT to to guide me through it, and it ended up being um a surprisingly small amount of code. It was like I don't know, probably less than a hundred lines of code. And it didn't even take that long to implement. Like I didn't really do anything, it was mostly chat GPT doing that. Um and then once I had it, I was like, okay, let me take this and try to understand it. And I read through and I was able to I was able to gain somewhat of an understanding of at least parts of it. I definitely could not understand the whole thing. Um but I would like to get to the point where I like deeply understand that stuff. Again, I'm not really making it a priority. Um I'm spending all my free time working on my product, my my CI platform I'm building. Um so I'm not giving it a very high priority. But there's something that like I just don't love about the idea of being only a developer who like takes the libraries that other people have built and like writing glue code that like takes something from one library and feeds it through this other library and stuff like that. Like, I don't have anything against that, and like that's most certainly the way to get certain things done, but there's something that bugs me about like doing that while not even trying to have an understanding of how those building blocks work under the hood, and so that's the part that I'm like I'm interested in both parts for sure, and it sounds like you're interested in both parts too.
SPEAKER_02:Correct. Yeah, I'm trying to remember is this saying that I cannot build what I don't understand, or something like that. I it's very hard to remember that the actual uh uh saying, but what exactly this? I mean, when I started to play with TensorFlow.js, which was basically one uh level up of uh basic uh code, I was always okay, but why is this like this, and why is this like this, and so on. And um, yeah, going to the lower level, uh uh why uh it helped a lot with understanding the randomness of these uh models because basically I don't know uh if you study the um um in uh university, uh there is in math there is this uh part where you are approximating functions. So basically, you know just the input and output uh of a function, but you don't know the actual formula of that function. And this is exactly how LLMs also, not sorry, AI models, not LLMs, work under the hood. So basically, they are designing the neural network in essence is just a uh estimation of a given function. And estimation is very important because exactly as in math, uh you could estimate um a function with uh specific uh accuracy.
SPEAKER_00:90%, 95%, plus a second and see if I can um understand what you're saying. Um I've read about what I think is is the same thing. Um like take a really basic example. Imagine like a Cartesian graph where there's uh it's like a scatterplot, I guess they call it. Um has a bunch of dots on it. And you can see like very roughly the dots kind of describe a sloping straight line. And so it's like, okay, like if you were to actually draw a line between all these dots, it would be like this crazy crooked line. And like obviously there's no function that will give that output. Um, but if you like took a ruler and drew a straight line through it, it would kind of like stab through the middle of all the dots, even though not all the dots are like perfectly on the line, the line would represent kind of the average of where they are. Exactly. And then you could figure out like, okay, this uh this function is like 2x plus 5 or something like that. Oh, yeah. And so 2x plus 5 is like the best approximation we can make of these points. And then, of course, the function can be more complicated, it could be like an exponential curve or something like that, or it could be any crazy kind of the spy evaluation.
SPEAKER_02:Yeah, whatever. Yeah, actually, also in the math, uh the estimation of the function happens from the visual representation of that function. So it's exactly the same thing. So you start from drawing a graph with the inputs and outputs, and based on that you um try to estimate the function. And uh I I don't remember like the full process. Uh, I can give you some links if you are interested. Uh, but the whole idea is that you can do this only with uh specific accuracy. So you are not uh uh as long as we have AI built with how it is built today, we it will be prone to hallucinations. You cannot make it 100% reliable.
SPEAKER_00:Yeah, so let's talk about that. Um a different term for hallucination is overfitting. And so, like the the visual example that I saw of overfitting is like let's say you have some some dots that are like kind of scattered, and so you can make a function that approximately follows those dots. Let's say it's kind of a wavy line, but then let's say there's a couple dots that are like outliers. Um, and then you can make a function that like it really hits every line, but it's like so sensitive. Sorry, it hits every dot, but it's like so uh precise that it like does too much. And so there's like some crazy spike in it that really shouldn't be there. And so it's like your function is like too good, it overfits.
SPEAKER_02:Yeah, yeah, exactly. It adapts more than it should to the examples and is not able to uh adapt to something that comes from reality, it only tries to fit as best as possible just the input data. So yeah, that's that's correct.
SPEAKER_00:Yeah. Yeah, um, and you know what? The this whole thing bothers me uh deeply.
SPEAKER_02:Uh you should try please continue.
SPEAKER_00:Yeah, yeah. Okay, so and I've talked about this on the show before, but like, are you familiar with Bayesian reasoning?
SPEAKER_02:Uh just a bit, yeah, yeah. Okay.
SPEAKER_00:Yeah, so like, dear listener, if you're not familiar with Bayesian Bayesian reasoning, let's see if I can even describe it, let's see if I even understand it well enough to describe it. But basically it's like a probabilistic epistemology. So you you you I'm trying to think of like an example. Um oh, I don't know. Let's um oh, I was gonna use an example of like sheep and like um uh determining the color of sheep, but that's maybe a bad example. Um let's see. Let's go with sheep, whatever. Um you see one sheep that's white. And it's like, okay, I've only seen one sheep in my life. It's white. Maybe sheep are white, but I'm not sure. So you assign it a probability, like a starting probability of one half. And so like the term. Probability in Bayesian reasoning has a different meaning than probability in statistics. Like it's not exactly the probability that something will happen. It's like the probability that something is true. So you start off like pretty neutral with just one half. Um and then sorry, that's that's maybe before you see any sheep. And then you see a sheep and it's white, and you're like, okay, that that adds a little bit of weight. And it's like, okay, so now I upgrade the probability to like a three-fourths. Then you see another sheep, and that one's white too. And so you upgrade it to like seven-eighths. And the more white sheep you see, the stronger you make the probability. Um but then of course you have um what they call the problem of induction. Um because when you when you see examples of okay, deduction is when you see examples of things and and come up with a uh general uh uh a general belief based on specific examples. That's induction, and when you make a prediction of specific instances from general knowledge, that's a deduction. And deduction is logically guaranteed to be correct. Um there's that famous log logic example of like all men are mortal, Socrates is a man, therefore Socrates is mortal. That's deduction. We can know with absolute logical certainty that Socrates is mortal if those premises of all men are mortal and Socrates is a man are true. But induction is not guaranteed to be correct because you can see a million white sheep and come away with a belief that all sheep are white, but then there's a black sheep and it's all out the window because you you used induction, which is not um it's it's not logically guaranteed to give you true answers. Anyway, um you can use Bayesian Bayesian reasoning and you can use induction to kind of fudge it. Like you can arrive at a lot of like incidentally correct beliefs a lot of the time. But that's not how like real human knowledge creation works. That's not how science works. Um we um okay, so if you apply this like Bayesian logic to the sun rising every day, for example, you might say, well, I assign a one-half probability that the sun rises tomorrow, and then it rises and you upgrade it to three fourths, seven eighths, blah, blah, blah. You see the sunrise ten thousand times, and you're like, okay, I've induced that the sun will rise tomorrow because I've seen it rise so many times in the past. But that's not how we know that the sun will rise tomorrow. We know the sun will rise tomorrow because we have an explanation of why the sun will rise tomorrow. Um you know, we know that there's the solar system and the the earth goes around the sun. Um and so this like statistical method of AI learning where it like fits the points or whatever, that is if I'm thinking about it correctly, that's induction and that's a totally different thing from explanation-based reasoning. And so forever, no matter what, there will be a ceiling on what AI can learn, and not until we get away from that learning model and switch to an explanation-based learning model that we can achieve true intelligence.
SPEAKER_02:Correct, correct. Yeah, yeah, that's uh for me the uh the this is also a fantastic point that you brought up. For me, uh a very strange uh moment with uh when I realized how strange and how uh peculiar these things are was uh when I discovered uh I was learning more about embeddings. So embeddings um they are used mainly for uh vector databases and uh uh LLMs and uh RAG uh retrieval augmented uh uh generation.
SPEAKER_00:Okay, sorry, what are embeddings and what are vector databases?
SPEAKER_02:Yes. One second, just to explain. So for example, you you you have uh let's say you have uh uh uh the the following words uh mouse, uh frog, uh whale, and the dolphin. So you have four animals, and imagine that you can you will put these animals in a 2D graph, in a bidimensional graph, where on one side you have on one axis, you have how big or small these animals, uh these creatures, because not all of them are probably not an animal, but how how how big or small these creatures are on on the other axis? Uh you have if they uh walk or if they swim. And most likely you will end up with the dolphin and the whale close, this axis, because they are both big, let's say, and uh they swim. And the frog and the mouse, okay, the frog is also swimming a bit, but still the frog and the mouse will be closer, uh, and while on the other side the dolphin and the whale uh will be again closer. So you have two these two clusters of uh uh words because if you represent them uh in uh uh bidimensional space and you can assign how big it is, okay, you'll assign most likely one to uh the the dolphin uh to the dolphin the and the whale. Okay, the whale may most likely bigger than the dolphin. So okay, dolphin has 0.9 while the uh whale has uh one. On the other side, the other and with the other features are smaller, and the same with the other axis. So basically, you are by embeddings, we are trying to uh refer the meaning of uh a word and put it in numbers, okay, and basically in a set of numbers. In this case, we just took into account how big or small is that thing and if it swims or not. So you can uh and uh now imagine for each notion, for each orb that we know, we have way more multiple uh meanings. So it's not just uh you can, I don't know, if it eats, if it's made of metal, it's everything, it's uh well imagine that this is exactly uh how LLMs are perceiving uh words, how they know that a mouse is closer to a frog versus uh a whale and uh a dolphin. But the fun fact now is that actually the embedding um spaces used by LLMs, we don't know the meaning of what the LLM understands by one embedding. If in this example I could say that, okay, this word represents how big that creatures sorry, this number represents how big that creature is, while on the other side I can say this number represents how much that creature swims or not, lives in the water or not. Well, LLMs are way more stranger in the idea that we don't even know what does what a set what that word means for the LLM. But we know that it works.
SPEAKER_00:Let's see if I understand that um it's like we don't know if the LLM is going by like how big it is, whether it swims, whatever. We don't know what those are.
SPEAKER_02:Yeah, exactly. Exactly, exactly. We know that it functions like that, but we don't know what it means for the LLM. One specific dimension, we don't know what the LLM understands by that. It it for me that was a bit concerning, let's say. A bit uh like we're putting uh we are putting so many uh so so much trust in this uh the uh all of this technology. Uh but actually we don't get the fundamentals. Like what are those numbers? What are those vectors of um of numbers? So um yeah, it's uh that that was the strange moment uh for me, one of them.
SPEAKER_00:Interesting. Yeah, yeah, okay. And so to come back around, um what you you said embeddings, I'm not familiar with that term.
SPEAKER_02:So embeddings uh are exactly this: you take a word and you try to explain that word through numbers, where each number uh represents a specific meaning. So when I say, for example, that a mouse is, I don't know, let's say uh 0.1 and 0.2, I'm referring so basically an array, uh uh an embedding is basically a set of numerical values, and each of those um values has a specific meaning. In the earlier example was how big and how uh well that uh that word, that creature that uh uh represented the word uh does. Does it make any sense? It's a bit more clear now. Yeah, basically, we're trying to put words into numbers because after that, yeah, computers are great with numbers and horrible with uh words. So this is an embedding. This translation based on a specific embedding space, for example, uh, I think ChatGPT uh 3 had something like 700, uh 600 or 700, I don't know by heart, but around there uh embeddings. Um so basically each uh token that you would put uh you'll take you'll take it will have around 700 values that it would represent different meanings. But now the fun fact is the fact that we don't know what those meanings are. So that's that's strange.
SPEAKER_00:Interesting. Okay. Yeah, yeah, and it's not as though the the LLM could explain to us what those meanings are. Exactly.
SPEAKER_02:It doesn't even that's the point.
SPEAKER_00:Yeah, exactly.
SPEAKER_02:It learns, it works.
SPEAKER_00:Yeah, interesting. Um, embeddings and vector database. I'm guessing a vector database is like uh something to do with persisting those embeddings or something like that.
SPEAKER_02:Exactly, exactly, exactly, exactly. It makes uh makes it very easy to search based based on uh these values, to cluster them together, to uh uh so on. So uh yeah, this is uh a vector database is basically a storage mechanism for uh an optimized storage mechanism for these embeddings.
SPEAKER_00:Interesting. Um and I understand LLMs use neural networks, right?
SPEAKER_02:Yeah, yeah, in a sense, yeah, yeah, yeah. The transformer is a neural network architecture. I mean the transformer, the thing that uh powered uh modern-day uh LLM three um L neural network in the end.
SPEAKER_00:Yeah, okay. Yeah, I made um my own attempt at building like a really rudimentary, crappy baby LLM. Um and the way I did it was like I ingested like a bunch of books and I saved the I I tokenized the content of the books and I like saved the words that were close to each or I saved the tokens that appeared close to other tokens. I tried to label the tokens with parts of speech and stuff like that, and then I had it generate sentences. And I got it to the point where it could generate sentences that were pretty much grammatically correct, but they never made any sense and they were never interesting. Like every great once in a while, like the monkeys typing Shakespeare type thing, like every great once in a while it would make a sentence that was like kind of funny or something like that. Yeah, but it never really made all that much sense. Um I've heard of people like building their own LLMs from scratch. I assume there's like tutorials out there that can help you.
SPEAKER_02:Yeah, exactly. Uh there is uh on Amazon uh the 100 there there are multiple uh resources now. So I um uh there is um uh what what's the name? The 100 page uh LLM book. Uh it it's quite good. It's quite good. Uh and also I I think if you Google on Amazon Build LLM from scratch, uh there is another uh book uh quite good in uh and uh also uh Isha Nanand uh he was at my podcast and uh you can see his talk on uh the AI engineering uh uh channel. If you search Ishaan Anand AI Engineering, uh has a fantastic course on uh how LLMs work under the hood. But I think overall for us, I don't know, as classical web developers, let's say, uh as long as we get a fundamental understanding of and uh demystify a bit this black box of uh this magic black box of LLMs, uh I think it's or at least this was the conclusion, this is the conclusion that I read, it's way better to uh uh orient ourselves to how to integrate this uh new technology and uh uh how to use it in order to build stuff. Uh so for example, uh by default, LLMs are uh stateless, they don't have any state. It's up to us as a developer to provide state uh and uh in the in this way the LLM will know um what it said in the last sentence, in the last uh reply. Uh so uh uh this is uh these are the challenges that uh we have to work as uh developers. Uh and I think this is this instead of building LLM scope sketch, which which is great in order to get some things rolling, uh it's also very important for us to know how to integrate them and uh how to deploy them and uh yeah uh everything in that.
SPEAKER_00:Yeah. Yeah, and to me, there's kind of two ways of using AI uh for programming. There's like you can use AI um in the machinery of your program, or you can use AI to generate the machinery of your program. Yep. Um I've done a bit of both, but the vast majority of the way that I've used AI and programming is just use AI to help me in coding, um, but there's no actual AI stuff happening in the machinery of the program. Um the program is just the artifact.
SPEAKER_02:Um but there's a yeah for me was a bit was a bit exactly on the other side. Uh uh I still like to program by hand when when it's possible. For sure, it's it's way more productive to use uh AI to code. Um but on the other side I tried as much as possible to add to this uh AI features in to what I was building. And uh I don't know, I think uh uh LLMs and AI in general will be like the databases. Uh each uh new product will have as it has a database, it will also have uh LLM or some AI stuff uh included. It will become also as a it will become as close as a default requirement for any software.
SPEAKER_00:Um yeah, that's interesting. So, like what are you saying? Are you saying like you think in the future like pretty much any software product that has a database will have an LLM that you can use to interface with the database?
SPEAKER_02:Uh no, no, not yeah, that this can be one, but in general, uh yeah, I think uh not not just for the database, but uh almost for anything else. You as we have right now a database layer for almost any application that we are building, or I don't know, 80%, let's say, of the applications that we are building, they have a database somewhere to store data, they will also have uh LLM uh included, or they will use uh an LLM. Because things like this, uh you have uh until up to this point, you'll have uh you have to know how to use the interface in order there to get the information that you need. Uh, but given the fact that LLMs have this uh natural language processing layer where I can say in words what I want, right? I can go to Excel and just say, look, make me um the average of the top 25% of the values of this table. I can say this literally in words and even more with my spoken. Exactly, exactly, exactly. And uh that thing can be translated by the LLM directly to what the user needs until up to this point you know or you need to know okay how to filter values in Excel. After that, how to rank the first 25% values, after that, apply the average function, so on and so forth. So uh basically AI and sorry, LLMs mostly I think they will become the interface to this.
SPEAKER_00:Interesting.
SPEAKER_02:I don't know if you have tried, for example, Mac Whisper. Mac Whisper is an application that uh you can use to just dictate to the uh computer what you mean. It's absolutely fantastic, you know.
SPEAKER_00:I think that I'm not heard of this.
SPEAKER_02:Yeah, yeah, yeah. There are multiple is built on the Whisper model from uh AI. Uh and I'm almost sure that in the again, almost not 100%, but uh in the future our interactions with uh machines will be through spoken words. Uh or it will be half of uh I think we will be moving more and more to uh I don't know, audio web or something like that where we will say in words what we'll mean instead of clicking buttons all over the place and doing shortcuts and uh so on. Because yeah, this is what LLMs are allowing us to do. They have natural language processing.
SPEAKER_00:Yeah. Yeah, I thought the exact same thing. Like, as I've been okay, so first when I was using ChatGPT for programming, I'm like, it feels so wasteful that I have to like switch between the browser and my editor. Like, there's no logical reason why this can't just all be in the editor. Um and then that came. Um, like there's things like GitHub, GitHub Copilot, cursors, super all that stuff. And now my thought is like, okay, now it's in the editor, but like, why do I have to type everything? Like, it would be so much more efficient if I could just say what I want.
SPEAKER_02:Try it out, try it out. It basically records uh the sound, and after that it uses to uh uh AI model up the whisper model to uh remove all the ums and uh all the bumblings and so on from what you said, and you will get uh very decent looking uh prompt at the end or anything else you can dictate. I using it right now. I'm replying to emails just by uh speaking, uh just by uh dictating by answer.
SPEAKER_00:Wow. Okay. I feel dumb for like never finding this before, but uh that's great.
SPEAKER_02:I started to use it three weeks three weeks ago, so things are uh don't okay. Things are moving way too fast.
SPEAKER_00:Yeah, and it seems like, for example, if I want to like manipulate Vim with my voice, um there would have to be some kind of mechanism to translate the spoken commands into Vim commands.
SPEAKER_01:Yeah.
SPEAKER_00:Yeah, like it's obviously like it's trivially easy for for the computer to like take your spoken words and just plop them into an email editor. Um a little bit more involved uh to translate your spoken words into into different kinds in into manipulating different kinds of programs. Although I understand there's people working on that kind of stuff. They're working on like um APIs that translate between language and um manipulating your computer. And so yeah, it seems like all the building blocks are there, like we just need to get them hooked up in a certain way, and and it's gonna get crazy.
SPEAKER_02:Yeah, I think right now we actually have more tools, more building blocks that we actually built of uh the vision API. It's also mind-blowing. I mean, uh uh the AI moder can actually understand what you are doing on your screen and uh also the audio, and uh, I think in the following uh years we'll have an avalanche of new uh yeah, I think that's absolutely right.
SPEAKER_00:Yeah, like right now my bottleneck is just like my ability to stay on top of all the useful new things that are coming out. Like I know that there's so much that there must be so much stuff that would be like so useful to me that I just don't even know about because like it's just too hard to drink from the fire hose.
SPEAKER_02:Yeah, correct, correct. And I I also always think uh like uh the Pareto rule, you know, like 20% of the uh 80% of the effects stamps from 20% of the courses, but in this case, which are the right 20% of the information you need to stay on top of because everything is uh all uh all over the place. But uh yeah. For example, now we say agents because uh 2025 was declared in multiple uh occasions the uh uh year of agents, which okay. But basically, agents are not like uh a new thing. There are uh if you think about them, there are just uh LLM that is running in the loop, in a loop, and tries to achieve a given objective. So this is what agent are there, there isn't like a real technology per se. You just you put an LLM, you give it some tools to interact with in general, you buy tools we are meaning to call some API or to do some stuff with your computer and to run some code basically, one way or the other. And after that, you put that LLM in a loop in order to achieve uh an objective, to buy you uh uh the best uh price it can get for uh plaintiff or uh stuff like that. And even if there isn't anything like fundamental new just to keep up with what's happening in the agent space, it's it's a lot. It's uh it's by going. And yeah, I don't know. I think you uh we'll have to specialize very soon, otherwise we'll be overwhelmed by uh what is happening.
SPEAKER_00:Yeah, man, it's so crazy. Um I and I'd like to go even deeper into the AI agent stuff, but unfortunately we're pretty much out of time. Um before we wrap up, is there anything that you want to touch on that we haven't touched on so far?
SPEAKER_02:Uh no, no, I actually there is uh this is a very small shameless plug uh about uh tomorrow. It will be the launch date uh on uh of a book that I've uh I've written for the past few months and was one of my most fun projects uh that I have done. It's a book on uh Landgraph.js, uh which is basically a framework that allows to build and uh manage AI agents uh using uh JavaScript, uh but you can also use Python with uh the Landgraph uh version of uh Python.
SPEAKER_00:So uh what's the library called again?
SPEAKER_02:Uh Landgraph. Langgraph it's a product uh is this is made by the same uh company that produced uh land chain.
SPEAKER_00:So basically there are uh so it's like L A N G G R A P H.
SPEAKER_02:Exactly, exactly, exactly, exactly. Uh if like starting from tomorrow, you go to my website, js-craft.io, uh you'll see the uh the book. Uh it will pop up in the top menu. Uh share. And uh it is uh basically a study case. Uh it's is made of uh six uh applications uh that uh you can put everything in practice, and uh in each application you learn something you know, like how to memory, how to add human in the loop, for example, how to uh request extra permissions from the human uh in order to do stuff, how to manage, because you can have these agents that can work together, like one agent can be responsible with writing the code, and the other to checking with what the first to checking the code written by the first agent, and so on. And you have to know how to uh orchestrate and integrate these agents. And uh these things are actually more uh simpler than they seem, uh they uh seem at the first glance. But uh yeah, I try to write the book as an easy introduction to uh this wonderful world, brave new world.
SPEAKER_00:Yeah, that's actually something I'm very interested in because I'm wanting to automate a lot of my podcast production work. Like, for example, um before I put our episode live, Daniel, I'm gonna go on Google Image Search and I'm gonna search Daniel Nastase, and I'm gonna grab a photo from your you know LinkedIn or Twitter or whatever it may be, um, whatever I think is the photo that you like just generally use for stuff. I'll grab that one and I'm gonna make an episode summary using Chat GPT, but that's kind of um um manual. And so I want to build a tool that will like just go and fetch all these different things for me, do all these little tasks that I'm doing manually right now, and then like interface with my podcast host and you know, upload that image and paste in that transcript and stuff like that. And so it sounds like your book at least overlaps with like that kind of exactly, exactly.
SPEAKER_02:Uh I would add uh as a competition. In your case, I think you are speaking uh mainly about a traditional chain. Uh basically you are moving from one step to another. Uh an agent, uh because what you do, you get a picture after that you do the transcription, and after that you do the summary, extract the links and so on. This is you move from one step to another. On the other side, an agent uh is when you allow an LLM to take decisions. In this case, there is there aren't too many decisions to be taken. I'm searching for an image with Daniel, and after that, I'm extracting the um what's the name, the transcript, and so on. But when you have a decision that the LLM needs to uh to do, like okay, uh Where does Jason want to go? He wants to go to New York, let's say. Uh, he what's the best way to get there? Where do I uh where I'm able to buy to find the best ticket, price ticket for uh Jason Street and so on? In there we are speaking about agents. But don't worry, uh uh I will also I will send you the link the or the book on uh Langchain that uh I I wrote it some a while ago, and maybe uh you will discover in uh which tools to apply.
SPEAKER_00:Oh, yeah, thank you for that. Um, so we'll put that in the show notes, of course, and then anything else that you want to share uh that people can check out.
SPEAKER_02:Yeah, this uh this would be uh thank you for having me. It was uh real real pleasure, and uh thank you very much for doing this. I did uh I don't think I told you uh when we met in Bucharest that uh you are the reason why uh I started my own podcast.
SPEAKER_01:Oh wow.
SPEAKER_02:Uh yeah, uh so uh yeah, yeah, you you you actually inspired me to do it, and uh I was listening to your episodes and uh enjoying uh I was actually really enjoying the conversation and said okay, I I also try to and you try to do this, it's it's fun.
SPEAKER_00:Wow, well that um makes me really happy to hear. Um and and um I'm I'm happy that you're doing that. Um and this has been a great conversation. We covered um, you know, AI, birds, bears, um all all that stuff. Um and yeah, thanks so much for coming on the show.
SPEAKER_02:Thank you, Jason. And uh keep it up.
SPEAKER_00:Thank you. I will.