31: Local LLMs: Good Enough Might Be Enough

Jim shares his adventure into running LLMs on his own hardware. For him it's less about saving money and more about privacy — working in healthcare, he can't send patient data to the cloud.

• App vs. model: Claude Code and Codex are applications, not models. Features like plan mode come from the app. (Wolf's "Opus Plan" is a Claude Code mode that uses Sonnet 4.6 for most work and Opus 4.8 for planning.)
• Ollama makes local models easy — ~15-min install, runs on macOS/Linux/Windows, and exposes a REST API. Not to be confused with Meta's Llama models. Example: `ollama run llama3`.
• Parameters & training: Think of an 8B model as "8 billion knobs." Training randomly initializes them, then refines predictions over billions of iterations. Wolf ties this to Markov models (parameters ≈ weighted edges) and the Bayes episode (random init = priors).
• Fitting big models in memory: Quantization shrinks 32-bit parameters down to ~4 bits. Mixture of Experts (MoE) keeps only part of a model active (e.g., Llama 4 is ~108B params but ~17B active).
• Jim's tests (M1 Mac Studio, 64 GB), asking why H₂O is liquid: Llama 4 Scout took ~10–15 min and maxed out RAM/swap; Llama 3 (8B) answered in ~31s; Qwen (36B) gave the best answer in just 34s.
• The open question: Is local "good enough"? Wolf's real test isn't trivia — can a local model write and iterate on an 8-page implementation plan? (Homework for Wolf's 128 GB MacBook Pro.)
• Build your own: Fine-tune an existing model or train from scratch. Jim's dream: a local model fine-tuned on his DB schema + 2,000 SQL queries so users could ask in plain English and get runnable Postgres — no cloud required. Browse Hugging Face for specialized models

Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm

Follow us on Mastodon: @RuntimeArguments@hachyderm.io

If you have feedback for us, please send it to feedback@RuntimeArguments.fm

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky

Transcript

Wolf: 0:00

Hey everybody, it is Runtime Arguments, episode number 31, and as always, off by one, because it's our 30-second episode. Um… This week, yeah, it's AI again, but this time it's Jim who has been running AI models locally.

Jim McQuillan: 0:22

Umm.

Wolf: 0:23

On his own machines! For free!

Jim McQuillan: 0:24

Uh, Wolf? Well, yes, yes, that's right, Wolf. Uh, you didn't introduce yourself.

Wolf: 0:30

Oh, crap. I'm Wolf, and my best friend here as always is Jim. Say hi, Jim.

Jim McQuillan: 0:40

Hey everybody, how you doing?

Wolf: 0:42

This, apparently, off to a weird start. I guess we're gonna… Let's see how it goes. Why don't we start with, how was your week? How was your week, Jim? Or two weeks?

Jim McQuillan: 0:53

Oh, I had a nice, uh, I, um, I go through periods of, of, uh, not being very productive. And then, fortunately, that's followed by weeks of real productivity. And I've had a couple of really good weeks, and I'm really enjoying it. And, um, you know, I had a thought, you know, we… this whole podcast came out of Wilf and I having lunch every Saturday. And this past Saturday, uh, we just had a blast. I think there was five of us there? Uh, the, the group, the group is growing, and, and, uh, it's, it's really a lot of fun. And it.

Wolf: 1:24

There were five. Our new friend Patrick came.

Jim McQuillan: 1:32

That's right, that's right. And I just got confirmation he's coming again this Saturday, so that's nice. But one of the things, as I was sitting there, I was sort of thinking about how. how much I enjoy my smart friends. I enjoy all my friends, but I enjoy having smart people to be around so much because I grow from that. And, uh, and Wolf, you're, you're at the top of that list. And, uh, the other guys that show up, um, it's just so much fun. And…

Wolf: 1:59

Oh, you make me blush.

Jim McQuillan: 2:05

And I learn and I thrive on that and it's great. So yeah, that was sort of my thought for the week. How you doing, Wolf?

Wolf: 2:14

Well, I'm never satisfied. Sometimes I'm happy, but I'm never satisfied. And there's a tool I use, and for people who aren't using this, I don't know why you're not. Uh, and that tool is… DIRENV. D-I-R-E-N-V. And, uh, DIRENV is a pretty simple tool, super useful, written in Go, but there's some shell stuff that goes along with it. And the idea behind direnv is that when you cd into a directory that has the magic file, and the magic file is named .envrc, E-N-V-R-C. Um, then DurM does stuff that's in that file, and typically it's easy stuff, like… exporting some environment variables. So, for instance, maybe what you have is a couple of different projects that access different AWS accounts. Uh, you could make it be that when you CD into one of those directories, it exports the appropriate AWS credentials for that account. Um, and then when you CD out of it, you don't have to do anything, they just go away, they're gone, and you CD into the other directory, and you get the new ones. Um, I love that. Um… I. write Python, so I'm always using some kind of project. Form shape and I have been for a long time in the old days we didn't use shapes but then there were like gosh there's a zillion but like poetry is what the two things that I lean on most heavily. these days, is, um, a normal Python project I will manage with UV. UV is a tool that's super fast, written in Rust, replaces a bunch of stuff. Um… It has a way of making sure you've got your dependencies, and they're the right ones, and either executing a tool as though you have activated a virtual environment. or just activating a virtual environment. You can do either of those. Um, and then Pixie is another, uh, tool that, um, manages projects. The main difference is. if all you care about is normal, ordinary Python packages for your dependencies, your requirements, you would use UV. If some of your dependencies are from Conda, from Anaconda, from that world, then you use Pixie, because Pixie knows how to get both. The annoying thing about dir env… Is. for almost any kind… and Jiraenv applies to everything. Uh, like, if you're building a Ruby project, or a… whatever. It has layouts. It knows what to do. for whatever language or shape project you're working with. Um, and you normally start your .envrc file with, for instance, if it's Ruby, layout Ruby. Right? Um… We have an existing one for Python, Layout Python, that assumes you're gonna use the old style of Python version and VN… anyway. They don't have layout UV. And that means I have to implement my own and keep it around in my config files and then I can reference it, but then my projects aren't as portable because my .envrc sample says. Um, layout UV at the top, and there is no layout UV. Ugh. friction. I hate friction. So, it's an itch, and I had to scratch it. Um, there were open tickets, there are people who want this, there's people who explicitly don't want it. People provided implementations that had really good stuff in them, and basically… It all stalled in 2024. and that, that's not good enough for me. So, I took everything good that the people who came before had, um, I wrote a modern. PR implementation and ticket that cited the old tickets, referenced the contributors who had thoughts, addressed the three different groups. The three groups were… NVARC doesn't… Durin doesn't do this, so don't. And one group was, this is great, but just activate the venv. Don't try to bring it up to date. Don't install any packages. And the third group was, yeah, do it all. And I addressed each of those three groups. what you should get or have. And I submitted the PR. Nobody has noticed yet, so it's been a while, but it's there. So I'm pretty excited about that. More exciting to me is we've been talking about DappMux. my Debug Adapter Protocol Multiplexer that lets you hook up as many. pieces of machinery as you want to do debugging. So, for Python, for instance. uh, IPython, plus the debugger, plus this super smart guy, um, named Sean Perry, wrote, uh, uh, DAP Observer. It might be being renamed to DAP Tools. Um, and your editor, whatever editor you want, and you hook them all together, and they're all working at the same time. We've talked about this. Well, Microsoft, the place where DAP comes from, has a DAP home page. where they list implementations. But an implementation is like, I have a debugger, I have an editor, and it does that. They didn't have a page for this. So I submitted a PR that says, here's a whole new category of tools, and the first one is DatMux. And they accepted my PR. Um, I'm not sure it's… it is merged! They did it, but maybe it's not published, so I don't actually see the page live as of when I last checked, which was yesterday evening or something, or else maybe I didn't understand how to make the menu work or something. I thought it was all automatic. Uh, we'll see, I'll keep checking. But yeah, um, it's a public thing now. Now if only I could show Julia Evans that it addresses the problem she was facing and blogged about. That would make my day. So that was my week or two weeks. Rewarding, but a lot about me. solving problems maybe I shouldn't have bothered with. But I did.

Jim McQuillan: 9:32

Yeah, you're on fire, and I love watching that. I see your little posts, I see your posts on Mastodon talking about this stuff, and it's exciting. I like that.

Wolf: 9:43

Um… Let's talk about feedback. I hear you have some with respect to IPv6.

Jim McQuillan: 9:51

Up. Uh, yeah, just a little bit. Back, uh, episode 11, back in September, we talked about IPv6, and it was a fun conversation, a lot of useful information, I think. And, uh, Google just announced, uh, within the last couple of weeks. that, uh, more than 50% of their traffic worldwide is now over IPv6. I think that's pretty good. I think it's being adopted pretty well. I know I'm using it here at home when I connect between my machines. I'm doing the dynamic DNS thing where I set up a machine. For instance, for this episode, I set up a machine called AI. So I can just SSH to ai.local and it finds it. I don't have to set up a DNS entry. I don't even need an entry in my SSH config file or anything. I just, I can just point at ai.local and it connects and the connection is IPv6. Now, there's nothing about dynamic DNS that says it has to be IPv6, that's just the one it chooses. before IPv4. Um, so that's… that's kind of neat. I… I like to see people adopting IPv6. Uh, I know there's a number of, um… Um, uh, internet providers that don't offer it. Uh, I think yours, right? Well, if you're doing, what's, what's the name? You've got a fiber.

Wolf: 11:11

Because I have carrier-grade NAT, IPv6 doesn't go through my, uh… is it an ONT? I always forget what the name of the fiber box is.

Jim McQuillan: 11:20

Well, carrier-grade NAT doesn't restrict you from doing IPv6. They wouldn't need NAT at all if they would just implement IPv6.

Wolf: 11:29

Oh, maybe it's not that I have, uh, CGNAT, maybe it's just that it's my provider.

Jim McQuillan: 11:35

Yeah, there's a number of providers that just don't supply it yet, and that's too bad. But hey, it's getting better. Over 50% of the traffic to Google is IPv6, so that's kind of neat.

Wolf: 11:47

I have a little feedback.

Jim McQuillan: 11:49

You do.

Wolf: 11:50

I do. And my feedback is, um, that someone was talking to me about the show. Um, and they said, uh, that the show, uh, gets them fired up about, um, software engineering, about solving problems. Um, that…

Jim McQuillan: 12:09

Nice.

Wolf: 12:12

boy, did that hit me in the feels. I mean, that's what I want. That was exactly a great thing to hear. I was very excited about that.

Jim McQuillan: 12:17

Yep.

Wolf: 12:24

And, uh, I'm happy that we're starting to make an impact. I'm not, um… I never assume that it's gonna… anything is gonna go as fast as you think or plan for it to go. Um, but the fact that we have listeners, and you know, we're… We're well over 2,000 downloads, 2,500 or something like that.

Jim McQuillan: 12:45

Uh, 30… no, 3,500.

Wolf: 12:47

Oh, 3,500, shit. Oh, crap, you'll have to beat that out.

Jim McQuillan: 12:48

Yeah, yeah, we're, I mean, you know, compared to some of the big podcasts out there, that's, that's, uh, that's, they laugh. Yeah, they laugh at that. But hey, we're happy. We're having fun. Our, our, our listeners are, uh, are, uh, happy, I think.

Wolf: 12:55

Yeah, that's what they get in the first hour.

Jim McQuillan: 13:05

I hope.

Wolf: 13:06

Yes. Um, why don't…

Jim McQuillan: 13:08

Some of them are.

Wolf: 13:09

Why don't we do this? Why don't you tell us about saving money? Oh wait, I changed the topic. Why don't you tell us about running AI locally?

Jim McQuillan: 13:19

Yeah. Yeah, yeah. Now, I'm gonna get into depth about that, and right off the top, for me, it's not about saving money, because I'm not hardly spending any money on AI. It's more about the ability to run things local, to keep my stuff out of the cloud. Um, and because I work in a… in the healthcare field, um, I can't use AI for any of the data that I work on. Um, so I'm looking for ways that I can. do that. Anyway, um… You know, AI is a controversial topic, to say the least, right? There are people that love it, and there are people that hate it, and you can't convince either one of them about the other's point of view. kind of somewhere in the middle. I like some of the power of it, and the things that I can do, yet I realize the… the evil corporations that are building data centers and, you know, doing all the things, harvesting data all over the internet. That doesn't bother me enough to stay away from it. Um, but that's out there. So, uh, I, I know Wolf, you use it for a few things yourself. Um, and, and I, yeah, I, I, I, I love hearing about that too. Uh, there's another episode, I forget the number, but Wolf talks about how he uses Claude for a lot of, uh, a lot of stuff.

Wolf: 14:41

A few.

Jim McQuillan: 14:53

And it's pretty neat.

Wolf: 14:53

Unexpected stuff.

Jim McQuillan: 14:55

Yeah, yeah. You know, I use it primarily as a coding assistant, and it's working pretty well for me to explore languages that I'm not very familiar with. I'm doing a lot of TypeScript now, and Claude is helping me a great deal with that. I appreciate it. Uh, you know, back, uh, episode 8, that was August of last year, uh, we did an episode on, on AI, and Wolf. talked about, um… you know, how to do programming with it, and it was pretty neat, and I think the title of that episode was, uh, Dragging Me Into It, and that's kind of what he was doing. Um… I wasn't using it at all back then, and now I am. And one of my real concerns was because I don't understand how it works. I mentioned during that episode that I understand how compilers work, and I understand how the shell works. And I understand all that stuff, so I'm comfortable using it. AI, I don't have a clue. So this episode is…

Wolf: 15:55

See, I want to insert words.

Jim McQuillan: 15:57

Yeah, yeah.

Wolf: 15:58

Because I think there's two things here. I think you do understand how it works, or you could easily, but here's the part. I think you don't understand why it makes some decisions. Knowing how it works, and knowing why it makes a particular decision. is two different things.

Jim McQuillan: 16:24

Sure.

Wolf: 16:25

And I think that's like being married, whichever side you're on, and I'm not going to pick one. I think it's like that for your spouse. Like, you know. How they work?

Jim McQuillan: 16:40

Yes.

Wolf: 16:40

You even know what they know, but you don't know why they say those things. Why did they say that? And I think it's the same with AI.

Jim McQuillan: 16:46

Or why did they do that? Yeah, probably. Although… although I… I don't know how it works. This episode is my adventure into trying to understand. It's just a small bit of how AI works.

Wolf: 16:49

I know how it works.

Jim McQuillan: 17:00

Uh, and it's, uh, it's getting really interesting and, and I'm, uh, I'm enjoying this little adventure and I'm gonna be on this adventure for a long time. So there will probably be future episodes where we talk more about how AI works. Um, yeah. a couple of weeks ago was the Apple Worldwide Developer Conference, and… that… the keynote and pretty much every talk was almost all about AI, to the point where I wasn't really that interested in listening to the talks, because they talked so much about AI. But one of the things that they did mention was Xcode, which I use for developing iOS apps. They're tightening up the integration with AI into Xcode, which is pretty cool. It works with Xcode now. I'm sorry, it works with Claude Code, and it works with, I think, Copilot and OpenAI's GPT. But, one of the things that they mentioned was, you'll be able to connect up local LLMs. to Xcode. Which I thought that was pretty neat. And they mentioned a few of them like Llama and Quen. And I think they mentioned maybe Kiwi. I didn't know what all those things were. I knew about Llama a little bit, but I didn't know what the rest were. So I started digging in and that's what I thought. this might be an interesting episode, because I'm sure other people are in the same boat, right? They hear these words, they don't really know what they mean. So today, we're going to get into some of that meaning. Now, uh, there are a number of commercial models. I've already talked about OpenAI, they've got their GPT, I think they're up to 5.5 right now? It's a very capable model. You use it with Codex, and I think ChatGPT probably uses it. Um, and Anthropic, uh, they've got a couple of models that, that I am a bit more familiar with. They've got, uh, I think the current version is Opus 4.8.

Wolf: 18:59

That's correct.

Jim McQuillan: 19:00

There's. Uh, and it seems like about every 2 months they come out with another one, so it probably won't be too long before they release yet another. And then there's Sonnet 4.6, which is the, how would you say it, the less capable, faster model. If you want to do deep reasoning.

Wolf: 19:18

Uh, believe it or not. I use 4.6 predominantly. Um, I'm, uh, when you pick what model you use, I have selected, uh, the word OpusPlan. It's an 8-character word, and what that means is.

Jim McQuillan: 19:23

Yum. Yah.

Wolf: 19:35

do almost all the work in Sonnet 4.6, but when you do planning, switch to 4.8. But, of course, Haiku, the really fast, cheap one, is always available, and inside any.

Jim McQuillan: 19:44

Okay.

Wolf: 19:52

skill you write, you can say, oh, by the way, this skill requires models such and such. And then you get that one.

Jim McQuillan: 19:58

Okay. Okay, um, you know, I just have mine set to Opus 4.8, because I don't use it that much, and to me, the reason why you would switch between one and the other, I believe Opus. Choose up your tokens faster?

Wolf: 20:16

It does.

Jim McQuillan: 20:16

Uh, and it's slower and, and I'm not worried so much about the speed and I believe I'm nowhere near hitting my limits on token usage. So I just leave it at 4.8 and don't worry about it. Uh, I suppose if I bump into the limits, uh, then the choice there is either pay more money and increase my limits.

Wolf: 20:19

It is.

Jim McQuillan: 20:34

or switch to Sonnet. which is uses tokens. uses fewer tokens. So, uh, there's a couple of choices there. Now, Anthropic also has a model called Mythos, and they have something called Fable 5, and this has been in the news the last couple of weeks. Mythos is interesting. I was confused about it. I didn't know what's the difference between that and Opus and Sonnet. Well, Mythos is highly advanced, specialized cybersecurity model that can be used. Uh, to both, uh, identify and create sophisticated software exploits. So this is something that it seems very, very dangerous. So the way they were licensing Mythos was a small, select group of people were allowed to use it. Um, I know like, um, uh, people who are in the security department of projects, uh, for instance, uh, Daniel Stenberg, the guy that runs, uh, uh, the curl project, he has access to Mythos. Um, and that lets him run it against his code, and I guess they tightly control what you can do with it, but he runs it against the curl code, looking for exploits, and he found a couple of exploits, uh, nothing major, because that code is so tight anyway, and people are looking at it so carefully. But he has access to that, and other people, like the people in the Linux kernel, have access to it, and corporations and stuff. Well, Fable 5 is a model that's basically Mythos, but they put some pretty strict guardrails on it, so you're really limited at what you can do. There's an ongoing feud between Anthropic and the U.S. government, and the U.S. government has basically put a stop to that. Nobody can use either Mythos or Fable 5 right now. Um, and there's a pretty interesting article on the Anthropic site, sort of explain… explaining their stand on this. Uh, and they basically, in order to comply.

Wolf: 22:44

They will be painting. They will be painting both of those models American Flag Blue.

Jim McQuillan: 22:46

I'm sorry? That's right. The article is pretty interesting explaining their stand on that. And they've basically made it so nobody can use either of those two models just in order to stay in compliance. with the federal government. And that's too bad. I think that'll probably change. There'll probably be court cases and all that kind of stuff. One of the downsides is the NSA, National Security Agency of the U.S. government. uh, was relying on Mythos for a lot of stuff they do, and they don't get to do that anymore. By…

Wolf: 23:29

That's a downside.

Jim McQuillan: 23:31

Uh, well, to them. Right? They no longer have access, and they're a bit upset about that, but OpenAI says, or I'm sorry, Anthropic says, hey, too bad, you said we can't license it? That includes you. And I think that's — I sort of admire Anthropic for being that way. But, you know, there are other commercial models as well. I haven't looked into them. But they're out there. You can find them easily enough by googling. But what I really want to talk about today is local models. You can actually install and run local models on your computer, and it turns out you don't need that much of a computer to do it. Now, I have a Mac Studio with an M1 processor, that's Apple Silicon. And, um, 64 gigs of RAM. It's not a huge machine. Um, it, it, it, right now you can get an M4. Uh, unfortunately, Apple is limiting the RAM to 64 gigs, I think, uh, because of the shortage of RAM out there. But my little M1…

Wolf: 24:42

Uh, I, wait a second, you're M4 Studio, you mean?

Jim McQuillan: 24:46

My M4 studio.

Wolf: 24:48

Right, like, because that you can get an M5 laptop.

Jim McQuillan: 24:48

Yes. You can now get an M5 MacBook Pro with 128GB of RAM. I looked at it. Boy, is that pricey.

Wolf: 25:02

And it's gonna be more expensive, Apple says.

Jim McQuillan: 25:04

I'm wondering if they already increased the price, because if you want a pretty well-decked-out MacBook Pro with an M5 and 128GB of RAM and 4TB of storage.

Wolf: 25:19

I kinda do want that. I'm not gonna ever have it, but…

Jim McQuillan: 25:19

Um, it, it, it, it's about. It, it, it's about eight grand. Yes. Can you imagine that? $8,000 for a, for a, a, a laptop.

Wolf: 25:29

Um, yeah, I can imagine that pretty hard.

Jim McQuillan: 25:33

Oh. I'll be in the market for a new machine this fall, hopefully. I was sort of hoping that the studio would have an M5, but… Uh, between the prices and the limited RAM, I'm not sure I'm gonna make any, any change at all, uh, for the foreseeable future, because, man, 8 grand, that's a lot of money. Anyway, let's get into the models. There's a number of models you can install and run locally. And it turns out, it's very simple to do. Extremely easy to do. I did it in 15 minutes. It was so easy. There's a program called Ollama. Now, I want to make a distinction here. Meta has models called Llama. That's L-L-A-M-A. There's a program called Ollama, O-L-L-A-M-A. That's a program for downloading and managing your local models. The Llama models are… some of those models you can run in Ollama. It's confusing. I'm not quite sure why they named things like that, but anyway, Ollama is the program you want to install, and it's a really simple install. Um, I, I won't get into the instructions on how to install it here, that's, it's all out on the internet. Uh, but you, you install Ollama, and, and all you gotta do is, is, uh, if you wanna run, like, uh, Meta's Llama 3. You type ollama run llama3. it'll go download the model for you, and start it running. And these models… there… there's some big ones, and there's some small ones. I think, um… Uh, I forget the size. It seemed like Ollama was 17 gigs in size or something. It downloaded pretty quickly. Um, there, there are, there's a, that's the, I think the 8 billion parameter model. Uh, there's a 70. billion parameter model that's, like, 47 gigabytes. Fortunately, I've got a nice, fast internet connection, so these download in a matter of minutes, so it's pretty nice. And they both run, those two Llama 3 models, they both run pretty well on my little Mac Studio. I was kind of happy with that. Umm.

Wolf: 27:49

I know you want to talk about a bunch of these models, but you did say some numbers.

Jim McQuillan: 27:53

Yeah. Okay.

Wolf: 27:57

And on behalf of the listeners, I feel required to ask. Eight billion. Parameters? What? 70 billion parameters? What does that mean?

Jim McQuillan: 28:09

Yeah, okay, um… Parameters are an adjustable internal value, primarily weights and biases, that the AI uses to process the language and generate responses. um, they're like dials, or a recipe, uh, measurement, um, that you encode into the model, or they, they, the people who built the model encode these things in. And the number of parameters that a model has is, is kinda like the. power of the model. It's not the only thing. Um, but it's, it's, it's how they tune, uh, these, these models. Think of, uh, think of an 8 billion model, uh, uh, uh, an 8 billion parameter model as having 8 billion knobs that can be adjusted. to tune that… that model, uh, to do what it's want. That's… that's what training is all about. Uh, when you hear about model training, that's where the real horsepower is needed.

Wolf: 29:04

I…

Jim McQuillan: 29:08

And you're gonna say something?

Wolf: 29:08

I sort of get what you're saying, um, and it sounds like the way you're saying it is way better for, uh, uh, uh… A non-expert to hear it.

Jim McQuillan: 29:19

Yah.

Wolf: 29:19

But, um… you know, my familiarity with how these things work is that they are not unlike a thing I've worked with for decades, um, called Markov models. And a Markov model is a graph, a big old.

Jim McQuillan: 29:33

Right.

Wolf: 29:36

graph. So there's nodes and edges, and typically the edges are weighted, um, and you'll move. between nodes by crossing the appropriate edge, depending on whatever thing it is you're letting guide you through the model. So if you're using it for compression, then as you get a new input symbol, you'll cross the right edge into the next. node and the edge you cross tells you, for instance, you pick that edge based on what the input was and the node you land in gives you some bits to emit or something. When you are looking at one of these giant. And there's several different ways to implement AIs. But when you're looking at one of these. It is not unlike those Markov models. And the thing that an AI expert calls a parameter? is what I would have identified as an edge. But in a lot of my models, from back in the day, compression and whatnot, the. thing that's on an edge, yeah, it's a weight, the number of times you cross it or whatever, is very simple. It might be a character or a word. It turns out that the thing that lets you pick which edge to cross, which parameter to use. is… Not even just one token, but some set of tokens, of which maybe you need any particular one, or some combination of them. And then nodes are parts of the answer, what things to put out next, because remember, these are giant predictors. So, when you say, knobs that need tweaking, that's the thing that hits me weird, and yet, it really is the right way to say it. And how you build the model? Because you start off with no nodes, no knowledge at all, um, and what you do is you acquire knowledge.

Jim McQuillan: 31:48

Mmhm.

Wolf: 31:53

In whatever way. You know, reading books, or stealing them, or however you got it. And you add new nodes, and as you… figure out the tokens of the thing that you're grabbing and pulling in? You build these new edges. And that's the knobs that get tweaked. That's the expensive part that you're talking about, where you need all the horsepower to identify the tokens, and pick which ones go together to make an edge, and determine what's going to be emitted at the brand new node you just made. So that's how I think of it. They're not all.

Jim McQuillan: 32:34

Sure. Go ahead.

Wolf: 32:39

They're not all shaped like this. Um, there's… there's many different ways to do it, and a neural net is not like this, and uh… you know, it all depends on what you're making, but the thing I just described is, um…

Jim McQuillan: 32:41

Yeah. Yeah, okay.

Wolf: 32:54

a way that these two things Markov models and the general form of one of these multi-parameter AI graphs is built. This is how they kind of correspond. They're not the same, but they're so much in the same family that you can think about them that way.

Jim McQuillan: 33:11

Yah. Well, this… this is… this is the part that I just still don't understand. And I'm… I'm… I'm gonna work my way towards it slowly. Uh, it… I… I do think it's pretty interesting. Um… anyway, think of, uh, you know, the smaller models are 8 billion parameters. Think of 8 billion knobs that… that can be tweaked. And, you know, there's… there's not a human, uh, looking at each one of those and saying, oh, I think this should be a 7. Right? It's all… it's all done automatically. Um… in… when they're doing it, they are… they first, they initialize. all 8 billion parameters to random numbers. So the output then is just gibberish. And then they, as part of the training, they go through something called forward propagation, where they make a guess at what each of those numbers should be. or where each knob should be set.

Wolf: 34:12

And that should sound super familiar to anybody who listened to our Bayes episode, because when you are doing Bayesian thinking.

Jim McQuillan: 34:18

Yes, yes.

Wolf: 34:22

and you don't have any priors, um, you start with arbitrary knowledge, you know, sometimes 50-50, or, you know, whatever kind of decision you're making. This is setting up priors when you don't have any.

Jim McQuillan: 34:30

Right. Right. Yeah. They'll feed it a sentence. For instance, the example that I saw on the internet was, they'll feed it a sentence like, the cat sat on the… and they leave the last word out, and then they use the model to try to predict what the next word is going to be. And before they've. tuned anything, uh, it's gonna just come up with some random word, like, imagine it came up with the word banana. Uh, the cat sat on the banana. Now, that doesn't make any sense. So they go through and they calculate how far off they are, and I think that's where they're using the Bayesian. thinking, and they keep working through this and working through this, they run these 8 billion parameters. They might run through this model billions of times, trying to refine the model over and over and over again. And this process can take months. I think this is why, like, like, uh, Opus comes out, you know, uh, uh… Anthropic comes out with a new version of Opus, like, every two months, because it takes a long time to… Retrain the model on, you know, they, they gather new data.

Wolf: 35:41

Lots of time, lots of electricity, and lots of GPUs!

Jim McQuillan: 35:48

Yes, yes, they run it over… Anthropic won't say just what the size of their compute farm is, but it's tens, if not hundreds of thousands of GPUs, maybe millions at this point, and it just runs over and over and over again. refining the model and refining it until they finally come up with something where it can predict what the next word is going to be. And I still don't know how they get from that to what I'm doing, you know, by generating code with Claude Code. It's still magic to me, right? Um… Anyway, so we sort of talked about, yeah, Llama, Meta has their Llama models, which are. Pretty capable. There's Llama 3 that I played with quite a bit and Llama 4, which is newer and larger. That the Llama 4 is like 17 billion parameters. In fact, it's really they did something. I've got it here in my notes. They did something called MOE. Umm. Um, I don't see that right. Anyway, um, this thing, um, um. I'm sorry. Anyway, they break the model up into smaller pieces, so that it's not really processing the whole model at once. it, uh, it, it, they break it down. So, um, for instance, uh, Llama 4 is 108 billion parameters, but there's only 17 billion active at any given time. So they can make much, much larger models that are much more capable, but they sort of partition it, and they look at small pieces of it at once, and those small pieces can fit into memory.

Wolf: 37:35

So this sounds like… You are starting to answer the very next question I wanted to ask you.

Jim McQuillan: 37:42

Yes.

Wolf: 37:43

And that is, you know, $70 billion of anything is a lot.

Jim McQuillan: 37:49

Sure. Yah.

Wolf: 37:50

Um, and we could do a little math, but that would have big numbers in it, and we already know how many episodes I've talked about hating big numbers. Uh…

Jim McQuillan: 37:57

Yep.

Wolf: 37:59

No matter what you say, if it starts with $70 billion, there's gonna be a lot of RAM involved. So my question is, how on earth do you get these giant models to run on your computer, your personal computer?

Jim McQuillan: 38:05

Right. Okay, you're right. You know, an 8 billion parameter model, you know, if a parameter was one byte, that's 8 gigabytes of space, right? Am I doing the math right? That's a fairly simple one. 70 billion parameters, that's going to be 6 billion. Uh, 70 gigs. And let me tell you, there's models that are much, much, much larger than that. Um, Anthropic won't release, uh, the, the size of their models. They won't tell us how many parameters there, but it's estimated, uh, uh, Opus is estimated to be between 800 billion. and 2 trillion parameters. Uh, now they're using this MOE thing so that they break it down into smaller pieces so that it can run and not, not have to load the whole 2 trillion in memory at once. And again, that 2 trillion is just a guess. Um, uh, uh, open AI with their GPT 5.5. That's estimated to be between 1.5 trillion and 9 trillion parameters. Um, so yeah, how do they, no matter what, what the number is, 8 billion, 17 billion, 70 billion, whatever, how do they fit it into memory? Uh, well, Wolf talked about these, these nodes, uh, in, in the, um, the, the 8 billion. parameters, or nodes in the… in what you call it, the Markov model?

Wolf: 39:32

Right, the parameters are the edges.

Jim McQuillan: 39:32

Up. Okay. Yeah, okay, so the edges, if you've got 8 billion of those, well, when they're building these models, those are usually a 32-bit floating point number. So imagine 32-bit, that's 42 bits. 4 bytes times 8 billion. That's a lot. That's an awful lot of memory right there. So they do something called quantization. I think I'm saying it right. Where they will reduce that number. Uh, mathematically, down to something much smaller, and it's very common for the, for the models you would run on your, on your own computer to, to use a 4-bit. to have 4-bit quantization. That means each one of those nodes, or each one of those edges, is only 4 bits.

Wolf: 40:18

They're putting it into buckets.

Jim McQuillan: 40:24

So, 8 billion…

Wolf: 40:24

Cause you're picking which bucket it is.

Jim McQuillan: 40:26

Yeah, okay. You know a lot more about this than I do. So anyway, that's how they do it. It's a method of compression. They take that 32-bit number, and they shrink it down to 4 bits, because that's enough to give you what you need for. for these models to do what they do. Now, if you had more bits, I think it would be a more powerful model. Uh, it's just one of the things, right? Maybe a more accurate model? Um, but that's… that's how they do it. They… they, uh, instead of a 32-bit number, it's a 4-bit number. Um… And, you know, because we don't have that much RAM on our machines. You know, like I said, my machine here is 64 gigs of RAM. And it's funny because I tried a bunch of different models and I tried the… Llama 4 17 billion parameter model. It's, there's a name for it. I'm sorry. Scout. No, no, Behemoth isn't available. Scout.

Wolf: 41:33

Behemoth? Scout.

Jim McQuillan: 41:38

is a 17 billion, uh, active parameter model. Maverick is a 400 billion. Uh, Behemoth, which isn't available to us, is 2 trillion parameters. Um… So I ran the Scout model. And… I pushed my system so hard. It's a 67 gigabyte download. So that took about 30 minutes to download. I was pegging my… I've got a 1.2 gigabit internet connection, but unfortunately my router maxes out at about 450 megabits.

Wolf: 42:12

Ouch.

Jim McQuillan: 42:13

Um, yeah, I've got to replace it one of these days. It's just a, it's a Netgate. It has great software on it, but it can't handle the full bandwidth of my internet connection. So I pegged that thing for 30 minutes to the point where I had a VPN running to a customer. That died. I had other things running that just wouldn't work. Uh… that was just the download. Once I finished downloading it, and I started running it, um… I pushed this little M1 64 gigabit machine so hard. Um, I, I. I tried asking a question. In fact, I did ask this question of several models. I was driving in the car, uh… last week, you know? When I drive out to Ann Arbor for lunch, I have, like, 50 minutes in the car by myself. So I have all kinds of weird thoughts, and I… somehow, this random thought entered my mind, and that is… Um, if water is H2O, that is, uh, two hydrogen atoms and one oxygen atom, and they're both gaseous at room temperature. Why, when they combine, is it liquid? I didn't understand, okay? So I thought, hey, there's a question to ask AI. So I did. I downloaded these various models, and I asked that question. And boy, I asked that question of Llama 4 Scout. And it took, uh, maybe… 10 or 15 minutes. I didn't count exactly how long. I didn't want to run it again to count it. It took at least 10 minutes to run and it maxed out my RAM to the point where I was using up almost all my swap. And I've never pushed my studio this hard. You know, I run. I run Xcode, which is kind of a pig, and that goes through short bursts of heavy usage, but this thing, that fan was running like crazy on that CPU, and it used up, like. every bit of the memory that it could, and it finally came out with an answer, and I was surprised. All these models came out with great answers to that question, and it describes covalent bonding, and positive-negative charges, and why it's liquid, and it was pretty interesting. But all these models were capable of doing that, all of them that I tried. Uh, I… I think the best model, uh, at least what I thought was the best answer, the most complete answer, was when I used QWEN. That's Q-W-E-N. And that's… that's a model with, uh, 36 billion parameters. And… I was surprised. Asking that question only took 34 seconds for it to answer. And that's on my M1. Now I also, you know, Ollama is not limited to Mac OS. It runs fine on Linux and Windows. So, I fired up a Linux virtual machine. And I ran Qwen there with Ollama, and it took 121 seconds. So, yeah, it's slower. Now, that machine is just a VM. It doesn't have access to any GPU at all. It's just CPU. And Ollama will do what it can with the hardware you have. Uh, now, you do have to have some memory.

Wolf: 45:29

So… If you, um…

Jim McQuillan: 45:32

But yeah.

Wolf: 45:35

I assume Quen is not the only. good, fast model. And yeah, it's big, you described big numbers, but if you can run a good, fast model that gives you a great answer on.

Jim McQuillan: 45:50

Okay.

Wolf: 45:53

Your own machine? Ummm… Why would you spend money on? ChatGPT or Anthropic or Copilot or why?

Jim McQuillan: 46:06

Well, you know the the. First of all, those models, the Sonnet and Opus and GPT-5.5 and Copilot, they all run out in the cloud. They don't run locally on your machine. Umm. And there's some benefits to running locally, but there's some downsides as well, and that is the sheer horsepower you need to actually run those. Now, the… the bulk of the processing is done when they train the models. And by the time we're using the model, it's trained. You don't care about that anymore. But the other thing that you need the horsepower for is inference. That is, taking what I type and trying to figure out what it is that I typed, and turning that into something that means something. so that it can process it with a model, and then come up with an answer, and spit that out. All that stuff takes a lot of horsepower. Um… And yeah, these models… you know, I did mention that I used the… it was the Llama 4 model that I ran that took, like, 10 minutes to run. Llama 3 is much, much faster, and Llama 3, the 8 billion parameter model, came out with a great answer. in 31 seconds or something, um, which is comparable with the Qwen model. Um, but back to Wolf's question, why would you… uh, pay, uh, Anthropic to use Claude Code, uh, to do your work. And, and I think… like I said, to do the real horsepower, to do the real heavy lifting, you gotta run the model on really big hardware. So if I had… I don't have access to a Linux box with a GPU. If I had access to that Linux box with lots of RAM and a GPU, I think you could do really well running these. Now, one of the things that I learned in this process, I was always it was a little bit muddy to me what the difference was between. Claude, Sonnet, Opus, uh, Claude Code. And then over the OpenAI world, ChatGPT, Codex using the GPT models. I didn't really understand what all those things really meant. And it turns out that, like, Claude Code is just an application. It's not a model. It's an application. That lets you… use models. Codex is an application, and it lets you use a model. Interesting thing is, you can use Claude Code, and sure, you can pick between Opus and Sonnet, and what's the thing you used again? Sonnet…

Wolf: 48:40

Uh, there is Haiku, as well.

Jim McQuillan: 48:43

Yeah, right, but there's a thing that you use where it sort of switches back and forth automatically.

Wolf: 48:45

Oh, so the name of the thing I'm using is Opus Plan, but that's not a new model, that just picks between models depending upon the situation.

Jim McQuillan: 48:50

Yah. Okay. Oh, it's… That's probably a feature of Claude Code that's doing that switching for you.

Wolf: 49:00

Yes, it is.

Jim McQuillan: 49:01

I'm guessing. Okay, well, it turns out that if you download these local models, you can tell Claude Code to use the local model.

Wolf: 49:02

It is.

Jim McQuillan: 49:09

So you can tell Claude Code to use Quinn or Llama. Uh, which means you can do your programming with that. I didn't… I did run it, just to see that it works, and yes, it does. Um, but I didn't try it in an actual programming exercise to see the quality of what it came out with. Uh, I think that would be a really interesting thing to do. Maybe I'll give it a try. And give, uh, some feedback, uh, next episode. But, again, back to Wolf's question, why would you pay? I think, right now, we're paying for the quality of the model. In my head, anyway, I think the… the Opus model is probably gonna be far, far superior to anything I could run locally. And, you know, I know, Wolf, you've got 128 gigs of RAM and an M4. I think… uh, running a model, uh, in the cloud, uh, like at Anthropic is still gonna give you better results. Maybe a local model is good enough results. Uh, I, I…

Wolf: 50:06

Well, to me, the deciding question is not, why is H2O a liquid? The deciding question is. can you write a plan and us iterate on it? Um, that's what Opus does for me, is I want to do some incredibly complicated task. that's going to have many steps, and I want to figure out what steps, some of which I'm going to do, some of which I'm going to let Claude do, some of which I'm going to decide shouldn't happen at all, some of which. Claude won't think of, and I'm gonna say, by the way, we also need to do this, and it's gonna take me 12 tries to get it, and the plan is, you know, 8 pages long.

Jim McQuillan: 50:49

Sure.

Wolf: 50:53

Um… I want to know, can a local model do that? Is it even in the running?

Jim McQuillan: 51:03

Okay, my understanding is Claude Code is what provides you with that ability to do a plan. That's a feature, a function of.

Wolf: 51:03

Umm.

Jim McQuillan: 51:13

your app, whether it's Cloud Code or Codex.

Wolf: 51:16

Well, the idea… well…

Jim McQuillan: 51:17

Um. The interacting, the…

Wolf: 51:19

It's a really big answer that came from the model. The fact that I can look at that answer, edit it, and iterate on it, that's a feature of the app.

Jim McQuillan: 51:31

Right, that's not the model, that's the app. So I think if you're using like Claude Code and you're used to, you know, doing like slash plan to start your plan going, that's Claude Code doing that. Now, if you have Claude Code plugged into Llama or Quinn.

Wolf: 51:31

Um…

Jim McQuillan: 51:46

I don't see why it wouldn't work just as well. Now, would the results be as good? Would the results be good enough?

Wolf: 51:52

That's what I want to know. Can it give me a sensible 8-page plan?

Jim McQuillan: 51:54

That's worth trying. Yeah, that's worth trying, and we should do that. And like I said, it's like a 15-minute install to get Llama up and running, or Olama up and running. You ought to give it a try with your big monster MacBook Pro. Because only you can really say whether it's good enough for you or not. I can certainly give it a try. So yeah, I just gave you some homework. Alright, so… I'm amazed at what these… what these, uh, local models can do. It's pretty… pretty neat. I just sort of touched the, you know, the tip of the iceberg here.

Wolf: 52:32

Well. This… this gives me… maybe this is my final question, because I don't want us to go so… so long, but…

Jim McQuillan: 52:37

Umm.

Wolf: 52:44

Um, what if these models aren't good enough? What if I want something that knows about the stuff I care about? Can I build my own models?

Jim McQuillan: 52:55

Turns out you can. Um, there's a couple of ways. You can, uh, you can fine-tune an existing model. Or you can start from scratch and build your own model. That's a huge task, but there are tools out there to let you… to help you with that. But I'm more interested in the first one, because, uh… And so after looking at all this, I'm kind of wondering, what does all this mean? How can I actually use this? What can I do with it? And, you know, I run a company where we do software for doctors. And there's a query builder where people can just go in and point and click and add what they want. But the serious people, there's a couple of them, they write SQL to do this and it's complex. And, you know, my customer, my big customer is a. They're a cancer place, a place that detects breast cancer. And the queries are pretty complex, and I love writing queries. I think I'm pretty good at it, and I help whenever I can, because I get so much joy from it. But I'm kind of wondering, is there a way that I could provide an AI model. to help them create their own queries. Now, like I mentioned, I can't give access to my database to anything running in the cloud. I just can't do that. Uh, but what if I could run my own model locally that has access to the schema of my database, and it has access to the library of over 2,000 queries that we've written to, to, to, you know. do cancer research. That's a lot of what they do there. They have tons of data, they have data on millions of patients. Wouldn't it be cool if I could say, hey, here's this AI model that can do that for you?

Wolf: 54:48

And can you?

Jim McQuillan: 54:49

Uh, well, it turns out, yes, I can. I have not tried yet, but I'm very, very interested in doing this, because imagine going into the report creator, and the point where you go to create a query. What if you could just ask. Um, my LLM to write the query for you with its knowledge of, and now my, you know, the knowledge isn't great. 2000 queries is not a lot to learn on, but it's a start. Uh, the database schema is obviously crucial. Uh, for it to understand. But I could fine-tune one of the models. to do exactly that. And then the neat thing about Ollama is, when you run Ollama, it sets up a server listening on a port, and you can connect to that port. It's a RESTful API. You can connect to that port with curl, if you want, and you can feed it a query. You know, you can feed it your the text of your query, you know, something like, write me an SQL query that will find the number of cancers in a patient population of 40 to 50 year old women. Uh, who are African American. That's my goal, is to be able to, through an API, ask that question, and the output should be an SQL query that'll run in Postgres. Um, the tools are there. Uh, I just have to find the time to, to play with it. And I think that's, that's where the power really comes, with your own models, uh, running them locally. Training your own model or fine tuning an existing model. And, you know, I mentioned, uh, a whole bunch of models. Ollama, or, uh, Llama, um, Quinn, uh, there's DeepSeek, that's a Chinese model that got a lot of attention several months ago. That's out there. I downloaded it, I ran it, it works. But there's a, there's like a whole place to go on the internet called huggingface.com, which is just chock full of models. And they have models for all kinds of things that specialize in all kinds of neat things. Like maybe you want to generate graphic images. There's a model for that, or there's a whole bunch of models for that. Maybe you want to feed it graphic images and have it describe what's in that image. There's models to do that. You can download those things and run them locally. It's… It's a huge world, and it's only getting bigger, and I'm kind of anxious to see where it takes us, and I'm really interested in seeing how I can integrate this local thing into my application. Any more questions, Wolf?

Wolf: 57:27

I am dry. You used me up. This was a good one. I was pretty excited by the stuff you had to talk about.

Jim McQuillan: 57:33

You said… Yeah, it's pretty neat stuff, and I don't totally understand it, but I'm trying to learn, and I'll tell you what, as I learn more, I'll be happy to share it with you and with the world through this podcast. So yeah, it was a lot of fun. I really enjoyed it. So… Why don't you take us out?

Wolf: 57:53

Let me thank everybody who made it this far. In fact, let me thank the people who didn't even make it this far. We're super happy you listened and we hope that we brought something to the table that you. uh, enjoyed. Uh, we love feedback. Please send us feedback. You can email us, feedback at runtimearguments.fm. There's a website. It is http colon slash slash runtime arguments dot fm. Um, you can reach us on Mastodon, at runtimearguments at hackyderm.io. Uh, Jim is on Mastodon, and let's see if I can get it right this time. It would be a first, after a year. Uh, let's try at jammcq at, uh, hackyderm.io? Is that right, Jim?

Jim McQuillan: 58:54

That's exactly right. That's it.

Wolf: 58:55

Oh my god, I got it right. And I am @yesjustwolf@hackyderm.io.

Jim McQuillan: 58:56

Good job.

Wolf: 59:03

Um, we'd love to hear from you. We want your opinions. What did we do right? What did we do wrong? What things would you like us to talk about in the future? Um… Anything you have to think about, anything we've done anytime in the past, uh… When you go to our website, you will see there's a transcript, there's show notes, there's gonna be links, um… Thanks for listening, and I think that's it from me. Goodbye, everybody. Jim?

Jim McQuillan: 59:35

Yeah, thanks everybody. Appreciate it.