24: Bayes' Rule - The Formula For Learning Everything

If you've ever debugged a program, looked for lost socks or tried to figure out why red spots are developing on your skin, then Bayes' rule was almost certainly used to help you on your journey. Even if you don't know anything about it. Humans have evolved to solve problems but along the way, we as a species sometimes fall for traps or fail to consider all the evidence when figuring things out.

In this episode, Wolf explains what Bayes' rule is, how we use it and how we could use it better to solve our mysteries.

One sentence
Bayes' Rule is the formula that tells you how to update what you believe when you get new evidence — it combines what was already true with what you just learned.

The math
The probability of A given B equals the probability of B given A, times the probability of A, divided by the probability of B

P(A | B) = P(B | A) * P(A) / P(B)

Key concepts

• Bayes' Rule — the formula for updating what you believe when you get new evidence
• Representativeness heuristic — substituting "how well does this match?" for "how likely is this?" (ignoring base rates)
• Base rate neglect — the tendency to ignore population-level frequencies when evaluating specific cases
• Prior / likelihood / posterior — what you believed before, how likely the evidence is, what you should believe now
• System 1 / System 2 — Kahneman's framework for fast intuitive thinking vs. slow deliberate reasoning

The Tom W problem
From Kahneman's Thinking, Fast and Slow, Chapter 14. A personality description that tricks you into ignoring base rates. The Sin of Representativeness — Unearned Wisdom

The cab problem
Also from Kahneman. A witness, a hit-and-run, and the surprising math of why 80% reliability doesn't mean 80% probability. Kahneman's Bayesian inference example

Books:

• Daniel Kahneman, Thinking, Fast and Slow (2011) — the Tom W problem, the cab problem, System 1/System 2, representativeness
• Sharon Bertsch McGrayne, The Theory That Wouldn't Die (2011) — the history of Bayes' theorem from its discovery through the frequentist wars to its modern resurgence
• Douglas Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid (1979) — a Pulitzer-winning exploration of how self-reference and formal systems connect mathematics, art, and music
• Ernest Nagel and James R. Newman, Gödel's Proof (1958) — a concise, accessible walkthrough of Gödel's Incompleteness Theorems for non-mathematicians

Historical:

• Thomas Bayes (1701–1761) — Presbyterian minister who first derived the theorem; never published it. Richard Price submitted it posthumously.
• Pierre-Simon Laplace — independently derived and generalized Bayes' work; arguably did the heavier mathematical lifting

Tools (if you want to go deeper):

• PyMC — Python library for Bayesian statistical modeling
• Bayes' theorem — Wikipedia
• Thinking, Fast and Slow — Wikipedia

Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm

Follow us on Mastodon: @RuntimeArguments@hachyderm.io

If you have feedback for us, please send it to feedback@RuntimeArguments.fm

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky

Transcript

Copy Transcript Copied

Jim 0:05

Welcome to another episode of Runtime Arguments. Uh this is the uh uh podcast where my buddy Wolf and I get to talk about interesting fun things. And uh today we've got a really good one uh for you. But before we uh before we get into that, uh Wolf, how are you doing? Um I uh I'm doing good. I feel good. And you? Yeah, I'm I'm doing great, and it's it's so much fun doing these things. I've been looking forward to this one for quite a while.

How was your week

Jim 0:37

Um challenging week for us. Yeah, yeah. We uh we we both had a little bit of fun uh debugging problems. I had a uh small one. I had a I had what was kind of weird. I had a JavaScript error in my framework. Now we built a framework for my system that I use for medical offices. We built it long before React existed and all these other frameworks, so we have our own little custom-built framework. And we had a JavaScript bug that's been there it got there two and a half years ago. Uh it was really simple. It was a matter of uh using the word true in uppercase instead of lowercase. And uh that code path wasn't executed very often, and uh it it it reared its ugly head um last week when uh Google released Chrome um I I think it was 146 uh users started having trouble. And we went into panic mode because we had to figure out what it was, and it all boiled down to uh when the upgrade happened, those users who upgraded were getting failures. And uh now where this code was executing, it was inside of a uh uh try catch thing. And and the uh the fact that the it was uppercase instead of lowercase, it was a bone-headed mistake, but it would be caught by that, and and our system still behaved okay. Um but with the with the update to uh uh Chrome, uh it didn't catch our error. It it just behaved it it it it behaved weirdly. And the weird thing was um when I opened the debugger, it behaved the way we wanted it to behave, uh although there was still a bug, but it it behaved uh uh as you would expect, uh instead of in this weird way. And it turns out that uh this newest version of Chrome has more aggressive optimizations, and when you open the debugger, many of those optimizations are disabled, so it it wasn't the same with or without the debugger. Uh and it was it was kind of fun. It took us like two and a half hours to figure it out. Once we did, we fixed our little stupid typo and and everybody's happy. But uh it really surprised me. This this is one of those cases where um uh monitoring the problem affects the problem, right? Um it was fun, but that was nowhere near as fun as uh the fun that Wolf Wolf had with a little project he's been working on uh for worky work, he says. So, Wolf, tell us about that little project and what happened.

Wolf 3:22

Um well, first I have to say this was a gigantic uh bone of contention between me and Jim. Let me tell you the problem, and then let me tell you what the contention was. Uh the can the the problem was that they were the the problem was this. I wrote an ORM. Um why? Why would anybody write an ORM? My ORM has different goals than anybody else's.

Jim 3:52

There just aren't enough of them in the world.

Wolf 3:54

Uh well, they all want to hide stuff from you. And they all want to do stuff behind your back, and they all want to use their own SQL. Um and I didn't want that. Um I wrote something that was good for our problem. And there's a lot of interesting things about our problem um that are different than Jim's problem. It happens that Jim and I use the same underlying database. We're both Postgres users. And uh Jim is an absolute Postgres expert. He's been using it for decades, he knows it inside and out. Um, and I don't. Uh I I I don't even qualify as intermediate when it comes to understanding the details of Postgres. So naturally, Jim is one of the people I turn to for knowledge. Uh but uh another thing that I do when I hit bugs is I write more tests and I try to figure out what went wrong. Um so the problem was uh because of my situation, um, which involves a couple of important things, that I want to update a great many rows in a table at once, um, and that the latency between me and the server is very high. Um, in in the case where I did some tests, it was 63 milliseconds. So this is different from Jim's natural um problem. This isn't what he's normally facing. Um so we both had different starting points of view on this, and the actual bug was that um to minimize round trips, uh, I would write as many rows in one SQL statement as I could, because that makes a difference to me. Usually that doesn't make a difference to Jim because A, he's in a situation where the latency is super low, and B, he's not working with 5,000 rows at a time. He's working with, I think everybody knows you do medical software, so one patient at a time, one row at a time. It's right.

Jim 6:10

Sometimes lots of rows, but never 5,000 rows in an update. The interesting thing about this update is like like you can do an SQL update and affect 5,000 rows with one simple statement, just all the rows that match your query. But that's not what you were doing. You had different data for each row that you were updating. And I and I didn't understand that.

Wolf 6:33

Um and that's that's a thing on my skills to communicate the problem. But uh yeah, uh it's just like when you uh accidentally uh delete or change all the rows in a in a database because you didn't supply a where statement. Yeah. Uh that's an example of the thing Jim just said. Yeah, you can change every single row to the same thing, no problem. But my problem was, yeah, I wanted to change the same columns in every row, but two different values. Uh, you know, like if it were for Jim, uh one row per patient, and I wanted to fix all their names. And I had 5,000 names ready to go, and I wanted to send them in, and I wanted to do that as a chunk. Uh, why does this make a difference? Um, it makes a difference because it doesn't matter how fast you are on the database side, if you are 63 milliseconds away from your destination, um just the handshake, just the turnaround at 63 milliseconds per trip, if you do it as 5,000 individual chunks, that's gonna be somewhere near five minutes, maybe a little over. Um and if you uh do it all in one chunk, uh you could probably do it in under a second. Um to me that makes a difference. If you're only writing one row, um then there's nothing to to bunch up, it doesn't help you. Uh okay, my problem was in the case where you are writing five thousand rows and one of the columns that you are writing happens to be nullable, like it's a nullable uh Boolean column.

Jim 8:26

It's a boolean that can contain null.

Wolf 8:29

Right. So for instance, true, the sign validation is accurate. False, the sign validation isn't accurate. Or null. We don't know if the sign validation is accurate or not. Um if it happens that you are writing 5,000 rows, and for whatever reason, in a specific column, that column is null in every row, then the way I was writing 5,000 rows, which is it's a regular update statement, but I'm passing in values, the values clause inside. Um, because of the intermediate evaluation that Postgres is doing on that particular SQL statement, it doesn't know what the underlying column types are unless you gave it an example in the value that you're updating to. And therefore, it makes a guess based on all those nulls, and its guess is text. And when you try to assign null into a text field, and this sounds like a very bad guess to me, if you try to assign null into a field that it has guessed is a non-nullable text, it is uh essentially a syntax error at the point of evaluation in the database. Um and I felt uh let me give you my overall view is that almost all bugs, once you're uh a a seasoned programmer, almost all bugs happen where two pieces of code rub up against each other at the API boundary. And the problem is almost always in these cases that one side has some specific model or advertises some specific n model, or you mistakenly assume it has some specific model, um, and your code doesn't do the right thing. It doesn't use that model as it actually exists. So um the A way and unsatisfying to Jim way that I have described this is that neither side has a bug. The bug is that there's an impedance mismatch. Um I had this problem, the fix is easy, I did tests, the fix um costs nothing, uh and so it was way easier to just fix my code than it was to uh get the Postgres guys to to do it. To me, the sub there was a surprise, and the surprise was I wasn't actually writing 5,000 nulls, I was expecting a round trip. I read an an entire table that was 5,000 rows long, and then I changed one column, another column, but Postgres is gonna write the whole row anyway, and so it's just easier for me to notice that the row changed, but not individual columns in the row. So um I just write out the whole row, and that meant I was writing a a column of nulls, uh, and this bug cropped up. Um and uh once Jim fully understood the problem, um he explained to me that my expectations were incorrect, um I had a bug. That was his feeling, and I believe it still is to this day. Um but I fixed it uh in my code so that now what I thought Postgres was going to do and what Postgres actually does um agree. Um I accept Jim's uh defin um opinion that this was a bug in my code. I don't think the documentation for Postgres made it clear that I couldn't expect a round trip in this case. Um so I feel cheated. Um but I'm okay. I'll live because you know I found it and it all worked. And and the nice thing is that uh my scheme I did get Jim to admit something. You know, Jim does it Jim does it one row at a time, um, and that's fine. And he told me that the right thing for me to do was to do the five thousand rows, one row at a time. And I told them, you know, that's gonna take five minutes. And when I do it as a bunch, it takes under a second. And he said, yeah, but he said, yeah, but it doesn't work at all. Therefore, you are wrong. Um and so the I admitted to Jim, okay, I have a bug, and Jim admitted to me, um, yes, I can see why there might be ways to do it other than the way I do it. For both of us, um, it was a road to get from our original positions to saying the words we said to each other. That was my exciting week in bugs.

Jim 14:09

Yeah, okay. I I I I have a little bit to add to this. Um uh Wolf uh to be fair to Pro Postgres here, uh Wolf described it as using the values clause. That's not actually what it was. The values values is a statement that produces data. And the rules in the Postgres documentation for values is very clear how it's going to determine the data type in that column. Uh, if all the values are null in that column, it will be text. And you can argue about whether that's nullable text. What? Non-nullable. Well, it's not. It thinks it's non-nullable. No, it thinks it's text. It has nothing to do with whether it's nullable or not. It thinks you're trying to feed text to a Boolean field, and that's the problem. Um, and the documentation's pretty clear, but uh nobody is gonna sit down and completely read the documentation and understand every little nuance of it. And I I get that. And uh Wolf did a bunch of benchmarking to prove that uh his way was faster, and it was way faster. Or his was milliseconds. Uh, my way of doing that in the same scenario was minutes, and and uh I I will always go for the milliseconds over minutes. In fact, one of the things I love doing that I I get great joy of doing is tuning queries, and it it's a geeky thing. I know a lot of people can't be bothered, but I love it when I take a query that takes a long time to run and I get it down to milliseconds. It just uh it's one of those things that really, really lights up my neurons and my brain. Um and uh we have a happy ending. He he uh fixed it by uh casting the values in his value, by casting the columns in his values statement, and then that feedback. Which was free. Yeah, it's free. It's I measured it all. Yeah, it's it's it's it's a good way to do it. Be explicit about what your data types are, and in this case, you he had to. Um, because this this kind of bug could have gone for a long time and not be noticed, you right? Because if any one of those 5,000 rows had a value of true or false in in that column, it would have worked fine. If 4,999 of them were null, but one of them was true or false, you wouldn't have found this bug. So it it was really good that you found it. Um it was a fun adventure because we both learned a lot. And and I'll tell you, the name of our podcast is runtime arguments, and that's not an accident. Right? We uh we we get into uh uh it it doesn't elevate to the level of debate or argument, but we we uh we have different ideas about how things should work, and and so uh the name of our podcast is quite appropriate.

Wolf 17:07

Um The only reason it doesn't become an out and out brawl is because of our respect for each other.

Jim 17:14

Yeah, yeah. And in the end, we we both uh uh reflect back uh about how much we enjoyed that that adventure, and this was a good one. This is one of the best. Uh anyway, we are 17 minutes into this. We should really get moving on. I I do have just a

Feedback

Jim 17:30

little bit of feedback. Our last episode, uh episode 23, was about uh containers like Docker and Podman and those. And one of the points that I I wanted to make and I didn't was when you're doing virtual machines, uh you can you can set up tens or maybe a hundred virtual machines on a server. I think a hundred is kind of pushing it. When I had a uh a server rack in the data center, I had like 30 VMs running on my host, and it ran pretty well, but uh you wouldn't go much higher than that. I mean you'd have to add a lot more RAM uh to do that. Um containers, however, it's common to run thousands of containers on a single host. Uh it's it's it's that much more lightweight. Uh, and I wanted to point that out. So if you're thinking about uh trying to decide between virtual machines and containers, think of that. You can run thousands of containers on a on a host. Uh the other thing that I wanted to uh talk very briefly about was uh episode 19. We talked about data centers, and and that kind of hinged around the idea that uh there's a new data center going in uh near Ann Arbor. Uh it's part of the Stargate project from Oracle and OpenAI and a couple of other companies. Uh and it's 1.4 gigawatts of electricity that they're going to use. And there's been a lot of debate about it in the state. Um there's another data center coming uh down in uh Van Buren, which I think is down near Detroit Metro Airport. Uh Google has been uh in negotiations with Detroit Edison Electric DTE we call it for a one gigawatt uh feed of electricity for a data center they want to build. Um so that was in the news this week. And um uh one of the points that came out of it that I read is Michigan is in an interesting position to offer uh uh uh a home for these data centers because Michigan sort of was built on the industrial revolution. We had the auto factories here and all kinds of other factories building things that used consumed great amounts of electricity. And uh we don't have that anymore. Yeah, we have a few factories, but nothing like we used to. You know, Detroit is the motor city, and we used to have tons of factories around here building cars. So Detroit Edison is sitting there with all this unused c electrical capacity. So uh for a data center to come in, they see that as a huge uh win. It's a chance for them to uh uh provide some of that capacity. So it's kind of interesting, and you can still debate whether it's a good idea or not, and whether AI is a good idea or not. There's so many debates, but the fact uh the fact is Michigan is a good place for these data centers. And uh I I think it's kind of interesting. So should we get on to the real thing? What we're should not. We should not.

Wolf 20:27

Yeah, because I I have a hot take.

Jim 20:30

Okay.

Wolf 20:31

Um something horrifying happened today, and I just want to mention it. Okay, and that is um I have some tools that I use that I think are unbelievable. Um I use a tool called UV. Um, UV, and uh I also use a type checker called TY. Uh, that and a third thing called Rough, they all come from this company called Astral. They apply to Python, but they're all written in Rust. They are so much faster than anything else. They are so much better. They make my life better because of how I how much I have to deal with the things that they do. Um, I love these tools, they have been free from the beginning. Um, and people immediately complained when they started to become popular. You know, what's gonna happen when something horrible happens to Astral and they get acquired or something? Um, and Astral was like, oh, we'll make a foundation, it's all gonna be fine, nothing's ever gonna happen, everything's cool. Well, today Astral got acquired by OpenAI to be integrated into their codex uh software, which is their clothing coding AI stuff. So I don't know. I mean, uh GitHub bought by Microsoft and they're still doing okay, maybe kind of.

Jim 22:13

And and it was open source, so somebody could fork it and keep going?

Wolf 22:17

Yes. Yes. Okay. And and maybe there was a foundation. They talked about that. I think that might have happened. I don't know. But please don't take these tools away from me. I don't want that to happen.

Jim 22:28

They are too important to you. Yeah.

The Meat

Jim 22:32

Now let's get into it. Let's get going. Uh today I didn't even say what we were going to talk about today. Um for a long time now, maybe a year. Uh I I don't think a lunch goes by where Wolf doesn't mention uh the Bayes rule. Um that's B-A-Y-E-S rule. And uh he's been wanting to do an episode on this, and today is the day. He's gonna tell us what Bayes rule is, what it means to us, why we should care. And um I I I've been looking forward to this because uh I I I still don't understand it. And he does. So here we go. Wolf, you ready?

Wolf 23:13

I am. Um, but I want I want this to be me convincing you. I know you believe in Bane's rule from what I've told you. True. Um, but this is about uh, and I said these exact words to Jim, a lot of the little stories I'm about to tell, um, I kept hidden from Jim. Normally, uh we sort of talk about what we're gonna talk about, and so we have a pretty good idea. Uh questions are left unanswered, of course, but I really held back in our in our uh discussion of what was gonna happen, so that everything I say is gonna be a surprise to Jim. And the thing I said to him is by the end of this, I know that you believe in Bayes, but I want it to be in your brain the way Vim is in your fingers. Um But probably most people don't even know what Bayes rule is or Bayesian thinking in general.

Jim 24:20

So I barely know what it is. I could not explain it to somebody, so today you're gonna explain it to me and everybody else.

Wolf 24:28

Uh well, let me just give you the elevator pitch. Bayes rule is the formula, and it is a formula, there's actual math that tells you how to update what you know right now. There's some model in your head of facts, and I don't care how it's represented, but it's the the knowledge in your head. How to update that when you get new evidence. What is your model before you touch the hot burner on the stove? Then you touch it, now you know something you didn't know before. Um what happened? Bayes' rule happened inside your head. But but you didn't know that name and you didn't do any math, it just happened. Um that's what Bayes rule is. Um but there's a lot of different ways that Bayes' rule interacts with the world. Um, and mostly, as as a lot of the people listening to this are programmers, almost all of you already use Bayes' rule or Bayesian thinking, but you don't know what the math is. I'm gonna try to avoid saying any math. I am gonna say uh some numbers later. But let me tell you how this all happened. Um at my jobby job, I work with an amazing team. I I don't think there's a person on my team um that I wouldn't choose myself to be uh a partner in uh in the work that we do. And one of them is our chief scientist. He is a mathematician, uh, has a PhD in mathematics, and every time I talk to him, I come away smarter. And uh he was talking to me about Bay's rule, uh, and I didn't know anything at the time. I guess this must have been quite a while ago, over a year. And he said, Oh, let me loan you a book, and he loaned me this book. The book is called The Theory That Wouldn't Die. Um, and the author is uh Sharon McGrain. Um it's it's not a math book, it's a history book, and it's about um Bayes himself, who was a pastor in the 1700s. Um before reading this book, I had kind of a pantheon of heroes. Um I'm sure they're totally different than anybody else's heroes or even the kind of heroes they have. But they were Richard Feynman, Claude Shannon, and Kurt Girdle. Um, you can see I'm a nerd. Um I think Feynman might be the greatest teacher who ever lived. Claude Shannon invented information theory, which is just gigantic inside my own mind, and Kurt Girdle sort of proved the limitations of formal systems. After reading this book, I added one name to my Pantheon. Um, and you'd think it would be Bayes, but it wasn't. Bayes invented the idea, uh, but he wasn't a hundred percent behind his own idea, and he didn't figure out all the details of it. Instead, that all fell to a French mathematician whose name you all know. Um it's French and long, so all I know is the last part, which is Laplace. Um, you know, like Laplace transforms and all that crap. Laplace did all the heavy lifting, and he figured out the details. Um, really, it ought to be called Laplace's theory. Uh but Laplace is like, nope, it's Bayes, that's all fine. But everything we really know about it is from Laplace. So now I have four, and the fourth one is Laplace. Um and my question for you, Jim, to start everything off is what do you think the most important mathematical theorem is? Ever, anywhere?

Jim 28:43

Uh E equals MC squared.

Wolf 28:45

CN knew you were going to say that. Did you? Um I love that one. E equals MC squared is uh very compelling because it's so easy to say. If I were to say Bay's actual rule right now, you would stop me in the middle. Uh which I don't even have time to do when you say e equals mc squared.

Jim 29:09

Well, I kind of like Pythagorean's theorem, too.

Wolf 29:13

Yeah.

Jim 29:15

Yeah, is that a squared plus B squared equals C squared? Did I get it right?

Wolf 29:21

Uh yes, uh if you say it the right way, and that is also the uh instance uh of Fermat's last theorem, um, that is the one we know works, and supposedly at this moment we know that no other one works. Like there's no A cubed plus B cubed equals C cubed. Um that's what we understand. Um my thesis is uh that Bayes' theorem is the mathematical formulation of how we learn anything. And to me, that's more important than E equals MC squared. It is every time you open your eyes and see something, Bayes is happening in you. Um that's what I think. Um for me before this it had to do with entropy and Claude Shannon, uh, and now it's Bayes. So let's start. I have a story that I held back from you, and I am going to tell you uh this story. Tom W. This is straight out of the book. Um and the book in question here is Thinking Fast and Slow by Daniel Kenneman, who is a psychologist by trade but won the Nobel Prize for Economics in 2002. Um this is all about Bayes. It looks like you're gonna say something, Jim.

Jim 30:59

I'm just gonna say we're we will include a link in the show notes for this and the other book you mentioned. Uh the first one.

Wolf 31:08

Uh so Tom W. He is of high intelligence, although lacking in true creativity. He has a need for order and clarity and for neat and tidy systems in which every detail finds its appropriate place. His writing is rather dull and mechanical, occasionally enlivened by somewhat corny puns and flashes of imagination of the sci-fi type. Uh he has a strong drive for competence. He seems to have little feel and little sympathy for other people, and does not enjoy interacting with others. Self-centered, he nonetheless has a deep moral sense. That's the original description of Tom W. from the book I just mentioned. And here's the question for you, Jim. Is Tom W. more likely a computer science graduate student or a humanities graduate student? Well, that sounds like a computer science guy. Doesn't it? Yeah. Um but it's a trap. The description screams computer science. He's the stereotype. But humanities students outnumber uh CS students uh maybe by a couple of orders of magnitude. I mean, it's huge. Uh it's easily uh well, maybe not a couple of orders of magnitude. Well, anyway. Yeah, they outnumber. Wait, when when you look at a college, uh, there are so many more humanities students, and I didn't tell you anything about Tom W. Uh that uh had numbers associated with it that could possibly adjust what in the world of statistics we would call the base rate. Um the base rate is how many students are humanities students, and that might be 90%. Um I didn't tell you anything that adjusted that base rate. No extra information. Um so the thing that you just experienced is described by Kenneman in Thinking Fast and Slow as representativeness heuristic. He looks like a CS student. So when you didn't know the base rate or wouldn't incorporate the base rate, you answered in your own head the easier question, which is not what is he more likely to be, it's what does he look like. He looks like a CS student.

Jim 33:54

Well, because I'm a CS guy, had I been a humanitarian, I may have said the exact opposite thing, right? Possible. He looks like people I know.

Wolf 34:05

So the fact of the matter is the thing you just did is something almost every single person on the planet does unless they've specifically trained how to do better. Even statisticians, even people whose job it is to know better don't know better. What you just did is what we all do. So I'm normal. Here's the second story. Um I'm gonna call this the camp.

Jim 34:38

Aren't we gonna get down to what he is? Is he a humanities guy?

Wolf 34:43

Uh guess what? They don't tell you.

Jim 34:46

Oh, that's most likely he's a humanities student.

Wolf 34:50

Well, yeah.

Jim 34:50

Yeah, because the statistics tell you he's a humanities guy. Okay, but they're not gonna they're not gonna tell us. Okay.

Wolf 34:57

Uh so this is the cab problem. Um a cab was involved in a hit and run at night. Um cab companies in the city. 85% of the cabs in the city are green, 15% are blue. Um, and of course, at night, green and blue. They look a lot alike. And maybe for you, they always look.

Jim 35:26

For me, they look more alike.

Wolf 35:28

Uh there was a witness. I mean, the cab did the hit hit him and then drove off. But there was a witness who saw it for a short time. The witness says it was blue. Uh, of course, the prosecution took the witness out and tested the hell out of him, as I would have done. And it turns out that the witness is 80% reliable. So, here's the question for you, Jim.

Jim 35:54

Oh, geez, you're gonna make me do math.

Wolf 35:56

What's the probability the cab was actually blue, as the witness said? 80% reliable.

Jim 36:04

Well, it's not 80%.

Wolf 36:06

Of course not. I wouldn't be asking you if it was 80%. Yeah. But of course, the truth is almost everybody says 80%.

Jim 36:16

Yeah.

Wolf 36:17

Yeah.

Jim 36:18

But you know, I know enough. I know enough that it's not 80%.

Wolf 36:23

Would you like to know what it is? I would.

Jim 36:26

You're gonna tell me this time?

Wolf 36:28

I am. It's 41%, and I'll walk you through it. Of a hundred uh cabs, fifteen are blue and eighty-five are green. Okay. The witness sees a blue cab. So fifteen times. That's eighty percent, is twelve correct. Or the witness sees a green cab and says blue. That's eighty-five times.2 equals seventeen mistakes. So total times. Right.

Jim 37:06

Okay.

Wolf 37:07

Total times the the witness says blue is 12 plus 17, that's 29. Of those 29, only 12 were actually blue. That's 12 out of 29, 41%. That is Bay's rule in action. Okay. You had a base rate, the base rate was 15% or blue. And now you have new information. The new information is identifying that the cab was blue, but with only 80% confidence. Bayes' rule is how did you update your model with that new knowledge? Um the thing that we knew, the 15%, that we call Bayes calls that the prior, statistics calls it the base rate. Um you and I might call it what we know for sure. Uh, and then we combine that with the evidence, uh, produces what Bayes calls the posterior 41%, um, that dragged the base rate down from 80 to 41. So the witness is mostly right, but there are so many more green cabs that the false alarms outnumber uh the true sightings. Uh it is a surprise. Most people say, oh, he's 80% right. It's 80%.

Jim 38:40

But he's 80% right. He's only 41% accurate in this case. The the chance of it being a blue cab is 41%. That's right. Yeah.

Wolf 38:52

Okay. Okay. There's a zillion different ways that you can apply Bayes. Um, but it all comes down to that fundamental what you know plus what you learn. So uh let me give you an example. This is something we have all done, but maybe not with socks. I'm gonna say it with socks. You lose your socks. And uh let's just say that we don't know anything, so there's an equal chance that your lost socks could be in any room in your house. Uh so you check the laundry room and you don't find them. You know what that means? That means that now they are more likely in every other room, and significantly less likely in the laundry room. Not maybe not zero, maybe you missed them, but they were there. Uh but you have changed the odds. Uh searching and not finding it is evidence, as everybody who has ever debugged a problem knows. I'll get back to that. Um, and this is Bayesian updating. You uh had knowledge, performed an experiment, updated your model. Uh doesn't that sound like the scientific method? It does. Alright, let me give you another example of a different way that people use Bayes, and this is getting into um a way that people fundamentally understand Bayes, and that is this. Um medical studies, we already know that um if you have the measles, then there is a well-known probability, based on your population, that you will acquire red spots. We know that probability. It's from medical studies. Bayes uh probab statistics has a way of saying that. They say, um, with symbols and whatnot, but they'll say it like this: the probability of red spots given measles. So that's P of X bar Y. Bar means given. Um if you have measles, what's the chance of red spots? But that's not what happens. What happens is a patient walks into the doctor and the patient has red spots. How are you gonna treat that? The doctor needs the opposite of what you know. You know, if you have measles, what the chance is that um you'll get red spots. What's the inverse of that? The chance that um if you have red spots, the cause was measles. They're fundamentally different numbers, but they overlap. They overlap in the group of people who have both. There is a group of people that has measles and red spots. Those people, that group, that overlap is in both of those two questions.

Jim 42:05

So there are people that have red spots, but it's not measles. And there are people that have measles that's could be poisoned ivy. People that have measles but don't have red so red spots.

Wolf 42:16

That is correct.

Jim 42:18

But there's an overlap.

Wolf 42:20

So it's exactly like Venn diagrams, two circles.

Jim 42:24

Yeah, that's what I picture in my head. I'm with you so far.

Wolf 42:27

Okay. Um the intra- So here's the thing that um that you need to know. Science uh gives you, uh or at least reveals, um, the arrow between cause and effect. From scientific studies, we have a pretty good idea of if this thing happens, then that will make this other thing happen with some probability. I just said that for measles. If this thing happens, you have measles, then this other thing, you getting red spots, will happen with some probability. Cause, arrow, effect. That's a thing that we can discover. Seems clear. Yes. But reality is the opposite. You don't get to see the cause. You see the effect, and you have to work your way backwards to get the cause. And human beings are so screwed up that in our minds we will make up every kind of story to be the cause. Oh, my sock, the fact that I'm wearing two different socks made the Dodgers win. So now, in a game, I'm gonna wear two different socks. That's just how it works. Um Superstition, everything that you can think of, it's the tiny stories in our head. Because we want to flip that arrow around. We want to know the cause. Human beings need a cause to make it all make sense to us. Bayes is the mathematical scheme, mechanism, plan for flipping the arrow around. That's what it does. You see the effect. Help me figure out the cause.

Jim 44:33

Okay, what's going through my head right now is uh hypochondriacs. They experience all kinds of symptoms, and they make assumptions about what the cause is. They need to learn from this, right?

Wolf 44:50

So let me tie this together. Um boy, I'm super tempted in a second to actually uh say the math out loud for the line. There's no numbers in it, but but let me say this last thing. For the people who are listening to this podcast, you probably care about debugging. I already said you almost certainly use Bayesian thinking all the time. Bayesian thinking is about seeing, you you know when you watch something in your code, you know what that's going to do. The problem is, a bug, you see what it did, and now you have to figure out what in your code caused that. You've got to flip the arrow around. You were born to do this. That's why you're in this job. What were you thinking? I'm gonna tell you. Um the bug manifests, that's the evidence. You think of all different reasons why it might happen, and you focus on them one at a time, that's your hypothesis or hypotheses. You don't have complete information, um, that's your underlying model, your priors. You check the logs, you run the tests, you search the likely places, like the laundry room. Um you don't find anything in some of those places. Oh, I guess it wasn't uh latency, oh, I guess it wasn't uh SQL, oh, I guess it wasn't the every one of those, even though what you found was nothing, the fact that you found nothing was a fact that updated your model, and eventually, um you find the answer. Um you are solving the inverse problem. Uh what code you have, arrow, bug, that's easy. What bug you found, arrow, which code, that's the hard thing. And that's why you do what you do, it's what you're good at. Um you are doing exactly what the ship hunters do in uh these shipwrecks. And I I'm actually uh gonna tell a story right now about um Air France 447. Uh what year did you say this was, Jim? 2009.

Jim 47:21

Yeah.

Wolf 47:22

Yeah. In 2009, um Air France 477 uh crashed into the ocean. Didn't did I not say 447? You said 477. Oh, I hate that. I do that sometimes. 447. Um, and it was horrifying. Um every every person on board the plane, every passenger, every um crew member, every airline uh employee, everyone died. It was a an utter tragedy. Um, and they found the wreckage for the plane very quickly, but what they didn't find was the black box, or maybe it was black boxes. I I think it was one or but um they had a million square kilometers of ocean to search. It wasn't just socks somewhere in your house, it was a black box somewhere in a million square miles, and of course they divided it up into grid squares, like rooms in your house, and they had a lot more information. Their information wasn't just um, oh, it's not in the laundry room. Their information was here's where the plane crashed, so that changed the probabilities to cluster more around the plane crash, and here's what the prevailing currents were, and that changed the probabilities based on where the currents would take it. And here's the underlying terrain of the ocean, and that changed the probabilities based on and using that, and two years of searching, and every time they did a search and didn't find it, bat information, and finally, they found the black box. Um and we have used Bay's rule everywhere for everything. It helped us solve enigma, it helped 538 solve political uh polling questions, who's who's gonna win? And it's surprising, because people want to make up stories, and the stories don't correspond with uh um the best predictions we can make based on the actual uh information. Um so uh when we actually uh do the stuff that we do, how do we know what's true and false in this world? We we use the scientific method for a lot of it. Bayes is the mathematical representation of the scientific method. Your hypothesis is your prior belief, your experiment is your new evidence. When you update your understanding, the result is what Bayes calls the posterior belief. This is why Bayes applies to everything. It's the mathematics of the scientific method itself. Everything we know, everything we learn, everything we see is about Bayes. You just don't know it. Um this is fascinating to me. Um, but I I I I I I have to tell you.

Jim 50:38

Go ahead. We've been doing this. You and I have been doing these kinds of things, this kind of debugging, for our whole lives, our whole professional careers. How does knowing that it's Bayes help that? I just do it. I I can do it without knowing there's math behind it. I can do it without understanding that it's called Bayes. What what what are you getting at here?

Wolf 51:09

Um Well, the knowledge that I'm giving you right now is not gonna make you a better debugger than you are right now. Because, first of all, um your priors are not quantifiable. Like you don't know, oh, 37 37% of the time if you log this, I blah blah blah. No, that you don't know that number. Um so the actual math can't help you. This is more about understanding how the world works. And also, although Bayes is a very good m understanding of how certain things work, there are plenty of times where Bayes cannot be a tool for you. Uh I want to make sure I don't accidentally deceive anybody. Um there's lots of times when Bayes just isn't going to help you. Like, for instance, if you already know everything there is to know, if your model is is good enough, new information doesn't help you. Just calculate the answer. This usually applies to math, but um Bayes is a lot about deduction and searching. If your model doesn't can't add new evidence, it can't help you. Um if there's a simple heuristic that will solve your problem, then you don't need to do math. Bayes won't help you. If your prior beliefs are insane, um, like, for instance, oh, the okay, I'm just gonna say it. I don't care who who hears me and says I'm uh I'm being um prejudiced or not. But if your prior belief is the earth is flat, um I don't know how good you can update your model to make your new predictions be better. I mean, I do know, but you won't do it. Yeah, right. Right.

Jim 53:16

Hey, um let's not conf let let's let's not let people think that I believe the earth is flat. I do not.

Wolf 53:22

So um here's the guide. Use Bayesian thinking like you're doing right now, everywhere, all the time. Uh, and you already are for lots of things, but more. Do it more. Um, like when you think about Tom W. Um, don't throw away Bayes. Don't throw away um the base rate. Uh do the right thing. And and like you are right now for debugging, but just debug the world. That's what I want.

Jim 53:57

Or when I think you're doing your SQL wrong. I didn't have a full model of what was going on.

Wolf 54:05

Yeah. Uh but use Bayesian tools uh when the problem is complex enough to warrant it, when there is math, when you do have an incomplete but fairly accurate model for the part that you do know. And stay humble. Stay humble about your models. Absolutely understand there's a confidence, and there's new facts, and they have a confidence, and you want to put it all together so that you can get the best outcome. You know, um my let me just go on and maybe I'll come back to the thing I'm trying to put words to. Um let me give you the big picture. Kurt Girdle showed the limits of formal systems. Um for those of you who aren't um uh familiar with Goethe's incompleteness theorem, uh, I highly recommend uh uh at least two books. One is called Gürtel Escherbach and Eternal Golden Braid by Douglas Hofstetter, where he gives a terrific metaphor about record players, which are just physical machines, and maybe nobody here has even heard of a record player, but um, because a physical machine exists in time and space, um, it has a harmonic frequency. There is a sound you could play if you played it loud enough that would shake a machine apart. A record player is just a machine, so there's a sound that can shake it apart. So there's a scale. Do you have a record player that sounds like crap, can't play every sound that there is? Or is it good enough to play every sound that there is? And therefore, it can play the sound that can shake itself apart. Girdle's incompleteness theorem says formal systems like math and logic, A or B, uh seven plus two, formal systems obey that rule. Either your system can't play every true, every uh possible sound, um, and for Girdle that means express every truth, or um your system can uh express um uh statements that are uh unprovable. Uh in other words, it can play the sound that shakes itself apart. So that's a limit of formal systems, Girdle's incompleteness theorem. Um Shannon quantified information. He explained how when you have two different models separated by some distant, uh you can make changes in one based on what came from the other. You can communicate ideas and what that means and how much news is in a piece of information, what changes in a model. Um Bayes formalized um the sort of one-ended version of that when you just have one of the two sides that you would have had in communication, but now you have new facts, how do you update that one model? Um I re-ask you, Jim, the original question with this knowledge. Do you feel like this mathematical formula, which I have never said yet, fits into the most important math that there is? Do you feel like Laplace, who did all the heavy lifting on this, is a worthy candidate for my Pantheon?

Jim 57:59

Yeah. Yeah. Um and and is uh is this math more important than E equals MC squared? Um I can't answer.

Wolf 58:11

E equals MC squared is important but invisible. Um this Bayes algorithm, we have made it invisible, but we ignore it at our peril. When you say, um, oh Tom's a CS guy, that's you ignoring Bayes. Um so um that's that's my feeling. I get it. Um I have um a couple of important uh lines to remember, uh statements about things. Um Bayes is how you learn everything a little bit at a time, and science gives you cause, arrow, effect, cause implies effect. Reality gives you effect, arrow, cause, and it's Bayes that flips the arrow. That might be the most important thing I said. It's Bayes that flips the arrow. Um If it wasn't for Bayes, the job of insurance companies could not be done. Oh, the actuaries. Bayes is the fundamental thing, and every one of them knows it. Um you are making guesses about can this bad thing happen and how much am I willing to pay? You know a base rate, you have uh extra information, insurance is all about Bayes. Uh there was this, I'm not gonna tell you. If you read this book, uh The Algorithm, uh The Theory That Wouldn't Die, um they will talk about the two camps, the Bayesians and the Frequentists, who disagreed, and the frequentists uh were in charge for 200 years. Um the frequentists had had some ideas that weren't as workable. Um, and if you went on strict frequentist ideas, insurance just wouldn't work. And uh finally, uh debugging is Bayesian reasoning. You just don't call it that. So the four signs that your problem might be applicable for Bayes, and then we're gonna call it a day. Uh if you're solving an inverse problem, if you're trying to work backward from effects to causes, it's gotta be that. If you're using that, Bayes might help you. You have incomplete information, but some evidence. You know some stuff and you see some new stuff. The outcome isn't binary. Um could be continuous, could be multivariate. Um, for instance, um, is it going to be uh any one of these six candidates who wins the election? Um that's not binary, that's um, you know, six or whatever. Um and if you are in a situation where brand new data should or could rationally change your conclusions, if all four of those things are true, maybe Bayes can help you. Um and I I think that might be all I have to say.

Jim 1:01:54

Oh okay, I've got a question. We we can't talk about Bayes without mentioning something that people have heard, especially people that have been in the business for a long time like us. Spam filters using Bayesian filters. Uh what what's the connection there? Is there a connection?

Wolf 1:02:13

Oh my god, yes. Um what Bayes does is it knows when you're looking at a problem through a Bayesian lens, as it were.

Jim 1:02:28

Okay, so we're looking at spam. We're looking at an email message. Trying to figure out if it's spam or not.

Wolf 1:02:35

Is this message spam and you know a base rate, and now you're gathering evidence from within the message, like does it say uh erectile dysfunction?

Jim 1:02:48

Umranty.

Wolf 1:02:52

Yeah. Some of some of these are extremely strong evidence that adjusts the pr the posterior uh value greatly upwards from the base rate. Um and that's that's all there is to it. It is the fact that um you're looking at it through this lens of knowing, yeah, there's a 80% chance that it's spam. Uh, but because I saw these words, it's a 90% chance. And if it gets to 92%, well, I'm just not going to show it. We're just gonna call it spam and and move on. That's right. Are you ever wrong? Of course. Uh sometimes the cab was green. That's just all there is to it.

Jim 1:03:38

Right. Sometimes it wasn't spam.

Wolf 1:03:42

Do you think it would be okay if I said the words of what Bayes is? Because when we discussed it at lunch, I was asked, but what does it mean? What's the math? I'm gonna say the words. What's the math? This is Bayes. Bayes says that the probability of A given B, and remember, think back to measles and red spots. The probability of A given B, the probability of measles given that you have red spots, is the probability of B given A, the probability of red spots given measles, because you happen to know what that is, we have a measure of it from scientific studies, um, times the probability of A, that is how many people in the population have measles divided by the probability of B. In other words, how many people in the population have red spots. Um I'll just say it out. The probability of A given B equals the probability of B given A times the probability of A divided by the probability of B. It's actually pretty simple, and in English it is flip the conditional, but weight it by how equal uh how likely each thing was in the first place. Uh once you know what the parts are, it's actually pretty easy and fundamental.

Jim 1:05:00

When you say it like that, yeah, it sounds easy. Uh will you include that uh that that formula in the show notes?

Wolf 1:05:08

If you like it, I will.

Jim 1:05:09

Yes, let's do it. Just in case anybody is wants to dig into it.

Wolf 1:05:15

I have such strong um affinity for Bayes that I got a custom license plate that says Bayes.

Jim 1:05:27

That's right. That's right. You got the Bayes license plate. So now you can drive around and people know you're a dork.

Wolf 1:05:37

No, they just think my name is Bayes. Because if they know what Bayes theorem is, um, then they're like, oh shit, that's a Bayesian. That's awesome. But if they don't know, it's like, hey, that guy's name is Bayes. Yeah.

Jim 1:05:57

All right. You can wrap it up pretty quick.

Wrap up

Jim 1:05:59

All right. Well, thanks. That's uh that that's that's a lot of fun and uh more interesting than I thought it would be. I was looking forward to this episode. I was really looking forward to this, and and that was uh that was a lot of fun. I think you did a great job explaining it. So uh thank you for everybody for uh joining us today. Um uh as always, uh feedback. Uh please send us your feedback. Uh we like to get it so we know if we're doing okay. Uh uh you can email it to uh runtime arguments. I'm sorry, feedback at runtimearguments.fm. You can visit our website at runtimearguments.fm. Uh you can find us both on Mastodon. Uh you can uh uh we we announce our episodes on LinkedIn and on Facebook and on Mastodon. And uh, you know, send us uh questions, send us ideas, send us feedback, whatever. We'd love to see it. Uh and we have transcripts. And and we we have a transcript attached to this. We have show notes uh that you should be able to see from your uh podcast player. And um, man, this was a good one. Uh thank you, Wolf. Uh I I think we can sign off.

Wolf 1:07:16

Yeah, this was a ton of fun. I had a good time. Maybe it wasn't quite as fun as the no problem, but it was pretty good. Yeah.

Jim 1:07:24

Yeah. Oh, oh, well, one more thing. We uh the last episode was on uh Docker containers or all kinds of containers. Uh coming up at the Michigan Unix Users Group, that's mug.org, m u g dot org. Uh the April meeting, I think it's April 14th. Uh it's it's on Zoom. Anyone is welcome. We're basically going to be doing that episode uh as a live visuals with visuals uh using Zoom where we actually get to show um examples of creating a container and running a container and interacting with a container, and you can ask questions and uh it'll be a lot of fun. So uh put that on your calendar uh uh and check out mug.org for more information. And um I I think that's it. So thanks, Wolf. See you later. Uh man, this was my pleasure.

Wolf 1:08:15

See y'all. Bye bye.