Runtime Arguments
Conversations about technology between two friends who disagree on plenty, and agree on plenty more.
Runtime Arguments
34: The Hardest Test You'll Ever Write Is Test #1
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Wolf is at the "relies on tests" end of the spectrum. Jim is closer to the other end, somewhere around "how do I start?". He doesn’t need convincing. He just needs to get from zero (non-trivial) tests to one. Today’s episode is a candid discussion where Wolf addresses Jim’s obstacles and tries to "start Jim’s testing engine". With other stuff along the way (as always).
Links:
- Pandoc celebrates 20 years
https://pandoc.org/twenty-years-of-pandoc.html - C-Kermit releases version 11.0, first release in 15 years
https://changelog.complete.org/archives/44456-celebrating-45-years-of-kermit-with-the-first-new-c-kermit-release-in-15-years-and-working-with-a-decades-old-c-codebase
Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm
Follow us on Mastodon: @RuntimeArguments@hachyderm.io
If you have feedback for us, please send it to feedback@RuntimeArguments.fm
Checkout our webpage at http://RuntimeArguments.fm
Theme music:
Dawn by nuer self, from the album Digital Sky
Welcome to another episode of Runtime Arguments, the podcast where we talk about tech things. Uh, and here today with me is my buddy Wolf.
Wolf:Hey, everybody. It's, uh… maybe I shouldn't say hey, everybody, because I think I always say that, don't I? Instead, why don't I say…
Jim McQuillan:It's fine to say that.
Wolf:Hello all listeners. Good morning good afternoon and good evening.
Jim McQuillan:Right, right. And, uh, and I didn't mention my name. I'm Jim McQuillan, as always. Uh, this is episode number 34, our 35th episode, and today we're gonna talk about testing. And, uh, this… this should be fun, because test… uh, Wolf tests, and I don't, and we're gonna talk all about that. Uh, should be fun. Um… let's see… How was your week, Wolf?
Wolf:Umm. Pretty boring, actually. Uh…
Jim McQuillan:Hey. Is that good?
Wolf:There were places, as usual, where I learned I was an idiot. Every day, in every way, I learn more and more about the things I don't even know I don't know yet. So, that's a thing. Uh, and I did experience some annoying issues. There were a couple of tailscale updates, uh, that I experienced.
Jim McQuillan:Yeah.
Wolf:Um… And I don't know exactly what I've done to ***** up my world, but, um… I have Tailscale installed via the Homebrew cask. Maybe that's not the best way, but I have it set up so that it, um… Tailscale updates itself. Um, not, like… Homebrew update upgrade? That, that doesn't do the thing. Uh… But it's not quite set to totally update itself, it says hey I have updates you want me to update and then I push a button and that button says why don't you go ahead and do everything it takes to upgrade except. actually work. Stop at the place where you were gonna trade out your, um, system networking extension. and tell me you can't go any further than that, and make it be that I don't have any network at all. That's how I have Tailscale installed. And to get past that, and believe me, I've tried everything except a different installation mechanism. But, you know, rebooting, reinstalling, blah blah blah. Here's the thing that works for me. I reboot into system recovery mode. And go through all that bullcrap. I get to the terminal. Uh, there, and I disable SIP. That's System Integrity Protection. That's a thing you don't want to do.
Jim McQuillan:Right.
Wolf:Um. Right. And then you have to reboot. And then you're in a world with no SIP. Which is bad, it's like those super cool power saws that, uh, you know, table saws that stop if they touch flesh. Um, it's like NOT having that. I don't want to not have that. Uh, and then I ask… it could be as simple, then, as asking, uh, Tailscale to do the upgrade again. Or, it might be that I have to go in and remove the extension by hand, and then start Tailscale, and it will automatically install the extension. Anyway, once it resolves that problem, which is fairly easy with no SIP. Um, yeah. Then I have to reboot into recovery mode, uh, do all the password ***** get the terminal up, and then re-enable SIP. Uh, and then reboot again, and now I'm back to normal. So that whole thing takes 15 minutes, or maybe 20 minutes. Um, it's annoying.
Jim McQuillan:All right, all right, so. I just searched, recommended way to install Tailscale on macOS. And one of the guys, I've seen him a lot, from Tailscale, he says the recommended way to do it is download the standalone.pkg file.
Wolf:Uh-huh.
Jim McQuillan:Uh, and install it. Uh, it… it does everything you need it to do, and there… he says, yeah, there are other ways to do it, which apparently you're doing one of the other ways, but why not do it the recommended way? Is there something you're… you're not getting by… by doing that? Is there something you hope to achieve by doing it the brew-cask way?
Wolf:A. No, I… there were a bunch of different ways to install it, and, um… I… prefer to install things either using cargo or using brew because then.
Jim McQuillan:Mmhm.
Wolf:um, I'm automatically plugged into updates. Whether that thing knows how to update itself or not, no matter what it is.
Jim McQuillan:Okay.
Wolf:It's gonna be that at least once a week, um… I'm gonna, by hand, make sure that it updates, and if it's the kind of thing that updates itself, oh, that's great too. So, that's just my default choice when given choices.
Jim McQuillan:Okay, so that's… that's one of the differences between me and you.
Wolf:But if this one's better…
Jim McQuillan:That's, uh, you prefer to use brew or, uh, or, or, uh, cargo or something. I prefer to use the recommended method. And you know what? I don't have a problem updating Tailscale. It just updates.
Wolf:Okay, so you made it sound like. The things I said? are never the recommended method, but in fact, brew is often the recommended method.
Jim McQuillan:No, in this case. And I do install a lot of stuff with Brew, and I'm getting much more used to it, and I keep things updated, and I really like how it works. But, you know, when I went to install Tailscale, I said, how do I install it? And there were the instructions, so I followed it. And I believe Brew was there as well, but I chose the recommended way, or the first way I saw. And I just don't have a problem. So, that's my recommendation to you. Uninstall it. from Brew and try it the other way and see if that doesn't solve your problem.
Wolf:I am going to do that. That is my plan.
Jim McQuillan:And if it doesn't help, well, hey, sorry.
Wolf:It can't be worse! I mean, in all ways, in all ways, I love Tailscale.
Jim McQuillan:Oops. Well, it doesn't seem like it. It doesn't seem like it could be worse, yeah. Yeah, yeah.
Wolf:This one facet is a point of friction for me, and I need to fix it, and maybe this will.
Jim McQuillan:Uh, yeah, maybe.
Wolf:How about you? How was your week?
Jim McQuillan:So I, um… I, uh… it's been a good week, busy as usual, I'm enjoying what I'm doing, everything is great. I finished reading, this was last week, I finished reading The Martian from Andy Weir. And, you know, there's the movie with Matt Damon, and it's a great movie. The book… it's probably my favorite book ever that I've ever read. It just… I enjoyed it so much, and I'm sad that it's over. You know, it's one of those kinds of books.
Wolf:Really? I do know.
Jim McQuillan:Because he's… you know, Andy Weir is a… he was a software developer, and he started writing this book. It's his first book. He started writing the book chapter by chapter, releasing them online for free. And people were reading it, and when he got all done, somebody said, you should talk to a publisher and get it published. And so he did, you know, published into a real book that you could buy, and then, you know, Hollywood came calling, and they made this fantastic movie. I just really, really enjoyed the book, and now I have to go back and watch the movie again, because I know there's stuff I missed in the movie that I didn't understand what was happening. And now, you know, the books are always better, right? They're always much more detailed. And I just really. I really enjoyed it. Uh, I also read, uh, uh, uh, two months ago, I read Project Hail Mary. That's another one of his books that got turned into a movie starring Ryan Gosling, and that was a great movie, and the book was fantastic. And, uh, here I am. I'm stumping for Andy Weir. I read his book, Artemis. It's about a moon base where there are… it's a whole colony of people living there, and that was a great book. In fact, that got optioned into a TV show back around COVID time. Uh, so it looked like it was gonna become a series, and then it didn't, I think, because COVID kind of wrecked everything. Kind of wrecked the world. Um, but yeah, if you're looking for a book, uh, I'm not that much into sci-fi books, but if you're looking for a book, uh, The Martian is a joy to read. I really, really loved it. Um, so yeah, and like I said, I'm sad now 'cause I'm done. Yeah. But, uh, you know, I've, uh, we don't have any feedback from our, you know, our last episode was, uh, Pearl Refuses to Die, and I sort of expected some people would have some comments about it. But, um, um, it, it, it.
Wolf:Number one of which would be. What's Pearl?
Jim McQuillan:Yeah. Pearl refuses to die. I thought it was dead. No, I love Perl. Uh, I use it all the time. But, uh, check out that, that last episode if you, if you want to hear our thoughts on, on, our real thoughts on Perl. Um, but I do have some newsworthy stuff, I think. Yeah, yeah, you share it all the time. I've got some newsworthy stuff. A couple of things. Uh, Sea Kermit. Did you ever use Sea Kermit, Wolf?
Wolf:No, you always hear my real thoughts. I used Kermit. Is C Kermit something special?
Jim McQuillan:Uh, it's, it's, it's, uh, well, I, I think it's all the same thing. It came from Columbia University.
Wolf:I mean Kermit is the protocol, maybe see Kermit is the tool.
Jim McQuillan:Yes. Yeah. Yeah, the, the, the software, the open source project, in fact, it was an open source project before open source was a term. Um, it, uh, they released… A new version, their first new version in 11 years. Kermit was C. Kermit was written by Columbia people at Columbia University, and in 2011, Columbia canceled that project. I think they just didn't see a need for it anymore. So, uh, they kind of spun off the people that were doing the work. They spun off into their own open source project. And I'll include the link to the project in the show notes. And, uh, this one's kind of interesting for me, because years ago, I'm talking 1993. I was working on a system. The, the primary system we were using was a, uh, Texas Instruments, uh, an operating, the operating system was called DX10. It was a business-type operating system, and it worked. It ran COBOL. Um, but I, uh, at that time, we were also. porting our software to SEO Unix. Well, first we went to Xenix, then Unix, and of course then we went to Linux. But at that time in 93, we were still heavily involved in SEO Unix and I needed to get data from a, from the DX 10 system to a, uh, Unix system. And neither system had networking. I mean, of course, you could do networking on Unix, but at that time, you had to pay a lot of money to do networking. And the DX10 system did not have networking. There was no, uh, Ethernet card. Uh, so I needed to get the data from one machine to another, totally different. Systems, totally different operating system. So, I wrote a terminal emulator, where I could log in from the Unix system into the DX10 system. And so now I can sit at my desk with one terminal and talk both Unix and DXN, and life was good. And then I thought, why don't I add file transfer to this little terminal emulator? So I did some research, and I found Kermit. And there was a Kermit client for Unix, but certainly nothing for DX10. So I wrote my own, uh, uh, Kermit. protocol program. in assembly language on DX10. And it was very, very minimal. I could download a file, or, you know, I could pull a file from that machine.
Wolf:Did did you have other choices? For like implementation language.
Jim McQuillan:Like what? Like, what? No, that pro- that- that system, uh, okay, the languages that system supported were COBOL, Pascal, and Assembly. And we did not have… we…
Wolf:Okay, so you could have written this in Pascal.
Jim McQuillan:We did not have a Pascal compiler.
Wolf:Oh, so you couldn't write it in Pascal.
Jim McQuillan:I think that would have cost, like, $10,000. So we did not have it. The system supported it, we just didn't have the compiler. So my choice then was COBOL. or assembly. And I did assembly and, and.
Wolf:So your choice was assembly.
Jim McQuillan:Yeah, it wasn't even a choice, the language I use. But I implemented, uh, the barest of Kermit protocol parts that I needed. to pull files off that machine. And, uh, and that's what allowed us to take data from that machine and put it on a Unix machine and start giving our customers Unix. And, uh, so I had forgotten all about Kermit. It completely wiped from my memory. And then I saw this announcement that version 11 was released. I thought, that's pretty cool. I went, poked around the website, and I'll include the link in the show notes. So that was pretty neat. The other thing is Pandoc. Do you ever use Pandoc?
Wolf:Oh my god, do I use Pandoc. It's in my daily toolbox. I guess I don't use it every day, but…
Jim McQuillan:I think you convinced me to use it. All right. Yeah, I've got some stuff out there in Pandoc and I know our buddy Scotty, he uses Pandoc and it's out there. Anyway, they just celebrated their 20th anniversary on August 3rd. So I thought that was kind of neat. I thought I'd mention it. Pandoc is a tool. Tell us what Pandoc is for since you use it all the time.
Wolf:Pandoc is awesome, and it solves the n squared problem of input file formats to output file formats. Instead of having a different one, a different converter for every. pair, uh, which would be horrible. Uh, instead, what you have is, uh, a system where it can. Read. any input file into an intermediate format, which is more like an AST. It's not an actual document, it's a structure.
Jim McQuillan:And an AST is…
Wolf:Uh, an abstract syntax tree? Like, it's not actually an AI… or maybe it is, but whatever it is. It's not a document, it's a… it's a form that represents everything that's in that document.
Jim McQuillan:Yeah, okay.
Wolf:but probably only in memory. And then it has the opposite thing on the other side. From that intermediate form, produce one of these. Uh, big players in my world are, uh.
Jim McQuillan:In a structured way, yeah.
Wolf:Markdown documents, HTML, and PDF.
Jim McQuillan:Hmm, yeah.
Wolf:Um, and I will convert back and forth between those, but it knows something like, what is it, 80 different formats? And I think the formats are plugins.
Jim McQuillan:Yeah, that's… I've done both. It's a… it's a bunch.
Wolf:Um, as long as you know how to, uh, produce or consume the intermediate format, um…
Jim McQuillan:Yeah, it's. Yeah, and like with… I know with Markdown, there's a lot of different flavors of Markdown, with extensions and things, and Pandoc, uh, seems to handle all those things.
Wolf:I think you're you're. There are. Knows them all.
Jim McQuillan:Yeah, yeah, it's kind of neat. Anyway, happy 20th anniversary to the Pandoc project.
Wolf:You know, um… Umm… John Groomer. I think, has an opinion about how many different flavors of Markdown there are.
Jim McQuillan:Yah. Yeah.
Wolf:Err.
Jim McQuillan:For him, there's one.
Wolf:I'm kind of guessing, but yeah, I think he thinks… I think he… he… and I believe he has a pretty good claim on this. There is one. So, it's called Markdown, and there's… there's rules for how it works. Now, there's some other things…
Jim McQuillan:Yeah.
Wolf:that have Markdown in their name. Uhm… Like, GitHub-flavored markdown, and CommonMark, and things like that. But…
Jim McQuillan:Sure. Oh yeah.
Wolf:Umm. You know, Markdown. The thing. It's not just John Gruber, it's also Aaron Schwartz. Remember Aaron Schwartz?
Jim McQuillan:Oh yeah.
Wolf:A tremendous loss to the community.
Jim McQuillan:Yeah, his name came up last week for something, yeah. Yeah.
Wolf:He… he was a genius, and he was, uh, significant input into the design and, uh, maybe implementation of Markdown?
Jim McQuillan:Yah.
Wolf:Um, and… And he should be remembered for that.
Jim McQuillan:Yeah, yeah, you should. He should. Um, so, anyway, happy birthday, uh, Pandoc. Uh, if you don't under… if you don't know that project, go take a look at it. It's pretty neat if you want to convert files, uh, document.
Wolf:You didn't say the most interesting thing about Pandoc.
Jim McQuillan:Ah, what's that?
Wolf:What is it implemented in?
Jim McQuillan:Um, okay, if you're asking that question, it's almost certainly Rust.
Wolf:It is not! It is Haskell.
Jim McQuillan:Oh, what is it? Oh, that's right! Oh, oh, that's right. I knew that. I knew that. Haskell. I forgot. I even had it in my notes. Um, it is… it is, uh, the most popular Haskell project, uh, according to the Haskell website. I… I wanna… I want to take a look at the source code, because I've never seen Haskell. Have you ever done anything at all with it? I've never even seen it.
Wolf:I have.
Jim McQuillan:I want to take a look at it.
Wolf:It is very interesting. A interesting thing to know, by the way. Uh… I believe it is the case that the primary implementer behind Pandoc? It's also the driving force behind the common mark. a variation of Markdown.
Jim McQuillan:Yeah, yeah, I think I saw that, uh, because Pandoc seems to favor CommonMark when you start looking at their webpages and stuff. the documentation, um, which is good. You know, I always thought plain old Markdown, the official Markdown, sort of lacked a few things. Doesn't it not do tables or something? There's some stuff that it doesn't do very well.
Wolf:That's one thing it doesn't do.
Jim McQuillan:And that's kind of important, I think, for at least the things that I do in Markdown. Oh, gosh. Alright, you know, why don't we get into this? Uh, like I said earlier, we're gonna talk about testing. And this is not… This is not to convince you that you should be doing testing. This is about if you, assuming you want to do testing, um, this is kind of how, how to get going with it. And I, you know, Wolf, Wolf talks about tests all the time. He, he, he, he's a real big, uh, proponent. for doing testing. And I… I believe testing is important. I don't do it. I'm talking about automated testing. I just don't do it because I just can't seem to get my **** in gear to actually make it happen. And I think today's episode is going to help me. Uh, to move forward. And I hope it helps, uh, all of you out there too, if you're, uh, if you're not doing testing and, and, and Wolf's going to sort of tell us why we should and how we can and, and all of those kinds of things. So Wolf, you want to, you want to take it?
Wolf:Yeah, I think there's a spectrum, like there is for everything. And at one end of the spectrum, there's people who rely on tests, who need them and trust them, and they're part of the normal workflow. I don't think I'm at the very, very, very tip of that end. Um… Because I don't always… I don't have 100% coverage, and um… I don't make sure I have every single kind of test. And, uh, I don't… I… I use TDD, which is kind of extreme, um… when it suits me, but that… that's not a force behind… who I am or me getting the code done. TDD is test-driven development. It's where you have an idea that you want to implement or a bug you want to fix, but before you write the actual code to do that, you write the test. And you watch the test fail, and then you write the code, and you know you're done when the test goes green. So that's one end of the spectrum, and at the far end of the other end of the spectrum. uh, side of the spectrum, are people who don't see any value at all in tests. They can't imagine why they would waste their time writing tests. Um… that's not you. Uh, closer in towards me, but still at that end, are the people who, uh, aren't opposed, and then even further in, are the people who realize. Tests do have value. It's obvious they're doing something for somebody. They've seen it. But, man, is there a huge gap between… and it's this way for almost everything, right? The distance from 0 to 1? is gigantic. And the distance from one to a hundred? Nothing like that, not even close. It's a long way to go from zero to anything. But once you're at one. It's just a hop, skip, and a jump to get to 100. Not a problem.
Jim McQuillan:And that's where I need to get. I need to get to one.
Wolf:Yeah.
Jim McQuillan:So that I could start filling in the tests that I need.
Wolf:And some people say stuff like.
Jim McQuillan:So, yeah.
Wolf:I'm gonna liken this to something inside the testing methodology, like when people use test-driven development. They say, oh, your first test should just be, you know, this function is supposed to return 4, so you test that it returns 4. And then you go write the function, and all you write is return 4. Oy vey. Um, you know, I, I like testing, but that, that **** is just needless ceremony. Right? That's… I don't… I don't know what that is. Um… But… Essentially, um… You need to take a couple of steps. One step is where you can write a test at all. And the next step is where running those tests and seeing if they passed or failed. is an automatic part of your workflow. Not automatic because it's second nature to you. Automatic because… It's in pre-commit, or it's in, um, you know, the script that you run that does all the things you have to do before you can deliver. For some languages, you always have a thing like that, a thing that's in between you and the customers. Like, if you're writing a C++ program, there's a stage where you compile.
Jim McQuillan:Sure.
Wolf:If you're gonna compile… Why not add another step that is, now that you've compiled, run the tests? If you're using a language where there's no separate or noticeable or distinct. compile phase, like Perl, or Python, or, uh.
Jim McQuillan:Mmhm.
Wolf:Maybe even.
Jim McQuillan:JavaScript.
Wolf:JavaScript, but maybe even Java counts in this, not 100% sure.
Jim McQuillan:Yeah, thank you.
Wolf:There isn't an easy affordance to that, although pre-commit's pretty good. Uhm… And then making testing unavoidable. And then the second nature part about… just writing the tests all the time, because, uh, you know, it's amortized. If you, uh, change the way some API is designed. And you fix the tests right then, and you notice right away, because you run the tests every time. Then you're only writing, you know, 2 or 3 tests at a time, um, and the work you do is small, and the… activity of seeing a broken test is rare and small and. You know, it's like when you do it the right way in some particular algorithm, like growing your hash table or not. Sure, it's expensive to grow the hash table once it is too heavily populated, saturated. Um, because now you have to rehash and copy all the keys, for instance, if that's the kind of hash table you have. But because, for… 15 minutes up until then, you'd just been adding keys, um, over and over with no growing at all. So that big balloon payment you made, um… You really amortize that over the… 2 hours, or 15 minutes, or whatever it was, of individual entries, and it turns out to be small. But in testing, it actually is small. It's amortized physically as well as logically.
Jim McQuillan:Umm.
Wolf:So. You've got something in the way. And the thing that's in the way is… You've never written a test. So there's that. You've never run a test, so there's that. And maybe Pearl will help you. Write tests. Like, for instance, in Python, there are many, uh… testing packages that will help you. I think the best one is probably PyTest. A lot of these testing packages came from, um, other places, like originally Smalltalk. Java had some very important contributions to the testing landscape.
Jim McQuillan:Hmmm.
Wolf:Did you look around? Do you know if Pearl gives you something?
Jim McQuillan:Oh, yeah, there's a… there's a… there's a well-known testing framework, uh, for Perl. In fact, if you install… if you build a CPAN package. Um, I, I had a platform before where I had to, I couldn't install CPAN packages using the CPAN tool. I could just download the package from CPAN and build it myself. And there's, there's a testing framework there and it's, and if you run CPAN install package name, you'll see the tests run. And it always impresses me how many tests some of those packages have. I feel like that's a quality package when I see 60 tests for that one thing.
Wolf:And I have. Okay, so you know of a package. Is the package itself an obstacle? Is it hard to use, really big, bad API?
Jim McQuillan:Sure. No, no, it's, it's, it's slowing, you know, I'm moving along at 100 miles an hour every day.
Wolf:That's right.
Jim McQuillan:In order to do the testing, I'm gonna have to stop. Yeah, I'm gonna… you hit this, and you dealt with it a long, long time ago.
Wolf:Me too.
Jim McQuillan:I just haven't… slowed down the freight train enough yet to do the testing thing.
Wolf:You and I are very, very different in this way, and I think we've talked about it before, in that, um… We both want to deliver for the customer.
Jim McQuillan:Sure. Sure.
Wolf:Um, and your way of delivering is to work directly on the problem, as hard as you reasonably can, in a way that produces the best result, and get that to delivery as quickly as possible.
Jim McQuillan:Sure.
Wolf:That's you. Uh, and my way is, um…
Jim McQuillan:Yeah.
Wolf:See, my way, I know all kinds of, um… uh, adverbs and metaphors for. Uh, I think it was Abraham Lincoln, maybe? Somebody? Who said, if I had, uh, 6…
Jim McQuillan:Oh, when he was talking about testing?
Wolf:Six hours to chop down a tree.
Jim McQuillan:Yeah.
Wolf:I'd spend the first four sharpening my saw, or axe, or whatever. Uh, so, for me, I feel like.
Jim McQuillan:Yep. Sure. Sure, sure.
Wolf:it's okay to stop and do the thing that I know, in the end, is gonna make me go faster, not just for this one delivery, but forever. Um… A difference between you and me is, and I've noticed this, um… You predict in advance. What the switching cost is going to be. You say to yourself, oh, it's going to take me 10 hours to get testing into my pipeline. Um, and you let that 10 hours be weighed against what 10 hours of honest work on the real deliverable would be.
Jim McQuillan:Sure.
Wolf:And so, you're taking a high confidence number, the work, the deliverable, against a super low confidence number. But that doesn't matter to you. You just say, well… It's not worth it right now. Let me do this one. Maybe later.
Jim McQuillan:Yeah, yeah. Plus, plus, I get paid for delivering stuff to my client.
Wolf:Right.
Jim McQuillan:Um, working on my infrastructure is not paid time, so that influences what I'm gonna spend my time on. And I realize it's flawed. I just need to need to slow down this freight train and do the work. We've gotten into a rhythm. We've gotten into this rhythm in my company where we go through periods where we're working hard, really hard to deliver something.
Wolf:Well, maybe. I mean…
Jim McQuillan:And then we, then we might get like two or three weeks where the pressure is off. And what I need to do is jump the next time that happens and, and commit myself to, uh, to, to start the testing process, uh, the next time I get that. two or three week lull. I think that'd be a great use of my time during then.
Wolf:Umm. So you just said something to me, and I'm gonna paraphrase it. It started with.
Jim McQuillan:Umm.
Wolf:I'm gonna wait.
Jim McQuillan:Yeah.
Wolf:And then it just stopped. Because there, you know, like, what's the right time to apologize? What's the right time to blah blah blah? What's the right time? There's never a right time.
Jim McQuillan:Oh yeah, yeah, yeah. Sure, sure.
Wolf:It never comes. So there's only two answers. It's just like there's only two sane numbers in the world.
Jim McQuillan:Right.
Wolf:Right? I guess there's 3. There's 0, there's 1, and there's infinity. And anything in between is probably magic. If you have to do something, there are two good times. There's right now, and there's never.
Jim McQuillan:Yeah. Hmm. Yep. Yeah, well…
Wolf:So if you're not picking right now, explicitly. You are picking never implicitly.
Jim McQuillan:Yeah, okay. Okay.
Wolf:All right, so. Umm. What I think is motivating. is, uh, first of all, a thing I can't promise you, but let's attach it to a thing I can promise you. A thing that is motivating is when you write a test. And either immediately… Or… Sometime. that test fails, for real, and teaches you something you didn't know before. Like, hey, guess what? This routine I wrote isn't thread-safe, and I thought it was! Like, that's a big deal. Learning that fact. is way better than learning it from one of your customers.
Jim McQuillan:Mmhm.
Wolf:Right. throughout your code, so that's the thing I can't promise you. Um, and also… Whatever your testing situation is. It doesn't tell you that you're bug free. What it does is… A test turning red tells you you do have a bug. But, this is the classic, you-can't-prove-a-negative thing.
Jim McQuillan:Sure, sure.
Wolf:All you can show is my tests ran. So… In your code. You deal with all kinds of things at all different levels. you have made it, uh, uh, public to people listening to our podcast that you write medical software. So, obviously, uh, the integrity and privacy of, uh, personal information for.
Jim McQuillan:Mmhm.
Wolf:Patience. is a top priority for you. You want to keep that information safe, private, and protected.
Jim McQuillan:Mmhm.
Wolf:So this could be a critical area where you could write tests. Hey, does this thing get exposed here? Or something. Maybe that's not the right place to look. I don't know your code. YOU know your code. What you are looking for is, um… Something that… Changes a lot, maybe. Some piece of code where the code changes, or faces new inputs, or something like that, but there's change, there's flux. Um, that… If something goes wrong with this code, the blast radius is big. Like, this particular thing going wrong makes 10 other things go wrong. And the cost to either fix it or repair it. is high. When you identify a place like that in your code, that's a thing. That might be a good place to start. All I want to do. is I want to find success for me for this episode. Episode is. if you and I can agree, you don't even have to tell me what it is, but if we leave this show where you know the place in your code. That's WORTH. Overcoming that inertia and writing a single test. If you can know that place. This episode is going to be a success for me.
Jim McQuillan:Sure, sure, okay.
Wolf:So the way we start is you figure out that place. Umm. And what you care about is that it's critical, not that writing a test would be easy. But… You can't choose something that's so hard to test. it turns into, uh, an obstacle. Like… You try and try, it doesn't work, you can't make a test, and so you give up forever. I don't want that. What I want is something that's a balance. It's critical, but reachable. And then, on our next episode, we talk about the one test. You wrote.
Jim McQuillan:Hmm. Okay. You're assuming I'm gonna write a test in two weeks. That's funny.
Wolf:All it has to be is one test.
Jim McQuillan:You're cute. Okay, let me… you know, I've been saying I don't do testing, and it's not exactly true. I have a library of functions that I've written in Perl. that, uh, I don't know, contains about 40 or… 50… Just little utility things, you know, convert things to upper case, upper and lower case, combine a first name and last name in an intelligent way. I do have tests for those. I wrote those tests years and years ago. And here's the thing that just amazed me. I wrote those tests. Uh, you know, validate a phone number, uh, validate a social security number, those kinds of things. I wrote those tests, and immediately, I found cases that, that didn't, didn't pass the test. You know, it was… it was eye-opening. I… oh yeah, I found, uh, cases where, uh, I gave it some input, and it did not produce the output that it should have. Those… those tests, I have really clear in my head how to do those.
Wolf:The thing I couldn't promise you.
Jim McQuillan:how to write that kind of test. You know, you give it a phone number, and you want it to produce an edited version of that phone number, or you want to validate that phone number to make sure it's a valid-looking phone number, right? Those are really straightforward to me. Uh, you give it an email address. That's primarily a big regex, right, to handle all the possibilities. So, then you feed it a whole bunch of email addresses, and we've got a database with probably 750,000 patients.
Wolf:Uh-huh.
Jim McQuillan:A lot of them have email addresses. So I've run my tests against the entire database, you know, to see. And yes, I've found email addresses that are not valid. So I do understand. that I need to test. I do have some tests, but I'll tell you what really bogs me down.
Wolf:And you've had success.
Jim McQuillan:Yes, I've had success, I've seen the value, and I do run those tests. That's a module that contains all those functions. It doesn't change. you know, it just doesn't change much. So it's, you know, I'm not constantly finding new cases where it failed. So it's great to have those tests, and I made that library better as a result.
Wolf:So you've got some experience, but it doesn't solve this problem.
Jim McQuillan:But, let me… yeah, let me tell you… Let me tell you… where my… where my problems are in thinking how to test. And then maybe we can discuss ways to do these things. a whole lot of my code touches the database. It updates the database, it reads from the database, it inserts new rows into the database, and I've got this 4TB database is my, you know, my customer data set, and we… I don't have a copy of that data in my development environment. But we do have a, uh, uh, uh, uh. a snapshot of the database to work with at the client. So I can run, you know, tests and stuff against that. But I don't really understand how to write a test. that does some database work. Uh, how do you do that? How do you… you know? Especially if I want to do it here, uh, on my little tiny database. I just don't know where to begin, uh, how to deal with a database, uh, a module that talks to the database, how to do the testing on it. It just, it… It's not in my head how to do that. Before we start to, you know, talk about a solution to that. Another thing I do is a lot of UI stuff in a web browser. You know, it's JavaScript, HTML in a web browser. I don't know how to test that. I know there's things like Selenium that you can use. I haven't tried it. These are, like, these are the hurdles that are stopping me from going forward.
Wolf:I hear ya.
Jim McQuillan:How do you do that, right? How do you… and, you know, it's… my code is not only Perl. Like I said, there's some JavaScript. Now I'm working in TypeScript, and a whole other platform, I'm doing Swift development on a Mac. And, you know, obviously. maybe not obviously, but certainly there is a whole testing methodology for Swift and SwiftUI, and I've largely ignored it, sad to say, but I really need to do it. But again. you know, there's a lot of network stuff going on. How do you test that? It's sort of like a database problem. How do you test things that go through the network? This is where I get hung up.
Wolf:Um, okay. So let's talk about a couple of things. Let's talk about if…
Jim McQuillan:Yeah.
Wolf:Hard problems, like these, are the right starting place. Let's talk about, okay, they are the right starting place. How do you solve it? Um… What things about… so, um, there's… Up. A test can be flaky. or not. A network test might be flaky. In the particular testing framework that I use. You can mark particular… you can mark any test you want with anything that you want, and then you can run tests. based on which ones have specific marks. So, for instance, um, a thing that I do is I mark tests that are slow. Um, and tests that are slow. I run in a full test. but I don't run as part of my, uh, pre-commit test runs. So, you know, I commit, commit, commit all the time, but I don't commit without running the tests. But when I run the tests for commit. I want those tests to complete in 5 seconds or less. And so if I have a slow test. I just omit it from the pre-commit tests.
Jim McQuillan:Sure.
Wolf:And then later, when I'm gonna do a release, or when it's on my CICD machine, or whatever, that runs all the tests. If a test is going to be flaky, well, that's a problem. That's a question about whether you're going to trust that test.
Jim McQuillan:U-huh.
Wolf:Did it fail because the networking failed underneath it, or because it actually did the wrong thing? Um, and either you can tell the difference between those two things. or you can't. If you can't, is it a reasonable test? Is it a test that gives you new information? Let's talk about the database problem.
Jim McQuillan:Umm.
Wolf:What part about communicating with the database is in between you. and knowing whether the thing you wrote works or not.
Jim McQuillan:Well, let me… Let me clear something up first. You know, we mentioned I'm in the healthcare business, right? I'm not writing… I'm writing, like, business software for healthcare. You know, scheduling, billing, workflow, stuff like that. I'm not writing software that examines digital x-rays and decides if there's cancer there or not. That, you know, so it's not… it's business software that I write. The fact that it's used in a medical office, that's just how it is, right? It's, uh… so I don't want people to think that I'm writing software that's dangerous in a medical. Uh, setting. Um… The kinds of things I think about that I would like to be able to test would be, we manage the patient ledger. That's the charges that are billed to the patient. you know, the claim is sent to the insurance company, the payment comes back, and we process the payment file. It's all electronic. We process a payment file and apply those payments to that patient's ledger. So. I start thinking, man, how could I test that? First of all. Um… I get a payment file from the insurance company that's expecting certain patients to be in the database. I don't even know where to begin that kind of a thing.
Wolf:Okay, I can already stop you.
Jim McQuillan:Yes.
Wolf:Um… You are thinking big. And tests…
Jim McQuillan:Yah.
Wolf:are useful when they're small.
Jim McQuillan:Okay, so that's that's a big part part of my issue.
Wolf:So, that big thing isn't one test. It might be six.
Jim McQuillan:Yah.
Wolf:Um, where one of those tests is, uh, am I getting the right thing out of a received, um, insurance, uh. posting to me? Number two is, um, can I make a change to the patient ledger? Number three is, um. You know when you put them all together. And you do a big operation that… uses a real insurance document, and knows about the patient being in the database, and does the entire ledger transaction, and uses the network. That's not a unit test anymore. Unit tests are small. A unit test says this function. Does. What it says on the label. Its API makes a promise, it's gonna do this thing. And the test you write says it did what it promised. When you make a big thing that needs a whole bunch of components together and isn't satisfied by pretending it's got to be the real things, that's called an integration test. Integration tests are valuable. But they're expensive. and they're harder to write, and they're harder to run. When you're using Selenium on a web page, that is much closer to what we think of as integration testing than unit testing.
Jim McQuillan:Mmhm. Okay.
Wolf:So, let's just take the tiniest part of. Um… that. Uh, database transaction. Uh, the database knows stuff about the patient's current state of payment and whatnot.
Jim McQuillan:Mm-hmm, mm-hmm, the balance.
Wolf:And you've got a function that is gonna make some changes. And then… Once those changes are made. Everything should be back in a consistent state that matches what you and I both refer to as the invariance of the patient, or the patient's database records. Um, like, for instance, uh, a bank transaction, um. It might be that in the middle of the deposit function, or the withdrawal function, or the transfer function.
Jim McQuillan:Mmhm.
Wolf:Inside any one of those functions, the invariant might be broken. Uh, the invariant being, um, that if you add up all the accounts of this customer, they must equal exactly the total amount of money they really have.
Jim McQuillan:Mmhm.
Wolf:But during a transfer, it might be the case that if you're transferring $50. It might be the case that, uh, temporarily, their total balance, if you were to calculate it, is $50 too high, or $50 too low, depending on what order you do it in. Um… So. What can you test? You can test… that. The state BEFORE you do your operation is a given thing. And then you do your operation. And then you test if the state AFTER the operation is consistent, meets the invariance, and is what you expected. And that's all. Did they get the $50? In my framework, pytest, and actually in all the Python testing frameworks that I know of. You. Uh… either pass or fail the test by raising an exception, an assertion, or whatever. So, you assert that the answer is such and such.
Jim McQuillan:Mmhm.
Wolf:that they only have $50 total or whatever. Uh, they actually describe this as the, um, three A's. The first A is arrange. the second A is assert, and the third A is… Oh, no, no, the third A is assert. The middle A is assert. effect or something, that's where you run your function that you're actually testing. Um, there's a couple different. Names, or… acronyms or whatever for this. But the idea is… Like I'm always saying. It's about the promises.
Jim McQuillan:Sure.
Wolf:Especially don't be tricked into testing the implementation. Test the promise. This thing says that at the end, you know, it's the bank.transfer function. This thing says that at the end. This account is gonna be $50 more, and that account is gonna be $50 less.
Jim McQuillan:Sure, the double entry accounting. Yeah, yeah.
Wolf:Test that. Yeah, don't test how it did it. Test that it did what it said it was going to do. That way, if you need to change the implementation.
Jim McQuillan:Yeah, sure. So…
Wolf:You can change the implementation, and the tests will continue passing.
Jim McQuillan:Okay, so… I hope I'm not getting back towards too much integration testing, but for me to test that a payment is applied. There's a lot of things I have to have. I have to have a patient. Right? That patient, I have to have a charge, I have to have an encounter, that's the visit between the doctor and the patient. Within that encounter, I have to have services, what are the services that were rendered, you know, an injection or, or an examination or something. For each of those services, I have to have charges. posted, uh, that are the fee for doing that service. And now, when I get to that point, now I can apply a payment to that charge.
Wolf:See, this still seems like multiple unit tests to me. Like, one unit test says…
Jim McQuillan:how do I… It does. Multiple unit tests, but not a… but not an integration test?
Wolf:Uh, that could all be one integration test, yes.
Jim McQuillan:Okay, okay. Umm. So. So, do I have to write… what… you said there's the three A's. What was the first one?
Wolf:The first one is a range.
Jim McQuillan:A range, range. So is a range, like, get your stuff ready? Um…
Wolf:Like. Yeah, make there be a patient, make there be a charge, make there be services, or whatever you need, right?
Jim McQuillan:with… Okay. Sure.
Wolf:Uh, and then, the middle A is, uh, effect, or something, I can never remember what the middle A is, but that is, run your function. Do the thing that, that you're trying to test.
Jim McQuillan:Effect right right. Yeah, yeah.
Wolf:And then the last A is assert. Um, do however many asserts it takes to, um…
Jim McQuillan:Ew.
Wolf:validate that all invariants are preserved, and the expected outcome. has occurred.
Jim McQuillan:Okay. So, um, there, there's so many things I should be testing, but if we just sort of stick with this theme for a few minutes. um, would it make sense that I create a… a test that creates a patient? It inserts a patient into the system.
Wolf:Absolutely.
Jim McQuillan:That would be a test. Okay.
Wolf:I mean, that's a thing you do. You do it all the time, right?
Jim McQuillan:Oh, yeah, yeah, yeah, and I've got a, I've got a function that does that. So, uh, write a test that does that, and, and, uh, it inserts a patient, and then, uh, the assert part, um… Well, arrange, in this case, I'm starting with the patient's not there, right? I almost don't have to do anything to arrange it.
Wolf:You don't, although if it were me, I would probably assert that the patient isn't there. Like, when you add a new patient.
Jim McQuillan:Then, uh, I… Okay, okay, yeah.
Wolf:Uh, like, a thing is, boy, that patient better not already be a patient.
Jim McQuillan:Sure, okay. So, uh, the effect here would be the actual inserting of that patient into the database.
Wolf:Yes.
Jim McQuillan:running the code that causes that to be in the database. Then the assert is to go read the database and make sure that that patient is there.
Wolf:That's right. And it might be that, um. You need to do a query. And maybe this is your real database, maybe it's not your real database. It is certainly your real schema. Umm.
Jim McQuillan:Yes.
Wolf:But if inserting a patient affects 4 or 5 tables. You need to do a query with joins in it to get exactly back what you need to validate. Um, if you're expecting these 7 things to exist now.
Jim McQuillan:Sure.
Wolf:and not exist before.
Jim McQuillan:Okay.
Wolf:Um, that's two different queries, one in the arrange part, and one in the assert part.
Jim McQuillan:So. Okay, so let's keep going with this. So now I've got this test that creates a patient. So the end result is I have a patient, right? So then the next test should be I need to create an encounter.
Wolf:That's right.
Jim McQuillan:That's another entry in a database table, right?
Wolf:Okay, I do want to say something before you go on, and that is one goal of testing is that tests are independent. They can be run in any order.
Jim McQuillan:Yeah. That's what I'm starting to wonder. Right. But… Um, if I'm gonna insert an encounter into the system. I need to know the patient ID that I'm going to insert that encounter for. And… seems to me that that would be the result of the earlier test. So all of a sudden, we're not independent anymore.
Wolf:That's right. So in my world, yes, you do test patient insertion, but also patient insertion is something it happens that you rely upon in what we call.
Jim McQuillan:Or do it.
Wolf:a, um… fixture. And so…
Jim McQuillan:Okay.
Wolf:Creating a patient is tested separately and independently. You don't have to worry about that other test.
Jim McQuillan:Yep. Okay.
Wolf:creating the new patient. And by the way, um, you're gonna wanna clean up the patient that you made, um, in the first one, so that it's not there anymore.
Jim McQuillan:Okay. Yes.
Wolf:Umm. I assume, depending on how you write it exactly. But a fixture is a thing that is gonna do exactly what I just said. It's gonna insert a patient.
Jim McQuillan:Sure. Sure, sure.
Wolf:you're gonna use the fixture, and then it's going to remove the patient when you're all done. At least, that's how it happens for me in Python. And so, that fixture, you would just name, um…
Jim McQuillan:Mmhm. Okay, okay.
Wolf:uh… Existing patient, maybe that's what you called it, in your argument list. And what that would do is, inside your function. the word existing patient suddenly is an ID. An ID maybe you never saw before, because it's serial or something, uh, and database assigned to PK. And. And now you know the actual ID of the patient to whom you can attempt this thing of, uh, testing that, um, an encounter is appropriate.
Jim McQuillan:So this… this fixture… Umm. I don't want to spend too much time on this, but this is one of my stumbling blocks. I've got the test that inserts a patient and then removes it when it's done. And it's got the code in there to call my function that inserts a patient.
Wolf:That's right.
Jim McQuillan:And, you know, ultimately, it does SQL to insert the patient into the database. Okay. So that's that test, and I can test that all day long. It inserts a patient, then removes it. Inserts a patient, removes it. And the way the database works is every time it inserts, it's going to sign the next ID, right? Because it's a serial col.
Wolf:That's right.
Jim McQuillan:So then I'm writing the test…
Wolf:But but this is important.
Jim McQuillan:Yep.
Wolf:Um… you don't need to test Postgres. Like, you know Postgres can do an insert. The thing you're testing is…
Jim McQuillan:Sure. I'm not… yeah.
Wolf:really your SQL. Did my SQL… because your insert might insert actually into three different tables to get a patient in there.
Jim McQuillan:Certainly it does. Yeah. Yeah.
Wolf:Right. So, it's not about, could Postgres do the insert? It's about, did I get the things in?
Jim McQuillan:Yeah, but the way you prove that then is to go read it from the database to see that it's in.
Wolf:That's right. So really you're testing your SQL.
Jim McQuillan:Yeah, okay, okay. So I've, so let's say, so let's say I write that test, right? I write that test, it inserts a patient, and I check to see that the patient's there, and it is, and then it removes the patient, and the system is back mostly where it started.
Wolf:Aha.
Jim McQuillan:Okay, now I want to test inserting an encounter. So, in that code, in that fixture for that code, I have to repeat some of what I did… the fixture in my previous one. That is…
Wolf:That's right.
Jim McQuillan:inserting a patient. Okay, those two tests have nothing to do with each other.
Wolf:That's right.
Jim McQuillan:Right? Okay. Okay. I was sort of thinking of how to chain these tests together, so that the output of one test becomes some input for the next test. That doesn't sound like… the, the, the, the smart way to do it.
Wolf:Umm. For unit tests? Hmm. It absolutely isn't. That is correct.
Jim McQuillan:Okay, alright, alright. Oh, this is good. This is good. So that, you know, we just keep going, right? Obviously, my first test is not going to be whether I can do a payment or not. It's… it's gonna be, can I create the patient where I'm going to put the… the… the… the encounter, the char… the service, the charge, and then ultimately the payment.
Wolf:That's right.
Jim McQuillan:Okay. I can, I can, I can start to see how that could happen. And do all this with a testing framework.
Wolf:So… This is key, um…
Jim McQuillan:Don't.
Wolf:The whole reason I use tests is because they reduce friction and lead to more robust deliverables.
Jim McQuillan:Sure.
Wolf:If… They don't reduce friction. You need to figure out where you can apply something extra. It turns out that AI is one tool that's really good at writing tests.
Jim McQuillan:Mmhm.
Wolf:Except… AI, most AIs at this moment. When you say, hey, go test this piece of code. they will test the implementation. Bug for bug. If your implementation has some messed up bug in it. The test and AI by default is going to write. is gonna make sure that bug exists. You don't want that. So…
Jim McQuillan:No.
Wolf:AI, if it's part of your workflow, if it's a thing that you find acceptable. can make the writing the test part. Easy. Um… You do have to read the tests, just like you read everything. Anybody who contributes to your code or output.
Jim McQuillan:Yeah, yeah. Sure. Sure, sure.
Wolf:produces. Whether it's a co-worker, whether it's a person who works for you, whether it's AI, it doesn't matter. It's you. The buck stops here. And because the buck stops here, you. Read and understand and take responsibility for every line of code that gets into a release.
Jim McQuillan:I totally agree with you. Absolutely. So.
Wolf:Um…
Jim McQuillan:Okay. I, I, I think I have, uh, uh, an idea of how to do what we just talked about. Uh, you know, starting with the, with the first thing and then, and then, you know, keep building the tests. step at a time.
Wolf:And don't get complicated. Don't worry about an automated workflow yet. Don't worry about…
Jim McQuillan:Sure.
Wolf:"Do these tests fit the shape of food or bark?" Don't worry about that. Worry about writing a test that just tells you, hey, this works, or hey, it doesn't work.
Jim McQuillan:Okay. Okay, well, kind of like the, you know, the tests that I did write years ago for those utility functions.
Wolf:Once you can actually write a test. Then we'll worry about calling them automatically, and uh, gating your commits on them, and all that stuff, and that will be useful to you!
Jim McQuillan:Yeah, okay. Sure. Yeah.
Wolf:But that's not the thing that's in between 0 and 1.
Jim McQuillan:Yeah, I think… I think I, I've got a, a million unit tests to write, right? But the real interesting stuff will, for me, be the integration testing, the way all of our things talk together.
Wolf:It always is. It always is. But, remember, when you say, I've got a million tests to write.
Jim McQuillan:Yeah.
Wolf:That's a mountain in between you and, like, it being okay.
Jim McQuillan:Yeah, I know. I haven't written one.
Wolf:That's right, and that mountain is gonna be extremely discouraging.
Jim McQuillan:Yeah, yeah.
Wolf:Um, you gotta set… you gotta set your sights on one critical thing. Hey, can I test this?
Jim McQuillan:It's exactly what it is. That one. Yeah, I think so. Yeah. Yeah, you're right.
Wolf:And maybe the first bite of getting to that one critical thing? It's a single unit test. Uh, that seems simple. Uh-huh. and then it's 6 unit tests, and suddenly you can string them together, and it's an integration test. Maybe… maybe that's where you are. Did I already say that Selenium is essentially integration tests?
Jim McQuillan:Yeah, that's true. Okay, we… Yeah, okay. When you start doing integration tests, are you just using your unit tests, or do you write a whole new test to do the integration thing?
Wolf:Integration tests are like human beings. They're as different between each other as human beings are.
Jim McQuillan:Okay.
Wolf:There are integration tests that are you. Using unit tests? But it's more interesting, uh, when it does the thing you were thinking originally. One thing feeds into the next, feeds into the next, feeds into the next. Unit tests don't do that, because.
Jim McQuillan:Yes. Right. Right, they should clean up after themselves.
Wolf:In your case, um, if it takes patient. Insert patient to prove insert encounter works. Then if INSERT PATIENT doesn't work. Insert encounter never works Of course, that's true if we use a fixture also, but um.
Jim McQuillan:Right. Right, right.
Wolf:Uh, but ordered tests, which are reasonable in integration, aren't reasonable in unit tests. Because your focus won't be narrowed when you see a bug to the thing that actually is the bug.
Jim McQuillan:Right, because I can see if I, if I did some work on, um, on, uh, uh, refunding a patient, uh, refunding a, a payment, for instance. I don't want to get bogged down in… All the other parts to that, right?
Wolf:That's right.
Jim McQuillan:Yeah. Yeah. Okay, cool. Cool, see? You're teaching me something. Which, every day you teach me something.
Wolf:The goal is…
Jim McQuillan:Some days, some days you'll turn, you teach me two things.
Wolf:The goal is that… This is a tool that becomes easy to use.
Jim McQuillan:Sure.
Wolf:but has a big ROI.
Jim McQuillan:Right, okay. So, I've got a couple of terms here, I just want to make sure I understand them. Unit testing, we talked about that. Integration testing. Okay, end-to-end testing, is that integration testing?
Wolf:Um…
Jim McQuillan:Or do people not call it end to end?
Wolf:It happens that any end-to-end… Yes, it is. End-to-end testing is like… like a user story. It's a big thing. It's, does the whole thing work for this scenario?
Jim McQuillan:Yes. Yes. Right.
Wolf:Um. Integration testing might be smaller than an end-to-end test.
Jim McQuillan:Okay. Okay.
Wolf:Sometimes when you say end to end, it has a specific meaning like encrypted at rest, encrypted in transit, encrypted at rest again. That's not this meaning. What this meaning is, is it's the size of a user story of a scenario.
Jim McQuillan:Right. Right, right. Okay. Okay, then, uh, regression testing, what is that?
Wolf:Regression testing is the greatest thing in the world! A regression test is almost always a unit test, um…
Jim McQuillan:Okay. Okay.
Wolf:But, what normally what happens is, um. You see some bad behavior. And, uh… It might be hard, at first, to figure out exactly the conditions it takes to recreate that bad… reproduce that bad behavior. But as soon as you can reproduce that bad behavior, you can write a test. Now, that test… isn't yet a unit test. It might be an integration test. Um, it might even be an end-to-end test. But it's something that fires when that bad thing happens. Now… that you know how to reproduce it, you can do that. debugging thing. Figure out where the actual bug is.
Jim McQuillan:Okay. Right.
Wolf:And that would let you narrow your new test down to something as small as a unit test. That new unit test you wrote for just this bug. It's a regression test. You're gonna turn it green with your fix? And as soon as it is green, and green forever, except not forever, um… You are ready for the next time it happens. You know, for instance, the bug is, you, um, were trying to validate email addresses.
Jim McQuillan:Huh-huh.
Wolf:But you didn't realize that a top-level domain could be two letters. So, when somebody's email address didn't end in .com, or .org, or .edu, you're like, well, that's not an email address.
Jim McQuillan:Right. Right. Right.
Wolf:That is why half the places I tried to register my email address with are like, I never heard of that! It's Christmas Island, you morons! Um… dot CX. Doesn't everybody use that? What about the crabs?
Jim McQuillan:No. Love the crabs.
Wolf:You write a regression test that says, hey, this was a valid, um. uh… URI, or email address, or whatever, URN, and uh… it didn't pass. Um… And you turned it green on the cases that you had. And then someday it turns red because somebody used a five letter TLD and. And now you know.
Jim McQuillan:Like dot store. Yeah, yeah.
Wolf:Yeah, a regression test. that sometime later turns red again. is one of the greatest feelings in the world. Because not only did you see something is wrong, and you saw it before the customer did. You saw exactly what was wrong and where.
Jim McQuillan:The key word there is before the customer found it.
Wolf:That's right. Regression tests that you have turned green. and then later you see red again, that is a test that has earned its keep.
Jim McQuillan:Yep. I… I… I can see that.
Wolf:Um, and I've got to remind you. that tests don't prove you don't have bugs.
Jim McQuillan:Right.
Wolf:But a red test proves you do have a bug.
Jim McQuillan:That's, that's like, that's like the number one thing to take away from this episode, right?
Wolf:so. I think it is. I mean, you know, they say the thing, and this is one of my favorite, favorite, um, expressions, experience is the thing that you get.
Jim McQuillan:Yah.
Wolf:when you didn't get what you wanted. That's actually a quote from Randy Pausch in Randy Pausch's last lecture. I hope I got his name right.
Jim McQuillan:Right, right. I loved it.
Wolf:Um, but the point is that, um, succeeding at something doesn't teach you anything. It's… that's the situation with entropy, where there's… there's no news.
Jim McQuillan:Yeah, yeah, we learned. Right, we learn from our failures. Yeah. Yeah.
Wolf:Failing at something. IS news. That IS a learning opportunity. So, green tests? Great. Fine. That's awesome. Uh, red tests? Yeah.
Jim McQuillan:Yeah. And it and it.
Wolf:Now I know what's wrong and I have something to fix.
Jim McQuillan:Yeah. And what do they say? If you're not failing, you're not trying hard enough.
Wolf:I actually have been thinking about that very thing in relation to practice and competition.
Jim McQuillan:Yeah, yeah.
Wolf:And I believe this, there's two modes. You need to practice right at the edge. You need to practice at the edge where, you know, 50% of your attempts, or 40% or whatever.
Jim McQuillan:Sure.
Wolf:failures, but they're only bad enough that you still understand why you failed, so you know what to work on. And then you go into the competition. And you do the thing that 90% of the time is gonna give you the best possible outcome based on your skills. So those numbers and those… that edge is always moving. Practice hard. Practice at the edge. Practice to learn. Compete? At a place where you're going to perform at your very best guaranteed performance.
Jim McQuillan:Well, what's the other saying? The difference between an amateur and a professional? The amateur works hard enough so that they can get it right. A professional works so hard they can't get it wrong. Right? Like a musician, for instance. Right?
Wolf:Uh-huh. Yeah, that's the… I thought I was gonna be a musician when I was young, and then I found out what it's like for a real musician, how hard they practice, how much they know about everything having to do with exactly the thing they're doing.
Jim McQuillan:They… Yes. Yep. Yep.
Wolf:And… I… that… that path for music and my instrument, which was the saxophone, um… That was… that did not have positive ROI for me. Um, on computers, it did. Um, but I will tell you, I am an old man, and to this day, and I think I said this to you before we started recording.
Jim McQuillan:Yah. Yep.
Wolf:Every single day, I learn more and more about just how much I don't know.
Jim McQuillan:Yep.
Wolf:about what I don't know.
Jim McQuillan:Yep, yep. But you, you're, you're learning. That's, that's.
Wolf:Always.
Jim McQuillan:That's the difference between you and an awful lot of people, right? There's a lot of people out there that are perfectly happy to be just exactly where they are, and you're moving forward, and that's good.
Wolf:I know.
Jim McQuillan:That's good.
Wolf:Jonathan Livingston Seagull. That… that is my role model. I don't… I don't know if you read it, but Jonathan Livingston Seagull, he's like, I just want to fly.
Jim McQuillan:Yeah. I have not.
Wolf:And every day, he just worked on how to fly better. And then, like, a crux of the book is… He goes up a level. Um, and to us, it looks like… Okay, I won't spoil it.
Jim McQuillan:Okay, don't spoil it. Don't… don't spoil it for me, because I should read that book. I… I…
Wolf:It's short. You can get through it in an hour. It's a great book. You should read it.
Jim McQuillan:I should, I am not a fast reader, but I mean, come on, it took me like five months to get through, what was the Asimov?
Wolf:The foundation. No, no, uh…
Jim McQuillan:The foundation. Yeah. Foundation. Yeah. It took me like five months to get through that. I enjoyed it, but.
Wolf:Well! I don't know exactly, but I… if I recall correctly, Jonathan Livingston's Seagull is, like, 60 pages long, maybe less?
Jim McQuillan:Ah, okay. I could do that. I could do that in a week. I'm not that slow. I tend to read late at night when I'm tired, and uh, you know, 4 pages in, I'm ready to sleep. But all right, what else do you have to say about, yeah.
Wolf:Let me say two things before we let it go.
Jim McQuillan:Yeah, yeah.
Wolf:Um, first… And. We gotta be honest about the timeline. You… nobody, not you, not the listener, not… not people who don't listen, nobody, is gonna go from, um, zero task… they're not gonna go from you to me in… a day or a week.
Jim McQuillan:You're right.
Wolf:Maybe even a month. Umm. I have described tests as a thing you will hate on day two. And you will love and can't live without a year or two. Um, I think that probably, uh, conservatively. Uh, making… integrating tests into how you do your job, into your workflow. is like a 6-month job. It's not a thing you just do when you're not doing anything else.
Jim McQuillan:Right.
Wolf:It's a little slice, a little slice, a little slice, and you make it help you. And when it's not helping you, that means you're not making progress. So the real timeline is longer than you would hope. I don't want to promise you things that you can't achieve and would.
Jim McQuillan:Yeah, it's…
Wolf:would be discouraging.
Jim McQuillan:Yeah. It's, it's not instant gratification. I understand that.
Wolf:That's right. And then I saved out a couple of quotes. Um, that I think are interesting. Uh, the first test is the hard one. The hundredth is easy. If I rewrote the inside from scratch. Should this test still pass? That's the quote and in case you don't know what the answer is. If you rewrote the inside, but it still satisfies the promise. It better be that the test you wrote still passes. Because if it isn't, if it doesn't, you didn't write the right test.
Jim McQuillan:Right. Then the test doesn't represent the promise.
Wolf:Um, I've said this a couple times, but coverage tells you what you definitely didn't test.
Jim McQuillan:Right?
Wolf:It can never tell you what you did. Maybe we didn't talk about coverage. A lot of, uh, test packages will actually, um, do an analysis, generally by running. to see exactly which lines of your code under test, the implementation, were exercised by tests, and which weren't. And then it'll do that across the entire test suite.
Jim McQuillan:Ah.
Wolf:And it'll say, "Hey, you effectively tested 60% of the lines in your test. package, or library, or application, or whatever.
Jim McQuillan:Umm.
Wolf:But that tool, coverage, um… innie. In just… Um… It just tells you what you absolutely missed. The stuff it says you tested? That just means it's a thing that got exercised. It doesn't mean that you actually validated it and proved that it was correct.
Jim McQuillan:Right. Right, it may have run that line of code. But it doesn't tell you whether or not you checked for the right answers.
Wolf:Yes. Um… This might be maybe the most important, a test that has never been read. Has never done its job.
Jim McQuillan:The color red.
Wolf:In test-driven development, you want to see the test fail first.
Jim McQuillan:Yeah, yeah.
Wolf:Because it's only a failing test that proves, um… It is definitely triggered by the bad behavior. And then it's only that test turning green when it no longer has the bad behavior that proves the other side of the coin. You want both.
Jim McQuillan:Umm.
Wolf:And finally, test quantity. It's free now. That's the AI thing. If AI helps you write your tests, you can write as many tests as you need to.
Jim McQuillan:Oh, okay. Yeah.
Wolf:But quantity was never the problem. Uh, it's, it's not about…
Jim McQuillan:Quality of the… yeah.
Wolf:Yeah, it's not about having a thousand tests, it's about testing the four things that are absolutely critical to you delivering.
Jim McQuillan:Yah. Sure. Sure.
Wolf:A quality product.
Jim McQuillan:Okay.
Wolf:And I think, basically. That's all I've got to say, but my hope is, do you know. In your program, some, any of your programs. A place to start. A thing to test.
Jim McQuillan:You know, I'm probably going to do what we talked about. I'm going to do the create a patient, create an encounter, create a charge, that kind of a thing, just to get started. You know, then, you know, create an appointment, create, uh, cancel an appointment, reschedule an appointment, all those things.
Wolf:Okay. Would it be acceptable to you? If at our next recording. We check in and we see if you wrote that.
Jim McQuillan:Yeah, yeah, let's try. I can't guarantee at this point that'll be ready then, but I will absolutely go looking at the test framework for Pearl, and see if I can…
Wolf:And remember, all I'm saying is…
Jim McQuillan:understand what's required.
Wolf:The test. I'm not saying use the framework, I'm not saying stick it into anything.
Jim McQuillan:Yep. I understand. Yeah. Well, and I do have tests already. Yeah, I do have tests already, like I said.
Wolf:You do what you need to do. Okay.
Jim McQuillan:But, um… Yeah. Yeah. Well, let's talk about it next time. Let's see what I've done. Try to hold me accountable.
Wolf:I can't wait.
Jim McQuillan:Go ahead, try. That's it for you. Well, that's a lot. Yeah, we, uh, boy, we had a lot of fun on this conversation. Uh, of course, it went longer than I expected, but that's good, right? It's some good information. They all go…
Wolf:That's it for me. You want to wrap it up? See, you need to reset your expectations.
Jim McQuillan:I really do. They all… we've done this 35 times now, and they all go longer than I expected. But that's alright. I just need to change what I expect, right? My fear is then they'll go even longer. So anyway, that was a lot of fun. We didn't talk much about it at the beginning, but feedback, you can always send us feedback. Wolf, what's the email address for feedback?
Wolf:feedback at runtimearguments.fm.
Jim McQuillan:Ah.fm, one of those two-letter, uh, uh, top-level domains.
Wolf:That's right.
Jim McQuillan:That the email address test better deal with. Yeah, you can check out our show notes at the end of this podcast, or in your podcast player. There will be show notes, and we're gonna have a couple of links in there, and maybe a little summary.
Wolf:And the transcription?
Jim McQuillan:Uh, yeah, we do a transcript, um… Golly, what else? Is that it? I think, uh… I think I have…
Wolf:Do people care about contacting us directly on Mastodon?
Jim McQuillan:Yeah, we'll have the links in, um, the, the addresses in the show notes or on our website for that. Certainly Wolf and I are both on Mastodon. Uh, we're on, we, we send out our announcements on LinkedIn. I no longer send out announcements on Facebook because. It's Facebook, and I just… I can't bring myself to do it anymore. So, uh, yeah. Sorry, Merlin.
Wolf:Forgive us Marilyn.
Jim McQuillan:I think he understands. Anyway, it's been a blast, Wolf, and I'm looking forward to doing this again in a couple of weeks. As always, thanks for podcasting with me.
Wolf:Me too. Thank you and thank you everybody on the other end. Talk to you later.
Jim McQuillan:Goodbye.
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
CoRecursive: Coding Stories
Adam Gordon Bell - Software Developer
Two's Complement
Ben Rady and Matt Godbolt
Accidental Tech Podcast
Marco Arment, Casey Liss, John Siracusa
Python Bytes
Michael Kennedy and Calvin Hendryx-Parker