Runtime Arguments
Conversations about technology between two friends who disagree on plenty, and agree on plenty more.
Runtime Arguments
37: Unix Year 2038 problem and the art of underestimating
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Jim walks through the classic 2038 problem — Unix's 32-bit signed timestamp (seconds since Jan 1, 1970) overflows on January 19, 2038, flipping negative and breaking anything that depends on file/time comparisons (e.g., Make). He compares it to Y2K, tracing the two-digit-year decision back to real storage constraints of the era — punch cards (80 characters each), early hard disks with fixed-size sectors — and argues those were reasonable trade-offs for their time, not simply negligence. That leads into a side discussion (Wolf) about engineering decisions made from measurement versus decisions made from feeling — and why the former holds up better over decades.
Also covered: the "DJ10K" problem (fear that 4-digit stock-ticker fields would break when the Dow crossed 10,000 in 1999), and how the 2038 fix has rolled out — 64-bit systems are fine, Linux kernel 5.6+ (2020) handles it, filesystem support varies (ext4 and btrfs/xfs are fine with proper config, older ext3 is not). They compare how databases store timestamps: Postgres' 64-bit timestamptz (good until year 294,276), MySQL's 5-byte datetime, SQLite's lack of a native date type (stored as ISO 8601 strings), and DuckDB's 64-bit microsecond timestamps.
The episode then runs through a rapid-fire list of real-world overflow/rollover bugs:
- GPS's 10-bit week counter, which has already rolled over in 1999 and 2019, and will again in 2038 and 2058 (moving to 13 bits)
- NTP's 32-bit unsigned rollover coming February 7, 2036
- Postgres transaction ID (XID) wraparound, and how autovacuum (added in Postgres 8, 2005) prevents it
- The Boeing 787's 51-day generator bug (all four generator control units can fail simultaneously if not power-cycled)
- NASA's Deep Impact probe, lost after a 32-bit tenths-of-a-second counter overflowed
- 16-bit limits on PIDs and TCP port numbers
- Discord's issues with Twitter-style 64-bit Snowflake IDs, since JavaScript numbers only safely hold 53 bits
- IPv4 address exhaustion and the slow IPv6 transition
- AACS DRM's 32-bit hard-coded expiration field, which can make Blu-ray discs/players stop working on a schedule nobody chose
Then a set of leap-year date bugs: Excel/Lotus's belief that 1900 was a leap year (it wasn't — a refresher on the "divisible by 4, except by 100, except by 400" rule), the Sony PlayStation 3 bricking on Feb 29, 2010 (which wasn't a leap year), and the Microsoft Zune's clock freeze on Dec 31, 2008.
Closing thought: Wolf ties it back to values — writing software that's fixable and maintainable by anyone, not code where you're the only one who can "pull the lever" (job security through obscurity), which Jim agrees is admirable but ultimately counterproductive.
Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm
Follow us on Mastodon: @RuntimeArguments@hachyderm.io
If you have feedback for us, please send it to feedback@RuntimeArguments.fm
Check out our webpage at http://RuntimeArguments.fm
Theme music:
Dawn by nuer self, from the album Digital Sky
Hey everybody, it is another episode of Runtime Arguments. I'm Wolf, and as always, I'm with my very good best friend, Jim. Jim?
Jim McQuillan:Hello, Wolf, how you doing?
Wolf:You know. Pretty good. Um, this is episode number 37. Our 38th episode. I think I'm going to get tired of that math. Maybe. Um, and today, we are…
Jim McQuillan:Yeah, it's tough adding one. Hell yeah.
Wolf:Today we are going to talk about.
Jim McQuillan:Yeah.
Wolf:A class of bugs. Umm. That really should be avoidable. Um, I can't wait to give my opinion on this, because I absolutely have one! Um…
Jim McQuillan:Of course you do.
Wolf:And I think in about 6 minutes of Jim talking, you're gonna know what my opinion is, and I won't actually need to say it, but I'm gonna say it anyway.
Jim McQuillan:Yeah, okay.
Wolf:And. Let's start with feedback. I don't think we have any feedback for this show today. I didn't get any.
Jim McQuillan:No, we had a, you know, we had a lot of downloads on the, on the past episode, um, but, uh, we didn't, didn't get any feedback. Uh, uh, must be, uh, back to school, right? Everybody's busy.
Wolf:Which is interesting, it was a. It was a non-technical episode. Um, you know…
Jim McQuillan:Yeah, kind of abstract.
Wolf:You're very quiet, Jim. That can't be me. Is it me?
Jim McQuillan:A thinker. Am I? Is that better?
Wolf:I don't know.
Jim McQuillan:I don't know.
Wolf:Um… I hope people listened to the last episode because I. Felt like it was really important stuff. Um, that I wanted to say. But that means…
Jim McQuillan:Let's remind everybody what that was.
Wolf:Uh, we talked about leaning your ladder on the right wall, and uh, that what you really have to spend is not time, it's attention.
Jim McQuillan:What was that episode?
Wolf:Uh, and attention has different values, depending on the activity, and you have a different amount of attention. Based on your personal situation. Are you married with kids? Are you on vacation? A million. Are you tired? A million different things control just how much of this you have to spend.
Jim McQuillan:Yep.
Wolf:And attention is a thing you run out of every day.
Jim McQuillan:Yeah, it's a finite amount.
Wolf:It is, and maybe, maybe you have a lot. Maybe you have 6 hours worth of attention that you can put into stuff. Maybe you only have 4. Um, maybe you only have 2.
Jim McQuillan:Oh.
Wolf:Ah.
Jim McQuillan:Or maybe 15 minutes.
Wolf:Could be. I thought it was an important episode.
Jim McQuillan:It was good.
Wolf:Not too late for feedback. Everybody send your feedback. But this means more time for…
Jim McQuillan:Yeah, please.
Wolf:Jim, how was your week?
Jim McQuillan:Busy, as always. You know, I mentioned this a few months ago. I'm working on a project. This is something that's been brewing in my brain for 15 years, and I finally started it back in June, actually working on it. Uh, it's, it's this project that involves Node and TypeScript and lots of stuff that I, uh, I'm, I'm not. I'm not a real TypeScript programmer, so I'm learning TypeScript along the way, and I'm having Claude help me, and that's been a lot of fun. But this application that I've been working on, I'm going to show the customer in early October, and it's ready. But what I'm working on now is the documentation. So that's, that's the point I'm at. It's, it's functionally ready. And I can't wait to show them, because it's really gonna… gonna impress them, I think. And of course, I'll, I'll, uh, I'll let y'all know how that turns out. Um, so that's been, that's been consuming all of my attention lately. So that's been good.
Wolf:Uh, I'm super glad to hear you stepping out of your comfort zone and learning new stuff.
Jim McQuillan:Um, yeah. Yeah.
Wolf:Which you do. If I ever imply that you don't, that would be a wrong implication.
Jim McQuillan:Yeah, I try. No.
Wolf:But I love it every time I see it. I have done some stuff. Um, one thing very important to the show is I've been working on automation to make posting the announcements, um. much more, uh, friction-free, and I know everybody out there is saying. Jesus, Wolf. Um… Posting one paragraph on Mastodon every two weeks, what friction is there?
Jim McQuillan:Now you sound like me.
Wolf:Umm. Yeah.
Jim McQuillan:I think that's pretty much what I said.
Wolf:Um, but to me, there's friction, and it gets in the way, and you've seen the result that I don't do the posts. I'm not done with the automation, but I'll tell you all about it when I am done, and I will post the code. So there's that. I took this whole week off. Because I am preparing for my very biggest. um, match, ever. Uh, USPSA Level 2 match, the Michigan Sectionals. And. I thought I was gonna practice every day, all day long, and boy, I'm not doing that. Uh, I got my vaccines yesterday, I got the mRNA for the flu, and I got my COVID booster, and I slept today until 4pm! That doesn't sound right! So I don't know about that. Um… But it's been pretty good, I think.
Jim McQuillan:Yeah, yeah, you are definitely, you are definitely not an anti-vaxxer, and neither am I.
Wolf:So a gentle. Uh, yeah, um, I like to do this thing, we have a special word for it, and that word is science. Um…
Jim McQuillan:Yes.
Wolf:And a lot of people don't like science. Because when you use science. The answers are changing. As you learn more things, you refine what you think the proper course of action is. And there's a ton of people who are like, well, the answer changes, that means it's completely bogus, science is stupid. Umm. I have opinions, I think you know what they are. I think the right thing for us to do right now is to get right into the meat of the, uh, episode, and Jim has a topic. Jim, why don't you go ahead?
Jim McQuillan:Yeah, so, um, some of you know this, uh, and, and I'm sure some don't, but there's a problem, uh, lurking in our future, uh, and that's the Unix year 2038 problem. UNIX has historically stored timestamps in a 32-bit signed integer.
Jim McQuillan: And, uh, what it is, that integer, is the number of seconds since January 1st, 1970, uh, 00:00:00 UTC.
Jim McQuillan:Um… we're gonna… we're gonna overflow that integer on, um… uh… let's see, what's the date? I have it here. January 19th, 2038. So we've got 8 years, right? 7 and a half years. We're gonna… we're gonna overflow that integer. What's going to happen when that happens? Lots of things might happen. Some things people are worried about maybe won't happen. But what happens with a signed integer is when you keep incrementing, in this case, a signed 32-bit integer holds a number that is 2,147,483,647. That's the number of seconds. When you add one to that. It does not go to 2,147,483,648. Uh, no. It flips negative, because the integer overflows, the sign bit gets, uh, set. Uh, and then all the other bits go back to zero. But that number is negative $2,147,483. 2,147,483,648. And then it starts counting down towards zero. Umm. It's… it's anybody's guess what's gonna happen at that point. Um… think of things, uh… you know, as a software developer, I use Makefiles. Make depends on file stamps on files, right? When you, uh… when you have an object file and a… and a… source file, and the source file is newer than the object file. uh, the, the, the code's gonna get compiled. Make will, will see that the source code has changed. Well. If, uh. Uh, the timestamp rolls over, or the current time rolls over, and all of a sudden, it thinks we're back in 1901? And you touch a file. Is that file newer than the object file that was last built in 2037? No, so your make is gonna fail. Now, that's not a huge problem, but there's all kinds of other problems that relate to timestamps and things happening or not happening, and age… calculations and stuff. So… It's a real problem. And this is, this is kind of a class of problems, and we're, we're gonna cover a few of them, a few more of them here. Um, but, uh, uh, there have been steps, uh, made to, to solve this problem. Um… It's, uh, on a, this is typically on a 32 bit system, a system that stores its time in a 32 bit timestamp, a 32 bit CPU, or even a 16 bit CPU that uses a long integer, uh, can have this problem. 64 bit systems. generally don't have this problem, uh, because an integer in 60… in a 64-bit CPU is 64 bits long. That's a much, much, much longer. Uh, uh, time. That's, that's billions of years, uh, worth of seconds. So that problem doesn't exist. Uh, but there's an awful lot of machines out there, uh, systems out there, uh, embedded systems out there that are running 32-bit CPUs. that, boy, they could run into this problem. But anyway, a lot of work has been done. As I mentioned, the 64-bit systems really don't have this problem. Linux, the Linux kernel for 32-bit CPUs, they took care of that. Uh, let's see, how's it being fixed? Uh, they took care of that in, uh, Linux kernel 5.6, which was, uh, released in 2020. So that kernel's been out for about 6 years. So even if you have a 32-bit system, as long as you've got a newer version of Linux, uh, you should be okay. Um, but also, and that's just the time T, uh, data structure, uh, field that, that holds the time. Um. if your software hasn't been compiled to take advantage of that, you could still have a problem. This problem… feels an awful lot like a problem we all ran through, uh, any of us who've been computing for a while, and that's the Y2K. problem. Remember the year 2000 bug? Wolf, did you have to worry about that much? See, I was working with medical information. I was working with financial transactions, patient data. The year was really important to us.
Wolf:I did.
Jim McQuillan:Uh, so… so we had to deal with this. Uh, the year 2K… the Y2K problem is really the same kind of thing. The field that was set up to store the year wasn't big enough to hold. a big year. And that was when the calendar flipped from 1999 to 2000. A lot of old software was written to store the year as a two-digit number. year. So when it went from 99 to 2000, when it went from 1999 to 2000, it really went from 99 to zero. So all of a sudden the year is zero instead of 99. So now age calculations were all off. They predicted all kinds of problems. Fortunately, engineers saw this for a long, long time coming. You know, they sort of knew it was going to happen. So an awful lot of work went into it. A lot of money was spent on this. And in the end, the problems were really minor. I don't know of any significant issues that happened when we flipped to the year 2000. Um… And some people will say, well, that was an awful lot of hype, an awful lot of work for nothing. And others will say, well, we didn't have major problems because we put all that work into making sure we didn't have that problem. I side with the latter, where I think, yeah, the work was done. Uh, in my case. I went to work for a company in 1985. And at that point, the software, it was all written in COBOL, and it had two digit years. So patient dates of birth, dates of service, financial transaction dates, they were all a two digit year. I was in a good place because, uh, it was 85. You know, we had 15 years to deal with the problem. And in around 92, we started redesigning the system. Complete, complete rewrite. This was all written in COBOL. Uh, complete, uh, rewrite, uh, so we could deal with that. Um… But, but let's talk for a minute about, about why. Dates were stored in a two-digit year. Uh, you gotta think about, uh, storage, uh, back in the day, back in the 60s and 70s, and even into the 80s. Storage was expensive. Um… and rather limited in size. Uh, you go back to the 60s and 70s, and storage, many times, was punch cards. Did you ever get to work with punch cards?
Wolf:Oh my god, yes. I'm dating myself, but I never dropped a deck. But I absolutely had to deliver decks to the computing center for them to run.
Jim McQuillan:No.
Wolf:And. Yah.
Jim McQuillan:Yeah, well, I had a little bit of experience with cards. I went to school at Michigan Tech, and at that time we were, you know, the Fortran class, we were learning, we were programming using punch cards. That was my only experience was in school. Uh, but boy, I remember, you go to the student bookstore and buy a box of punch cards. It was 2,000 punch cards. And… The big fear was you'd drop that box. And, you know, a program, you know, you know, classic Fortran program for a college student. Uh, you know, the early days of, uh, programming classes, our job was to create a payroll system that calculates payroll, you know, regular time at 40 hours, time and a half after 40 hours, so we had to write a program like that. And that program might have been 100 lines of code. But even that, 100 lines of punch cards, 100 cards, uh, because each punch card was 80 characters wide, um…
Wolf:Not counting JCL.
Jim McQuillan:Well, yeah, you had to, you had to add the JCL at the beginning. So those, each line of JCL was another punch card. So you might've had six or seven or eight lines or eight cards in front of your code and then a, a, a card or two at the end to, to indicate that's the end of the deck. Uh, but boy, occasionally you'd see somebody in the hall, and they tripped or something, and they dropped their cards, and uh, yeah, there's their, there's their 110 lines of code… scrambled. I, it was, uh, I, I, I shouldn't laugh 'cause it did happen to people. But anyway, you, you take that 80 character card, that's 80 characters of information you can store on it. And a card sometimes would represent, um, a student. You know, it'd have their name on there, and, uh, their date of birth, um, maybe, um… I don't know. Maybe their address? You know, you can't fit a whole lot into 80 characters. Uh, so what you would do is you would try to store things as tightly as you could. You wouldn't store 1980 as the year. You would store 80. Because that extra 2 bytes out of 80, that was important. If you had a couple of dates in the row, in that record. There was a lot of space, so people just would use two-digit years. Umm. Fast forward a little bit to hard disks. Um, you know. back in those days, well, in college, we didn't have hard disks, but when I first started working for real in software development, we had hard disks that were on the order of, like. five megabytes. We had floppy disks, you know, that were, that were, uh, 512 K. Um, you didn't have a lot of room. So, so every byte you could save was a lot. So again, uh, uh, the designers of software, they would just. skimp and go with the two digit years. The other problem with hard disks, even as the hard disks grew. Um… storage was… it's not like it is now. Everybody paid attention to storage, the layout of the data on the disk. So if you were creating a table with a bunch of records in it, you know. patient records, or transaction records, or something like that. Um. The way hard disks worked was you have a platter or a set of platters. You know, if you've ever taken a disc apart, you've seen them, you've probably seen pictures of them, right? You have this set of platters and each platter has these concentric rings and those are called tracks. and each… and there might be, I don't know, 63 tracks on a platter, and each track was a set of sectors. It was broken down into, like… I remember the old IBM hard drives were, like, 17 sectors for a track. And each sector was… There was varying sizes, but there were, like, 256 bytes, or 512 bytes, or, uh, newer drives were, like, 4,096 bytes. The systems were so crude back then that when you'd store data, you could only write a sector at a time, or read a sector at a time. It had to be a full sector. So if you wanted to write one byte, it was going to use up a full sector. It was going to write to that sector. Uh, so if you wanted to write a patient record, uh, and maybe it's, uh, you know, 240 bytes of information, it was gonna use a sector of 512 bytes. No matter what. If you wanted to write a second patient, it didn't get tacked onto the end of that sector, it used the next sector. So there was a lot of wasted space. So everybody was concerned with packing that data in as tightly as they could. Uh, boy, I'm glad we don't have to worry about that anymore, because that, uh… you really had to consider what you were doing. Sometimes you didn't get to store the data you wanted to store. If you had a 512-byte sector, and you had 513 bytes of data, you would use one full sector, and then you'd use the next sector just to store that one byte. See, you get where I'm coming from, right? Uh, storage was at a premium. So, decisions were made in the design of software.
Wolf:Uh-huh.
Jim McQuillan:to cram as much as you could into as small a space as you could. So two digit years were chosen. And that's what led to the Y2K bug. The year 2038 thing is a little bit different. That was a real hardware limitation. Well… They had chosen to store the timestamp in a 32-bit integer. They didn't have 64-bit integers at the time, and they didn't want to go with an unsigned integer, because in 1970, every number going forward was a date in the future. Every number that was negative was a date and time in the past. You know, some people said, well, you know, can't we just change the time t data structure to make it an unsigned integer? Well, then you lose all the dates before 1970. So… It was an issue. Umm. I don't know. It had caused a lot of trouble, and a lot of money was spent back on Y2K, and now I think we're going to start hearing about this Y year 2038 thing. Fortunately, like I said, 64-bit systems really don't have the problem. 32-bit systems that are recompiled with a 64-bit time field don't have the problem. But all of those embedded systems out there. There's things, you know, things you don't even think about. Traffic lights. Um, I don't know, you got some idea of, uh, embedded systems, Wol.
Wolf:And you? You want me to name some?
Jim McQuillan:Uh, yeah!
Wolf:Oh my god, so many, everything in your car, everything in, you said, traffic lights, everything in. Utility management like water.
Jim McQuillan:Yeah, that's the big thing. Those things that have been out there for a long, long time, monitoring water flow, oil pressure, all those things out in the field that nobody can even get to. Those systems could fail.
Wolf:Yeah, okay. And, um, I have a take on this. Uh, you said… Thankfully, we don't have to care about this anymore. I would say…
Jim McQuillan:Yeah.
Wolf:Um, we don't HAVE to care about this anymore. But we should. There are things about storage and locality and size and packing that remain important. But they are no longer urgent. Uh, but we still need to consider them, and because we don't HAVE.
Jim McQuillan:Yah.
Wolf:You have to. People want to skip that, and I think skipping it is wrong.
Jim McQuillan:That's where… that's where bloat comes from, right? That's… that's the cause of bloat. People not paying attention to how they're… how they're storing their data, whether it's on a storage medium or in RAM, uh, in the system. Um, yeah, it's, uh, it's interesting. Okay, here's another, uh, similar situation to the year problem, uh, whether it's the year 2038 problem or the Y2K problem. Did you ever hear something called the DJ10K problem?
Wolf:DJ 10K, I have not?
Jim McQuillan:DJ 10K.
Wolf:When is that?
Jim McQuillan:Dow Jones Industrial Average. Uh, back in the 90s, uh, the, the Dow Jones, uh, Industrial Average was in the thousands, maybe, uh, 6,000, 7,000. It's a, it's a scoring, or, uh, uh, uh, of, uh. It's…
Wolf:Sure, I see it all the time. It is constantly listed.
Jim McQuillan:It's an indicator, yeah, you hear it in the news all the time, it's. yeah, it's an indicator of how the market is doing, right? Well, back then, it was around 6,000, 7,000, uh, and people started to worry, what's gonna happen when it hits 10,000? A lot of software out there was written to only allow 4 digits for the Dow Jones Industrial Average. So, uh, trading systems?
Wolf:I can't wait for you to get to the part where I get to say, oh, nobody is ever going to need more than 640K of RAM.
Jim McQuillan:Yeah. Ha ha. Right, right. Well, nobody's ever gonna… nobody's ever gonna worry about the Dow Jones average going above 10,000. Nobody's ever gonna worry about a year hitting 2038, you know, back when it's 1970. That's 68 years in the future. Nobody's possibly going to be using this software in 2038, are they? You know, in the 60s, they were writing healthcare systems, and nobody was going to worry about them still using that software in the year 2000.
Wolf:Right. Yeah, this… This just points out to me a thing that I have said many, many times on many, many topics, which is there's two different kinds of decisions you will make as a software engineer. One is, you'll make a decision based on measurement. and everything you know scientifically. Is this going to need x? Um, is this the slow part? Um, is this a thing that would make more sense if I put an index on it? Blah blah blah blah. And the other decision is when you make a decision based on what you feel. Now, I don't want to discount feelings. But the fact is, the first one is better. Everything you feel is suspect. When you feel that no one is going to need more than $640K, when you feel that your software is not going to be used.
Jim McQuillan:Hmmm.
Wolf:for 12 more years when you feel that a two-digit year is enough. Umm. Maybe that decision is right, maybe it's wrong, but your reason for making that decision absolutely is wrong. Um, you should have measured. You should have figured out what the reality was. Um, and maybe you would have come to the same conclusion. But maybe you wouldn't.
Jim McQuillan:Yeah, uh…
Wolf:That's my feeling.
Jim McQuillan:Yeah, but if we go back to the 60s, you know, when people were… I mean, computing was in its… just its infancy. Right? And storage, like I said, was at a premium. I don't really mean to be defending what they did, but they kind of did what they had to do, right? Uh, and, and, you know. The software industry was so young, they didn't think it was at all possible that these programs they were writing would still be there.
Wolf:I hear you. I hear you. But there are things in this world that are hard.
Jim McQuillan:But, and here we are.
Wolf:But you still have to do them. For instance, um…
Jim McQuillan:Sure, sure.
Wolf:Two things that I value are integrity and courage.
Jim McQuillan:Mmhm.
Wolf:And it turns out that to do the thing that satisfies integrity. sometimes requires courage because you have to do a thing.
Jim McQuillan:Sure.
Wolf:That was harder, that you maybe had to defend to your bosses. But it was absolutely the right thing to do, even though it felt to them like it wasn't the right thing to do. Right now.
Jim McQuillan:Sure.
Wolf:Umm. And it was harder, and that's just the way it is.
Jim McQuillan:Yeah, well, you know, some… yeah, sometimes you're just trying to solve today's problem without thinking about, uh, the problem… what problem's gonna create in 20 years. It's… it's…
Wolf:I agree with that, but that's a junior programmer.
Jim McQuillan:It's… yeah. Sure, sure. And, you know, in the 60s, I don't think there was much other than junior programmers. You know, there weren't a lot of senior-level software developers.
Wolf:I agree with that.
Jim McQuillan:Right? Anyway, so we'll get back to this DJ 10K problem real quick. Imagine the Dow Jones Industrial Average is ticking upward, upward, upward.
Wolf:Yeah, I agree with that.
Jim McQuillan:And it gets to the 9,000s. Uh, 9,900. It's getting really close. Um… If the market is depending on that number to decide what to do. And all of a sudden, that number is more than 5 digits, and it can't be represented in a 4-digit field. They didn't know what was going to happen. The possibilities were the leading digit would get truncated. So now 10,102, let's say, is the number. If it gets truncated, it could become. 102, because that first digit got chopped off. Well, that's like the market crashing, right? If the industrial average goes from 9,999 to 100, major sell-offs could happen. That could be catastrophic.
Wolf:It is. Especially because of all this automated software that buys and sells on its own.
Jim McQuillan:Um… Yeah, and, and, uh, well, the number did cross that threshold on, um, March 29th, 1999. And fortunately, there were no real big problems, because the engineers had. Had gone through and did the analysis, and they updated software to make sure it wouldn't happen. But, uh, you know, with automated trading, I don't know how much automated trading they were doing in 99, uh, but now it's a lot of automated training. But imagine, uh, the automated training all of a sudden sees that the number dropped down to 102. and it starts this automatic sell-off, well, now there are triggers in place that will stop. It'll just freeze the market if that happens for… I don't know how long it has to happen for. I think it's in a matter of seconds. The markets will actually halt. And that's a problem! You know, now you have unstability in the market. Big problems. Fortunately, they were able to, uh… not experience any issues, because they did work on it, but a lot of money, I'm sure, was spent to deal with that, and a lot of… it created its own instability, just knowing that that was going to happen. I remember this in 99, this was happening, because everybody was focused on Y2K. And then it's like, oh God, now here's another thing. But fortunately, neither one of those turned into much of a problem. Up. So I did mention how this problem is being fixed in Unix systems 64 bit systems don't have the problem. Newer Linux kernels don't have the problem. As long as your program is recompiled with the larger time, the larger time T. Uh, data type, uh, what is it a structure? I, I, I, I don't really know how it's stored. Um, I, I think it's just a, just a, a, a, an integer that, like I said, it is the number of seconds, uh, file systems. are a problem. Fortunately, the ext4 file system, which is probably the most popular file system on Linux, it handles it. Although there's a choice you can make with the ext4, whether you want to go with large inodes or small inodes. And if you go with the small iNodes, you're stuck with 32-bit timestamps. So that's a problem. But I think the default is the 64-bit timestamps for that. So that's good. But then there's other file systems, BTRFS, XFS. I can't even think of the others right now, but they all handle it just fine. The older EXT3 file systems? No, it's a 32-bit timestamp. Uh, those will be problems.
Wolf:I usually go with ButterFS.
Jim McQuillan:Do you know?
Wolf:I do, um, because supposedly it's better.
Jim McQuillan:Yeah, I hear it's faster. I think it's only recently that you could install the operating system on a ButterFS. Was that you clicking?
Wolf:Accidentally, yes.
Jim McQuillan:Okay. We'll just… we'll tell the editor to leave that in. Um, uh, it's only recently that a ButterFS, uh, you can use a ButterFS file system as the root file system. Uh, I think, uh, Ubuntu 2024 added that? Um, before that, your only choice was, like, an EXT4.
Wolf:And for everybody listening who doesn't know what I mean when I say ButterFS, I mean BTRFS.
Jim McQuillan:Yeah, yeah. Uh, BTRFS is pretty interesting. I think it supports snapshots. XFS supports snapshots. There's all kinds of neat features with these file systems. Didn't we do an episode on file systems? I think we did.
Wolf:I think we did.
Jim McQuillan:It was… yeah, that was like a year ago. Um, anyway… Let's talk about databases for a minute. You know, I'm a big fan of Postgres. I use Postgres all the time, for everything. And we store timestamps in the Postgres timestamp TZ data type. Well, that's an 8-bit, uh, an 8-byte, 64-bit number. That will store a timestamp all the way out to the year 294,276. So… You know, it might have been shortsighted to think the software would still be running after 2038. I think it's, I think it's pretty likely you're going to be okay by the year 294,276. So that's how I store all my timestamps and I store my dates in a date field. So I don't have to worry about two-digit year, four-digit year, whatever. Um, that's Postgres. MySQL, um, they've… They've got a few issues. Their timestamp is a 32-bit timestamp. It's a signed number, so it suffers the same problem that, you know, the original Unix 32-bit timestamp has. MySQL does have a date/time type that stores the number as a 5-byte byte. packed binary, uh, and that'll take you through, all the way through the year 9999. So that's, that's pretty good. Um, I… I was really interested to learn this. SQLite doesn't have a date or a time. Type. uh, you make a choice. You can store your, your timestamps as a, uh, ISO 8601 string, which is, you've probably seen this before, the YYYY-MM-DD, uh, hours, minutes, seconds, and then, um, up to thousands of a second, milliseconds. Um, you store it as a string in SQLite. Um…
Wolf:I wonder what DuckDB uses.
Jim McQuillan:I don't know. I don't know. I'll bet there's an easy way. Uh, I did look, uh, uh, SQL Server, uh, they've got fancy types for that, uh, type. Does DuckDB use for timestamps?
Wolf:It's funny to me that you group SQL Server with databases.
Jim McQuillan:Well, it is a database. I mean, so is Oracle. They're just not databases that I use. I have used SQL Server. Um… Uh, DuckDB stores… they've got a timestamp and a timestamp TZ, uh, field that stores it as the number of microseconds since, uh, 1970-01-01, uh, the Unix epoch.
Wolf:And how big is it?
Jim McQuillan:Um, you know what? It's not telling me. Uh, let me ask you.
Wolf:I don't like it when it doesn't tell you.
Jim McQuillan:How many bits, uh… this is 64-bit. So that's good. So, uh… Uh, that'll… that'll get you out pretty far. Way, way farther than a 32-bit. So that's… that's pretty nice. Umm. So, yeah, uh, if you're gonna store a date in a system, uh, use a database, and use their date. field, if you can. If you've got a timestamp, store it in a timestamp field, but if you're not sure what the storage format is, check it out, just to make sure it'll support what you're trying to do. I gotta think that databases in this day and age, you're gonna be. pretty good. You know. Back when I was doing the software in when I first started in 1985. Um, I said before, we were using COBOL. COBOL didn't have any kind of… COBOL didn't have types. You had numbers and text, and that was it. Uh, you had no date type or timestamp type. Um, so that's, that's, uh, that's, you know, that's much better now, when we actually have, uh, programming languages that actually have date time, uh, uh, types, uh, and timestamp types. Um, but, uh, this, this whole conversation leads us to a whole bunch of overflow issues like this, because that's what it was, an overflow issue, right? So, I, I've got a list of them, I'm just gonna run down them, uh, pretty quickly. Uh, you know, uh, GPS. has an issue. I actually remember hearing about this, uh, this happened in 2019. Uh, GPS systems, you know, those satellites up there, uh, you can't go up there and change them very easily, uh, but they use a 10-bit. Weak counter. That's week, like, like, you know, this week, next week, counter. Um, uh… 10-bit, so it runs out roughly every 20 years. And when that happens, I think a reset has to happen, and your GPS devices may lock up, you might have to reboot them, and then they start working again. Well, it happened in 1999. It happened in 2019. It's going to happen again in 2038. I don't know if it coincides with… I think it's completely independent of the year 2038 problem. And it's going to happen in 2058. They are changing it to a 13-bit weak counter. But, of course, that involves software updates.
Wolf:Oh, good.
Jim McQuillan:Well, 3 more bits, that's a lot more, right? That's pretty good. Um, isn't that 3 orders of magnitude larger?
Wolf:It is.
Jim McQuillan:So. Yeah, so instead of, uh…
Wolf:Three orders of magnitude, where an order of magnitude is base two.
Jim McQuillan:is a power of two. Yeah, yeah. Anyway, it's much better. Um, did you know NTP, the Network Time Protocol, uh, has a rollover issue? Uh, they use… they do! They use a 32-bit unsigned field. Uh, in this case, it's unsigned. Uh, it holds the number of seconds since 1900.
Wolf:I did not.
Jim McQuillan:Uh, it rolls over roughly every 136 years. So guess what? Not only do we have the 2038 problem, there's also gonna be a 2036 problem. And all of our systems, all of my systems use NTP to keep the clock set correctly. Uh, your systems probably do it too. If you don't even set it up, I'm sure it's there, right? And all these network devices, uh, routers and stuff, they all use NTP to keep the time going. Well, February 7th, 2036, look out, because they're all gonna roll over, and who knows what's gonna happen. I mentioned many times, and I even mentioned it earlier, I'm a big Postgres fan. Well, Postgres has an overflow issue. They've had it for years and years and years. Fortunately, they deal with it pretty well now. But they have a transaction ID, it's called the XID. And every time you start a transaction, this counter gets incremented. Uh, and any records that you write get stamped with that counter. as part of that transaction, so that they're multi-version… what is MVC-C? Multi-version Currency Control. Um… Uh, make sure that you get the right snapshots.
Wolf:This is weird to me. Like, why do transaction IDs, which don't get stored en masse, uh, have to be integers? Why can't they be orderable, um… U U I D's.
Jim McQuillan:Well… They can. That's just not what… in the 90s, that's not what they chose. That's, that's the problem. So, and there's a lot of Postgres databases out there. Uh, Postgres actually, uh, the way they solve the problem is, uh, with Postgres, if you, um, well, first of all. if you run out of these transaction IDs, these XIDs, it's a number that's, you know, 2 to the 31st minus 1, is that how it works? So it's 4.3 billion, roughly. When you hit the end, it wraps around. And any data that you have that's stamped with that…
Wolf:It's very surprising to me that a transaction ID would be signed.
Jim McQuillan:Yeah, thank you. It's unsigned. Did I say signed? Yeah, it's unsigned.
Wolf:Okay.
Jim McQuillan:Um, 4 billion. If it were signed, it'd be 2.1 billion. Uh, anyway, it's, uh, uh… What happens is, when it wraps around, any rows that had, uh… some value… it's weird, I tried to understand it. The, uh… the transaction ID, it gets split into… it's this rolling thing of 2 billion, so everything that's… greater than 2 billion is in one half, and everything that's less than 2 billion is in the other half. It's complex. Anyway, those rows, they're sitting there on disk, but your select won't find them. So that's a real problem. So Postgres detects that, and when it sees a transaction ID rollover, uh, or wraparound, uh, it'll halt the system. for fear of losing data. It will just halt the system.
Wolf:Mmhm.
Jim McQuillan:So, they solved this in Postgres 8, which was… 2005, so this problem was really fixed 20 years ago. They solve it by running vacuum. In fact, the problem has always been solvable by running vacuum. In Postgres, you have this. verb you run, it's called vacuum, that will sweep through the tables and clean things up. And it will clear out the transaction IDs for really old records that are absolutely no longer part of any transaction. Those transactions had committed long ago. So it clears the transaction IDs.
Wolf:And when I'm adding millions of records to specific tables, I absolutely do a vacuum on that table.
Jim McQuillan:Yes.
Wolf:At the end.
Jim McQuillan:You absolutely want to do that. That's for a different reason. The reason why you do that vacuum is to update your statistics, so that your query planner can do the right thing.
Wolf:Yes. Right.
Jim McQuillan:But, uh, and you're also… you're doing one transaction, I hope, when you do that, or many transactions, but not a transaction for every row? Cause that would be, that would be not optimal.
Wolf:Correct. Generally, if I'm doing millions or billions of changes to a single table, I am using multiprocessing, um, but I will break it up into.
Jim McQuillan:Umm. Yep.
Wolf:you know, 160, uh, transactions, where a process handles each one.
Jim McQuillan:groups. Sure. Sure. Yeah, yeah, you would do that, but you wouldn't do each one without… you wouldn't do it at all without transactions, I believe. But either way, um… uh… what Postgres added in 2005 was autovacuum, because… because sysadmins would forget to run vacuum, and then they'd run into this wraparound issue. So they added autovacuum, and it's this daemon that runs. when the system is not terribly busy, and it'll vacuum the tables for you. And that pretty well takes care of the problem. They are working on increasing the size of that transaction ID for some future version. I think since they added auto-vacuum. 20 years ago, it isn't as much of a big deal. Okay. Here's one that I think is really interesting and maybe a little scary. And I heard about this several months ago. The Boeing 787, that big airplane that they fly. Uh, it has a 51-day bug.
Wolf:Uh-huh.
Jim McQuillan:If the generator runs continuously for 51 days without a power cycle, an internal counter overflows, and all four generator control units go into a failsafe simultaneously.
Wolf:Oh, I'm never gonna fly again.
Jim McQuillan:I mean, it's easy, you just reboot the generators, or you cycle the power on the generator, but do you want to be doing that at…
Wolf:Uh-huh. Sounds like my infotainment system in the car.
Jim McQuillan:Do you want to be doing that at 36,000 feet? I don't know. That's an interesting one. NASA's deep space probe, sorry, deep impact space probe that they sent up, I don't even know where that thing was going.
Wolf:I do not.
Jim McQuillan:probe the, uh, the far reaches of the universe, I think. Um, it has a 32-bit, uh, counter that counts tenths of a second. Uh, it overflowed, and NASA lost contact with the probe entirely. The probe is no longer reachable because of this counter.
Wolf:That was totally predictable. They knew how long the trip was. Absolutely they knew.
Jim McQuillan:Yeah. Yeah. Yeah, yeah. I don't know if, uh, you know, you know, NASA designs these missions, and they put a lifetime on these missions. You know, like the Mars rovers, you know, they might say it's gonna be there for 18 months, and then everybody's surprised, because it's still going 10 years later. And this, uh, deep impact probe, it might have been. Way beyond its lifetime. When it, when it ran into that, uh, when it overflowed the counter, but, uh, it's kind of embarrassing when that's the problem, right?
Wolf:Yeah, I can only think of one reason. And that reason, I mean, you know, technically junior programmers, but that reason is the person who wrote the code didn't think like this is going to last longer than X what they thought was.
Jim McQuillan:Yep.
Wolf:I'm not going to be here in X.
Jim McQuillan:Yes. I'm sure. I'm sure that happens. It's not going to be my problem.
Wolf:Right.
Jim McQuillan:Um… you know, in UNIX and Linux, we've got PIDs, the process ID numbers, and we have file descriptors, and we even have port numbers, and they're all in fairly small fields, like a PID. I think Linux.
Wolf:Uh-huh.
Jim McQuillan:Uh, now, on a 64-bit system, it's not a big deal, but on a 32-bit system, the PIDs, I think, were limited to, like, up to, uh, 65535. That's a 16-bit number, right? I think a process ID could be no bigger.
Wolf:Right and.
Jim McQuillan:Then 65,535.
Wolf:Like a reasonable thing about that size is PIDs reset when you reboot.
Jim McQuillan:Yeah, they wrap, and when a process dies, it can become available, you know, for the next one. So, I mean, it's a limit, but is it a big one? I don't know. Port numbers, you know, port numbers were, um… I think the TCPIP spec. Uh, it says that the port number is a 16-bit number. Um… And when you have a system that can take millions of connections, well, guess what? You've got a problem. You know, and these systems these days, they can take… an awful lot of simultaneous connections. Um, and… but the port number is limited, so, uh… it can be a problem. Now, normally, incoming connections, they're all gonna connect to, like, port 443, or port 80, or, you know, whatever the port happens to be, so it's not a problem. It's… it…
Wolf:22 would be the one for me.
Jim McQuillan:22 is the one for you. Yeah, but if you've got a system with millions of connections coming in on port 22, that's an interesting one. 22 is SSH.
Wolf:Yes.
Jim McQuillan:Anybody doesn't know. Uh, so that's an issue, right? Um, here's one that you've probably never heard of, the Snowflake ID Overflow.
Wolf:Oh, I absolutely never heard of that one.
Jim McQuillan:And you might be thinking, what the heck is that? Well, it was created by Twitter. It's the ID number that they use for posts and all kinds of things. It's a 64-bit number. Um, and… It. it can wrap around. I don't know that Twitter has had this problem, but Discord followed Twitter, and they used this 64-bit snowflake ID, and they've had serious concerns about it. But here's where it gets interesting. 64-bit number's big. You're never gonna run out of whatever you can store in 64 bits. Well, I say never, but I shouldn't say that. Here's the problem. A lot of this software now is being implemented in JavaScript. Did you know that a JavaScript 60… an integer in JavaScript is not 64 bits? it's 53 bits, or I guess 54 bits. It can hold up to. 2 to the 53rd minus 1. It's about 16 digits.
Wolf:I'm a JavaScript expert, and I, you know, I worked on the original JavaScript interpreter at Netscape, and I did not know that.
Jim McQuillan:Yes. Yeah. I know. You didn't know this. You didn't know that. I didn't know that. Um, it just really, really surprised me to find out that the largest integer that can hold is roughly 16 digits long. Um, so software that's expecting a 64-bit number, and really it's getting a 54-bit number, that could be a real problem. And that's, uh, what Discord is, is battling right now, because they're using the Snowflake ID, uh, which is supposed to be 64 bits, but JavaScript. It stops at 54 bits. So, as long as you're dealing with it, you know, JavaScript is kind of funny, because it's sort of typeless in a lot of ways, and you can store numbers as strings, and as long as you're storing this 64-bit number as a character string, you're okay. But if you try to store it as a real integer, and maybe do some arithmetic on it. Even something as simple as adding one to it, that could be a problem. Um, another one, we talked about this, uh, a year ago, um, IPv4 exhaustion. Uh, IPv4 address is, again, a 32-bit address, and it's not even like we really get, um, uh… what can you hold in a 32-bit number? Well… 4 billion, right? It's not like we get 4 billion addresses, because the way they've segmented that address space up, we get quite a bit less than 4 billion addresses. Uh, and as we talked about in, uh, episode 11, the, the solution to that is, uh, IPv6. There's some other solutions. They, they, they broke address space up into CIDR blocks, and they got private addresses and stuff, and that's how we got this far, you know, because everybody's still using IPv4. But, um, there's an awful lot of IPv6 out there. In fact, Google, I think I mentioned this a month or two ago, Google announced that more than half of their traffic is IPv6. Um, okay, finally, I've got one. Um, the AACS DRM specification using TLS certificates. Um, it's a 32-bit hard-coded expiration field. Uh, that at some point, and this is not every device, but some, at some point, it'll start rejecting valid discs when it hits that expiration date. So you may have a Blu-ray player, and you might have a Blu-ray disc, and it might be working just fine today. But tomorrow, it might just decide to stop because it decides that you're not licensed to play that video on that DVD. Basically, your smart device becomes e-waste on a schedule that nobody planned. Uh, so I… I do have a couple of more that aren't exactly overflow issues, these are date bug issues, and… and these I've heard about before as well. Uh, you know, the Excel and Lotus spreadsheet, they've got a date issue? Did you know? Uh, they both think… I think Lotus started this. They think that the year 1900 was a leap year. And it wasn't. Do you know the rules for leap year?
Wolf:I do, um, uh, and when you come upon a date that is, uh, 00, that is NOT. a, um… Debate.
Jim McQuillan:Well, here's the rules, in case anybody's wondering. Uh, if the year can be divided by 4, evenly divided by 4, then it's a leap year. Unless the year is evenly divisible by 100. So, like you said, a year that can divide by 00, or that ends in 00, is not a leap year. So, that's true. So, like, the year 1900 was not a leap year. The year 2100. Uh, will not be a leap year. Uh, that rule has another UNLESS if the year is divisible by 400. So, for example, the year 2000. That was a leap year. Right? So, you said a year ending in 00 is not? Well, the year 2000… the year 2400 will not… will be a leap year. So this leap year calculations are kind of done like this, so that the calendar stays aligned with the real solar orbit of the Earth. Uh, because, you know, we have, like, leap seconds and all that kind of stuff. It has to do that. Otherwise, we'll be celebrating Christmas in July, which, I don't know, maybe not be so bad. But, um, the Sony PlayStation… Um, uh, it, it treated in, in, on February 3rd, 2010, uh, it treated 2010. As a leap year. Even though it wasn't. Right? Because 2012 would be a leap year, 2008 would be a leap year. 2010 was not a leap year, but the Sony PlayStation treated 2010 as a leap year and bricked itself, trying to process a non-existent February 29th. I think that's, uh, kind of funny. Uh, the Zune, you remember the Zune?
Wolf:I do.
Jim McQuillan:The, uh, the, uh, Microsoft… that was Microsoft's answer to the iPod, wasn't it? a little music player that, uh… it was, like, twice the size of an iPod and nowhere near as cool. Um, it had a leap year calculation bug. The internal clock driver froze every, uh, on every 30 gigabyte Zune. worldwide on December 31st, 2008, and it stayed frozen until the clock ticked past midnight. So, it's New Year's Eve, you're celebrating with your friends, you got this music playing, you got Prince playing, uh, 1999, everybody's having a great old time, and the music player stops. It was the Zune, uh, uh, 2010 bug. Anyway, uh, that's, that, that's my list of, of, uh, bugs that. Should have been avoided. Um, I, I hoped you learned something. I, I think, I, I think we actually taught Wolf something today, so that's always a good day.
Wolf:Uh, yeah, some cases I had never heard about, but, um…
Jim McQuillan:Yeah.
Wolf:A lot of these. Choices that people made were. What do I have to do that I don't have to fix it? Um… And that's not right. That's what I feel.
Jim McQuillan:Yah. Yeah, it's, I mean, you know. Life is a series of compromises. But there are some compromises you should really be careful about, right?
Wolf:Yeah, I have feelings that. I want to write software such that that software is fixable. Um, by every programmer going forward. Not just me. Um… Yah.
Jim McQuillan:Well, see, you raise a… you raise a really interesting point, and this is one of the things I admire about you, the way… the way you said that. You want to write software that is fixable. You're implying that you know that your software may have bugs. But you want to make sure, above anything else, you want to make sure that that software is maintainable by other people.
Wolf:I do.
Jim McQuillan:That's admirable. I like that.
Wolf:Yeah, that is part of integrity, transparency, honesty. I absolutely don't want to do a job.
Jim McQuillan:Uh, it's part of. Yes. Yes.
Wolf:where I have written the code in a way that. It's a lever only I can pull.
Jim McQuillan:Oh, yeah. Yeah, under the name of, uh, job protection.
Wolf:uh… Yeah, that is… Yeah, that is a form of job protection, but it's totally counterproductive because it means your job going forward.
Jim McQuillan:Yah. Sure.
Wolf:is you're gonna pull that lever over and over and over again.
Jim McQuillan:Right.
Wolf:I don't want to pull that lever. I want to write the software anybody can own.
Jim McQuillan:Yes. That's good, that's good. Alright, why don't you take us out.
Wolf:Well, first of all, thank you so much, Jim. And as always, total pleasure podcasting with you. For all the listeners, the show notes contain Mastodon addresses for Jim. for me, and for the show. We absolutely love feedback. You can send feedback to the email address, feedback@reddit.com. runtimearguments.fm if you need to get to the website. It is not SSL. It is, um, HTTP colon slash slash. runtimearguments.fm. Um, absolutely. Tell us what you thought of this episode. Tell us what you thought of ANY episode that you have listened to and have thoughts on. Um, especially the one we just did, uh, earlier about leaning your ladder, because I really want to hear thoughts about that. Um, our show comes with a transcript. Which you can see in the show notes, or maybe by going to the website for this episode. Um, we want to hear what topics you care about. Are there things you'd like us to talk about? Did we go too far on this episode? Did we not go far enough? Um. Do you like How Was Your Week? Did we spend too much time on it? Not enough! Um, it has absolutely… Been a pleasure for both of us. I think I speak for Jim here as well.
Jim McQuillan:Yes.
Wolf:Um, talking to you all about things we think are important. Um, I just hope. that, uh, at least occasionally, we find the topic you think is important. Um, and uh, until then, uh, my feeling is, thanks so much for listening and getting to the end. I can't wait to talk to you next time. Jim, what about you?
Jim McQuillan:Uh, thanks again, uh, Wolf, for podcasting with me. It's always a pleasure, and thanks to the listeners for listening, and uh, hey, have a great couple of weeks. We'll see you next time.
Wolf:Awesome. All right. See you later, everybody.
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
CoRecursive: Coding Stories
Adam Gordon Bell - Software Developer
Two's Complement
Ben Rady and Matt GodboltAccidental Tech Podcast
Marco Arment, Casey Liss, John Siracusa
Python Bytes
Michael Kennedy and Calvin Hendryx-Parker