Waves Interview: Sailing Through Cyberspace with Dave Taht's Guide to Bufferbloat

Have you ever wondered what magic lies behind the seamless flow of your online conversations, or how a tune could tell the tale of tech evolution? Networking virtuoso Dave Taht joins us to unravel the mysteries of internet technology, starting with his pioneering work on the embedded Linux wireless router. Dave, with his knack for making complex topics relatable, takes us back to the early days of the internet, from Fido to Gopher, and breathes new life into Usenet. He tackles Bufferbloat with an accessible funnel analogy, making sense of how data should move to keep our digital interactions smooth. And for an unexpected twist, Dave's fusion of music and technology adds a rhythmic pulse to our discussion, capturing the essence of internet challenges in a way that resonates with both tech veterans and novices.

In a captivating conversation about the digital world's legacy, we delve into the aftermath of 2008's outdated embedded devices, and the scarcity of expertise in bringing these relics up to speed. Dave sheds light on the unintentional impact BitTorrent had on stirring the network neutrality debate by saturating buffers, which inadvertently highlighted the critical need for smarter network algorithms. We then pivot to the power of firmware revamps via OpenWRT, discussing the transformative potential of updates on bufferbloat, security, and IPv6 integration, despite corporate hesitations. As we conclude, Dave serenades us with a song celebrating SpaceX's achievements, a fitting end that blurs the lines between the once far-fetched dreams of space travel and the unfolding reality of our technological prowess. Join us for an episode where inspiring melodies meet insightful tech talk, all wrapped up in a galactic bow.

Transcript

Speaker 1: 0:30

Okay, I'm going to start recording here in like 3, 2, 1, somewhere around there. Hey, what's up everybody? It's Drew Lentz, the Wireless Nerd, and I have a special guest today. So on this week's episode we're doing interviews. As you know, I've got some fun interviews lined up and Mr Dave Todd, a guy that I got the opportunity to meet in Cancun at the Wi-Fi Now event. Thanks, klaus. I had a good conversation with him there, talking about buffer bloat and all these wonderful things, but what makes him unique is you always find him walking around with a guitar. So, without further ado, I would like mr dave to introduce himself. Tell us a little bit about who you are, what you do and why you have a guitar, and feel free to sing it, brother live from fort myers.

Speaker 2: 1:09

It's dave totten of bufferblownet. It's great to be your show.

Speaker 1: 1:13

Drew thanks for having me yeah, man, absolutely what. So? So who are you? Who are you and why should people listen to you?

Speaker 2: 1:22

good question. Uh go googling for who invented, who invented, the embedded linux wireless router. There you go. Uh go looking for the word buffer bloat. Go looking for quite a few other things I'm in Wikipedia.

Speaker 2: 1:36

The thing was is that I don't even know where to start. I mean, I go back to the 80s. I got on the Internet with things like Fido and stuff, and so I've been around for a while. I'm getting close to retirement, I think, and I'm trying desperately, so desperately, to educate the public as to how the Internet really works, or how it should work. I have someone else come by to my uh, to my place in the asteroids, and fix my hover chair and explain to me what's going on in the world. Um, so that's been my mission now for 14 years is to explain how the internet really works. I've used jugglers, I've used funnels, I've used tubes it's been been a long thing and these days I use guitars.

Speaker 1: 2:16

Well, well, that definitely explains the guitars you mentioned, fido. Golly, I mean just immediately. I remember getting on one of the first, one of the first uh, vax clusters set up, connected to the internet and say what's this gopher thing, you know, and using gopher, fido, all those, all those wonderful, oh, I mean, who uses news anymore? That's what I really want to know Is there still a market for news Newsnet is still alive.

Speaker 2: 2:42

Man, it lives. I love it and it's a great place for all of this old farts. In fact, email is still a good place for the old farts. If you want to find the elders of the internet, you'll find us on email.

Speaker 1: 2:53

Altbinarieswirelessnerd Only in find the elders of the internet you'll find us on email altbinarieswireless nerd.

Speaker 2: 2:57

Only in my dreams, only your dreams, yeah. But, for example, there's a mailing list I hang out on called internet history, and if you want to talk to people like vince surf and other people in the internet, join that. Uh, and us old guys want to pass along the knowledge to the new.

Speaker 1: 3:10

I remember the one chance I got to meet vince surf. I remember the first time he responded to an email of mine on some list serve and I was like, oh, I'm like you know this is awesome. I'm talking to the internet. I love it.

Speaker 1: 3:24

Just Vint, oh yeah, oh yeah, man, you know. Speaking of what's interesting, I saw nomatics. You know, just this week or a couple days ago, nomadics announced that there was an acquisition that took place and I thought that was interesting. Asa Abloy purchased Nomadics, and Nomadics has one of the fathers of the internet working there too, so that's kind of interesting. Okay, so tell me about. I mean, people have seen you comment, I'm sure, on thread after thread online, and they've heard you talk about Buffer Bloat or they've heard you mention the name. Can you give me, either in song or in regular format, what is Buffer Bloat first? And then I'm going to ask you why it's important second. So take it away.

Speaker 2: 4:09

So I've done a number of demos on the Internet Again. I've done one with Chuck Wurst, I've done one with uh funnels and I like to use the analogy of the funnel. The thing is I didn't really set up a funnel for a test today, but basically, when you try to get, if you try to pour something, here you go. I'm gonna do this here.

Speaker 2: 4:27

There we go my monsters right, and if you try to pour a liquid into it, right, if you pour it, it's really hard to get in there.

Speaker 2: 4:35

But if you put a funnel in it, yeah, okay, um, it's a lot easier to get into it. Okay, buffer blow, really really overly sized funnel, like gargantuan. And if you pour data into it, it takes forever to get into where you're going and you can overflow the can, um, before you any signaling back. So we, the founders of the Internet, goofed in figuring out how to size and scale that funnel properly for the range of bandwidths that we have today, notably in Wi-Fi and wireless technologies, you have a difference in speed between oh I don't know a megabit and gigabits. Now you need to have really the right size funnel for each of those to make sure you can pass the data through while still getting interactive traffic through fairly well. I mean, we're doing video conferencing here and recording and you and I can interact. But if there was a really big funnel between the two of us, it might be. You go, oh, I do faces, you know, yeah, yeah, yeah.

Speaker 1: 5:35

So let me ask you, is it just the size of the neck of the funnel, like the part where it comes down? Is it that neck, or is it the size of the actual funnel, the bowl? Because it seems like as the internet speeds grow, the funnel might get a little larger, but it only gets larger as the bigger bowl gets larger, and thereby it doesn't solve the problem by having a bigger funnel, because the neck never grows, it never turns into a column. It's always this choke point here that goes out. Is that a fair assessment or am I?

Speaker 2: 6:08

wrong. That's a really great insight. You've been hanging out with me too long. The second part the neck of the funnel has to be right-sized for the size of the hole. You have too small a neck, then you can't fill this thing up as fast. So, yeah, pouring in too much data and getting enough data out, yeah, very insightful. And so they just think that.

Speaker 1: 6:29

So the internet just thinks of the funnel. You know, chip manufacturers, whatever just think well, if we just make a bigger funnel, we won't have the problem. But it's not the funnel, that's the issue. It's that choke point overall.

Speaker 2: 6:40

That's exactly where we went wrong. Okay, 2008, 2006. Okay, the Linux community and our communities in general made three big mistakes all at the same time. We oversized all of our buffers to be the right size for a gigabit, okay, okay.

Speaker 2: 6:59

And you need to have a bigger funnel for that yeah and then we shipped all these products to be right size for a gigabit and your typical internet connection is 10 megabits up, so that's off by a factor of 100. Wow, and that is why your systems misbehave when you try to do a big upload, because you have way too big a funnel for the actual link. And it was a mistake, man, we didn't know. It's been replicated billions of times and you'll see it absolutely everywhere, in almost all the gear made after 2008,. That just was the wrong size. That just was the wrong size. So my mission, with the help of all of you volunteers out there in the OpenWRT community and the Linux community and VST community, was to right-size the funnels and somehow stop billions of new devices that are being shipped with wrong-size funnels and then also retrofit all the other devices that are in the field. So the question is?

Speaker 1: 7:56

is it an issue that can be solved with software, or is the funnel hardware that any new hardware that comes out has to have the ability to change the size of the neck of the funnel? You know what I mean it's like. How do you fix the problem?

Speaker 2: 8:16

Well, so starters, we definitely oversized the buffers in the first place by a factor of at least 5 to 10. That's a big mistake, and we can just change one number in all the routers of the world and get a massively better result. And then you have all the people developing fancy schmancy algorithms like my project has, that will actually dynamically size the buffer to be roughly the right thing, no matter the speed you're running at. It's a Wi-Fi issue mostly, and we can also apply another technique called fair queuing, which is really probably one of the most important techniques that didn't deploy in 2008, except on small, select pieces of hardware.

Speaker 1: 8:51

And that's the? Yeah, it's fair queuing. That's like at the airport where you have where you've airport, where you've got five lines to get through. Security and fair queuing is making sure all five lines are open at the same time so everyone can move through, instead of just having everyone blocked up on three lines and TSA agents talking on the phone. Is that the fair queuing you're talking about, or is it something different? That's?

Speaker 2: 9:12

pretty good, except that with computers you can have a lot more lines. The FQ model algorithm has 1,024 lines. So if you have 1,024 people, everybody gets service. It's called Q-theory Everyone gets service roughly equally. The thing is I try to explain that large bandwidth applications.

Speaker 2: 9:32

So let's say we're doing voice Okay. So these are Nicorets. I used to use cigarettes for my demos, so these are Nicorets. I used to use cigarettes for my demos, now I use Nicorets. Voice is this one 80-bit little packet every 20 milliseconds? Okay, okay. And Netflix is pouring out, Okay. So if you have two different queues you have a queue for Netflix and you have a queue for voice your voice will never see the Netflix at all and now I have to pour all these damn Nicorette's back from her Bad demo. So we try very hard to provide equal service to all the different flows from everywhere, and that qualifies for most stuff VoIP, the DNS, interactive audio, all that kind of things. It's very small, it's the really fat packets like a big download or a big upload or netflix. I want to pick on netflix any video.

Speaker 1: 10:20

No, no, no streaming video. I I think I think everyone who's listening this is smart enough to figure that one out.

Speaker 2: 10:25

Yeah, so wi-fi has its own problems.

Speaker 1: 10:29

Go ahead so the so the question is it sounds like if you, if you know what the problem is and you know how to fix the problem and it's not difficult to do, then large-scale adoption shouldn't be difficult. But I think you've found that it is difficult. Well, and then that begs the question if we know this is an issue and it's affecting everything, why hasn't anybody done anything about it?

Speaker 2: 10:54

Well, there's about six things. For starters, the concept of bandwidth and latency in the public mind, in most minds, is completely conflated, okay, um. So one of the nice things that uh buffer blow did is it enabled uh isps, uh to go and sell more and more bandwidth and keeping this buffer size the same. So they reduced, they increased the bandwidth and reduced the latency and they sold the customer higher end thing. Okay, um, another one that happens. Could you repeat the question for me, drew?

Speaker 1: 11:23

I'm sorry what if it's so, if it's, if it's such an easy thing to fix, if we know? That's going to affect everything. It's such an easy thing to fix. Why haven't more people fixed it? Why does it? Take open wrt to go, guys, let me show you what's going on. Let me show you.

Speaker 2: 11:37

What's so much better? That's something I've been trying to fix for a very, very long time. Again, most of this work on embedded devices like these was largely competed by 2008, and there was a recession then and all the companies that were making products shipped their code and then went out of business and laid off everybody. So the people that had the skills to put in upgrades kind of vanished to the four seasons.

Speaker 1: 12:05

And.

Speaker 2: 12:07

Then we just started. The internet exploded and we just started copy-pasting.

Speaker 1: 12:12

Yeah, because it was working right. They're, like the funnel's, big enough and we're not seeing any problems. Let's just keep on keeping on. Well, we were seeing problems.

Speaker 2: 12:22

We just misidentified them. Some of this goes back to BitTorrent. There was a big brouhaha over network neutrality, with BitTorrent versus VoIP. Bittorrent was not an evil application, but it ate all the buffers and made VoIP badly, and the thing became a legal issue rather than a technical one. Oops, the buffers were too big, and then the whole concept of network neutrality came out of an application that saturated the buffers on devices designed and built in 2008. What we found out? We figured out a couple of really advanced algorithms. One of them is called FQ Cuddle RFC 8290, I think.

Speaker 1: 13:02

On your license plate behind you, over your left shoulder. There you go, gotcha.

Speaker 2: 13:06

Another one called Pi that can dynamically size the buffers and do fair queuing really well. Another one called Cake, which is what I work on these days, and with those in play and with new software this is a box here, it's a, a microtech H, a, p, a, c, light, okay, so you can just put new software on millions upon millions of devices, open WRT, dd, word, yeah. And the buffer blow problem is not only fixed, but you've also fixed the security problems and you've also added support for IPv6. Those are the three things that have been really lagging. So I've been proposing for years that we do a rip and replace of all the firmware in the world to something like that and we would be able to upgrade the entire edge of the internet to better software that works better across the board. And, uh, that campaign is not going so well. It really does require a guy with a guy or a gal with a degree and I don't know fiddling.

Speaker 1: 14:04

So to do so, then you would think also that if maybe not, maybe not everywhere, maybe not the d-links in the netgears of the world, maybe it's not something that they want to do, because they're just stamping out device after device, but what about the larger guys? What about Cisco? What about Juniper? What about them? Are they looking at this going? Hey, here's a better way to increase efficiency in our equipment and make it work better.

Speaker 2: 14:26

What's the story?

Speaker 1: 14:27

there.

Speaker 2: 14:29

Increasing efficiency means you sell less gear.

Speaker 1: 14:35

Okay, all right there.

Speaker 2: 14:36

increasing efficiency means you sell less gear. Okay, all right, I see where you're going there. Well, I guess. Example juniper, in particular. Um sells big buffers. Uh, they're not needed and they cost more, but they sell them as an advantage and the first thing I do when I get a piece of juniper gear is go tune the buffers down to something reasonable.

Speaker 1: 14:49

Um, and and they work better, interesting. So oh, that was my next. That was my next question. If these, if these groups haven't done that, if they haven't gone out and fixed the problem, is there something I can do on my home network, on my enterprise network, to change this? When is there a simple solution? Is it just turning buffers down?

Speaker 2: 15:06

oh, so we've been at this for 14 years now. I mean, I I would prefer everybody ran my algorithms, but you can just start by tuning the buffer sizes to what's the square root of bandwidth delay product for what you're doing. That's a complicated thing, but it's simple. You can go out and, I said, buy newer devices that do the right thing. There's a lot of devices. The Euro 5 in particular was spectacular in this respect and so was. So was even around a half dozen other companies actually adopted newer code, in particular, microtic adopted newer code and boom. All their devices from past to present all do the right things.

Speaker 2: 15:45

Um, the junipers and the ciscos of the world are also in more in the big iron stuff where the buffer blow problem really doesn't show up anywhere near as much, it is mostly along the edge and on wireless devices like 5G and Wi-Fi. So, normal persons, there's a wonderful piece by hold on Avery Penron. You should have him on your show. He's great. He's now the CEO of Tailscale and what we did when we were working at g fiber together is show that once you've got a gigabit network, all the buffer bloat moves to the wi-fi, and the wi-fi was really performing horribly and it still does in a lot of cases.

Speaker 1: 16:24

okay, so yeah, yeah, yeah, so you're, I mean, you're leading me right into it and, as anyone who's watching this can see, my remaining meeting time has popped up because I have not upgraded to Zoom Pro, so we've got about nine minutes left but it's all right, I would love to talk about some art and music and LibreOS and whatever.

Speaker 1: 16:43

Absolutely. Before I get into that, the Wi-Fi question. Right, Because Wi-Fi 7, MLO you know, we were at WLPC a couple of weeks ago and we enabled MLO and some equipment. We turned it up, we got connected at 2.8 gigabit per second between the device and the access point and there's all this awesome 8 gig, all these great things that we're going to do with Wi-Fi. Except what you got, except what happens to the buffers on Wi-Fi with buffer blow. How does this, how does this, how's this going to affect wi-fi 7 and and further all?

Speaker 2: 17:43

right, we're stepping sideways. Um, again, most people don't seem to understand that 2.5 gigs right next to the access point does not happen on your front porch. And if your buffers are not being dynamically sized to be the right size for your front porch, you might have a good download experience, but you'll have a lousy video conferencing one. So there's a paper that we wrote with Linux kernel drivers called Ending the Anomaly. It's the masterwork of my career. I wish more people understood that. One I want to touch MLO.

Speaker 2: 18:12

Mlo is a form of fair queuing. You get three queues and that's great. But if you can get more que more cues, you can handle a lot more traffic. Also, wi-fi has a problem in general with more than one station on the list and it is the worst station on a wi-fi link. That will drag the whole thing down. So, uh, people keep. Oh, I upgraded my access point. Well, I should maybe take that tv in the corner with a single antenna and run a wire to it. You know you can't just fix the access points, you have to fix the devices. So I like the idea of mlo, except that it's peas, all you know, I guess. Can I say that, michelle?

Speaker 2: 18:49

it pees all over the spectrum, drew and uh, we have a limited amount of spectrum available and we should be trying much harder to uh get smaller channels with less bandwidth but more range and run them without interference. Instead, we have people over-focused on getting the maximum bandwidth, standing right next to the access point and you want to have good bandwidth to your front porch.

Speaker 1: 19:14

I love it you just gave me the sound clip of the interview right there. Man, that's fantastic, okay, so awesome. Now moving on, because I did ask you about this earlier and I am curious, genuinely curious the art behind you. Tell me a little bit about that art that you have in the background over there.

Speaker 2: 19:29

My mom was a programmer for IBM in the 50s and 60s Wow and she then moved into making art after she got tired of raising us, so she mostly does impressionist art. She's 86 years old Wow and she's a little political sometimes. I love this painting. Behind me it's Uncle Sam, surrounded by dollars, being supplied by both the red and the blue sides, and pointing at you and crying. And she did this one in 2016. Um, she's planning on redoing it for 2020 and 2024 and we're going to put that one out on the cards. So, uh, anyway, she's, um, one of my favorite people in the world.

Speaker 1: 20:12

That's awesome did she have any influence on you, not just with the art, but did she have any influence on you in music? How did what's the story with you in music and how did that get?

Speaker 2: 20:21

started. It is kind of hilarious. Um, she loves gershwin, I love gershwin but can't play it. Um, I love rock and roll and yeah, we like we have like almost no music in common whatsoever. So she plays piano, I play piano, guitar, bass and drums and they sing, and so we've been together, hanging out together for the last few months and, after 40 years of trying, we have like three songs in common Nice, and I'm hoping to put together a great show at some point, and we'll do. I don't know things like this Land is your Land, if you've heard that. And there's a wonderful Gershwin tune that she does. There's one that D dazzled and bewildered, that we do. I play that on bass. It's really wonderful to, uh, be collaborating with her after so many years yeah, that's awesome, man, that's cool.

Speaker 1: 21:12

My mom, my mom, I a little bit of music in my family, nothing too crazy. My mom played viola in the in the local orchestra, and I grew up with, I grew up with music. Everywhere I've been, you know I've, there's always music playing somewhere at some point and you know you walk around, you can, you can definitely find some music. Well, that's awesome. Well, look, I know we're we're gonna get a little short on time here. Tell me about real quick, tell me about liber qos, because you know, uh, if it, if, it wasn't for mr frontisek, mr you know.

Speaker 1: 21:40

if it wasn't for Mr Frontisek, mr you know, I'm curious. What's going on there? Tell me a little bit about it.

Speaker 2: 21:45

So we solved the buffer bloat. Van Jacobson and Kathy Nichols solved the buffer bloat problem automatically for all forms of devices in 2012. They are geniuses. And that's an FQ Coddle algorithm and there was a successor algorithm to that called Cake. So FQ Coddle algorithm and there was a successor algorithm to that called Cake. So FQ Coddle is pretty well deployed now. It's universal in iOS and OS X and on Linux. It's a pretty good algorithm. It's just taking forever for the copy-paste guys to roll it out across the devices. But it wasn't quite good enough.

Speaker 2: 22:14

So I went to work on Cake after that. Common applications kept enhanced. It's widely available also, but not anywhere near as well deployed, and I was frustrated. We actually built it, as you know, a queue management system that even an ISP could configure, and it turns out I was wrong. I want to apologize for that. Isps have lots bigger problems than just BufferBook Billing. Billing for one billing is one of their big ones. Yeah, um, and they need to do that in order to survive anyway. So I developed with a great team john morton, etc. Cake and we're scratching our heads as to why people aren't using it. And, uh, this one wonderful isp guy figured out how to deploy on his network, saw great results, shared it with another guy and it's an open source. The core of it is an open source project which would make any ISP in the world free of buffer bloat, have really low latency, work really great with gaming and streaming at the same time, and you can deploy it almost overnight using a very cheap middle box. It's about 30 cents per subscriber or you could run Fiverr to everybody at $30,000 a

Speaker 2: 23:22

mile, or you could just put a little box in a different way and the network gets better. Now I know we're running low on time, but there's some really wonderful applications that I wanted to appear on the Internet, which was principally. I wanted to be able to plug my guitar into a wall and play with a drummer across town. It's totally feasible. The speed of light and speed of sound are a factor of a thousand different, and we still can't quite do that yet, and I'm trapped at home with no one to play with. And I learned so much for us to be collaborating in high VR and AR environments, which we really need to do after we saw Buffer Boat Universally Nothing in there. Anyway, that's my plug. They break you less. For about two years. We have well over a million devices on it. The base is free software. We do have a teeny business model, but if anyone out there has an ISP that's bothersome for gaming in particular, call us.

Speaker 1: 24:21

So, that being said, how far do you think we are from having something, having the problem solved? I mean, it sounds like you know where it is, we know how to identify it, we know what to do to fix it. What's it going to take?

Speaker 2: 24:35

Oh, I don't know. That's why I'm doing these podcasts, trying to raise awareness. You know, if I could get Biden to say my fellow Americans, america has a buffer bloat crisis. All you need to do is install this software. Buy some hardware off of eBay and you too can have a higher, can have a much cheaper much. I've been working on trying to get the bead program straightened out here. Oh, that's a whole different A couple of dollars of federal money to upgrade the internet. That just needs better software.

Speaker 1: 25:06

Like you said, just run Fiverr to everyone. That seems to be everybody's answer. No one wants to hear the wireless guy talk about not running Fiverr to everyone. That doesn't go over very well. Well, Dave, I'm going to let you outro us with a song. Man, we got about 60 seconds left. It has been an absolute pleasure to talk to you. Show me what you got, man. Give me some jams to walk home with.

Speaker 2: 25:25

Oh, you want chords. I was a song by the Mad Caddies that I love playing on bass. It's not in front of me. This is an intro to a song I wrote about SpaceX and it's on YouTube. When I was a kid growing up, buck Rogers always landed tail first, and yet we'd never had a landing on the cool green hills of Earth and I dreamed of O'Neill colonies and spinning in the stars and I thought we'd begin the asteroids and moving on to Mars. You want more?

Speaker 2: 26:09

We tried regalo wings and other things and nitrogen tetroxide Real stuff. All I dreamed was a awesome.

Speaker 1: 26:36

Awesome. Ah, come on, join back man, Join back. Hey, what's your cell phone Question mark? I joined back in the meeting, but you haven't popped back in yet.