Mike and Erik Go to Pasadena!

This week we are on location in the city of Pasadena, CA for Scale 22x, the Socal Linux Expo. We talked to people and went to talks and drank a lot of coffee. We do a debrief of some of the amazing work we heard about in the first few days of the conference.

Stay tuned next week to hear what we learned from Leslie Lamport!

Check out some stuff we learned about!

- The Open Source Rover (from JPL)

- Solomon Hykes Keynote: "Robots Building Robots"

Transcript

Music Hello, welcome to Picture Me Coding with Eric Aker and Mike Mull.

Hi, Mike. 

Hey there. 

Mike, we sound a little different today. You want to tell people where we're at? 

We are in Pasadena, California at the Southern California Linux Expo or scale. It's pretty cool here. We are in the convention hall right now, and we are not in the Van Halen building.

Yeah, I'm kind of disappointed we're not in the Van Halen building.

There's a building with a plaque on it that talks about how Van Halen came from Pasadena, and they donated like a convention hall, like a performance center. 

Something like that. And they're also doing America's Got Talent auditions there, but I don't think there's a podcast category.

Yeah. Every time Mike and I want to meet after going to sessions, I go, hey, let's meet in front of the Van Halen building. I don't know if that's a normal Pasadena thing to call it that. They should. So we got here on Friday, midday, and we talked to our friend Rob, and we had to print our own badges. That took a few minutes, and the hotel we're staying at wouldn't let us check in until 4 o'clock.

The registration computers were Raspberry Pis, which was pretty cool.

Yeah, they have Raspberry Pis all over the place. They ran all their own network lines. They've got their own routers. There's an open router that they have OpenWRT on in the exhibit hall. I talked to the guy who works for the company that makes those routers. It was pretty cool. We learned about that talking to Rob last week, actually, or two weeks ago. So conference runs from Wednesday through Sunday, March 6th through March 9th. It's a cool conference. One thing I've noticed is there's a lot of kids around. You notice that? All the kids? 

Yeah, and they said something about they encourage people to bring kids. I mean, these are like mostly middle schoolers, late elementary school kids, so it's not toddlers.

Kids and teens, there's a teen track with teenagers giving talks. The last conference I went to, big conference would have been KubeCon 2019. This conference has a little bit of a mixed personality, I think. That's not a bad thing. There's just a lot of different, a lot of variation in the talks that are offered here. Yeah, there's like hardcore systems guys who are writing schedulers. And then there's people doing scientific computing and distributed computing and pretty much everything in between. So first talk we went to on Friday would have been, what did we go to first on Friday, Mike?

It was a talk from somebody who works here at the Caltech Seismology Center, a guy named Ryan Tam.

That's right, yeah. He's been building this sort of cloud-based system to collect seismic data and do predictions or analysis of the seismic data. Pretty cool. That was cool. I learned about P waves and S waves. I didn't know about these things. earthquakes. They're sensing earthquakes. They're not predicting them in advance, but they're detecting these waves of energy. And it's an early warning system for earthquakes.

Yeah, essentially, you know, give you a little bit of time to know that it's coming, but they're not actually predicting that it's going to happen.

Yeah. So he has like a pretty typical telemetry data science problem where he's got a huge amount of signals coming in from all over California, all over the West Coast. They want to process them really rapidly and get them in front of people who can say, hey, there's an earthquake that's about to happen.

Yeah. And they've been trying to move this into the cloud, into AWS specifically. And, you know, so it's kind of interesting to see the tools that they're using on this problem that's not like your normal business problem.

One thing I thought was funny is he was using in his architecture diagram, he was using a lot of the like AWS icons of the logos, the product logos. But for S3, for some reason, he had like real pixel art buckets. yeah he didn't use that i don't know why we knew it we knew what it meant yeah exactly he also talked about this thing i had never heard of that they uh was invented i guess at caltech called the das system i don't remember what that stands for

distributed acoustic sensing i believe

yeah that thing was rad can you describe what that did

not exactly but the idea is that they are utilizing the fiber optic network that's run throughout California. And I don't understand the physics of this exactly, but apparently the seismic effect on these cables is detectable. And so they can use this to, you know, use this existing fiber optic cable that's being used for networking to detect seismic events. And it's just an amazing and kind of surprising idea.

Yeah. He said they have a thing called the interrogator, which sounds great, pretty sci-fi, right? Kafka maybe, Franz Kafka novel. The interrogator sends repeated pulses of light down the cable. And then they watch as the signals get received. They're looking for were they shaken. Right. It's such a cool idea. Such a simple idea.

Yeah, I really enjoyed that. And he talked about some of the tools that they use in that field. There's this data acquisition called Earthworm. Earthworm systems. Open source software for that. Yeah, I guess it started off with the USGS, but is now sort of maintained as open source. The system as he's building it, they bring in all this telemetry data from all over the state. You said something, was it five miles apart? They have sensors every... 

Oh, I don't know if I wrote that down. Yeah, I think you're right. Five, six miles, something like that. A lot of sensors, a lot of data. 

Yeah, and so that's coming into this server that collects everything together, and then they push it up to Amazon. It goes into Kinesis. And then they've got Lambda functions that are operating on the data going into Kinesis. So it's a cloud architecture that you would probably see in other businesses or other realms, but applied to this very interesting seismology problem. 

Yeah, it's cool to see the crossover, the usage of these tools in science too. It makes me think, wow, if you contribute to open source, you may be helping people do scientific discovery. That's pretty rad, actually. I also learned the term dark fiber. Had you ever heard that before? 

Yeah. In fact, the last company that I worked at uses dark fiber to connect between the two buildings that it occupies. 

So you've heard that term. I'd never heard that. What is dark fiber?

 So there's a lot of fiber optic cable out there that's been strung for potential use that doesn't actually have any data being transmitted. 

Is it abandoned, like a mall in a zombie film? Not exactly, but they call it dark just because there's not a signal going. 

Oh, I thought when somebody described it, I thought, oh, this is like abandoned. Like somebody stuck it in the ground and then it's just this leftover, you know, remnants of a civilization that once used that fiber optic cable. 

Some of it probably is abandoned, but I don't know the full story behind it. But I know they, you know, they ran cable along old railroad tracks and stuff. And some of it's just never been lit up. 

I like that term, dark fiber. It's great. Great term. Good term. That's a good one for you. What do you think about Pasadena? I'm kind of enjoying Pasadena. It's a cool place to have a conference.

Yeah, I like it more than I thought I would. You know, we've been here before. Both been in California for a long time, but I've never really spent a long time in downtown Pasadena. But it's a nice little town, and it's kind of cool to have the mountains off to the north, and the weather's nice. 

Yeah, it reminds me a lot of the desert. You got this stark picture of these mountains looming right behind you. Everywhere you look, if you look to the north, there's these mountains in between buildings, and they just shoot right up into the sky. It's a cool spot. Yeah. Yeah, so we've been enjoying Pasadena. There's good coffee here. We went to a place called Hello, You're Welcome. They have fantastic coffee if you're in Pasadena. 

They have good donuts too. 

They have gluten-free donuts if you want a vegan gluten-free donut, if you're gluten-free like I am. But honestly, they have phenomenal coffee. So what else did you do on Friday, first day we were here? 

So I went to a session by a researcher named Armstrong Phone Jim. He's from Quebec, I think, somewhere in Canada. The title of his talk was Leveraging Open Infra and Open Source Gen AI to Address Climate Change. And I didn't really know what to expect from it, and the content was a little bit different than what I was anticipating. they've developed a lot of metrics about like are you utilizing the data centers efficiently like they talked about this metric of power usage effectiveness which is sort of like what percentage of the power being used by that data center is actually going to power the servers the other interesting part that i was not expecting was he talked about this study they'd done about developer burnout. And the idea being that there are certain signals that indicate that a developer is suffering from burnout or, you know, lack of focus because... There's signals for that. 

Like my wife's car, it starts beeping at you if it thinks you're falling asleep. 

Signals that the developer generates. So, you know, excessive number of commits. 

Oh no, I do that. 

I didn't write down all of the things that they were tracking, but he calls them socio-technical factors, things to do with the number of commits and the number of unsuccessful CICD builds. So the theory behind this being that developers who are more productive are actually using the resources in the data center more effectively, and so they're trying to look at the factors that affect the productivity of developers so that they can sort of develop some sort of plan to make sure they're not working on weekends or doing stuff late at night to catch up on because they had too many meetings during the day or that kind of thing. 

So if they have like a bunch of commit messages, which are like, fix, fix, fix, it's going to turn a signal and a little flashing alert is going to be like, this person's burned out. 

Yeah, or they're writing a Docker file.

 It sounds interesting. It also sounds in the direction of the corporate spyware stuff I've heard about, where companies will install software on the computers to keep track of what developers are doing. And the reason why that rings a bell in my head, it rhymes with what you're saying, is these people may have noble intentions, but when you read the marketing materials for the corporate spyware, it's like, let us help you not get burned out by paying attention to all the work you're doing at all times and saying, hey, this is how productive you were on this day. This is our effort to help you not be burned out by maniacally tracking how much work you're doing. 

Yeah, there's definitely an opportunity for unintended consequences. If I were a developer working in this situation, I don't think I would want this to be applied to me for the rest of my career. But I suppose if they're tracking it for a year or so in order to do this study, that's probably acceptable. 

Okay, so short-term study. We don't want you to waste all of our energy. 

Yeah, there's potentially confounding factors because you're sort of assuming that the developer is under normal circumstances, you know, competent and productive. And so I don't know exactly how they set a baseline for the developers, but it's an interesting idea in any way. In any case, I'd never really thought about the idea that, you know, developer productivity and, you know, the thing we've talked about recently in the past, the psychological safety factors kind of have an effect on not just how good the work is you're producing, but actually the amount of energy that goes into doing that work. 

The second session I went to on that day was by a guy who built a tool that I have used. His name's Kyle Quest, and he had a shirt on that said, the Docker Slim guy. And then he got up there and he said, I'm the Docker Slim guy. And I didn't know in advance, and I was like, oh, I've used that tool. Docker Slim is a tool that you can use to compress down Docker images so they end up smaller, useful if you're building sometimes accidentally large ones. And you go, oh, no, this is a 2GIG Docker file. It's going to take a while to get that thing onto my Kubernetes cluster. darn it. The Docker talk I went to was about building containers from scratch, the talk that he gave. And he talked all about what goes into building a container, looking at the manifests. And he had deep knowledge about the stuff. He was like, here's stuff that's part of the spec by the Open Container Initiative, OCI. They have a spec and there's this other stuff that's Docker stuff that's eventually going to go away. He built containers from scratch and it was illuminating, but it fit this theme of other sessions here, which is, hey, here's the thing you use every day, Docker in this case. Let me tell you about what's going on underneath the hood. It's a different perspective than let me show you how to be more productive or use a different tool or let me sell you this new product. It's like, here's a thing that you use and we're going to go deep into that and look at how it's put together and look at what it's really doing, that demystifying process is satisfying to be in the audience for, I think. 

Yeah, I kind of wish I had seen that one. That stuff is a mystery to me. 

Apparently, Docker Slim now is, there's a whole toolkit that he has. It's called MinToolkit. He was building using Podman, which is an alternative to Docker Desktop. He was talking about different ways to build Docker containers. And one of my favorite moments in the talk was when he was like, has anybody in here heard of Heroku? And I mean, there were a lot of people in there who had heard of Heroku. Has anybody in here deployed on a Heroku? There weren't very many though, actually. I

s it skewed toward like your generation or is it? I

 mean, yeah, Heroku, I think is, there was a moment in time, right? Pre-2010, I guess, where Heroku was so easy to deploy stuff to, but, and this is, so he brought up Heroku because Heroku had this notion of a build pack and you can still use build packs to build Docker images, Docker containers. And so he was contrasting other ways of building Docker containers. And what he said about build packs was exactly what I remembered about Heroku. Heroku was great if you're running Rails, if you're running Django, and you're not doing anything too weird. They would have all this pre-knowledge about the software stacks that you might be deploying. And you would write this proc file, this like five line file. It could be, it was very minimal. It was like, deploy this. And I want this many, and they called them dynamos. I want this many workers. And I want a database. And it's PostgreSQL. And that's it. And it was amazing. And it was scalable and serverless. And it was expensive, but it was best in show for deploying software. And I have since seen that. I've never had an experience that amazing. I've since seen people over the years say, yeah, that was as good as it ever got. We've never really gotten. And when I've seen other people trying to build infrastructure tools, they always reference Heroku. They say, oh, if we could get back to that, that developer experience was amazing. The big downside, again, is if you want to do something custom, if you want to do something a little bit off the beaten path, it's pretty non-trivial. It's very difficult. It's a lot of effort. So that's really what was hard with Heroku because our industry, the things people want to build changes so rapidly. The variations are the norm almost. 

I guess now too, if you want to do something that's fairly typical or like a standard web app or a job to do some computation, you've got a lot of serverless options that are relatively easy to deploy. Yeah, especially with web apps now with like the Cloud Flares and the Nexts and the, who else goes in there? FlyIO and what's the, there's a React one I'm forgetting. But yeah, there's all these options. He gave a number of different tools for building Docker containers, MinToolkit, BuildPacks, NixPacks, Nixery, and ContainMe. I was kind of hoping to learn a little more Nix at this conference, but I've been remiss and gone to not Nix talks. Sorry, Nix friends. Sorry, Rob. We did get to visit with our friend Rob a few times. That was fun. He's in the NOC. We went and hung out with him in the NOC. 

Yeah, he seems like he's super busy. 

He's very busy. He doesn't look particularly stressed out. That's probably owing to his tenure in dealing with these situations. He's got a lot of people who are nervously trying to, not nervously, but they look like they're sweating over the network, trying to make that network solid. 

Yeah, he seems pretty chill. 

Yeah, it was fun talking to him. It's great to see him. I think they're having a successful conference. I mean, there's a lot of people here. and the stuff I've seen so far has really impressed me. 

Yeah, I was really surprised by the size of the conference. There's, what, a dozen tracks, different tracks, and it's four days long, so it's a pretty large number of speakers. 

Yeah, me too. The only disappointment for me is we're really spread across two buildings and we don't even get to go in the Van Halen building. 

Technically, the Van Halen building is the Pasadena Auditorium or something like that, but we're going to call it the Van Halen building. 

On Saturday, so the second day of the conference, we went to the keynote by Solomon Hikes. It was called Robots Building Robots, What AI Agents Mean for the Software Factory. I got to tell you, when I saw this title, I was like, oh, I don't know. It's a little too early in the day for AI hype for me. I don't know, I'm really looking forward to this. 

I was thinking it was going to be like literal. Robots. Robots, you know, Robbie the Robot type of thing. 

I thought so too. We're going to build robots. We're going to put AI on the robots. And that's not at all what it was. They introduced Solomon Hikes as, here's the guy who brought you Docker. And he was super cool. He gets up on stage, first thing out of his mouth, sorry for all the Docker bugs. It's like I like this guy already. 

Yeah, he was a good speaker as well. He has this company now called Dagger.io. You know, I think the name probably comes from the fact that it's another DAG-based workflow tool. But I think it was intended for doing builds, right? 

That's what it sounds like. Doing builds, it's DAG-based, so their whole idea is we will help you build, run, deploy, and run a directed acyclic graph composed of containers. And that's exciting to me because you can put whatever you want in the containers. And if it's a DAG, you can decide to go with on success, on failure. You can do scatter-gather. You can do all kinds of really interesting patterns with that. 

Right, and so what they have started to do and the point of this talk is that people who were using this tool realized that they could use it for other types of workflows. Now they've actually incorporated LLM into their technology so that you can leverage LLMs to help you build workflows. 

Yeah, his demo and talk was definitely the most surprising thing I saw at the conference so far. It was surprising. 

And he also has an interesting take, which is the AI is not this, you know, universe brain that surrounds everything. It's instead this sort of tool in the middle of your workflow that you can set up a loop with to get useful results from. So that, you know, it's, as he said, agents are still software. And so I think this idea of, you know, this is just another tool in the toolkit to build systems is kind of encouraging. And I hope he turns out to be right. 

Yeah, at the beginning, he said, I was sort of skeptical, you know, like any engineer, when I see a lot of hype, my skepticism meter goes up. I mean, those aren't the words he used, but that was kind of what he was saying. And he said, I'm here to tell you, this is real. There's something real here. It's not like everything's going to be solved right away. I'm not going to swallow all the hype, but there is something real going on here. And so he seemed very grounded. He seemed to have a pragmatic perspective. And his idea, like he said, agents are still software, means that we can plug them into a workflow. We can plug them into a DAG. And I had to confess that this kind of made sense to me. It was sort of pre-setup, I guess. I don't know. I was already ready to accept an idea like this because this had been proposed at my work where we were running an LLM to process unstructured data. These are just PDFs. And we don't know if it succeeded or failed. And so it's like, well, why don't we feed it back into the LLM if it fails and feed it back into the LLM if it succeeds? We want additional questions asked and we want to do a hands off thing. And I said, oh, well, you're describing sounds like a DAG. So we'll build like a DAG workflow and we'll have on success, on failure, on error. And we'll feed it. You can kind of draw the tree out of what will happen. So when he said this, I was like, oh, yeah, this is pretty much what I was trying to build at work in a much more bespoke. And, you know, I mean, it's the first pass out of it. So I wasn't that happy with what I was building. It was just a first attempt, I guess I would say, a rough draft. And what he was demonstrating was pretty cool because he was like using their Dagger I.O. tool. They have a terminal. They can use the LLM and they can go back and forth between Dagger functionality and LLM functionality. Dagger functionality is very container based. So he could say to the LLM, get me a container that's based on Alpine and put me in a shell. And then boom, he's in that shell environment and he can start typing stuff and then he can exit that. And then he could say to the LLM, can you read that container? And then he could say, can you dump out a program for me? And I think his original program was like Bash or Python or something. No, it was Go, I think. His original Python program was in Go. Yeah. Yeah. And so he had a Go program and he had the computer, he had the LLM compile it. And then he did something really surprising. 

Yeah. So there was a bit where he's trying to get the LLM to write a program and it comes up with a compiler error. You know, it had been trying to install tools and the software that it needed to compile this thing, and it failed. And so he just said to the LLM, "dude, you figure it out". 

Yeah. And the surprising part being that it did. It went out and figured out what the issues were and downloaded the right things and successfully compiled the program, which was both funny and surprising. Yeah. I look at what he's doing and I go, oh, that's kind of cool. There is a human in the loop part. And he said, you can take this whole flow and you can dump it out and have alum help you automate it so that you can turn this into a repeatable process. And I almost feel like I don't have enough creativity to then go, oh, I have this problem that I want to apply this solution to. Like there's something, there's a missing step for me there. 

Yeah, I'm not sure. I could not immediately think of how I would use it. I think if you, so for my own purposes, given the stuff that I do, if I had a tool like that, that would sort of help me build a workflow around like a data pipeline, I could see that being pretty useful because... 

What do you mean there? Tool around a data pipeline? Like what would you imagine? 

Well, I mean, say that you start off in some typical business situation where you say, I have this pile of unstructured data, and what I really want to know is how many things I sold at each location during the last year or something like that. I think if, you know, now you'd have to go through this process of doing various steps. You know, you might have to do an OCR step. You might have to do some natural language processing or something. And if you just had something to sort of give you guidance on the best way to go about it and the best tools to use, I can see that being valuable. 

Mm-hmm. He had a diagram. It was a diagram. It was a directed cyclic graph. And he had on the left, he had human, and there was an arrow going to the LLM. And then from the LLM, there was an arrow going to environment, which I thought, oh, that's a little fuzzy there. And then it would go back to the LLM. And then there was a stop node below, like maybe at some point we decide to stop. He substituted human on the left and environment on the right for functions. And so that was his argument like this is still software. You're going to write the software that feeds into the LLM. You're going to write the software that takes output from the LLM. You're going to have the LLM in the middle. It's not going to wrap everything. It's going to be somewhere inside the software someday. And then you might take that thing that we just described, function, LLM, function, and make that a module or component in a whole massive structure of the same. So function, LLM, function, Maytol, to another module function, LLM function, or another component function, LLM function. And it was surprising to think in that way. Yeah, it was a really good talk, though.

I'm kind of glad we went to that one together, because it was fun to sort of hash out stuff afterwards.

I'm glad we went to it, too. And I was totally a fool for being like, I don't know if I want to go to this keynote about robots and AI. It was a fantastic talk. I'm stoked to be there. I guess after that, I went to a handful of sessions throughout the day. Anything notable that you went to, Mike?

I spent the morning kind of nerding out on Linux kernel stuff. Oh, is that when you went to this GetX talk? Yeah, so there was a track that had to do with BPF, Berkeley packet filtering stuff. So this is not something that I'm super expert in by any means, but it's basically like a way to make extensions to the Linux kernel, which may run in user space. So the first one I went to was a talk about a tool called BPF Trace. And it was interesting. It was mostly about how they have evolved this DSL that they've built. So BPF Trace is essentially a domain-specific language for doing queries of system things using this Brickley packet filter technology.

So you're querying?

Yeah, so you're looking for things like, you know, tell me by process, tell me how many threads each process is running. So that's the sort of thing you can run against the kernel using this tool. 

I've heard stuff like that before. We're tracing with eBPF so you can get some increased observability in your system using this. 

Right, yeah. And he contrasted it to how difficult it is to write. Standard BPF programs are usually written in C, and they're reasonably complicated. It's fairly cryptic-looking code, and you have to pass things back and forth from kernel space to user space these buffers. 

But you can actually write those in Python and other languages too. 

Yeah, people, there's people do a lot of them in Rust now. So the second thing I went to in that track was a really interesting talk that was considerably over my head. 

Wait, I didn't get to tell you. The first time I heard of eBPF, I was at KubeCon 2018. I sat down to it next to a guy who turned out had like a whole networking company and he was trying to sell something for Kubernetes. And I didn't even know what I thought. I was just learning about Kubernetes networking. And it was an eBPF talk. Actually, it was a psyllium talk. And he started talking about eBPF. And I turned and I was like, Oh, I've never heard of that just under my breath. And the guy next to me goes, you've never heard of that. And I was like, Oh, shit, I guess I should have heard of this thing eBPF. What am I doing with my life that I haven't heard of this thing? 

shameful. 

Yeah, it was horrible. So but I know what it is now. Well, I've heard of it. I know what the letters mean. And they have Berkeley in there. That makes me proud. Okay. Sorry. Go ahead. So the next talk you went to, was that the SCED-X one? 

Yes. So the next one was two guys from Meta talking about this technology, which is either pronounced SCED-E-X-T or SCED-X-T. And I don't know because they pronounced it in both ways. So I guess either is acceptable. T

hey were not consistent in their terminology of what you're saying. 

Right. The pronunciation, maybe. But you can think of it as being scheduler extensions, which is essentially what it is. It's a way to write your own CPU level scheduler. So they talked about how that works and work that they have done at Meta to build their own schedulers. One of the interesting things to me is, and this is almost like magic, it seems like, but they write schedulers in Rust, which work in user space. That is weird sounding. Because the schedule is a kernel level thing, right? 

A kernel has to decide what processes, what threads to give CPU time to.

Exactly. And, you know, I sort of think of it as being, and we've talked about schedulers in the past on a couple of episodes, but, you know, when I think of the Linux kernel scheduler, I think of it as being something that's really, really deeply built into it. And so the idea that you can sort of extend it and do it in user space is fascinating to me. But anyway, they talked about work that they have done at Meta to build their own extended schedulers. 

Okay, but I got a dumb question. Why do you think they need that at Meta? 

Well, I think it's because they are so massive, and they run so many different types of workloads, that if they don't have, you know, a scheduler that's optimized for their particular jobs, then they would have to probably use more hardware than they do. You know, they're already hugely massive infrastructure. And so writing a scheduler, which is optimal for the particular workloads that they're working on probably saves them some hardware. 

So they do it to save money. Yeah, they probably do it to save money. 

They probably do it to, you know, optimize certain workloads because otherwise they would take too long to run and wouldn't, you know, be effective in giving feedback to users. But in any case, the work they were doing is very sophisticated, and they talked quite a bit about how they have to take into account things like trying to make sure that a thread is getting scheduled on the same core if it's in a certain level of cache. And they can't have queues that span what they called sockets, so they don't want processes going between two different sockets on these big computers with multiple CPUs on them. 

Because it's wasteful. 

It's wasteful, and they said it just brings things to a halt because the latency involved in swapping threads is too great. But anyway, they're looking at all this really fine-grained stuff about how to optimize the workloads on their machines. And of course, it varies. They built this scheduler called SCX Layered, and I guess they say it's super configurable because they have lots of different types of workloads and something that's a web app is going to be a lot different than something that's running GPU workload. 

I guess I have two responses to the SkedX schedulers and user space thing. That sounds super cool. And honestly, it sounds amazing engineering. I'm not sure if I would use that, what I would use it for. And let's say I decided to use it. My second response is, wow, this is pretty well over my head. It's pretty complicated, but it adds another layer of mystery and complexity to my massively complex stack of whatever I'm doing anyway. I'm writing a Rust program. I'm talking to a database. I need to talk to the protocol of the database. And I've also, oh, yeah, I've also got the scheduler happening. And maybe it's going to change how my sockets are working. It sounds a little scary to me. 

Yeah, they talked a lot about how they have to debug and deploy these things. And it is something they have to be super careful with. And I guess there's like in this SketX community, there's rules about like which version of the Rust compiler you can use and which version of, you know, the Clang you can use or CLang you can use. Do people say Clang? Is that a word? I've heard people say both. So I sort of prefer C-lang. But yeah, so they have to be super careful about configurations because even a minor change in compiler version or kernel version can break things hideously. 

After the keynote, I went to a talk called Source Available is Thriving but Open Source is Dying. The reason I went to this talk is, we've talked about this before on the show, I have this theory that, or belief or suspicion, I don't know how to call it, it's probably just a belief, that the world of software that you and I operate in now where we get so much free stuff that we build on top of that eventually that massive waterfall of free stuff is going to dry up. There's something like a renaissance happening where somebody gets excited about an idea, they build and release a library, and then people get inspired by that and build on top of it. It reminds me of the Italian Renaissance where somebody learns some landscape technique or perspective technique, and other painters get excited, and they do that. And these ideas, they rapidly propagate softwares like that, but I just don't think that can last forever. Something will change. So when I heard the title of this talk, I thought, oh, I'll go to that. Hazel Weekly is the director, I think, or at least on the board of the Haskell Foundation. Haskell Foundation is the nonprofit wing of the Haskell community. They provide a couple of services to the Haskell community. One is they support compiler software development. Somebody has to work on the Haskell compiler. It's a complex compiler with a lot of moving pieces, and they have to continue to do patches and releases and make that available to the community. And the other thing they do is they run the package manager, Hackage. So if you want to download a Haskell library, you would go to Hackage and you could download it. There are other places you can download Haskell libraries as well, but Hackage is like the biggest well-known one, equivalent to, you know, what's CPAN and Perl or PyPI and Python or cargo packages, something like that, right? So Haskell Foundation. And Hazel Weekly started to talk about how there's a, and I didn't get to ask when this happened or why, but what she said was being on the board of the Haskell Foundation one year at some point, there were a dozen or more sponsors for the Haskell Foundation. These are big companies donating money to the Haskell Foundation, saying we like the work that the Haskell community is doing. want you to keep working on Compile or whatever. And one year, every single sponsor decided not to renew out of more than a dozen. They all said, not going to do it next year. I'm still curious why. But she said we went from a funded organization to no funding whatsoever and suddenly had to figure out how to operate in this world. And this turned into a talk about open source licensing and what it means to be an open source maintainer. And there's this political philosophical ethos that exists below the surface. And we don't talk about it too much, but this emerges in things like software licenses. So GPL or AGPL or GPL version 2 is more of a free software license. And there's a political philosophy that informs and inspires that license. You look at MIT or BSD licensing, these are much more like do with this software whatever you want. And Hazel Weekly is saying as open source maintainers, we need to try to assert some power and we need to kind of come up with strategies. The overall argument was you can't just sort of say, okay, everybody is going to pay for things. Because, for example, if you say everybody has to pay a certain amount of money for the Linux kernel, you're talking about trillions of dollars in installs of a Linux kernel, right? That's just one, I mean, that's the most widely deployed open source software in the world. Hazel Weekly is saying if you operate a resource, say like a Docker hub or something like that, you may want to rate limit large companies and customers who are using a lot of your services. So she was talking about strategies to, she called this legibility, like make it obvious how much people are extracting and building on top of. I think there's a degree of controversy in this. And before I kick it over to you, what you think about this, I want to say that going to a Linux conference like this reminds me of when I first got into Linux 20 years ago. There was a lot more, maybe it's just the people I'm around now, but there was a lot of advocacy. There was a lot of political ethos below the surface of what, like Linux meant something philosophically. It still does, but it really meant something philosophically at that time in the early 2000s. If you're going to use Linux, you're like choosing a poor operating system experience. I don't think people would take issue of that in the 2000s, but you're doing it, you're taking a stance. A lot of people were taking a stance when they were running Linux or installing Linux. Do you remember that? 

Sure, yeah. There definitely was sort of a connection to sort of the hacker ethic idea, I think. 

What would you describe, hacker ethic, like what would that mean to you? 

Well, I think it's not just that it's open source and dissociated from like evil corporations, but also running your own operating system gave you the opportunity to understand how the underlying hardware worked, which is kind of one of the key aspects of the hacker ethic. The hacker that I'm talking about is not the sort of people who break into the CIA. I'm talking about the hackers who want to figure out how their computers work and how their phones work and how the electric grid works, and they explore it and do sort of paralegal things to figure that out.

Well, but the free software movement had as one of its tenants, if I'm going to use your software, I want to be able to modify it myself. I want to be able to change it, extend it, fix bugs, whatever. And I want those modifications to be participating in some larger pool, larger community. And there's this communal perspective that informed that. And Hazel Weekly's talk and at least one other talk I went to, that spirit was still alive. I don't know if it's died. Maybe it's just the people I hang out with. But I just I don't hear that much anymore. 

I wonder, I think people describe it as being sort of a generational thing. So, you know, I hear a lot of stuff now about how people are worried about Linux maintenance because the only people still doing kernel maintenance are graybeard people like me who are on their way to retirement. But I also wonder if maybe there's an aspect where when Linux first was new, it was still pretty much the personal computer era. You know, you were running that on your own computer, probably in your house or your office. And now, since everything has moved to the cloud, it's a little less personal, I guess. And I don't know, I've been following this, what they call the local first movement recently. 

And where'd you hear about that? 

Yeah, Martin Kleppman is associated with it. He in particular has worked on this database system or a CRDT called AutoMerge. You know, there's this sort of movement called local first. They have a conference every year and there's a podcast about it that I've been listening to. But I just wonder if maybe the incentives are slightly different, but also the sort of attitude about I'm modifying something for my own use is not really a thing for the current generation of developers.

Yeah. I imagine most people just think of it as these are tools. They're tools that are out there. It's like fruit. It grows on trees. I'll pick one. I'll eat it. Yeah, and it's great that it's free-ish because you wouldn't want to have a proprietary operating system on the million servers that you're running. Yeah, I mean, it's nice to have a free text editor. I can't imagine paying for a text editor. Do people do that? 

Not in my world. 

Some people do. Some people do. But on the supply side, it relies on the fact that there are a lot of people giving away labor for free. And that can't last forever. It can't last forever. 

I think there's a degree to which there's sort of like a selection going on where Linux is going to survive because even if all the people doing the work for free move on, Google is still going to support it and that is still going to support it and so forth. But things that are kind of on the fringes, scientific software and utilities that are, you know, for older hardware and things are probably just going to disappear. 

Things that big companies aren't necessarily interested in that don't help them deliver their products. 

Yeah, that don't have any direct bottom line impact. 

You know, the next talk I went to was called Advanced SRE, Advanced SRE Networking. And it was by a guy named Patrick Schuff, who works at Netflix. Great talk, very well delivered. I was pretty pleased that I went to the talk and I really liked how he gave the talk. He works for the CDN reliability team at Netflix. I wish I would have gotten a chance to ask him a couple of questions. My networking knowledge is not as yours is. And I used to come to you with questions. I still do sometimes. My networking knowledge kind of stops at the socket layer. I've done a tiny amount of socket level programming, but most of my knowledge is way far higher level and above that. So he started with this curl command. He did curl-v, www.netflix.com. And he said, we're going to walk through what this curl command does. And he went through it at moments in time. So you've got DNS, you've got TCP, you've got TLS, you've got the actual HTTP request, and then you go to response, and then you close your TCP connection. So that's like what happens in time. And he also talked about the OSI model, layer one, layer two, layer three, whatever those are, layer four, layer seven, the ones I tend to pay attention to. He talked about every single layer of the OSI model, and he was able to work this in, weave this in, in a talk that overall was very well structured and very well delivered. The utility for me in the talk, I think, is two things. There's holes and patches in my knowledge. Somebody mentions he talked about BGP at the end, border gateway protocol, internet exchanges. Somebody mentions this, and I'm like, yeah, that rings a bell. I kind of remember what that means. You've got networks talking to networks, right? He was able to fill in the patchiness for me, but he also delivered his material in a way that gave me a lot of, I think, tips and instruction on how to transfer some of this knowledge to other people. Because I find that I'm working with application developers often at work. There'll be some network problem, and this is just probably just an experience thing. You know, it's not like anybody did anything wrong. it's not like they're doing a bad job at all, but they often are used to thinking, well, I imported this library in my programming language. I ran this function call, which is the equivalent of it sends out this HTTP request. That failed, and it maybe gave me this inscrutable error, TLS failed, or timeouts, or whatever. And that's sort of where their debugging process stops. And often I find myself in this position of going, okay, but let's think about the conditions for why this network request might have failed because we need to know if there's something we can fix here. I mean, network requests fail, but is there something wrong? Are there resources constrained on the server, resources constrained on the client? What actually failed? What was happening at the moment that it failed? And being able to have a better picture to describe this to the people I work with is worth quite a lot because I often fail to communicate. Again, my knowledge is patchy, so maybe that's not why I'm a bad teacher of this stuff. But having somebody demonstrate how to walk through it in a very logical and very graspable way. I mean, he was not trying to treat this stuff as rocket science. It was really cool. I liked it a lot, actually. 

Yeah, I would have liked to have seen that. That's kind of my... 

That's your jam. 

That's kind of my jam.

 But I got to tell you, the whole time I was in the talk, I was like, here's another thing I bet Mike knows. 

So ironically, you went to a talk on SRE and I went to a talk on RSE. 

RSE? What's that? 

So there's this idea that is sort of growing. It's probably more so in Europe than the United States of what they call a research software engineer or an RSE. The idea is that, you know, software is now a very impactful part of science. The speaker in particular gave this citation about how the Hubble Space Telescope has been extremely successful and has prompted the writing of a lot of papers, you know, thousands of papers on new astronomical research. And also, what's the new telescope? The Webb? James Webb? James Webb. Yeah. Anyway, the new telescope, he said there's something like 6,000 papers that have been written on stuff that has come from that telescope, which is, you know, sounds phenomenal. but he also pointed out that in a much shorter period of time, there have been far more papers that have cited NumPy. Wow. The idea here being that the research software engineer is somebody who can work as a software engineer, but their work is not devalued in the scientific environment because right now, if you're not producing papers, you're not doing the thing that is equated with science or equated with success in science. And so the research engineering track is supposed to give people a career path where they can work on software and still have that contribution be valued, even though they're not writing research papers about it. 

Can you do that without getting a PhD? Is there like a job where you can just go and be? I mean, like your supercomputer center experience always sounds like a pretty rad environment. I want to go and help scientists build and deploy software. That sounds kind of cool. 

Yeah, it's very much like what I was doing in the early days of my career. but I think probably a lot of the people doing it now do have PhDs. And they probably don't have enough funding to just hire a software engineer to come and help them build whatever they're doing. It probably depends a bit on the field too. So if you're writing software for astronomy or quantum physics or particle accelerators or something like that, you probably have to have enough domain knowledge to be able to be productive with that kind of software. 

You've got to know what the hell all those I, Js mean. Yeah, you got to understand the equations. Sorry, that was a bad scientific computing joke that I made. The scientific programmers always are using these variables I and J and junk like that.

 So it was an interesting talk. He mostly talked about, you know, like the history of science and what has been considered to be science throughout the ages and how there's sort of like this revolution now where software is sort of the backbone of the scientific enterprise. Kind of quietly, not a thing you think about, but it's totally true. Yeah, it was an interesting talk, kind of promoting the idea of having this more in the United States. I guess it's not really caught on here as much as it has in Europe. 

When we went to that seismology talk the first day, I was looking at the guy giving the talk like, wow, his job seems rad. He works at Caltech. He helps these people doing earthquake predictions, earthquake early warnings, I guess. That's awesome. That's useful work. I would feel good about contributing to something like that. 

Yeah, I mean, it's interesting science, and it's also valuable to the general public. 

The other talk I went to after the SRE one was John Mad Dog Hall. I have naively followed John Mad Dog Hall's career because the first Linux book I bought was this doorstop book. I talked about it when we talked about our favorite technical books. I love that book. I thumbed the crap out of it. And there was a blurb by John Mad Dog Hall on the cover of the book. And because of that blurb, in my mind, he had written the book. And so I spent the next 15 years trying to be like, oh, yeah, I read this book by John Maddock. And like following his, paying attention to his career. But he gave this talk about how, the title of the talk was like, let's make programming fun again. And then he got up there and he was like, I was going to give that talk, but I'm pissed about the election. So now I'm going to talk about like free software and what it means. And it was, I mean, going back to that free software movement, the philosophy of free software. This is a guy who's been in the industry since the 60s. He worked at Digital Equipment Corporation. He worked at SGI. And he was up there still railing at Microsoft and Windows and Mr. Gates. And I was like, wow, what year is it? This is taking me back. He was talking about some interesting stuff. He has this program that he, it's a nonprofit. He has been building this foundation so that people who are college students in Latin America, They're studying CS. They can work as frontline support people in software and hardware. They can use that to learn business skills and also earn money so they can continue to fund their studies. It was really interesting. I haven't asked you yet how people, the questions that you've heard at the talks you've been to, if you've heard any good questions. I remember at John Hall's talk, he's one of those guys who's just very straightforward. And somebody gave this question at the very end. I didn't understand. And they were kind of connecting all these things. He said, you're making this a little too complicated. This is a guy who doesn't mince words. 

I've heard some really good questions. There's been this one person who's been in, it's a woman who's like of my generation. She's been in several of the sessions I've been in and she's asked really brilliant questions in all of them, but they're like very diverse fields. So I've been trying to find an opportunity to talk to her because I'm really curious about what she does. 

The John Hall talk was interesting. He was saying things were supposed to get easier once we had the cloud and once we had smartphones, but they're not getting easier. There's like a kind of easy. There's an easy in quotes, he said, and a hard in quotes. And this is a guy who firmly believes you should be able to modify the software on your machine and the operating system on your machine. This is a guy who firmly believes in Linux, the philosophy of Linux, the free software, Linux. And he said, if it were so easy, why would you have the genius bar at Apple? Why would you need someone else's help to fix things? That's that hacker ethos. He talked about critiques people often have for his ideas of things he's talked about over the last however many decades. He said people say, well, this doesn't sound very capitalistic, building this thing and giving it away. And he was like, it's not. One thing I thought was interesting, he started talking about land grant universities. Do you know that term? I'd heard the term. 

Yeah, that is well known to me for a number of reasons. Purdue University in Indiana, where I grew up, was a land-grant college. And University of Arizona, where I went as an undergrad, was also a land-grant college. 

He started talking about it because he was like, you could go for free to college. It was part of your local area college. And we've lost something in the last few decades. He said this is like a, he actually kind of almost got a little angry. What was the phrase he said? This is a war on knowledge. He would be an interesting person to talk to more, I think. His opinions, the things he's seen over the years. Someone said to him, well, if you could ask China to put Linux on computers before they ship them over here, people could still put Windows on them, of course. Then what if some government around the world said, oh, we don't want to install Linux. It's dangerous. We don't want people to use it. He was like, okay, well, you could still use it. I'm sure. I don't care about that. is what he said. And then he responded by saying, what would happen if somebody dropped a nuclear warhead on Redmond, Washington? And the guy just kind of, he was talking to, he was like, I don't know. And he was like, it would take them a long time to ship a new version of Windows. And he was like, how long would it take? And nobody answered it. He said, I don't know how long and I don't care. Jeez. It's very strongly worded. He had one other really interesting comment too. He said, Somebody asked him about AI. If you have AI, are you going to reduce the need for free software engineers, people working on free software? He somehow connected this to going to the store. He said, when you go to the store, what do they sell at the store? If you go to a computer store. People were like, computers? He was like, no, they sell shelf space. If someone goes into that store and has to ask the staff of the store even one question, the store is losing money. They want you to go in the store. They want you to take the thing off the shelf and pay for it. So it's like a different idea what motivates, what inspires the free software movement. It's like that candle still is burning. I don't encounter that perspective very often anymore. I don't know why that is. Yeah, that was kind of my question too. Here's my question for you. How many conferences have you been to in your career? 

That's a good question. I don't have a precise number. Probably somewhere in the vicinity of 30. 

Okay, 30. That's a lot. I've probably been to five conferences in my career. 

A lot of variety in there, though. I remember going to a conference in Cincinnati in the late 80s or early 90s, pre-World Wide Web. It's a conference on doing documentation and education in the high-performance computing world. I remember it being kind of funny because people were talking about the state-of-the-art then was things like Gopher and Anonymous FTP. and people were talking about how, I wish we had some internet thing that you didn't have to log into and that you can just grab information from. And little did we know.

Wow. I do get that feeling when I'm at a conference. Like people are talking about stuff that's state of the art, that's interesting, that's pretty new, but just around the corner a year, two years from now, all of this could be completely different in a completely different light because there's something else. And so you go to a conference, it feels like you're on the edge of something, But being on the edge, my natural inclination is to kind of continue to look behind me. Like if I'm on the edge of the cliff, not facing the ocean, I'm like, well, what's out there? Obviously I can't see the future, but it's hard to ignore the narrative that this stuff rapidly changes. 

Yeah, I mean, there's a lot of people here doing very interesting, very cutting edge things. But the reality probably is that people are doing the stuff that is going to surprise us in the future is probably like four people in a room somewhere. 

Yeah. Some university. And we're talking like VC people now. We've got to go and find those people and invest in them. That was a joke. Didn't sound joke enough. That was sarcastic. After that talk, you left. You sold out the conference. That's a bad one. 

I went to West Hollywood to hang out with the cool people. 

Oh, okay. I did not do that. I stayed in Pasadena, my new favorite LA region city. And I went to a talk called Unpacking Git by Nick Lewis and Tanya Ulyanova. They work for a company called Phlox or Phlox Dev. Phlox promises to package up and ship virtual environments for you. It looks like they do this on top of Nix and Nix packages. But their talk wasn't about Phlox. Their talk was about Git. And this is another one, like the Docker talk, like the networking talk. Here's one about Git where they're saying, here's a thing you use every day. Let's go deep. It was a great talk. One of the speakers, Tanya, she did a lot of the description, the object model, how things work. And then she would pass it over to Nick, and he would do the demos. They did a huge amount of research. They had been working on this project using Git internals for over a year. So they were pretty well prepared. But they were saying at the beginning of the talk, like, I'm still learning features of Git every day. I think one of my favorite moments in the talk is an example of a lower level git command called update ref, potentially pretty dangerous command. This is me saying caveat emptor if you go and decide to look at this command. And so he runs his git update ref command, and then suddenly his demo repo that he's using for the talk, it no longer wanted to respond to the, like all the commands, the answers he was getting got very confused. So he took it really well at stride. He was like, I don't know, I broke something using this command that changes the parent ref of this commit. It reminded me again of what a sharp tool Git can be. Great talk, though. 

I don't really understand Git that well. Probably the most complicated thing I've ever done with Git is Git bisect. But it's something that I don't think I really want to understand better than I do now. 

I kind of feel that way, too. I just want to keep using it. Maybe that makes me the bad person. And on the way into that talk, I was walking from one building in front of the Van Halen building to the other building where that talk was held. And the whole time I was walking behind this guy with this little six wheeled rover with a water bottle on top. And I was walking behind the guy at the same rate he was walking. And I couldn't see if he was controlling it or just remotely following him. And everyone passing us, they were just like eyeballing this rover as they passed. I could see their eyes glued to the rover. And so he got to the door. he pushed the handicap thing so he could get the rover in without having to use his hands. And I saw he had a controller. And he turns to me and he goes, oh, sorry, I'm in the way. I was like, no, no big deal. I've been watching you walk. I've been watching people look at the rover. And I was like, it looks like a super efficient way to transport your water bottle. And he was like, no, no, no, I just didn't want to forget my water bottle. And I was like, no, no, I'm kidding. And so then he had a QR code on the top of the rover. And I started asking questions about the rover. He's like, yeah, you can build this yourself. It's called Open Source Rover. If you want to see the repo, you just scan this QR code. I scanned the QR code and was blown away by what I saw in that GitHub repo. We'll include a link to it. It's Open Source Rover. It's actually produced or put out by JPL, Jet Propulsion Labs. I didn't know the guy was with JPL. And they have this purchase list, equipment list, and instructions on building this thing. It was very cool. It's a miniature version of the Mars rover that you can build yourself. I was blown away by this project. 

Yeah, I suppose people know that JPL is here in Pasadena, along with Caltech, since a large part of our audience has probably watched the Big Bang Theory at some point in their lives. 

I have never watched Big Bang Theory. I think of Caltech, I think, oh, so Feynman walked around these streets. Maybe that was his house. 

That's a good point. We should look for his house. 

Well, so here we are on the last day of the conference. We're about to attend keynotes. I think we're going to get to see Leslie Lamport speak today. Yep, that's the big draw. All right. So this has been Picture Me Coding with Eric Aker and Mike Mull at the Scale Conference, Scale 22X. This is a 22nd Scale Conference. Thanks for listening. See you next time.