23: Containers - What's in the box????

Containers have become the standard way for deploying applications on servers and the web and sometimes even on the desktop.

In this episode we dive into what containers are, how they work, how to build them and what you can do with them.

Whether you are using containers in your development environment, deploying on servers in your data center or as a cloud service, containers save time, handle dependencies, increase security and just make things easier and better in so many ways.

We discuss several commands to build and run containers and we've included examples here:

Dockerfile example:

-------------------------------------------------------------------------------------------------------

FROM ubuntu

RUN apt update && apt install -y apache2

ENTRYPOINT [ "/usr/sbin/apachectl", "-D", "FOREGROUND", "-k", "start" ]

-------------------------------------------------------------------------------------------------------

Build the image using the above Dockerfile:

    docker buildx build --tag my_container ./

Run the container:

    docker run -p 8080:80 -d my_container

Now, point your web browser at http://localhost:8080 (assuming you did this on your desktop)

Display a list of running containers:

    docker compose ls

Attach to a running container and get a shell:

    docker exec -it [container name] /bin/bash

Stop a container:

    docker container stop [container name]

Start it running again:

    docker container start [container name]

Remove a container (after stopping it)

    docker container rm [container name]

Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm

Follow us on Mastodon: @RuntimeArguments@hachyderm.io

If you have feedback for us, please send it to feedback@RuntimeArguments.fm

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky

Transcript

Wolf 0:05

Hey everybody! Welcome to another episode of Runtime Arguments. Yes, we will never stop. Okay, I can't say never. Um, but this is episode number 23, our 24th episode, because we started at zero. Uh I'm Wolf, and as always, I am joined by my very best friend, Jim. Say hi, Jim. Hello, Wolf. How are you doing? I'm doing good. And you? Good. I'm doing great. Uh I'm gonna talk uh in advance about uh what you've done uh research on and have been using. So, like there were lots of things you didn't need to research, you already knew, but you got to dive deeper. This episode is gonna be about containers. Um now you don't have the suspense, but uh let's start with how was your week? How's your week, Jim?

Jim 1:05

Oh, my week was great. Um uh it's actually two weeks between episodes. So my two weeks have really been good, uh, busy, uh, which I like, uh doing a lot of good work. Um, you know, we've we've talked about this before. We meet for lunch every Saturday at a uh sushi place in Ann Arbor. And by the way, anybody's welcome. We we are there at 11 a.m. Biwako. Um if you'd like to show up, please do. Uh so it it uh our little group has grown. We had seven people at lunch uh the other day. What used to be just Wolf and myself uh was seven people, and it's a lot of fun. We get into a lot of great conversations, and one of the things I love about it is I always learn it it's a it's a great place to to listen and learn. And uh this week I learned something new, and it just floored me. I'd never heard of this before. Uh HDF 5. A hierarchical data format 5. Uh Wolf and a couple other guys were talking about it, like, oh yeah, it's obvious. Use this for some things he was talking about. And I had never heard of it. And I I I don't know where I've been for the past uh uh I mean HDF uh format started in 1988, so where have I been? So I completely missed out on it. So uh I I wrote it down at lunch and I came home and I I dug into it a little bit. It's quite interesting, it's very good for scientific data. Um NASA uses it for information they get from all their missions, and I think it's pretty cool. How did I get this far in life without never hearing about it? So yeah. I love learning new things. Sometimes there are surprises. So yeah, how was your week?

Wolf 3:00

Uh my week was awesome. I wish I could swear. I mean, sometimes I can, but if I swear too hard, then Jim has to do editing work. And so we'll try not to. But let's just say it was effing awesome. Um uh in one way and kind of weird in another. Uh I just use AI more and more and more and more for different things. And I am at the point now where with the help of AI to make this frictionless, I'm able to uh just instantly record when I get out of bed and when I start my actual work day at my jobby job and when it ends, and how long I spend uh in deep work on particular tasks. And I have learned numbers um that make me in a way sad. Um and so I'll tell you this part and then I'll tell you the happy part. So the with respect to these numbers, how long do I spend on average in a day for my jobby job? Um, it is 10.5 hours. Um That's a lot of time. It it is a lot of time. It's probably more than I ought to be spending. Um so I don't know. That that doesn't feel good. I also don't feel good about the fact that you know you spend all that time, but a lot of it uh is wasted. Like the actual work that you do, that's what I'm calling focused time. What other people call deep work, where you really are, you're you know, you're in the zone or doing the thing. Um my average for that on a day is 4.5 hours. Now, when I first heard that number, I was horrified. Oh my god, I'm spending over 10 hours and doing under five hours of what I would consider the work. And then I did some research and found out that the average amount of deep work that a person, a knowledge worker, does in a day is two hours and thirty-eight minutes.

Jim 5:27

Wow.

Wolf 5:28

So um I so this isn't the happy part. I'm just telling you, it's not as bad as I thought it was. I I'm doing okay. Um definitely needs to be addressed somehow. I think what I what I have here is incorrect work-life balance. Um, and I need to fix that. So something about that. Here's the good thing. Um I uh I love programming. I love algorithms, I love making things go faster. Um and uh I'm I'm not braging. I I'm good at making things go faster. Um I make them go faster in a different way than the guys on two's compliment. They're like doing i individual instructions and figuring out what i y listen to their podcast, it's great. I I make things go faster in a different way. Um and a la and I s kind of specialize, one of the things that is super interesting to me is information theory, which touches um compression and encryption and text search and um indexing and uh AI in fact. Yeah. Yeah, it's I happen to love information theory, and Claude Shannon, the founder, the father of information theory, although he called it communication theory, he's one of my four heroes. Um a long time ago, I wrote a search algorithm that does something pretty amazing that other search algorithms can't do. Um and I lost the source code and it wasn't of immediate utility. Um, and how fast do you really need to search anyway? One of the great things, uh you know, for instance, in your editor, uh, if I can find the thing you're looking for in your editor uh 17 times faster than the thing you're using, guess what? We're both under a second and you can't tell the difference. So it doesn't matter. Uh but here's a place where I matter. Most search algorithms fall down in cases that are described by information theoretists as small alphabet scenarios. Uh you know, English text has i i if it were all case folded 26 characters, or maybe you'd call it 256. But DNA only has an alphabet of four characters. And so when you hit a care most of these fast search algorithms, look at a character and then decide based on if it's in the needle, the pattern for which you are searching, how far can you slide the your comparisons so that you line up again with the next possible place? In normal text, there's lots of mismatches. You you hit a Z and you're looking for cat, you can slide all the way past cat. Um of course we're talking about much longer strings, but in DNA there's only four characters. When you probe your first character, it's almost certainly going to be in your needle. Um you don't get to slide very far. Um and the sad truth about these algorithms is they just keep going not very far over and over and over again. They're acceptable in human text and they're awful in DNA. Now there's some good algorithms out there, um and uh competitive with the thing that I do. Uh, but my magic is that I can make the best mathematically uh the mathematically best choice for jump given all the knowledge you have. Every character you've seen so far, that's my secret sauce. Um and this weekend I rewrote it. I have it running again. And not just my code running again, but I wrote it in Rust. Which is super exciting to me because I want to be better at Rust, I want to learn Rust, and I also at the same time implemented um in Rust all of the reasonable competitors like Boyermore, Horsepool, and um the the one that is a really good competitor for me. Um it's uh it its name is four letters starting with D. I can never remember what it is. I'd have to look it up. I'll put it in the show notes if it matters. But I got it running. I have a lot of work to do. I am further along with it now, in modern times, in 2026, than I ever was in the past. And uh to me, this is a really exciting thing. This is a good week. I feel great. This energized. That's my week.

Jim 10:57

This really energized you. So that's cool. That's uh that's uh I'm anxious to see where you go with that. Uh we talked a little bit about last night, and and it sounds really, really neat. So I'm sure you're gonna hear more about that from Wolf uh coming soon. So yeah.

Wolf 11:16

Um we should. There uh there is feedback, possibly. Is there feedback?

Jim 11:23

I have not received any feedback. We're still getting lots and lots of downloads of the podcast, but uh I didn't get any feedback this time.

Wolf 11:33

For our last episode, I didn't get any feedback either. Um, other than I think you and I both heard talk about our audio production quality.

Jim 11:46

Yeah, there was a mention about it. Um we we fixed that. Yeah, do we sound good now? Give us some feedback, let us know.

Wolf 11:55

I would really like to know if we sound good now. Because in my well, not exactly my headphones, but in the audio that I'm receiving, it sounds too loud to me, a little crunchy, like it's going over the edges. If you all hear something, please let us know. Yeah. Now so we can make it better. Yeah. Now I think it's time to get onto the meat, and this is Jim's expertise today. So, Jim, tell us all about it.

Jim 12:23

Uh, thanks, Wolf. So, like Wolf said, we're gonna be talking about containers. Uh, a little disclaimer here. This is not a step-by-step tutorial. Uh, we just can't do that in this environment. Um, this is more of a description of what containers are and what you can do with them, and a little bit of uh command line stuff mixed in. Um so be ready for that. Um, the if if we're gonna talk about containers, uh you know, obviously the first thing we have to do is explain what a container is. So let's get into that. Uh one more thing. I'm gonna be talking a lot about Docker because that's the container platform that I use, but a lot of what I talk about uh it is the same for all containers. Um uh so what is a container? Uh a container lets developers package their applications along with anything they need libraries, uh utilities, all all the things, all the dependencies for your application. It allows you to package it into a a format, um a blob, let's call it, uh, that uh can be run on uh on your machine or on any machine. Um it's it's kind of neat because you don't have to worry anymore about uh uh differences in environment. Uh you you've got everything you need to run that application uh right there. Um containers can have their own IP address. Uh uh they are completely isolated from the host OS. Uh they just sort of run on their own on your on your host, and and it's really neat. And they've been around for quite a while. Um I'll get more into that in a little bit. Uh, but let's say you want to run a service uh on a system, you want to run something like PostFix uh to handle uh your mail, your mail transfer agent. Uh let's say you're doing Postfix, it listens on port 25. Um you can install that right on your host, and that's fine. But uh I think a better way to do that is to build a container and run PostFix in that container. Now that Postfix mail client or mail server is completely isolated from the rest of the system. Uh if anything bad happens in that container, it's isolated to that container. Uh and when you're exposing a service on the internet, uh, you know, Postfix runs on port 25. If you're if you're exposing that to the internet, bad things could happen. Uh uh I I like Postfix. I I think it's uh it's a very nice secure thing. Uh but running it inside a container is a great idea.

Wolf 15:11

Could happen.

Jim 15:13

You said could. Yeah, yeah, right. Right. It does happen. Uh so if you can isolate that uh uh away from everything else on the system, um it's an attack vector that if somebody gets through, they didn't get much, right? They didn't get they didn't get access to your entire system. They're stuck inside this container, they can't do anything else. Uh so that's a a reason you might want to run a container.

Wolf 15:42

Um uh so So my question immediately is oh okay, isolation, but uh in general, modulo isolation, why would you want this?

Jim 15:58

Well, like like I started to mention, consistency. Uh you can build up this environment, uh stick it into a container, and now that container can be run on your development machine, it can be run on your your testing machine, your your staging machine, and finally in production, the same container can run in all those places. So you're you're rolling something out to production that you have presumably tested really well, and all the all the libraries are are there and and it everything's nice and consistent. Um you can deploy this thing on on hundreds or thousands of nodes, and it's the same thing everywhere. Uh same version of everything.

Wolf 16:42

So uh maybe this question is nonsensical, but uh what if uh the production environment is big endian and uh my development environment is little endian?

Jim 16:57

Multiple architecture. Uh I I cover that a little bit later. It's it's fascinating. I when I learned that, I was like, whoa, that's uh that's a superpower right there. So we'll get the multi-architecture support. Yeah, don't jump ahead for me. 32-bit, 64-bit CPU, right, instruction set, all of that. We can handle that. Um portability. Uh like I said, you can run it on your uh as your dev environment or your production environment or anything in between. You can also run it on your desktop, your laptop, uh, or or a server, or up in the cloud. Same image can run everywhere. Um it's really cool. And uh the the third reason you might want to do this is scalability. Uh you can set up your containers to uh uh uh start as many as you need. You know, if you're running PostFix, you probably don't need a lot of uh uh servers running PostFix, right? It's it's handling mail. You know, if if you've if you're running your own mail server, if you're Google handling mail, that's a different story. But if you're running your own mail server, um uh you you need one instance, right? But let's say you're running some big uh number crunching intense application and you're having uh thousands or tens of thousands or even millions of users hit it, you want to scale that up. You want to run more instances of the same thing and maybe have a load balancer in front of it so that uh all the requests come in and they get pushed out to the different uh instances of your container. Docker and containers handle that beautifully, and and we'll talk more about that uh as we go. So, does that answer your question?

Wolf 18:44

It does. I wa I was concerned. I I absolutely like you said something, uh and I know you're gonna talk about this a little later, so uh, I don't want to ask any questions now, but I'm very curious about the case of actually doing your development inside one of these.

Jim 19:03

Yeah, I do that, and it's neat. Um so let's talk a little bit about uh you may be thinking uh a lot of these things we could do before with virtual machines, right? What's the difference between a container and a virtual machine? Well, a virtual machine is running a full operating system. There's a kernel there, there's all the libraries, there's all the daemons, there's all the utilities, there's everything. Uh it's a fairly heavyweight thing uh running in a hypervisor. And I I've been using virtual machines for for gosh, since the late 90s. Uh in fact, virtualization has existed since what the 60s with IBM uh 360s. They were virtual.

Wolf 19:47

I'm falling behind, but you said a word that makes me feel underinformed. Yeah, um, other people have said this word. Um I'm just gonna ask, could you go into a little depth? What what the hell does hypervisor mean? What is that?

Jim 20:02

That's uh well, VMware is a hypervisor, virtual box is a hypervisor, um uh XCPNG uh hypervisor. That's the thing that runs your virtual machine and it interacts with the CPU to uh deal with you know the CPUs now they have virtualization built in, but somebody has to manage that. That's the hypervisor. It's typically a stripped-down uh OS. Uh many times uh it's it's Linux. I think VMware is still using uh some version of Linux. Um XCPNG um is using Linux. Um it's just a very stripped-down version of the OS that really just um uh handles the managing of all those virtual machines, starting, stopping, task switching, all that kind of stuff. So it's sort of a common thing. Different than what I was thinking. What what did you think?

Wolf 21:02

Um I thought it was something in the CPU that okay, the thing I'm gonna say now is obviously false, and if I had thought about it at all, I would have known I was an idiot. But when when that somebody said hypervisor, I thought, oh, that's some you know mode in the CPU that controls addressing or something like that.

Jim 21:28

Um Yeah. Yeah, that's uh that's that's the whole virtual address translation tables and all that kind of stuff.

Wolf 21:35

It's almost completely unrelated.

Jim 21:38

Yeah, right. Although it's still important. Um but you know the CPUs have had virtualization uh extensions since the 386, you know, the CPUs that we use. Um the the Big Iron have had it much longer than that. Um so yeah, uh a virtual machine is that. It's a full operating system running uh as a as a thing uh on an on a host uh you know containers uh are a thing running on a host but they're they're much lighter weight um they they they share the a container will share this the kernel with the host uh operating system so uh you don't need to fire up a new kernel to do this you you know if you have uh 50 containers you don't have 50 kernels running you have one and and it's sort of sharing its itself with all of your containers um and it does that with the weird memory stuff that i just said it does yeah that the virtual memory manager is important in this case oh good i'm glad i was at least in the ballpark yeah yeah yeah i was selling hot doors uh i still don't know how either one of them really work but so so yeah um so uh that that that's kind of what virtual machines are uh let's talk a little bit about how containers work i um i did a little digging here i i've been using containers for a long time and i have a pretty good idea how to how to make it do what i need to do but i hadn't really dug into really what makes them work um and a container um it it's built upon several features of the Linux kernel uh there's something called cgroups or control groups uh something called namespaces and then there's bind mounts so we'll talk about each of those uh cgroups is a Linux feature uh that organizes processes into groups uh so that you can manage limit and monitor their consumption of resources like CPU memory disk IO um thing about C groups is um your container running in its own C group has its own idea of what process ID one is because you know in a Linux Unix system process ID one is important that's generally the init program or uh uh systemd um that that sort of handles all the other running of all the other processes uh inside of a container each one's got its own process ID one um it you know it gets its own process table it gets its uh its own uh memory its own disk IO uh and that's all through c groups uh the next thing is namespaces and uh before I describe that I'll uh uh uh you know I talk once in a while about how uh I I I worked on a project I created this project called LTSP Linux Terminal Server Project Thin Clients on Linux and way back in the beginning 1999 I think before we even gave it a name uh LTSP I I used to go to the Ottawa Linux Symposium in Ottawa Canada and uh my buddy Ron and I uh we both created LTSP uh we went there and the Ottawa Linux Symposium is uh a bunch of Linux kernel people there's I I don't know there's probably 200 people there uh giving talks on kernel things and this is you know 1999 the the current Linux was still fairly young um Linus Torvalds was there and and Ted Cho and Steven Tweety and all the guys that are really into the kernel the guys that really make it all work were there and Alan Cox was there and he was uh uh sort of uh Linus's uh uh right hand man uh for maintaining uh stable kernels I I think it was the stable kernels uh he was he was kind of like the uh uh release manager for the kernels he you know Linus would sort of decide what goes into a kernel what you know what what patches go in and stuff and Alan was the next guy in charge and I happened to talk to him I like you know the thing about conferences the best part of the conference is the hallway talk so we happened to see him in the hallway and Ron and I walked up to we started talking to him I had some question that about network booting I don't remember what the question was uh but I had asked Alan what he was working on and he said he was working on namespaces in the kernel and of course I I didn't know what he was talking about and I didn't understand the value of it this is 1999 he was doing the work uh to put namespaces into the kernel uh and it it really wasn't until eight years ago before I understood what what namespaces were in the kernel. But anyway namespaces are are a key feature of the Linux kernel that make this possible. What they do is they they partition global system resources uh providing each group of processes with its own isolated view of those resources uh it gives you isolation uh so that processes from one namespace can't can only see processes in that namespace they can't see any other processes from any other namespaces or the host uh there's several types of namespaces there's mount uh that's you know file system stuff or processes IDs uh there's networking uh IPC interprocess communication stuff uh uh UTS which I didn't really understand this one uh until I read a little bit more about it it's isolated host name and domain name allowing each of the namespaces to have its own host name and I thought well host name that's an I you know that's a DNS thing and I you know a TCP IP thing but actually the Linux kernel has a concept of what its host name is um so what happens if you have lots of uh containers running all sharing the same kernel well because of namespaces they can each have their own host name so that's kind of neat uh uh other things users uh the whole you know user ID thing uh what's the what's the root user ID user zero well yeah you can have a user zero in the container that has nothing to do with the user ID zero in the host uh you know because you need the kernel treats user ID uh zero specially it's got like permission to do anything um uh so containers can have their own user ID that's isolated from the rest uh control groups like I mentioned you can have control groups inside of the container that don't know anything about other control groups and finally time each uh uh kernel or uh each uh uh container uh has its own concept of time when it was booted uh what time it is currently what time zone it's in those are all unique to that uh container so you could have one container running in the Eastern Standard time zone you could have another one running in uh Pacific time zone and it's completely independent of each other and of the host so the third feature that uh that that the Linux kernel provides is this idea of bind mounts and that's a feature where you can map specific files or directories on the host uh directly into a container uh or into another directory tree you it doesn't you don't have to have a container to do this but bind mounts mounts make containers possible um it's not like an NFS mount uh where it's a remote file system it's actually the same file system uh or a piece of the file system that you're making available inside the container like a hard link um yeah it's like a hard link or a sim link um but it's uh each kernel has their own view of it um and you can control that through through the bind mount command uh it's a mount uh I forget what the argument is there's an argument to mount that that you can do that anyway the kernel platform or the uh container platform will will take care of that for you we'll talk a little bit more about that in a little bit so um it's uh all of what basically what I'm getting out of this is that what enables the containers as we're describing them here is a Linux kernel which they all share and which they have easy access to because the host has one um but what if the host isn't a Linux box what if I'm trying to run it on my Mac what if I want this on a Windows machine um how how does this even work on those devices? Well it it clearly works because we're both doing it right you're running uh Docker on Windows aren't you for your for your day job I mean yeah I am yeah yeah reluctantly but you have to uh and I'm running Docker on my Mac uh uh for development and like I said the container shares the kernel of the host well you can't do that when the host is a Mac or Windows uh there's no way that a Linux process can use the the the Mac kernel to to do anything everything's different uh so what they do is is uh your your container platform uh on those machines sets up a v a virtual machine running a Linux kernel uh it's a very very stripped down it's not a Linux distribution that's running in the VM it's just the kernel and it makes that kernel available to the containers that run uh that that's how you're you're accessing a Linux kernel from your container it's and so that is why bind mounts don't work on Mac which you'll talk about later but still yeah makes me so angry I can't stop myself from mentioning it. Angry but it's it's nobody's fault right it's it's uh we'll get into it in a little bit. Um so basically if you want to run uh a uh a Linux container on a Mac desktop or laptop or on a Windows machine there's a Linux kernel running in a VM uh making that happen. And it's it's seamless. You don't even realize it really it's it's it's pretty neat. There is the concept or the ability to run a Mac container a Mac application in a container on a Mac system. And in that case you're sharing the kernel of the Mac you're not you're not running a Linux kernel in the middle there somewhere and you're not running a VM uh with a kernel in it. You're just working direct. The same capability exists under Windows you can have a Windows container uh running on a Windows machine uh what I haven't seen yet is the ability to run a Windows application in a container on a Linux server or or the same for a Mac. I don't think you can run a Mac app on a Linux server.

Wolf 33:23

Maybe somebody's made that happen but I don't know Linux on Linux is just containers. Mac on Mac or Windows on Windows um is that so you used the word Docker before Docker's uh an application slash system slash facility that gives you access to these containers but most of the machinery of containers is built into Linux. Is it still Docker to give you Mac on Mac?

Jim 33:56

No Docker as far as I know Docker is going to have Linux in there someplace. Apple has Apple containers if you want to run a uh an Apple Mac on an Apple machine I believe Apple containers is going to be involved in there. And Windows has its own container thing container platform for running Windows apps on Windows systems. And we we keep throwing around the word Docker. There are other things that you can use besides Docker. Like I said earlier Docker's the one that I use so that's the one I'm gonna talk about uh there are others are are you gonna tell us about the others? I am I am I've got a list of them coming up uh so let's talk about what you can run inside a container what you can do with a container um they're great for running services like I talked about uh postfix uh you can run that in a container you can run uh a web server whether it's Apache or Nginx or whatever your web server is you can easily run that inside of a container you can run applications maybe you've got a a Python Flask app or a Node.js app or or really any kind of app you can think about you could run that in a container um it just sits there and runs and you connect up you know I'm talking about like network accessible services right um it just runs and you connect to it and it it just works. The user has no idea that it's in a container it's just there um but you can also use a container to run just a single utility I had a problem several years ago uh you know we use uh uh an application called WKHTML to pdf it's this thing that lets you convert uh HTML pages to pdf and it it's kind of marvelous how it works uh and I kind of hate it because the the output isn't usually isn't what I want because HTML pages don't often map to a PDF page you know there for instance there's no instruction in HTML to do a page break. Um it it it's a little bit awkward but it's necessary for some of the stuff that we do. So at one point we were running Ubuntu 2004 and we upgraded the client to 2204 and you know we we were ready to go live but during testing we realized uh WK HTML to PDF hadn't been packaged for uh Ubuntu 2204 yet so now we're stuck what are we not do the upgrade or figure out a solution so what I did was I took uh that application I built a Docker container uh including that app and we run it and that app doesn't run as a service it's a you run it and it's done you run it again you know and it's done so we created a container for that and I create a little shell script that uh and it was called WK HTML to pdf I put it in user local bin and when you run that it just runs my container uh I send standard into it it spits standard out back and it worked perfectly so now I had a uh the version of WK HTML to PDF uh for Ubuntu 2004 running on an Ubuntu 22.04 in fact I could take that same container and I could run it like on an Ubuntu 16.04 if I wanted or now uh 2404 it would be the same container it would run everywhere uh so it that got me out of a bind and it's not that uncommon for people to do that it's it's really kind of neat I I that seems like a really solution that seems like a really good solution um the and that it makes me feel kind of guilty to go backwards in time to the point where you said HTML has no page break um I I really thought it did I could be wrong it's been a while but there's some tricks you can play with CSS to kind of make it happen but it's it's difficult. And you know what is a page size on an A HTML page you know how does eight and a half by eleven map to an HTML page I see what you're saying yeah it it doesn't okay uh or A4 if if if you're from that side of the pond right uh it it it doesn't really fit um all right interruption over sorry that's fine I I like the interruptions you you are the audience you're asking for them right all right so we've been talking about Docker uh and you even asked uh are there other things besides Docker and yeah there are uh Docker's the one I use it's sort of the the the big name in town uh for this uh it was created in uh 2013 so it's like 13 years ago now it's it's gotten really mature and it's really cool. Uh there's another one called Podman and I've heard a lot of people talk about this and uh when you use Docker there's this background daemon running that sort of manages the running of your containers um you start and stop your containers and and uh the the Docker engine is running in the background orchestrating the whole thing Podman doesn't have a daemon podman is a command line thing that you use to start a container and it starts and then you kill the container it's not running anymore it's sort of a lightweight version of Docker um and it's pretty neat it was created by Red Hat um and they use it and it's really cool. I played with it a little bit and and it's neat um and I also mentioned Apple containers that's a real new one they announced that at 2025 wwwdc that was back last June so it hasn't been around all that long and what's what I found interesting about that is uh you know Apple is fairly closed um but their container thing is on GitHub if you want to install Apple containers you fetch it from GitHub to run it and I and I thought that was pretty neat. So I installed it and I and I ran it and it's it's neat and it's kind of a almost one for one uh replacement for Docker. A lot of the commands have the same arguments a lot of the whatever you learn in Docker uh you can carry over to Apple containers it's pretty cool uh I I ran it um and and then there's some hybrid things that that can run containers and they can also do other things like uh LXD uh it's actually pronounced legs um that was something created by Canonical for Ubuntu uh and they created that back in 2008 so that predates uh Docker by five years uh but it it runs uh it uh lexdy can run virtual machines and containers um so you can run a full operating system in Lexty or you can just run a single simple container uh another one uh uh is Incas and Incas was forked from Lexty in fact uh it was forked by a couple of people one of them uh we know uh Stefan Graber uh he used to work for Canonical and in fact he was working heavily on Lexdy uh when I guess canonical decided to take a different direction and they laid off all uh many of their uh the the team including Stefan and so he left and they forked Lexi and created Incas and it's the same kind of thing it can run a complete uh virtual machine or it can run just a container um but it it's out there and and and it works I played with it a little bit and I and I be before this during my research I talked to Stefan about it and he sort of explained the the purpose of it and the fact that you can really run full virtual machines with it. And it's kind of neat. And then there's a couple of things that I I thought were containers but they're not they're they're sort of like containers they share some of the same attributes uh and that is uh Ubuntu snaps Ubuntu seems to have sort of gone crazy with their snaps and it's a packaging method to package up the the application and all the libraries into a blob uh and and then you run that uh only it's not really a container um it it seemed interesting you know you you you don't have to deal with the dependency issues and it's kind of neat and it's working really well for Ubuntu but I think it's kind of strictly Ubuntu it's something that they created I I think there are uh snap uh um things for other OSs but I don't know if it's really caught on all that well and then there's flat pack uh which again is is uh a blob containing um uh your libraries and your applications uh and they're they're closer to a container because they do use c groups and namespaces and bind mounts but somehow they also share things between uh the flat back uh flat pack apps uh I don't really know how they do that but uh oh and one more that I I forgot to mention in the hybrid is Proxmox. Uh I know a lot of people are using Proxmox for their virtualization virtualization. Um you know especially since uh uh uh VMware uh got bought out by Qualcomm and kind of angered everybody with their licensing changes and stuff so people were looking for other solutions a lot of people went to Proxmox that can run full virtual machines and it can run uh uh containers so that's that's kind of neat all right so there's a zillion different ones and whenever there's a zillion of something

Wolf 44:00

Um a thing that's important is how how hard is it for me to go from one to the other? Um, and a thing that makes that easier or harder, depending on the quality, is uh are there standards?

Jim 44:15

Well, that's the neat thing. There is, and there has been for quite some time. Uh, there's this organization uh called the Open Container Initiative, OCI. Uh, and it was established in 2015, so that's like 11 years old now, and it defines the format of these container images. And all the ones that I mentioned earlier, they can all uh they all do use OCI compliant images, which means uh I can create my image using the Docker build tool, and I can run it with Apple Containers uh container platform, which which is just amazing. It's it's it's beautiful. So now, you know, you might be familiar now with Docker, but later on you want to move to something else, all your containers will move. The technologies are all the same, it's just the the tools you use uh to to run them, to start, stop, and even build. Um, it's a different set of tools, but it produces the same thing. It's it's pretty neat. Um when I yeah, go ahead.

Wolf 45:21

Well, you just used a word that you had not used before. Okay, and because I happen to know something about containers already, I'm gonna ask a question about that word. Yeah. Uh and I think it leads into uh a thing that you're gonna want to talk about. You use the word you have been saying containers all along, and then suddenly you use the word image.

Jim 45:48

Image, yes.

Wolf 45:50

And the very next question I was gonna ask you, even before you said the word image, was you keep talking about these containers. How do you get one? How do you put stuff in it, like my post fixes in my container? How does that happen? And now you said images. Can you put that all together? Tell me.

Jim 46:10

Yeah, I can try. I I I think of it an image is the think of it as like the file that contains everything. It's not running, it's just the thing that that contains everything. You build an image, a container image, okay? And then a container, a running container, um, that's where you can have many, many instances of the same image running. Um it so for instance the terms are f are are are used almost interchangeably.

Wolf 46:42

But like an image is like the binary on your disk if you're talking about a program. Think of it like that as executable.

Jim 46:49

It's it's more complex than that, but think of it as if you wanted to really simplify it, yeah, it's the binary thing that's sitting there, right? And it becomes when you run it in memory. I think uh the way to describe it is the container is the running instance of that image. Does that make sense? I guess it's sort of like an object in a class, right?

Wolf 47:12

Yeah, and just like on a Linux or Unix system, if you have some binary like LS or something, um eighteen different people, uh uh logins, accounts, shells, terminals, whatever, can run LS at the exact same moment. So one binary, or yeah, I guess it's a built-in, so maybe it's not the perfect example, but 20 different images in memory of it going.

Jim 47:41

10 people or 20 people running Vim on the same server.

Wolf 47:44

Yeah. Yeah.

Jim 47:46

Okay. That's that's yeah, think of you know Vim as the image and the running instances of it as the container.

Wolf 47:55

Okay, and so now you've moved the problem down one level. We were at how do you get a container? Oh, you run an image. Now the problem is how do you how do you get an image?

Jim 48:10

Okay, we're gonna we're gonna do that now. Um now this is an audio only podcast. I'm gonna be saying some commands. Um I don't expect you to remember them. I don't want you to stop and write them down. Uh, I I will include these commands in the show notes so that uh you have them to look at later. And uh there's only a handful of commands I'm gonna be talking about. But to build a container image, uh we're gonna talk about how to do it with Docker, because it's the only one I've ever actually used to build an image. Um uh and by the way, if if we were a video format or or something, we could talk a lot more about this. And uh in April, uh April 14th, uh, we're talking 2026 this year, uh, in case you're listening to this epic episode far in the future, uh, April 14th at 6:30 p.m. at the Michigan Unix Users Group meeting, we're gonna be talking about containers. Uh, and the neat thing about that is it's over Zoom and we can show examples and share screens and stuff. So if you really want to get further into this, uh show up at our mug meeting. It's free, they're always free. Uh it's on Zoom. And uh you can get more details on the mug.org website. Uh the details are not there right now because it's too early, but uh we will be doing that. And also the we record those on for YouTube, so you'll be able to watch the video later. It's gonna cover all the stuff we're talking about, probably in greater depth. So I I suggest you do that. But to build an image, you start with something called a Docker file. It's just a flat file. Uh you edit it with your favorite editor. And the simplest container I could come up with, or the sim simplest Docker file I could come up with, is simply one line from Ubuntu. Um I could say from Red Hat. Uh I don't know if it would just be Red Hat or R H E L or uh from um um uh I've I've forgotten the name of the what's the Red Hat free version? Uh uh I'll think of it later, right? Um uh so anyway, uh I have in mind from Ubuntu, and that's it. And I build my image by saying uh Docker build x build dash dash tag my container and then the directory where the where the Docker file is. So in my case it's dot because it's in my current directory. Okay, you run that and you have just built a Docker image. It's that easy, and and that will build pretty quickly. Uh it'll fetch from the internet the Ubuntu Docker image and suck it down and and build your image on top of that. So your image is based on on that image. And that image, when I say it comes from the internet, it actually comes from a place called Docker Hub. That's this repository of all kinds of images. There's everything out there. You can pull down Apache, an Apache image, you can pull down oh so many things.

Wolf 51:22

Uh so anyway, this I just this gives me a horrible, horrible question. A question I'm I'm so sorry to ask you. Um how did they make the Ubuntu image? Or let me turn it around. What if I want to make a brand new image that is the root? A brand new image from nothing? That's what I want.

Jim 51:46

Uh I don't know. I don't know. Certainly you can do it. Somebody did it. I mean, it's it sounds like a chicken and egg problem, right? How did Ubuntu do it? I don't know.

Wolf 51:58

Um maybe you can answer that in feedback next next time.

Jim 52:02

Maybe, yeah. I'll I'll I'll I'll do the research so you don't have to, right? Uh yeah. So uh I use from Ubuntu. A really popular one uh that you can base your images on is Alpine Linux. And Alpine is a very, very small image, it's a kernel with uh it uses BusyBox. I don't do you know what BusyBox is? It's uh I do because I use Alpine all the time. Yeah, it's a it's a single binary with a whole bunch of hard links um to to uh various entry points within the binary. So you want LS and CD and a in a simple shell, and it's all in this thing. Anyway, an Alpine image is something like five megabytes in size, so it's nice and small. I think an Ubuntu image is probably gonna be closer to 80 to 100 megabytes, so it it's somewhat bigger. Um uh so yeah, you can use uh you can use Alpine if you want. I use Ubuntu because I'm so comfortable with the packaging. I know the apt command, I know how to install things. So uh we built it using the build x command, the docker build x command, right? That created the image, and it did it in just a few seconds. Now you can run it. So you here here's how you run it Docker space run dash it my container slash bin slash bash. And what's gonna happen is it's gonna start up that container and you're in the container. You're actually at a shell prompt in the container, and you'll notice that your your uh shell prompt changed. It'll include like the the uh uh an MD5 uh hash, uh just a short little one uh of the uh container image. Um and you're there, you're running uh you're running a bash shell in in um basically in Ubuntu. Even if you were on a Mac, you could do the commands I just said, and you'd be sitting at a Linux kernel um uh Linux shell. And it's kind of neat. You can do anything you want in that shell, uh, including install more things. So while you're while you're there, you can type uh apt install uh Vim and install that, and it'll suck in all the dependencies and stuff. You can do all those kinds of things, right? Um when I ran it, I said uh uh it's Docker space run spash space dash it. Um the IT parameter, the the I will make the container interactive, and the T will allocate a pseudo TTY, which you need if you're gonna run a shell, because you you need to interact with it and you need a TTY to do it. If you don't add those two commands, you you're gonna end up with a running container that you can't do anything in. It won't respond to your keystrokes or anything like that.

Wolf 54:59

Um I I need you to tell the bad news though.

Jim 55:03

Yes.

Wolf 55:06

You install stuff.

Jim 55:07

You installed stuff, yeah.

Wolf 55:09

And then what happens if you terminate that image that container?

Jim 55:17

Um it's really pretty cool, okay? Because if you just uh if you do that container that I said, you you know, you run bin bash uh and you're at a shell. And if you hit control D or type in exit, you're out of it. And the container stops running. But you haven't lost anything that you did inside that. It created a virtual disk for you or virtual file system. You haven't lost that. If you run that command again, if you run the docker run uh dash IT my container bin bash, whatever you had installed earlier is still there.

Wolf 55:53

It's wait, what?

Jim 55:55

Yeah, it's still there, okay? It's not until you do the Docker RM command or remove command that it blows that away.

Wolf 56:05

I think I don't believe you. Okay. Here's what I think, and you tell me where the thing I'm thinking disagrees with the thing you're describing. Because maybe you're saying a thing that does work and I'm not getting it. Yeah. I think if you ask Docker to run an image, you get a container with like it's brand new. But if you ask if you ask Docker to run that image again, you get another one that's brand new and it's different. But if you ask Docker to rerun, to restart, and you have to say it a different way.

Jim 56:50

Yeah, I think, okay. You wouldn't say run that time, you'd say start Docker start and then give it the the instance name. Okay, we're gonna talk about that in a second. But you can tell it to restart that instance and and tell it to run bin bash, and whatever work you had done previously is still there. If you'd installed packages and stuff, it's all still there.

Wolf 57:12

So, in other words, it's exactly like uh, you know, except it's sort of a different world, but it's exactly like working on a detached head in Git.

Jim 57:24

Yeah, I guess you would know. Like you start talking detached head, and I start thinking zombie. No, uh, I I guess it's the same. Uh to me, it's more like if you had uh when you exit, it it shuts down the system, and when you start it, it starts up the system, and everything you you wanted is still there, and it'll stay there until you do Docker remove or Docker Docker container remove, Docker container RM. And then you give it that instance name, and that will throw away that virtual disk that had all those changes. Okay, so then the next time you do it, you can't do a Docker start again because it's not there. You'd have to do a Docker run to create a new instance, a new run, a new container. Okay. Uh so that image that we created isn't all that useful. It's it's it's something, you know, it's an educational thing. You could play with it, and it's kind of neat. Uh, but if you want to create an image that does a little more, it's really simple. You you you just edit that Docker file and you add another couple of lines. You add a line, uh, and here it is. I'll I'll just write it out or say it to you. The line you would add would be run apt update ampersand ampersand apt install minus y appache2. Okay, so now we've just told it to install the Apache 2 application. And apt is gonna install all the dependencies that it needs. Uh, and then finally you need to set an entry point. So you create a line, uh, the the the last line of the file. It doesn't really have to be the last line, but our file's only gonna have three lines, right? And the entry point is uh it's actually using uh JSON syntax, the word is entry point, and then there's this JSON thing that's just an array of the arguments, uh, the first one being the name of the com the applications. So if you're gonna run Apache, uh you're gonna use Apache CTL. And then you're gonna pass dash D foreground dash K start, that's gonna cause uh uh Apache to start up when you run that uh container. Uh you notice I added dash d foreground because uh the container stops when the command that was running stops. And if you start Apache, the normal way you'd start Apache is it would start in the background. So the command that started Apache stops and then your container stops. So you're gonna be wondering why you keep starting this container and then it's not running. Uh, it's because you want to do dash D foreground to make it run in the foreground. Um Apache by itself isn't terribly useful uh if you didn't give it any network access, right? What good is a web server if you can't point your web browser at it? So to run this container, you're gonna run this command. You're gonna say Docker space run space dash p uh and then give it a port number. Uh 8080 is a good one if you're playing around, right? 8080 colon 80. And what that's saying is port 8080 on my host, it's gonna map to port 80 in the container, and then you're gonna add another option that I haven't mentioned yet, and that is dash D. That's that's for detach. And then you're gonna name your container, my container, right? Um, that's gonna run in the background Apache, listening on port 8080 on the host. So now go point your web browser at localhost, uh, assuming you're doing this in your desktop, put your web browser at localhost uh colon 8080, and you're gonna get the default Apache homepage. Apache running. It's pretty cool. You can run as many instances of that as you want, just make sure you specify a different host port because you can't have two processes listening on the same port. Um so now you've got a Apache running in a container. Uh, it's in the background. How do you deal with that? Uh well, there's a couple of commands. Docker container ls will show you all of the dock all the containers that are running. Uh, there's a shortened version of that, Docker PS. You run that and it'll show you the containers running. And one of the things that's pretty neat is it if you didn't give it a name, now I know you named the image. That's that's to tell it which container you want to run. Uh, but if you didn't actually give your container a name to refer to it later by, uh I'm probably confusing some people. We gave the image a name of my container, but we did not name the running instance of that container. If you don't give it a name, Docker will allocate one for you. And it and and they're always funny. Um when I ran it, um, I ended up with the name of youthful underscore hopper. And the way it derives these names, I've seen other things do this too. Um, it derives the names as um uh uh it's a randomly selected adjective, an underscore, and a notable scientist or hacker. So in this case, youthful is the adjective, and hopper being Grace Hopper, uh, that's the name it gave it. If I were to run another instance of it, it would give it some other completely random thing. And uh, you know, it might be uh uh fast Einstein or something like that, right? So the way to see what name it gave it is uh uh with Docker PS, and it'll show you that. Now if you know that name, you can do things. Uh one of the things you could do is uh stop it. You can say Docker uh container. Um I always get this confused whether the the stop comes first and then the container name comes last. I think it's the other way around. I think you'd say Docker container youthful hopper stop, and it'll stop that container. Um another thing you can do is is uh actually uh get a shell in that running container. So you can do uh Docker space exec space dash IT space uh the name of that instance, uh in my case, youthful hopper space slash bin slash bash. I've now got a shell on that running container. So while it's running, you can you can do a ps command inside the container, and you can see there's only a couple of processes running. There's your shell, and and then process one is going to be um uh Apache CTL. And then there's gonna be all the you know the the Apache process itself. I think it's HTTPD, and then usually Apache will fire off like 10 instances of of the daemon, so you'll see those processes, and that's it. Uh and and you happen to be root inside that container, even though you're not rooting.

Wolf 1:04:27

There is actually a really awesome container command. Um so let's say you have made um a container and you've installed a ton of things in it, and you really like this, and you want to make others, you can take a running container and you can save it into a brand new image that is the updated version that contains all your changes.

Jim 1:04:56

Yeah, and and that's pretty neat. Well, um I don't like doing that because that's not repeatable. You know, you no longer have the Docker file that generated that. You you had a Docker file that generated your base image, and then you started modifying the base image, and now where are you? You can't really recreate that later. You can just keep rerunning that that image if you want, and it's pretty neat. Uh, there's also tools for copying things in and out of a container. Uh there's Docker Inspect that lets you inspect the container.

Wolf 1:05:32

Um and there's multi-stage. When you uh build a Docker file that's complicated, it's not worth talking about here. But the point is you can make an image. Oh, we will. Oh, awesome. I won't spoil it then.

Jim 1:05:48

Yeah, yeah. So I I gave you a recipe for a really simple Docker image. Um and you know, we've been talking about image. And it's really a complex thing. It's not just an image, it's a series of layers, a whole bunch of layers that layer on top of each other. And then there's a manifest that that tells what each of those layers is and how they fit into the hierarchy, which layers come first and second and third and so on. And every time you run a directive in the Docker file, the only ones you've seen are um from uh run and entry point. There's a lot of other containers, a lot of other uh directives. But some of those directives, every time you have one in your file, that creates a new layer. So if you have a whole bunch of runs, like let's say I want to I want to I have a run that does apt update, and then I have a run that installs Apache, and then I have a run that does an apt install something else. Uh one of the things I like to always install is Net tools. That gives me things like Netstat and and uh um uh ifconfig and I think the IP command and several things. I like to install that in my images. So every time you do this run command, it's another layer, and those layers add up. They make your image bigger and bigger. You can get bloat, especially if you do something like in in one layer, you do your apt update, and then you um uh something I I do, uh I like to build my own Postgres. So what that requires is I do an apt install of all the tools, the compiler, the you know, GCC and a whole bunch of libraries, and basically I install build essential, which gives me everything, gives me make and the compiler and all that kind of stuff. So if I do that in one layer and then I build Postgres in another layer, even if I apt purge and and apt remove a whole bunch of things, like all of build essential, if I remove those things, they're still in those layers. They get drug around from you know, every time you you use that image, all of those layers are there.

Wolf 1:08:04

And and it's just like it. Um all those commits in in history that had giant files you didn't mean to get.

Jim 1:08:12

They're all there. Uh yeah, so you want to be crafty in your Docker file. Uh, so you want to combine things on one run line, like the in the the example I gave. Uh uh apt update and and apt install minus y Apache 2. Um that combined two things on the same command line, so that's only one layer. You can you can use a backslash at the end to continue on to the next line, so you can chain these things together as long as you want. So it's all one layer for all that work you did. Um, and one of the really handy things, this is maybe it's advanced. I do a run with a here document. So you do run space less than less than some token and then greater than um actually you don't need the greater than on this, but uh then all of the lines you give after that are commands part of that run statement, and then you terminate with whatever your your token was. Let's say it's the word EOF or something. So all of those things run and they're part of that one layer. And that it turns out to be nice and readable, and I I really kind of like it.

Wolf 1:09:23

So it is a balance because layers are reusable, so you can have two different images that share layers up to point X, and then they add new stuff on top.

Jim 1:09:37

Yes. Oh gosh, there's so much you can do. We could talk for hours about this stuff. Um, you you you like in my case, I uh my company is called Averis, and I create an image called Averis Base, and that has the bare minimum stuff I need. The Ubuntu, uh, like I said, I like to install the net utils and a f and a handful of other things. So that's my base image. It's got the basic things that I need. And then I create another Docker file, and uh for base, I use uh for base, for the from line at the top of my file, I uh it says from a veris base. So I just built a new image based on my old image, and I've got this little hierarchy, like I've got uh um uh uh an image called uh a veris emr, which contains all the all the Perl stuff that I use. That you know, my my system is an electronic medical record system, and so I've got this image called Avarus EMR, and then I have an image called Avarus Web, which contains the the Apache, the web server, and a handful of other things. So it's this this built-up layers of things, and I try to keep each layer as small as I can uh so that I can reuse it. Because I've also got other images that that rely on the EMR image, and it and it works out really well. Um man, we could talk forever on this stuff. Uh, but like I said, whatever you do inside that container is gonna be persistent until you remove that container from from uh the system um from the Docker engine. And you do that with Docker uh container RM uh image name. So uh be careful of that. Don't go installing a bunch of stuff uh inside a uh like an ad hoc image. If you really need something, put it in your Docker file and build it. Then it's repeatable, and that's important. Uh and then that you know, save your Docker file in Git so that you've got a history of what you've been doing. Um yeah. So that's that's kind of how to build an image and and how you can run it.

Wolf 1:11:45

Um so that then makes me wonder, um, didn't didn't you I mean, the whole reason I feel like we got into this, and maybe it's not the reason that inspired you, but what put me down this path was uh us talking about you not getting the right thing on a Mac. You're using Docker, yeah, and it just wasn't doing the right thing.

Jim 1:12:14

Okay, so let's um let me explain what I do. I've got my application running in the cloud on Azure uh uh using Docker. Um I'm handling a couple of clients that way. I want I I decided to go Docker for this or containers for this because I didn't want to buy multiple virtual machines on Azure. I wanted to run it all in one machine. So I use containers so that I could separate my clients from each other. Uh I can have instances of my application running for each client uh with their own host name, and then I have uh my DNS set up to set up you know these different clients, giving them and their own version of your source.

Wolf 1:12:59

They get to pick which version.

Jim 1:13:00

Yeah, the their own version of it because uh you know uh when I do an upgrade, I don't upgrade everybody all at once. I stage it. I I you know I upgrade one, make sure it's okay, then I upgrade another, and I make sure it's okay. So they are separate instances all running on the same machine, and it and and containers work perfectly for that. But in development, I I used to do all my development on a Linux machine uh on a separate machine. I would SSH over to that machine. In fact, I get this big, beautiful Mac Studio machine, and all it was was a uh it ran a web browser for me and it ran an SSH client, and it was kind of a shame. So I I would SSH to a Linux box and I would do all my work. Um and I did that for years and years and years. Uh so recently I decided I want to take that local. I want to uh because I want to get into artificial intelligence. I want to start using Claude and I want to have the tools available to me that I can that I can use on my desktop. So I want to start doing my development on my desktop, but I don't want to pollute my desktop with all the things that I need, like all the Perl packages I have to install, and I have Node and all the things you have to install with npm and just everything, all the tools I install, all the libraries I install. I didn't want that polluting my Mac machine.

Wolf 1:14:17

I totally hear you. I'm a Python programmer, yeah, and uh Python is not on my Mac. I mean, I have it everywhere I want it. Every single virtual environment that I have, and that's what we use when you're developing Python programs. Yeah, everyone has Python in it. That's the right version for that project for what you're doing.

Jim 1:14:42

And uh and you use uh uh what do you use? UV in Python for your virtual environments.

Wolf 1:14:47

Oh my god, I love UV so much. I'm using I would marry it, I would have UV's baby.

Jim 1:14:56

I um I'm using containers that way. Only in my containers, I'm running more than the Perl code. I'm running JavaScript code, I'm running Apache, I'm running all kinds of things, and that's all running inside of a container. So here I am on a Mac, and I want to I want to build my system. Uh and I want to make sure it works exactly the same way in production, which is on an x86 server. Um so I learned something really, really cool. You can create multi-architecture images. Very, very powerful. When I learned this, uh it was only a few months ago that I learned this, and it's like, oh man, that's that's cool. Um, with one build command, I add the dash dash platform argument and I list the platforms I want to build for. So in in my case, I build for Linux slash AMD64 and Linux slash ARM64. I I list those both separated by a comma. And when I do the build command, it builds both. And and it's really cool. Even though I'm on an ARM uh you know, max silicon chip, it's building for both environments. It uses QE QEMU to build for uh the other architecture. Like I'm on an ARM chip, so it's gonna build the ARM image very, very quickly, but it's gonna use QEMU to build it for uh x86. And uh those aren't the only platforms you can build for. You can build it for PowerPC, you can build it for S390, uh, all using this this tool, and it's really neat. So I figured out how to do that. So I'm at my Mac and I'm building images to do that. And then it's like, well, gosh, can I do the same thing on Linux? So I go over to Linux and I take this, I I use a make file to run my build command. I just ran it, and it worked. It built it built the x86 image very quickly, and it used QEMU to build the ARM uh image. And and you know, I keep saying image, these are actually layers that are being built, and so you end up with this one multi-architecture image that's a bunch of layers, contains x86 layers and ARM layers, and it's and it's really neat. And so now I can build an image in one place and I can run it anywhere. I can run it on my desk, which my desk, which is an ARM chip, I can run it on the server, uh, which is an x86 chip, I can run it up in the cloud on Azure, and it same image works, and it's just fantastic and really, really useful. But I think what Wolf was alluding to was I ran into a problem. Now I don't include in when I'm doing development, I don't include the source code in that image. I set up my source code in a separate directory, and I I I fetch it with git, you know, I do git clone into a directory, and that's outside of my Docker container. And then I do something called a bind mount. I tell the Docker engine to mount that directory inside of my container when it's running. So now that directory is available to my running container, and it's really great. I can I can go in there and I can I can get a shell in that running container and I can see the source code. I can be outside of the container and I can interact with the source code, I can edit it. You know, my editor is running uh on the on the Mac uh on Mac OS. Uh and and if I make changes to it, they they show up uh inside the running container. Uh well they're supposed to. Uh and if I were on Linux, if I were on straight Linux, Linux host, Linux containers, everything works beautifully because it's all the same file system. What the client, what what the host sees, what the what the container sees, it's all the same. But when you're not on an x86 uh Linux server, things get a little bit messy. It still works, but it gets a little bit messy. You you can't share the file system. You have to use a remote file system. You don't use NFS, you use something called uh there's a fuse module, which surprised me. I didn't know that fuse worked in Mac OS, but somehow it it kind of does. Um, so that's how it bind mounts the the file system for you. The problem is it doesn't stay synchronized very well. So I make changes on on the host OS and they don't often show up in the container, or they will give it enough time. But you know, when you're in when you're in development mode, you're editing files, you're running them, you're editing more files, you're running them, you're compiling, you're doing all kinds of things. You want those changes to show up immediately, and they and they didn't. And I thought, oh man, I'm screwed. This is not gonna work. I did a whole bunch of digging around and I found out it's it's not a problem with my system, it's not a problem with the way I'm doing it, it's a problem with the bind mounts don't really exist on Mac. And they don't really exist on Windows either. So if I were using a Windows desktop, I'd have the same problem. Um, but then I found I'm using Docker Desktop. That's how I installed Docker on my machine. It's a commercial product. You can run you can install Docker other ways. There's an open source thing you can install to run Docker on a Mac, but there's a free version of Docker Desktop, and that that makes things really nice and easy. Well, it turns out that there's this synchronization thing, um, synchronized file shares that you only get if you pay for Docker Desktop. You can pay for the like the the the whatever the lowest tier is beyond free. You pay for that, and now you get synchronized file shares. It turns out that works perfectly. So I don't like the fact that I gotta pay like I don't know what I'm paying 19 bucks a month or something. That annoys me. But it works. So I I'll you know, I'll pay for something that works. So if you're if you're running on a Windows desktop or a macOS desktop, be aware that your your your mounted volumes may not work the way you want them to immediately. Um eventually they can work, but uh be be wary of that. I think is that what you were sort of getting at, Wolf?

Wolf 1:21:01

Um Well, yeah, and uh I'm a I'm a j I not I I like to think I'm a nice person. I like to think I'm gentle. I I like to think I don't lose my temper. But the fact that this is synchronization and not a hard link or symbolic link or whatever you want, the fact that it is two different file system objects, you know, foo.py on my host and a different bit set of bits, foo.py inside the container, the fact that those are two different sets of bits, I'm filled with rage. And then then and an insult to injury, like, yeah, they're different bits, but we can make that faster for money. Yeah, is there the best rage?

Jim 1:21:58

Because that's when so what you want is awesomeness for free, is what you're saying. Right?

Wolf 1:22:06

Well, then it's when I provide one ever thing giving it to me.

Jim 1:22:10

Do you? I never know. Maybe I mean they have to make money. And there's ways to work around it, but the w the the choice I made was to pay the 20 bucks a month uh to do uh Docker desktop and and make it work. If I were on a Linux uh desktop, that problem wouldn't exist, it just flat out works. Um so keep that in mind. If you're on a Mac, you've already paid a lot of money. Maybe 20 bucks a month isn't that much more, right? Um so anyway, um, I talked a little bit about the the layers and stuff, and when you create a multi-architecture image, um, you're creating lots of layers. Uh you're creating like twice as many layers because you got to create an x86 layer and an ARM layer for every layer. Um, so it sounds like your images get pretty big. And they can. And um, you know, earlier I mentioned this thing called Docker Hub. And you might be wondering what that is. Uh remember the from Ubuntu line that I put in my Docker file as the first line, it pulls down an image from Docker Hub. Well, your from line could say anything. It doesn't have to, it doesn't have to use the default of Docker Hub. You can list your your your registry name. So Docker Hub is a registry. Uh it's the default registry for Docker. Uh that's where all these images exist. Um but you can create your own uh container registry. Uh and I've done that. Um well I I do a couple of things. First of all, I don't uh I grab things from Docker Hub, but I don't push things to Docker Hub. I don't have an account on Docker Hub. I would have to pay if I wanted to push things to it. Uh and I didn't want to do that. Uh but I do have a GitHub account, and GitHub can act as a uh container registry. Uh GitHub packages, it turns out, is a perfect place to put your your containers. Uh so I do that for my production containers, I push them up to GitHub, and then up in Azure, when I run my my system, uh it pulls from GitHub, and and it works really quite well. Uh but you know when I have this multi-architecture image, it gets pretty big. Lots of layers. And if you uh the way you do it, you do a uh Docker push in order to push thing up something up to a registry. When you do the Docker push, all these layers take time. And over the internet, it can take time. Uh I'm not talking hours, I'm talking a few minutes. But when you're in development mode, a few minutes is forever. So what I did is I I run my own container registry locally on my home network. Um, and and it was really simple to set that up. You know, you know how I did it? I installed the registry container. I just I just did uh basically Docker Rud uh registry and it pulled down that image and and started it running. I had to add a few parameters in order to map uh some network ports and and file system space and stuff, but it's really pretty simple. So I've got my own local registry as well. So I have two registries, GitHub and my own uh registry on one of my machines here on my network. So I push my images there, and these multi-architecture images are quite large. Uh, but when I go to run an image somewhere, like if I go over to a Linux box and I tell it I want to run that image and pull it from my container registry, it only pulls the layers it needs. It doesn't pull the layers.

Wolf 1:25:44

Like you don't have to say something special.

Jim 1:25:46

I don't say anything. I say Docker run uh and I give it uh uh Cr.wdc.avaris.com. That's the name of my my registry. And I uh I think colon uh and then the container name. Um and it and it pulls it down from my container registry and it only pulls the layers it needs and it starts running. Now if I stop that container and I start another container running, or if I start multiple images, it cached all those layers, so it doesn't have to pull it down every time. Uh it'll actually make a check from the it'll check the container registry to see if anything's newer or not, and if and if it passes the right checksums and stuff, it doesn't pull down the layers because it's already got them cached locally, and it's really nice. And all those layers are shared between um additional containers that that have the same layers. So, like I said earlier, I create a a container that's got the Apache server, and that's a layer on top of all the other stuff that I have. So when I run that, it just pulls down the Apache layer. It doesn't have to pull down all the base layers because they're already there. And it's and it's pretty neat. So that's that's there for free. It's it's it's automatic, it's fantastic.

Wolf 1:27:01

Everything you want and nothing you don't want.

Jim 1:27:04

Right. And and um it just amazed me. It just it it really upped my game, I think. For now, I can develop on my desktop and I can deploy in production, and it's all the same thing. Remember years ago in like 1995 when when Sun came out with Java, they said write once, run everywhere. And it and it turned out what they really meant was uh write once, run nowhere. Um this uh I can write once. I can create this image once, and I can run it anywhere on any machine. As long as I built it for the right architectures, uh it it'll it'll do the right thing. Um so I I advise uh well you want to use a container, I think. Unless you're just sitting at your desktop and you're playing around and you create an image and you run it, and you create more images and you run them, you don't need a registry at all. But if you want to start sharing. Carrying images between machines. Okay. Yes, you can use it. There's a Docker command where you can export your image to a flat file, and then you can copy it to another system, and then you can import it into the Docker engine. You don't want to do that. Just use a container registry. Much, much easier.

Wolf 1:28:20

Okay. So yeah. I feel like you've you've talked about um individual images and uh building them and the places you can run them and how they can be the same and um how they can share. I I feel like I've got that part. But you intimated earlier, and you use the name of uh a particular app application. Well, maybe it's not an application, but tool um that makes multiple of these things work together, and you said some other things, and I want to know more now. The word you used was Kubernetes.

Jim 1:29:01

Um I was confused about what that was. I was using Docker, and everybody was saying, Oh, you should be using Kubernetes. And I'm thinking, instead, you know. Uh but a while ago I did some digging, and uh Kubernetes is a tool to basically deploy and manage your containers. Uh, before we really jump into that, let me tell you how I run my containers. There's a feature in Docker called Docker Compose. It's you create a YAML file. It's a file called uh Docker Compose dot yml. Maybe Y A M L. I don't remember. But you create this YAML. Another markup language. Yeah, another markup language. It's a it's a format, right? Like it could have been JSON, it could have been XML, but it's YAML. So in that file is just a uh you you you basically lay out your your dependency list. Like I run Postgres as a container, I run my EMR as a container, I run um Cups as a container for printing, I run uh cron because I have cron jobs to run. So these are all my my different images that I need to run. So I set up a container, uh a compose file, and in there I list the dependencies. So my first dependency I have to have before anything else will work is my Postgres database. So that that's listed first. It doesn't actually have to be at the top of the file, but that's listed. There's no dependencies for that. But then all the other things depend on that. So I say that in the in the uh compose file.

Wolf 1:30:40

Uh let me make sure I understand. Yeah. Your compose file isn't talking about Postgres the app, it's talking about your specific Postgres planned container. Yes. That's the thing that it knows about. Yep. I get it. Okay.

Jim 1:30:59

Yep. The Docker file is what defined what that is. The compose file is how we lay out all the dependencies of these different images and how to get them up and running. So you know, earlier I was saying this here's how you run. You do Docker space, run space dash IT space, image names, blah, blah, blah, blah. Right. You you do all that stuff in the compose file. There's separate fields for all those things. There's fields for what ports you want to map, there's fields for uh what volumes you want to mount, the fields for the name you want to give that container, um uh what I what host name you want to give it. All those things are in your Docker Compose file. So now you have this Docker Compose file, and it's really, really simple. You s you type Docker Compose up, and it and it starts everything in the right order. And it and it when that thing, you know, it starts up, and and now you've got your running complete stack application running. And it's it's really, really cool. And for my use, I I don't have uh you know tens of thousands of users hitting it. I don't need scalability. Uh so I just use Docker Compose. It's really quite nice. Now, if you take that further, like let's say you you do have uh an application that has uh lots and lots and lots of users where you want to scale, you wanna you want to automatically spin up additional instances of your application. Kubernetes is what you want.

Wolf 1:32:27

Or or high availability. You only have 50 users, right? But they've gotta have the service.

Jim 1:32:35

Yeah, right. And and the way Kubernetes works is it's not just running on one machine. You set up a a a whole bunch of nodes, a whole bunch of machines to run your application on. And Kubernetes tracks that it starts up more instances, it'll start up an instance on another machine because this machine's already too busy. Um it's fantastic. Um if I had the type of application that needed that, I would definitely be using it. And maybe you do. Um you know, use the tools that you need uh for your thing. Um it's really a neat thing. Uh and all the cloud service providers, they will run Kubernetes for you. Like for me, running my containers, I have a Linux VM running in Azure, and then there I install Docker, and then I have my Docker Compose file, and I do that that stuff. If you want to run Kubernetes, you just use the Azure or AWS or Google Cloud Services implementation of Kubernetes. You go in and in the setup screen in the web portal, you you define what you want to do, and it runs. You don't have to worry about it.

Wolf 1:33:48

Which they are happy to do for you because for them, it is a money faucet.

Jim 1:33:55

Yes, it is. Be careful because if you tell it, you are you know, if you tell it you're allowed to scale up to 10,000 nodes, you're gonna pay for those. You know, me, I I pretty well contain my costs because I have one simple virtual machine. Um and it's only gonna run what I tell it to run. It's not gonna spin up additional things. But you know, one of the examples I heard, which was really interesting, was uh like CNN on election night. They need more nodes than you can imagine for their website for election night. But a week later, they don't need that many nodes, so they just spin up the the instances of their their web application that they need at the time, and in fact it spins up automatically based on load. And when the load dies down, those instances die. Amazon at Christmas, Amazon at Christmas is another one, right? All these things uh where you might have spikes in your in your resource needs, Kubernetes and containers is the way to go. So yeah. So I I think I'm very near the end here. One of the things I want to point out is the the the idea of containers, uh, if you're running it the way I run it, uh and you I don't want to say you should run the same way I do, but one of the things you should keep in mind is one container should run one service. Don't have a whole bunch of services in your container, right? Keep them separate. So, like my containers, I've got a Postgres container, an HA proxy container, an Apache container, a post fix for doing mail, I've got a cron. I think I mentioned I've got a cups uh container. Those are all separate containers. I've got my isolation, they're easy to manage. If I need to change something, I can bring down that container and bring it back up, and it's it's really kind of beautiful. So uh yeah, that's what I do. You know, I thought this was gonna be a short episode every time, you know. We always think that. I'll I'll I'll uh I'll I'll let you guys in on a secret. We shoot for 50 minutes for our episodes. Five zero achieve that. Yeah, five-zero minutes. And here we are, a minute and thirty an hour and thirty-six minutes. We're way past. I'll do a little bit of editing, but I'm not gonna take much off. Uh so I get I guess the end has come. Let me get I just got a couple of takeaways. Uh, one of the things is pick the container platform you want to use. Uh my suggestion is uh pick one. If you're new to this, pick one that where you can get lots of help. I think I think Docker is a good first choice, it's a good starter platform because it's a lot of different things. Might not be where you end up. Yeah, you might not end up there, but if you're gonna get into this, start with something where where if you Google for an answer, or if you ask uh Claude for an answer, use the thing that everybody is using. Later on, when you've when you've been working on it for six months or a year or something, then you can switch to something else. If you want to go to Podband, great, it works perfectly. If you want to instead go to Incas or LexD or something else, switch to that. At least now you'll have a really good idea of how it all works. You've you you'll be well immersed into the technology. Then you can make the choices. Uh, so I think that's a good way to go. Uh, and the other thing is learn how to build the containers that you need to solve your problem, right? Learn about layers and how to how to make a small container and how to make it do exactly what you need to do, and how to separate your things, your problem into separate things and make those separate containers.

Wolf 1:37:41

Now you're talking my language.

Jim 1:37:43

Yeah.

Wolf 1:37:44

Say the words.

Jim 1:37:45

Say it. Say it. Put the ladder. What is that the uh make sure your ladder is leaning against the right wall? That's the words. That's them. Yeah, I don't think an episode has gone by where you didn't mention that one. Uh so anyway, as I mentioned way back, uh, we will be doing a live version of much of this information at our Michigan Unix Users Group meeting on Tuesday, April 14th at 6.30 p.m. It's free. Uh, there's no cost. You just show up, you go to the mug.org. That's m ug.org website. Uh, there's no information there now because uh we're in early March. But uh at some point. We are gonna link this in the show notes. I'll put a link to the show notes uh in the show notes to this. Uh if you wanna if you want to ask questions, the mug meetings are a great way to do that. I get to show examples, I get to run containers and share my screen and all that kind of stuff. So come on by. And if you miss it, uh let's say you're listening to this sometime in the future. Uh, this is me talking to future you. Um uh there'll be a YouTube video of that meeting so you can catch it.

Wolf 1:38:57

I don't want to scare you, Jim, but um they are listening in the future. That's that's yeah, no matter what this works.

Jim 1:39:06

Right. Every single person listening is in the future. Okay. We're just from the past. All right. So we are managing 40 minutes. I'm I'm yeah, I'm worn out.

Wolf 1:39:19

Wolf, take us out, would you? Um, I want to thank everybody for listening. Um, it is our pleasure to uh do this. We wouldn't do it if we didn't have people listening who told us that they got something out of it. If you got something out of it, or you think uh we need to change, or we made a mistake, or you just want to say hi, send us or you just want to say we're bonehead. Boneheads. Could be. We like feedback. We like feedback. Feedback at runtimearguments.fm. Uh in fact, runtimearguments.fm is our website as well. Uh that's HTTP, not HTTPS. Um, but whatever your podcast player is, it's gonna show you the show notes, which will include all of that. Um I guess that's gonna be it from me, Wolf, and uh Jim. You want to say goodbye? Anything last?

Jim 1:40:24

Yeah, goodbye, everybody. Thank you for uh sticking around this long. I realize this is a long one. This might be our longest yet. So uh thanks for sticking around, and I appreciate everybody who shows up and listens. And uh have a great day.

Wolf 1:40:38

It's funny because I'm the one who talks too much, and yet you're the one whose episodes go too long.

Jim 1:40:47

I'm the one that keeps complaining about how long the episodes are, and then mine are the longest. Anyway, goodbye, everybody. Bye bye.