27: SSH and how we got here

From rsh to certificates: The story of how we learned to log in safely.

SSH has been around for a long time, over 30 years in fact. In this episode we talk about what came before and why SSH is such a huge improvement. We talk about choices you can make in using SSH, such as choosing the right key algorithm and how to securely store those private keys and how to distribute the public keys.

Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm

Follow us on Mastodon: @RuntimeArguments@hachyderm.io

If you have feedback for us, please send it to feedback@RuntimeArguments.fm

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky

Transcript

Copy Transcript Copied

Wolf 0:05

Hey everybody, it is another episode of Runtime Arguments. I'm Wolf. Here's my best friend Jim. Say hi, Jim. Hi, Wolf. How are you doing? I'm doing good. Uh hi everybody. I guess we're gonna uh talk a little bit more about that in just a second. Before we even get into that, I wanna make sure it's not a surprise. Today, we're gonna talk about SSH, um, which is basically fundamentals to your life unless all you use is your phone. Um Jim, how was your week?

Jim 0:42

Uh, you know, uh the the the the past week was uh just crazy busy. I I I like that. Nothing really remarkable going on, just uh, you know, trying to keep customers happy. And uh and and that's working so far. How about you?

Wolf 0:59

Uh last episode uh when we did How Was Your Week, um, I said the best part of my week was yet to come, that I was gonna be in a competition. That's right. I was in that competition, and I went from unranked to ranked. Now, admittedly, I have the worst rank you can have, D, but the step from zero and all mathematicians will disagree with this, but the distance from zero to one is way bigger than the distance from one to two. So that little tiny step for me was gigantic.

Jim 1:39

All right, you're you're being a little bit ambiguous here, and I think I know why. But everybody's wondering, what are you talking about ranking? What what did you do?

Wolf 1:51

I am a so my hobby is uh competition shooting in the USPSA, that's the United States Practical Shooting Association. Um and I am, I've only been to three official matches so far. Um and the good thing is my accuracy is awesome. 82% alpha zero misses across the entire match. My speed is awful. So I am a D because it's uh total score divided by time. Um and uh coming up uh uh very soon, I am going to take uh uh eight-hour seminar uh along with a bunch of self-study that you have to do first to officially be qualified as a range officer, which lets me uh help ensure the safety of the people who are engaged in a match and uh make sure all rules are followed. Um it's a participation sport, so everybody who can do something can do some part, does. And uh there's a little side benefit, which is a typical match what might cost you $25. And if you're a range officer, uh you you could get a $10 discount typically. Score. So that's what I've been doing. Um I've been doing shooting for a very long time. I've only just started competing. I know it's very divisive and controversial. Lots of people think it's terrible, lots of people think it's fantastic. Um, and uh I am getting a tremendous amount of satisfaction from it. Okay, happy Yeah.

Jim 3:43

I mean I knew what it was, but uh you can't just leave it hanging out there, a competition without telling me. Well, I didn't want to scare people. Oh, sure. Sure.

Wolf 3:54

I don't think you did. We didn't get any feedback on our less last episode, which is kind of weird to me, because editors seems like the very most controversial topic we've ever touched. Uh except for one tiny thing, which I have noticed on Reddit, and that is the difference between spaces and tabs. Um If you make a post about that or or how big your tabs are, you are just asking for trouble. Um so no feedback.

Jim 4:32

All right, so um so yeah, true. There there was there was no real feedback. We did have a few people tell us they really enjoyed it, and the numbers are are astounding. The downloads were great for that episode, and I I don't know if that means the episode was good or not, but there it was very highly downloaded. So I'm gonna take that as good. Right?

Wolf 4:53

I think I will too. I felt like we struck the right tone because we weren't telling people what to do. We were just saying, hey, look, by the way, there's a whole world out there. Here's some stuff.

Jim 5:07

Yeah, yeah. And you know, I've been hanging around you for a long, long time, and I've you know, uh stuff rubs off, right? I I I hear you talking about editors and stuff, but I learned from that episode, and and hopefully others did as well. So that was great.

Wolf 5:22

Uh I think it's very, very interesting the specific things you called out. Why don't you uh say those?

Jim 5:31

Uh what did I call out?

Wolf 5:33

You said tree sitter? Oh yeah, LSP.

Jim 5:38

Uh is is very interesting to me now. Um and I I said I think I was gonna switch to NeoVim and give it a try. I haven't done that. Too busy with real work. Um but that that's one of the things that was very interesting to me. Uh and the LSPs, uh language service protocol.

Wolf 5:58

Language server protocol.

Jim 6:00

Yeah, language server protocol. Um that that is interesting to me as well. I am getting that both of those things from uh when I use VS Code and when I use Xcode. Um although Xcode, I'm not sure it really uses that or its own thing, but um I am getting those benefits in those editors, although I still spend most of my time in Vim. And I've I don't have it configured to to give me those. But I should. But it was a great episode. Go back and listen to it, um episode 26. It was just two weeks ago, uh, and it was great.

Wolf 6:35

Oh, uh you said number 26. I should point out that this is episode number 27, our 28th episode. Um and the more we have to say the two different numbers, the more I wonder if starting at zero was more trouble than it was worth. But I'm kind of a more trouble than it's worth guy. Um I I did not give my oldest boy his name, which is Benjamin, with a Y instead of an I, but I will tell you that is another more trouble than it's worth decision. So I'm very familiar.

Jim 7:20

Um in both cases, the decision has been made.

Wolf 7:24

Yeah. So uh why don't we uh talk about SSH? Um, this is gonna be you doing a lot of the talking, Jim, and me sticking my nose in every place it's not wanted.

Jim 7:36

Uh and maybe a few places it is. So yeah, we're gonna talk about SSH. Uh, and before we get into SSH, let's talk briefly about what came before. Um, you know, I I've been working professionally as a software developer for an awful long time. Uh back in the days before we were even doing networking, back in the mid-80s, uh, you know, we were doing serial communications, so we would use uh uh you know terminal emulator programs that did serial. So there was no SSH back then. Uh but then when we started getting getting into networking, uh tools came up like Telnet, uh FTP, um uh RSH, not the restricted shell, but the remote shell uh program and our login and our CP. Uh and gopher? Does Gopher count? Uh what wasn't was Gopher its own protocol or was that over FTP?

Wolf 8:32

Uh uh Gopher, I think was its own thing. I'm not 100% sure. However, Gopher was the very first network protocol I ever used. Yeah. And the person who held my hand and showed me how to use Gopher specifically to download scientific papers was John Gianandrea, who um you guys may have heard of as being uh the the I guess head of all AI at Google, and then Apple had him for a little while, and Apple didn't know what they had and let him go, uh, which was a huge mistake on their part. But um yeah, he I I was uh friends with John, uh who we call John G all the time, uh for many years, and Gopher was how we met.

Jim 9:22

Interesting. Um, you know, I I barely used Gopher. Uh you know, like I knew of its existence. I tried it a few times, but I never really used it. Um uh so I don't have the experience with it that you do. Uh, but I certainly use Delnet and FTP and uh uh RCP and our login. I use those a lot. And what's wrong with those programs? Why why don't we?

Wolf 9:48

I know, but why don't you tell us? We don't we don't use them because they're like sending a postcard.

Jim 9:55

Um well, yeah. There's that's a great way to put it. Um uh you know the thing about sending a postcard is the entire message is on the outside of the postcard, so anybody can read it. And uh Telnet, uh I'm just gonna talk about Telnet. The others are have the same problem, and that is there's absolutely no encryption or authentication of any kind. Uh the the the text that you type, including your username and your password, are sent in in clear text. Um, anybody snooping the line using uh TCP dump or Wireshark or whatever, um, they can see what you typed. Uh they can catch your password that way. And uh so it wasn't very long before it was realized that those are not suitable uh for any kind of real communication on the network. Um just not good. Um so uh um a a new protocol was developed. In fact, uh uh you know I'd been using Telnet in in the 80s, and it wasn't until 1995 that uh uh the University of Helsinki, uh it's the Helsinki University of Technology in Finland, uh they experienced a password sniffing attack. Um that basically using the the method we just talked about, uh uh TCP dump. Um, and that prompted I'm gonna try to get this name right. Uh it's a Finnish name. Uh Tatu Lonin.

Wolf 11:30

I'm not even sure you should try. I would not attempt that.

Jim 11:33

Yeah, okay. It's uh it's Y-L-O with an umlat uh N-E-N. Uh anyway, he was a uh I think a graduate student at uh the the University of Technology in Finland. Um, and he developed uh the first version of the SSH protocol. And it kind of surprises me that that was only in 1995, uh 10 years after I'd been using Telnet.

Wolf 11:58

Um it's him it's important to note that while he developed the protocol, he didn't make up the math or the Diffie-Hellman key exchange.

Jim 12:11

Oh, sure. Sure. In fact, Diffie-Hellman wasn't even used until like a later version of the SSH protocol. Um that there's an awful lot of math that goes on, and and he just harnessed uh some libraries that did encryption. Um but the goal of SSH was to prel uh replace our login, telnet, FTP, and the RSH protocols uh so that we could have some form of secure communication uh when we're logging into a remote machine. Um it it wasn't until 1999 that OpenSSH was released by the OpenBSD folks. Um so at some point I was probably using um uh Tatu's uh uh version of SSH uh on my system because I know I was using SSH before 1999. Um you switched to it right away. Um as soon as I heard about it, I think. Um that is amazing.

Wolf 13:09

But hearing right, hearing that you switch to something right away is like a big deal.

Jim 13:15

I was younger then more nimble. Nimble, yes, I could I could switch and keep on moving. Um yeah. Uh uh so yeah, 1999 OpenBSD came out, uh largely because um uh when Tatu created uh SSH, it was uh uh you know the term open source uh wasn't there at the time, uh, but it was freely available. And slowly he started to close the source. They built a company around it and um made it less uh available. Uh you'd have to pay money to get it. So open BSD was created, and of course that was free. And uh it it it is now the most popular uh SSH uh implementation out there. Like every Linux distribution has it. Uh Windows added it um to Windows version 10. Uh uh, I guess version 1803 or build 1803, or I don't know how how the Windows thing works, but uh Windows 10 added actual Open SSH. I think that's pretty impressive. Um uh before that on Windows you had Putty uh for logging into a remote machine and uh Win SCP for doing copying. Um so you know, at least that was that was there and that was good. But open SSH is the way to go.

Wolf 14:44

Yeah. Do you happen to know what happened to uh tattoo's um original company closed source product thing?

Jim 14:54

I believe the company is still out there. Uh I I don't I don't know who would be using it.

Wolf 14:59

Are they are they living off t-shirt sales or I uh merch? I don't know.

Jim 15:07

Maybe it's two guys doing a podcast about it. I don't know. Uh uh when I when I did look them up, uh I I believe the company is still around, but uh I I don't know because OpenSSH is so awesome. Um you know, uh when I think about the commands that I run every day, and and I know you run it every day as well, it's probably the most frequent thing I type at the command line. I probably run between like 40 and 80 uh instances of SSH every day. Uh I haven't.

Wolf 15:47

I have to say I I absolutely love and rely on SSH. I use it on every machine, on every platform. I use it to connect between machines and to servers. And um I am reminded, because I've just been setting up a new machine and some connections to some new servers, that the annoying part is the part where you put keys in different places to make the like uh I will I will authorized authorized well you'll get to it.

Jim 16:24

Uh yeah, we'll get to that. I'll show you some some some ways to simplify that, uh, which which I think are pretty cool. But what can you do with SSH? Besides logging into remote machines, which I do all the time, uh you can run a command remotely. You don't have to SSH to the remote machine and start up a command. Um you can just r use the SSH command to you you name the machine you want to you want to run the command on, and then you name the command, and it runs that command on the remote machine. Uh of course that command has to exist on the remote machine, but can uh can the remote command be tmux?

Wolf 17:05

Sure. Sure. Boy, that's useful. Yeah. Because I don't care about the outer shell.

Jim 17:12

Yeah. A lot of people I think do that. They just run uh tmux remotely, and that that gives them their terminal multiple uh uh multiplexing and and gives them their one or more sessions uh through tmux. I I don't run remote commands, I SSH to the remote machine and run whatever I need to run. Uh but a lot of people they'll just run tmux directly that way. I'm gonna try that tonight. You can even run like Vim remotely if you want. Um you know, if you know the name of the file, you want to edit. Uh although some people, I guess, when they launch Vim, they just launch Vim without giving it a file name. And then once they're in Vim, they select the file they want to edit. Uh again, that's not the way I do it. But uh so yeah, you can run commands remotely. Uh you can copy files using SSH. Uh you and here's the interesting thing. Well, first of all, um there's two ways to copy files. There's SCP, the secure copy program, and there's SFTP. It's sort of like FTP, only it's over the SSH protocol. Um so it's it's an interactive thing. When you run SFTP, it's interactive. You you SFTP to the remote machine, you can browse the directory, you can uh get files and put files, just like FTP. Uh I rarely use that. I use SCP, which is really much closer to the original copy program. Uh you you name your source file, you name your destination file. Um so if your source file is on a local machine and your destination file is on a remote machine, uh you you just name you you say that you it's the uh the host name, colon, and the file. So whether that's on the on the low uh the from or the to, uh, that works. And then Wolf just showed me something a couple of days ago, the fact that you can copy from one remote machine to another. I I didn't know you can do that. In fact, the reason I I thought you couldn't do that because is because years ago I read in the man page for SCP that that wasn't supported. So I just thought it always wasn't supported, but at some point they did add that capability. So I can I can name a remote file on uh as the from and a remote file as the to, and it'll copy the file uh from the remote machine, from one remote machine to another remote machine. So I started looking at that more closely, and I thought, well, boy, isn't that nice? It doesn't go through the local machine. Uh by default, it does go through the local machine. Like normally I would SCP the file from the remote machine to my local machine, and then SCP that same file up to the different remote machine. I I do that not terribly often, but enough. Um and uh by by naming a remote uh for both the sides, uh, it just handles it for you, but it comes through your local machine when it does that. Unless you specify the capital R flag, which causes the re the the first remote machine to actually open a connection to the second remote machine and transfer directly between those two machines without the data flowing through your machine. Now, to make that work, those machines need to be able to talk to each other. So I do a lot of things where uh uh I'll have a file on one customer's system and I want to copy to another customer's system. Those two systems can't talk to each other. There's there's no setup where they can do that. Uh so I I have to uh go through my machine to get there. But if the two remote machines are maybe in the same network or at least able to talk to each other, it works fine.

Wolf 20:53

I I thought that was most importantly, um, those two machines need to um whatever words you use to name them need to be in the uh like when you say the destination, the name you use for the destination must either be fully qualified or else appear in the.ssh config for your user in the second machine, the from. Or in the host file of the second machine.

Jim 21:30

Well, actually. Actually, it's in the first machine that that needs to exist because it needs to be able to talk to the remote machine. It needs to be able to get there. It needs to be able to resolve that name to an address.

Wolf 21:39

Aaron Powell That That's what I'm trying to say. Like there's a from and a to, and there's you. And if you're going to use minus capital R, then the to must be known to the from in the way that you say it.

Jim 21:55

That's correct. Um Capability exists, I didn't know it was there. Um, another thing you can do with SSH is create tunnels, and these are really powerful. You can create an SSH tunnel uh between two machines, uh, and it can go through other machines, but uh create a tunnel between two machines so then all traffic can flow through that. Um, and it's not really like all traffic, it's like traffic for certain ports. For instance, you could um set up a tunnel that uh basically proxies uh port 80 on one machine to a totally different machine. So people think they're talking to one machine, but port 80 is going to be tunneled through the the SSH tunnel to the other machine without them having to without a hole in your firewall. Yeah, yeah. If you're t if your firewall supports uh uh port 22 uh or whatever port you're running SSH on, um you can set it, set your firewall up to say, yeah, but port 80, I want it to go through that SSH tunnel to some other internal machine. That that's one way of doing it. Um or it could be port 443, it could be any port. Uh the communication will flow through that tunnel just by talking to that port that the tunnel is configured for. Um and you can set up local tunnels and remote tunnels so that uh port 80 on a remote machine will come to you. It's I I I don't want to get too deep into that. It's all in the man pages. Um uh but it's it's quite handy. Um and one of the places SSH tunnels uh not so much anymore, but used to be used a lot, is for the X Windows uh protocol, uh for for running uh uh an X Windows client uh to uh a remote machine over SSH so that that connection is secure because by default um uh X the X protocol, there's no security in it. Uh there's certainly no encryption uh and and really the uh the authentication isn't even uh secure. Uh but if you run that through SSH now it becomes uh secure, which is really handy. I used to do that all the time. Um in fact one of the cool things is you can you can tunnel your X over it and you can turn on compression uh over that uh that tunnel and uh it it it works quite well. I I I have run uh X clients across the internet to uh machines in other countries and it's not as good as local, but it's still amazingly good.

Wolf 24:42

Um this instantly gives me a question. Yeah. Because my Ubuntu um 24.04 LTS system is not using X, it's using Wayland.

Jim 24:58

Um it's using Wayland as its default protocol, but there is X Wayland, which is an X uh nested X server that runs under Wayland, so you can still run your X applications uh through SSH uh if you want. Okay. Um the same thing. It's almost seamless, yeah. Even though, even though your your machine, your your Ubuntu machine uh running Wayland, um uh is talking it it basically in that case the software is talking directly to the hardware through a thin layer of drivers and stuff, uh, is not using the X protocol. But by running X Wayland, now you can use the X protocol to talk to other machines. Uh so it's entirely possible and and people do it.

Wolf 25:49

I specifically had to um install WL clipboard to get WL copy and WL something else, uh, which is what let me do copy and paste that got out of the shell and into the system as a whole that's running Wayland, so I could paste it in my browser or whatever. Uh the point I'm trying to get to is this Mini X uh system inside of Wayland. Do I need to separately apt install that? Uh I don't know.

Jim 26:28

I don't know. Okay. I I am so far away from uh Linux desktops. You know, it used to be my life back when I was doing LTSP. That's all I did was was X on the desktop. But I am so far away from that now, I haven't even tinkered with Wayland at all. Uh we we did get some feedback after our uh X Windows uh episode several weeks ago that maybe we should do an episode on Wayland and maybe we will, because I I really don't know very much about it at all. Um but uh back back to our topic today, uh SSH. Uh you can run X over it, uh you can tunnel all kinds of things over it, and it's it's kind of neat. Um but let's um let's talk about uh uh uh how people use uh SSH. Um I'm gonna sort of talk in order of uh least secure to most secure. And and I want to point out that even least secure is way, way better than telnet or FTP, right? So when I talk about least secure, I don't want to say it's a bad way to run it because it is secure. Uh but most people, when they first uh start using SSH, uh they don't want to be bothered with setup or anything, and they just uh use SSH to completely replace Telnet. So they type uh SSH space uh remote machine. And uh it it assuming the connection uh succeeds, uh it'll pop up and ask for a password. So they type their password in and they just they they're now they're now uh running on the remote machine and they can run whatever they want to run and everything's secure. Um, but it's it's to them, it other than being secure, it's really not much better than telnet. It's it's just like using telnet. Uh, but there's better things. Um I I like I uh you know, Wolf talks a lot about reducing friction. I like to reduce friction on the things that I use often, like SSH. Uh I I don't want to be prompted for a password. So I generate a public-private key pair, which uh everybody should do, um, and and I put that public key on the remote machine in a special file. We'll talk about it in a minute. Uh, but by doing that, if I copy my public key over to the remote machine, now I can SSH to that remote machine and it won't ask me or ask me for a password. And uh, Wolf, why does it work that way? What's so special about a public and a private key pair that makes that work?

Wolf 29:06

Uh so this is asymmetric encryption. Um, and I'm sure you're gonna point out in a second that the SSH protocol uses asymmetric encryption just for key exchange for the symmetric key that is gonna drive the rest of the session, because key exchange is the big big deal. And asymmetric encryption means that you have a pair of keys. There's actually general cases of this that are N, but we don't care. What we care about is two. And when you have these, if you lock with one of them, you unlock with the other. And it works both ways. Um, if you have the private key, you can lock something with the private key, and then anybody who has the public key, which is everybody by the way, can unlock it. And since you're the only one with the private key, they know you did the locking, and that's called a signature. And if it happens the other way, somebody with the public key locks it with the public key, then only you, the person with the private key, can unlock it. So when you have this key pair, the remote machine knows your public key specifically. Um, just like everybody knows your public key if they want to. But the remote says, by the way, this is one of the people who's my friend. I'm gonna let this person log in. And they know it's you because you do the connection with them and sign your request to connect with the private key. Because they can unlock it with the public key, they know it must be you. It can't be anyone else, and therefore the connection can proceed. Asymmetric.

Jim 31:09

Yeah. Um it's my understanding that asymmetric uh encryption is rather slow, and symmetric encryption is much, much faster. So we're doing this kind of one-time thing uh by uh uh authenticating with the remote host and negotiating what the symmetric key is going to be. And that's that's relatively slow. I mean, it's still very, very fast, but it's it's it can be slow. It's okay if it's slow, because you're only you're not doing it all the time. You're only doing it to establish the connection. Once the connection is up, then you switch to symmetric key uh encryption and everything's fast.

Wolf 31:54

Um and symmetric, we didn't define symmetric, but symmetric means you only have one key, and that one key locks and unlocks. You share that key. Both that's right. Both sides have the same key. Um that's what the symmetri the asymmetric part was for, was to generate that together. Both both sides have it. And now um when you want to send information from one side to the other, you lock it with the symmetric key. Um, the encrypted information crosses the unsafe path between the two machines, and then the other machine uses the exact same key to unlock it. That uh fact that it's the same key on both sides is symmetric.

Jim 32:45

Yes. Okay. So uh uh you really should generate a key pair. And when you do generate the key pair, by default, the algorithm that's gonna use, is is is that the right word for it? Algorithm or the the um I I guess it's the algorithm, right?

Wolf 33:03

There's multiple different kinds of keys, right? And uh it is the algorithm that is the key type.

Jim 33:10

Yeah, and the the default algorithm, if you don't ask for one, it's gonna be RSA.

Wolf 33:15

Which is it still RSA or is it DSA?

Jim 33:19

Uh oh DSA's been been uh uh deprecated for a long, long time.

Wolf 33:24

Yeah. Obviously, I never use either of those, and you're gonna say why in a second.

Jim 33:30

Yeah, well, they're not they they they haven't stood the test of time uh as being uncrackable. Uh these these are all secure, but there's levels of security and the newer algorithms are better. So you really don't want to use, well, you certainly don't want to use DSA because a lot of machines you connect to won't allow it. So you you can't use that anymore. Uh RSA you can use, but there's better. Uh there's EC DSA, uh, which is better. Uh but the one everybody's talking about now that that you probably should be using is ED 25519. So ED25199. That's what I use. And that's what I use too. And you know, I was always confused about that. Which one should I use? Which one is better? There's choices here, and nobody was telling me which one I should be using. So I I mean you should ask me.

Wolf 34:24

I definitely wouldn't tell you.

Jim 34:26

Yeah, maybe I maybe I didn't ask you in that case. Uh, but my friend Marlon, uh the the the the the wonder kid at uh I mean he's hardly a kid anymore at at Meta, uh he cleared that up for me. Um ED uh 25519 is the one you want to use. And it's real simple when you do your uh your key generation, which by the way, it's uh SSH dash key gen is the utility you want to use. Uh you just say dash t and then you name the the algorithm you want to use, whether it's RSA, EC, DSA, or ED25519. Uh so choose that third one, ED25519. Um that that's the best one at this point. Um and you want to avoid DSA completely. Um so yeah. And at this point, I think uh if you start if you do use RSA, I think you might get a warning about um uh quantum attacks. Uh I I I know you've seen that, Wolf. What what's that all about? Do you know?

Wolf 35:34

Um so the computers we use today are um uh discrete. They're they're interesting. A bit is either a yes or a no. Um and so you use bits and transistors that can flips back and forth and let some things pass and not to solve one problem at a time. And when you solve that one problem, um the inputs are discrete, they're ones or zeros, and the answer is ones or zeros. And if uh you have a hard problem uh where you need to you know look at 150 million answers um with a discrete computer, you have to solve whatever underlying problem it is 150 million times. That's how my computer works, that's how your computer works. But we've had this idea for a long time, and now we have some computers that can do it. Quantum computers. Quantum computers don't have discrete bits in them. What they have are a thing called qubits. And a qubit isn't a one or a zero. Instead, a qubit is sort of like a curve of a whole bunch of different values at the same time. So where a discrete computer has to solve 150 million problems one after the other, in a quantum computer, it can be arranged such that you solve 150 million problems at once, using the same bits. Because the bits don't just say one problem, they say all the problems. And then when the answer comes out, it's all the answers at once arranged in a curve so that you can see which is the rightest one. Um the quantum computers we have right now aren't great, um, but they're the worst we're ever going to have. So uh you want it to be like a big thing in uh these uh network sessions, uh, aren't necessarily about people overhearing you and instantly uh decrypting your messages, they're about them overhearing you and saving your exchanges and solving them later. And uh the idea of taking part of what you've made and reusing it immediately, or taking part of what you have and decrypting it later, um there's l different words for this. Replay attack is one of them, etc. So we're not just concerned with the problems um not being hard enough. We're worried about them being durable over time. So mathematicians who know stuff um have been working on how to build uh algorithms that are uh quantum strong, quantum secure, and uh withstand replay attacks. Um right now, you can use RSA and you're okay. But if somebody overhears that or saves it, or if you're still using RSA five years from now, well, um probably not. So that's what you want. Something that's quantum strong.

Jim 39:23

I I think with newer versions of OpenSSH, when you talk to a remote machine using RSA, it will warn you that it's not quantum safe. That it's correct. Something to that effect.

Wolf 39:37

And that sounds now you can turn off the warning, but by default it's on.

Jim 39:42

It's still there, yeah. I mean it's still happening, right? It's still not quantum safe. But so you you mentioned uh discrete computers like the CPUs we use that you know, maybe you have uh 20 cores or eight cores or whatever. Um and then there's these computers.

Wolf 40:01

You're a GPU and you have 10,000 cores.

Jim 40:05

That's where I was going next. You sort of talked about one end and the other end, but GPUs are sort of in the middle. Are people using GPUs to try to uh crack these things?

Wolf 40:15

Yes, in fact, that's how the two main ways to crack passwords um and uh you know these interactions um is uh GPUs and um uh field programmable gate arrays, FPGAs, or uh there's a couple different names for things where you make the machine that you want it to be. Uh RS unlike um ED25519, uh RSA keys, you specify the length of um the particular numbers that take part in the underlying equation. So for instance, you might have a 512-bit key or a thousand twenty-four-bit key or a two thousand forty-eight-bit key. Uh in ED25519, you don't. In I think in the elliptic curve ones like EC DSA, you don't. But RSA, you set the size, um, and for a long time now, we've been at a place where setting the size smaller than 2000 is not smart. Um, but also using RSA is not smart. Uh at this time, you should be using ED25519.

Jim 41:32

Because we have choices. We have better choices. And that's the same thing.

Wolf 41:36

And you should point out you should point out that both sides um have a collection of which algorithms they are willing to support, which depends on their age and what's installed, and they negotiate and agree to use a particular one. Um, and if you only supply one key, that agreement is going to be a very short discussion.

Jim 42:02

Yeah. And you might get turned down, right? That is correct. I've been in situations where I had to talk to very, very old devices. Uh old Cisco routers come to mind that only supported uh uh I think in that case only supported DSA. And so you have to jump through a few hoops in order to use DSA.

Wolf 42:23

Who was the comedian who said um I wouldn't attend uh I wouldn't be a member of a group that uh that allowed me, yeah. Yeah. Like whatever it is. Like if if DSA is the only way to talk to the server, I'm happy you're not talking to it. I know that's not the situation you're in, like you had to talk to it.

Jim 42:47

Yeah, yeah. So uh and and I'll still say that it's even with DSA, it's still better than Telnet. Um but it you know, we had to do it. It was a it was a business thing, we had to connect to it. Um okay, so we talked about uh you can generate a public-private key pair, and and the question that always came up to me was what algorithm to use. And now I know uh ED uh 25519 is the way to go. Um if both machines uh are are new enough. Uh and and that algorithm's been around quite a while, so uh it chances are you're talking to a remote machine that can handle that. Uh so um you've got this key pair, the private key stays on your machine, the public key can be known to anybody. Uh you can put that out uh on on remote machines and not worry that somebody might see it because it's just your public key. Anybody can look at it. Uh but without your private key, they can't do anything with it. So, how do you store this? What what what do you do with this thing? By default, it's just a file in your.ssh directory. Um if you're gonna just store it as a file, um Another uh uh choice you have to make is whether you want to put a passphrase on it or not, so that is that private key is locked until you enter your passphrase. Uh and you know, for the longest time, uh when when Keygen came up and asked me for the passphrase, I would just press enter and not even put a passphrase in. And I always felt a little guilty for doing that, like I was doing something bad. Uh, but it turns out that's pretty common to not lock your your your uh key pair, your private key with a passphrase. Um and I think Wolf, you and I talked about this the other day. You you you have situations where you don't protect it, right?

Wolf 44:41

Uh that is correct. There's layers. Um am I doing the best thing possible? Um no, uh, I'm not. Sometimes I'm doing the thing that's slightly more convenient. The general case for me is that I don't actually have my private keys in files on the disk. They live inside one password. And one password has a you'll have to talk about the SSH agent, um, but one password has a custom one that will grab them out of one password um instead of grabbing them from files like the one that comes by default with OpenSSH. So they for me, they actually are encrypted in the general case because they live inside one password, which I have to unlock.

Jim 45:33

Yeah, so in your case, the the passphrase is on one password itself, not on the individual keys. Yeah. Correct. Um and that's that's good. That's good. So you can use software uh to hold these these the the private key, uh like one password, and uh I'm guessing the other password managers support that. I I'm not sure. But I know one password supports it.

Wolf 45:57

Um I I'm pretty sure that um LastPass connects with your contacts app and sends your private key to all your enemies. Yeah.

Jim 46:12

I I I don't believe that's true. I think you're joking. I'm I'm gonna assume you're you're you're just messing with me. Anyway, uh I I've sort of found that there's no reason not to put a passphrase in. Because while Wolf said that it's uh uh you know there's a level of convenience by not putting it in, if you have an SSH agent running on your machine, and you probably do, um uh uh using uh uh uh a passphrase uh on your key pair is uh on your private key is is quite simple because you only have to unlock that once. Uh and then that private key gets cached in the in memory uh in the SSH agent. Uh so from that from that point on, when you connect to a remote machine, uh OpenSSH will talk to the agent to get that private key to start using it. And you don't have to enter the passphrase every time. I'm talking about desktop Linux. Uh if you're sitting. Yeah.

Wolf 47:17

But um you should also note that the SSH agent uh normally is configured to only cache it for a fixed amount of time. And this also is a thing that doesn't necessarily happen at connection time. If you haven't set it up exactly right, you have to enter that password at shell launch time. And finally, uh, and that's bad, if you have configured it to launch at shell launch time instead of when you first connect, um, then it will also be the case that if you have six different private keys, you must unlock all six. So you want to make sure that you don't do those things.

Jim 48:06

Yeah. Again, if it's if it's configured properly, uh it's fairly easy to use with a with a uh passphrase. Um if it's not, well, then it's not gonna be so easy to use. Um the the the next thing, after you've uh uh uh locked your private key with a passphrase, uh the the next level of security is a hardware key uh or uh a smart card. You can store your private key on uh like a UB key or any of these other uh uh private key token things. Uh they can be configured to store your SSH private key so it doesn't exist on your disk. And the other thing that I found out uh is again a lesson from Marlin, uh, is uh in the Apple environment, you've got the secure Enclave. And it's really simple to store your private keys in the secure Enclave. In fact, the the the tool you use to do that is called Secretive, it's an open source project. Uh you install that uh on your Mac, and you don't actually uh import your private keys to it. You use Secretive to generate your private keys and the uh in fact generate the whole key pair. The private key will be stored in uh in the enclave. You will never actually see the private key. There's no way to extract that private key out and and get a copy of it as a file on the disk. Um it's in your enclave. Um SSH uh will talk to an agent. Secretive is that agent, and it knows how to get the data out of the enclave and use it for your uh uh uh connection with a remote server. I set that up. It took me just a couple of minutes, uh very easy, and it works really well. And the neat thing is if you uh you can tell it. In fact, the default is uh biometric uh unlocking of the of the key. So uh for instance on a Mac uh if you have uh touch ID or what what it is it touch ID that the little button on the keyboard is is it it is uh yes. So if I SSH to a remote machine, it prompts a little a little dialogue prompts up telling me uh to unlock it with my fingerprint. And so I do that. And if I I at that point I have a choice to not use my fingerprint, but instead uh type in my passphrase, and it will uh unlock it for me. Um or type in my my system password, whatever I use to log into the the Mac with. Um so I'm talking a lot about uh the Mac with the Secure Enclave, but um uh Linux and Windows has the same thing with a TPM, a trusted platform module. You can do the very same thing. You can't use the secretive program, but there's other programs. There's something called SC auth that you can use uh to store your keys in in the TPM. And I think that's fantastic. So now your your private key never exists on disk, so nobody can steal it. Uh it's in the uh the secure uh either the the enclave or the trusted platform uh module. I think that's pretty neat. So all my private keys are now in my uh Enclave, and I'm I I never felt uh insecure before, but I I I feel pretty good about that. It's it's working quite nice. Um so that's a lot of talk about keys and and key pairs and stuff, but there's one more way uh to authenticate and to set up a connection between two machines using SSH, and that's certificates. Um uh if if you think about managing those private public keys, uh the private key you just keep local, that's no big deal. But uh using the public key, you have to get that out to all the machines that you're gonna log into. Uh and you know, well, for you and I, we don't deal with that many machines.

Wolf 52:14

Uh and you know why? It's not we wrote a whole episode about this. And the episode is called You Are Not Google. That's right.

Jim 52:23

So we, you know, I I have I don't know, maybe 15 or 18 or 20 machines that I have to log into occasionally. It's not that big a deal for me to put my public key out on those machines. But imagine you are Google or you are Meta or or Slack or any of these other big companies. Managing those public keys is impossible to even think about. Um so there's another way to do it, and that is with uh SSH certificates. And the way that works is uh it's very much like uh uh SSL certs, you know, that you would use for a web browser uh to secure a web server. Um you sign your public key, your SSH public key with a certificate authority. And this is all done through the SSH key gen key key gen utility. Um you you create a uh uh a CA uh key pair and you can use that to sign uh your public keys and you configure the remote machine to trust anything that's signed by that CA. So now when I SSH to a remote machine, that remote machine looks at my public key and says, Oh, this is signed by a CA that I know and I trust, so I'm gonna let you in. Now, these big corporations uh they they don't do this stuff manually either. They have software in place to handle all this stuff so that when you go to log into a remote machine, uh you might go through some identity provider that that gives you a a certificate or a token to use for that session and it's all nicely managed. And it's excuse me, it's pretty neat, I think, how that works. Um your uh uh identity provider uh can be configured to um uh allow you or not allow you. Uh if if somebody leaves an organization, you want to make sure you turn off their SSH access to all of the machines. You don't want to have to go through and and delete his public key off of all those machines. You want to just flip one switch and instantly that person can no longer log into those machines. And you know, I'm talking about an awful lot of machines out there that that you might have to do that with. But by using a certificate uh mechanism, it it's it's a much simpler deal. And I I tried it, I I actually succeeded. I set up a remote machine, uh, I set up a SSH uh CA, uh a certificate authority, uh, to sign my key, and it all works quite nicely. Uh uh it's not that hard to do. I'm not gonna explain the intricate details here how to do it. It's all in the documentation, and and of course you can search the web for it. But I uh I wanted to point out that it is possible, and the large corporations are definitely doing this because otherwise they'd they'd go crazy trying to manage uh new hires, uh you know, eliminating old um employees, that kind of thing. Uh it's it's a real neat way to do it. Um let's talk for a couple of minutes about the files involved. Because if you start looking, uh, like for instance, if you look in your.ssh file uh directory uh in your home directory, uh, you'll see a couple of files there. You might not really understand what they do. And also if you look in uh slash etsy slash ssh, there's some configuration files there that might seem scary at first. Uh and we're just gonna talk real briefly about what those things are. Um you know when you when you uh log into a remote machine the very first time, you'll always get this prompt that tells you you're connecting to a remote machine and here's that remote machine's uh uh host key fingerprint. And I don't know, Wolf. Do you ever do this? Do you ever actually compare the fingerprint from the remote machine uh with something you know to make sure that you are indeed talking about it?

Wolf 56:38

To be brutally candid about it, um I have faced this question oh at least a hundred times, probably in the multiple hundreds, and I think I did compare one once. Once yeah.

Jim 57:00

Uh usually I'm responsible for both sides of the conversation. I build a new virtual machine, I install Linux on it, I turn on SSH, and then I SSH to that machine, and uh uh I trust it. So I say yes to that question, and from that point on, that machine's uh host key, public host key, gets put into my uh known hosts file on my machine. So there's an entry. Uh if you look in.ssh, there's a file called uh known hosts, and it contains the public key of all the machines you talk to. And that's how the next time you connect up to that machine, it doesn't ask that question anymore because that uh public key exists in your file, and and now it knows uh I trust that that machine is who they say they are. Uh so it you sort of forget about it. But that file just contains a whole bunch of those public keys. There'll be at least one for every machine you talk to. Uh, if you talk to that machine by multiple names, you can talk to that machine by typing in its IP address or a shortcut name or the fully qualified uh domain name. Um and there'll be an entry in that uh in that file for all of those. Each of those. Each of those, yes. Um uh what gets scary is I I I I've been known from time to time to build a machine. Uh I call it test one. It's a virtual machine. I'll I'll log into that machine, I'll do some testing, I'll I'll do all kinds of things to it. And at some point, I'll throw that machine away. I'll delete that virtual machine. Some number of months later, I'll need another test machine, so I'll create a new virtual machine, I'll call it test, and I go to log into it, and I get this big scary message about man in the middle attack. What that's telling me is that machine presented a host public key, and it and it's got the same name as as as the old machine. It looks up in in the uh uh the known hosts file, and it sees a machine with that name, only it's got a different public key. It's like, whoa, what's going on? Uh, you're trying to talk to a machine, but the host key has changed. And they throw up this big scary message about man in the middle attack and be careful, and it and you can't get any further. And it tells you, I've seen different things throughout the years. Uh sometimes it'll tell you the command you need to type to remove that from your hosts, your known host file. Otherwise, it'll tell you the line number that it is, and you have to edit that file manually to remove that old entry from your known host file. Um, that's what's going on there. It it's not necessarily scary. Um, if you understand what it is, it's just telling you that the host key has changed. Now, if you're logging into a remote machine controlled by somebody else and you get that, you might want to watch out because it could be a man-in-the-middle attack. And Wolf, what is a man-in-the-middle attack?

Wolf 1:00:01

A man-in-the-middle attack is when you instead of reaching the destination that you wanted, you reach some enemy in the middle. And the enemy in the middle pretends to be the destination, decrypts your traffic, and re-encrypts it uh to go to the destination. So, yes, you get whatever you were trying to get from the far end, but the guy in the middle sees everything and uh can make changes and do bad things. So uh you'll see this sometimes as M-I-M with the M's capitalized, man in the middle. Uh you'll sometimes see it as um, you know, Al and Betty and Charlie is the man in the middle, or you know, something like that. Um uh but it is one of the things that uh these uh people try to design the algorithms to uh prevent. Um and this is a place where replay attacks come in. Um, but man in the middle, that is a thing that happens all over the place. You don't want it. If you see that dialogue um or text or error message or warning, please take it seriously. Stop and consider. Could this description be the truth?

Jim 1:01:35

You know, this this sort of makes me think of something that's really the same thing, and that is these card skimmers uh people are putting on like gas station pumps where you scan your credit card and it uh uh you know it's and it still works. You scan your credit card, but there's this little device in the middle that captures your credit card information before relaying it on to the actual gas pumps uh credit card scanner, it's the same kind of thing.

Wolf 1:02:01

It's a man in the middle. That is the perfect simile. Um, yes. That is the real life hardware equivalent of a man in the middle attack. Absolutely. Good good observation.

Jim 1:02:12

And here's the case where somebody just went and slapped up something over the top of the credit card scanner, and it looks to you like it's normal. And I've not not actually encountered one of these, but I have heard of people hitting these things. They don't know the difference. It's just a credit card scanner, and they tap their card or or uh uh you know scan their card and and go on. Uh and then somebody comes by later and grabs that uh that little device, and now they've got all these credit card uh numbers and stuff.

Wolf 1:02:40

That's that's why I I mean maybe this is paranoid, but that's one reason I really like Costco, because when you go to Costco, they've got the tape over the um connection to where you insert your card. So you can see that they have inspected it and there was no skimmer.

Jim 1:03:01

Yeah, and and nobody's ever made counterfeit tape, have they? Um yeah. Well, anyway, I I I I agree. Uh anyway, okay. Um uh so uh that's the host keys file. Um the authorized keys file, that's a file you'll typically have on the remote machine, and that's the file where you put your public key so that you can log into that machine. And and that's I talked about it earlier. That's what allows you to log into the remote machine without entering your password because you're supplying your public key as part of the connection, and it looks it up in the authorized keys and says, Yeah, we know him, he's okay. Let him in without a password. Um that uh uh like I said, that file's on the remote end. Um so that presents a challenge, and we'll mention this earlier is how do you get your keys into that file? Um there's you can you can SSH to the remote machine, enter your password, go into your.ssh directory, edit the authorized key file, and paste your public key into it. That's a real clumsy way to do it. A lot of people do it that way. I I did it that way for a long time. Uh the the easier way to do that is with the SSH copy ID command that exists on uh I know it exists on uh well it's part of OpenSSH. So it exists on uh on a Mac, it exists on a Linux box. So what you do on your local machine is you just type SSH-copy dash ID and you list the remote machine and it copies your public key over there for you and inserts it into the authorized keys file, and then it tells you success. And so then the next thing you do is you SSH to that machine, no no password required because that public key got installed on the remote machine. And that's really handy. And then the next step is what, Jim? Okay, the next step is uh something I do. You can take your public keys, you might have a couple of them, you might have a whole bunch of them, and you can put them up on a service like GitHub. I I you you have to put at least one up on GitHub if you're using it, in order for for you to use SSH to uh uh authenticate against GitHub, you know, for doing pulls and pushes and all that kind of stuff. Um, but you can put more uh public keys up there. You just log into GitHub, you go to profile, you go to settings, and then there's a thing about SSH keys and GPG keys. Uh in that screen, you can put all your public keys. And where I find this really, really handy is when I build a new virtual machine, uh, it's usually Ubuntu. Uh and when I stall install Ubuntu, as I'm going through the installation process, there's a step where it says uh uh do you have any public keys stored on a server that you want me to fetch. And uh I I think there's two choices. Um maybe three. There's GitHub, there's launch pad, that's uh Ubuntu canonicals thing. And I think it might be GitLab. It asks uh if you have any keys, you tell it yes, you tell it GitHub, you give your GitHub name, and it'll go fetch those public keys for you and install them in your.ssh slash authorized keys file. So right off the bat, I build a virtual machine and I can SSH to it without it asking me for a password. I really like that. Um if you fail to do it at that point during install time or you want to redo it later, there is a command that you can run on on uh Ubuntu machines it might exist on Debian I don't know if it exists on Fedora and Red Hat and and that family. But it's uh SSH-import dash ID dash gh. You run that command on a on a remote machine and it'll you uh and the argument you give it is your GitHub name and it'll go fetch those keys from from GitHub and install them in your local machine. So it's really so when I I I mentioned I switched to using secretive so that my my keys are stored in uh in the uh secure enclave well when I did that of course I had new keys new new private keys new public keys and I wanted to get those out there so I put those up on GitHub then I logged into each of those machines that I log into uh and I ran that import uh SSH import IDGH command and it went and fetched all those keys and installed them in my file in my authorized keys file.

Wolf 1:07:39

So you can't do this if you have turned off the SSH capability to log in with a username and password.

Jim 1:07:50

Oh we didn't even talk about that remember the first way of using SSH is to have it ask you for a password. Well what I always do is I tell uh I configure SSH to not allow password authentication. And that gets pretty sticky if you're building a machine and you turn off password authentication and you need to log into it for the first time and you can't. Now when I'm building a virtual machine I have some kind of console so that I can I can actually log into that machine through the console. It'd be like if the actual machine was sitting on my desk uh and there I install my my keys. But I always turn off password authentication in SSH. That's a setting uh that you would put in the Etsy slash SSH slash SSHD underscore config.

Wolf 1:08:40

Put that in there me too I always do the same thing.

Jim 1:08:43

Always turn that because if you ever watch your logs if you have a machine that's on a public facing network uh if you watch your logs you're just gonna get bombarded with requests to log in to SSH. It's just a nonstop barrage. It's like I don't know how they know that you built a new machine and put it on the network but almost immediately somebody is trying to attack that machine um and so some people get around that by uh running SSH on a different port. Normally SSH is run on port 22. So they might run it on a different port. And that's just that's just sort of hiding the the the problem although the problem still exists if it's out there in public. So I don't worry about that. I run it on port 22 but I never turn on password authentication. You have to do that.

Wolf 1:09:33

It's important to know that these attacks these probes are not people trying to log in with an SSH key. These are people logging in with SSH trying to crack your password like a text password not a key. Yep.

Jim 1:09:51

Yeah go go look at your log files like var uh what is it var log uh syslog or var log uh I think it's var log syslog where this stuff goes you'll see all kinds of attacks on that port.

Wolf 1:10:04

Like multiple per second.

Jim 1:10:06

Oh yeah it's crazy. So uh turn off password authentication that if you if you take away anything from this episode use key pairs and turn off password authentication so that you don't have to worry about somebody uh brute forcing uh your machine um and also you are not Google you are not yeah yeah yeah use certificates if you're Google right so there's a bunch of other config files uh that you can look at um uh uh one that I I edit all the time uh and I use is my dotssh slash config that's where you can put in things like uh a short shorthand name for a machine you're gonna log into uh maybe you want to use a specific uh private key when you talk to that machine um maybe you want to turn on uh uh X forwarding um log in as a different user log in as a different user uh what's the number one re time or the most popular time you do that uh when you want to be git yeah yeah and and it seems weird when you use git to talk to something like GitHub you're not logging in as yourself you're actually logging in as git uh the username you're gonna be but it knows who you are because of the key pair yes yes um so uh in your config file in your.ssh directory that's where you say that uh and Wolf has told me many times that's the number one problem he sees from users in in his organization or organization or anywhere anywhere um anywhere I've ever been people who have trouble using git uh with uh upstream repositories it's because they didn't configure their SSH properly um so that that file take a look at it uh there there's a man page on on what can go in there um the primarily what I use is uh like I'll set up a new machine I don't really want to bother with DNS um adding that machine to DNS it's just a temporary machine so I'll edit my config file I'll create a host entry for that new machine give it a name something simple uh and then under host name I I'll list the IP address and and now I don't have to worry about it uh uh the hosts file or uh DNS or anything like that. I can just SSH to that machine because there's an entry in the uh config file.

Wolf 1:12:43

But there's a ton of things to put in there. Oh there's keep pretty much any option. Yeah yeah everything you want SSH to do that file is where you say it or you know if you don't want to say it on the command line. And and by the way you don't want to say it on the command line.

Jim 1:13:01

No no that's nonsense to be specifying those options on the command line you can put them in that file. That file is just for you just for each user it's in their home directory in their in their home dot ssh slash config. You can also put those same kinds of entries in the global config file that's under etsy ssh slash sshd underscore config most of those entries you can put there as well uh like if you want to turn on x forwarding or you want to turn on uh agent forwarding or uh you know all kinds there's there's probably a hundred things you can put in that file uh read the man page it's all there it's it's pretty neat uh one final thing to say about the config files it's absolutely important that you get the permissions right on those files you got to get the permissions right on the dotssh directory i i think i always set mine to uh 700 which is read write execute for the owner myself and nobody else can see that nobody else can look in that directory they can't write to it they can't do anything and then the files within that directory uh it ssh is very very picky about those permissions you want to make sure that the files are owned by you and that uh only writable by you uh you don't want anybody else uh writing those and your private key file especially don't even allow anybody else to read that file if they can't get into the directory they can't read it but you still have to have the permissions correct on the files themselves that to me uh is uh that's the problem I run into most with people that need help with SSH is because they don't have their permissions correct yep so for private keys 0600 that's the way you would say it in change mod right um and for public keys it is uh 0644 yep 0644 is fine um you know nobody's gonna get to that file anyway because it's in your dot ssh directory which is protected with permissions uh but you're gonna put that file on other machines and that's the permissions you you want you don't you don't want anybody else to be able to write to that file but they should be able to read that file um yeah permissions are very important uh SSH agents uh we talked a little bit about it before that's where your credentials can be cached uh really handy thing what I do a lot is I'll SSH to another machine at a customer site and then I have once I do that I have to SSH to a different machine within their network um and then on the There's a special name for that uh jump host or bass host yeah or jump box yeah jump box or uh um I'll uh I'll SSH to that second machine and then I need to run git on that second machine and if I don't have private keys on those machines which I don't want anyway but I need to talk to Git I need like to GitHub um I need my public key I I need this whole encryption thing to come back to my local desktop in order to establish the connection and that's done with SSH agents and with forwarding the SSH agent that's a config entry in your in your config file it's disabled by default so if you're doing this like jump host kind of thing you're gonna want to turn that on so that uh uh the encryption the the um um the keys uh get processed correctly so that you can actually uh get to your private key on your local machine and remember these private keys are never copied to any other machine ever they're they stay local it's this communication negotiation thing that happens and that's what the agent is doing for you and if you ever change your your public keys you've uh you you've got them stored in your agent the cached in your agent if you ever regenerate your keys and you're wondering why they don't work it's because you need to flush uh the agent cache after modifying your key pairs I've been bit by that one in fact recently when I started playing with with secretive um you want to flush your there's a command to do it uh you can Google for it um flush it and then you'll re-uh import your your keys into your agent uh and and then that'll work so gosh we've been talking for like an hour and 15 minutes I think we're about to wrap it up I want to mention a couple of alternatives to SSH um there's something called MOSH it's mobile shell m o s h um and that that's an interesting terminal emulator not an emulator but a terminal program that lets you run remote commands uh it's interesting because it uses ssh for to establish the initial connection but then the connection reverts to a UDP connection and it's it's very resilient for uh like if you're roaming you know let's say you're on a laptop or an iPad or something a tablet and you're moving around and your IP address might change um Mosh handles that really well um and it's secure it and it's it is kind of interesting. If you're more of a graphical thing there's RDP the remote desktop protocol that like Windows uses that's a remote connection thing. WinRM and and PowerShell they allow you to run remote commands. PS exec I guess is one command that people use to run remote commands. And then VNC that's secure that's um that runs over the over the internet in a secure way uh it's a graphical thing. And then if you're using tail scale we mentioned that in in our episode a long time ago about uh VPN my favorite i i love tail scale you do don't you i'm I'm so happy I turned you on to that that is such a great uh great thing I could talk for a long time about how great that company is uh but they have a SSH uh built into tail scale which uh I I it I don't know exactly how it works. I'm not using it but it's like if you have this tail net that's that's a little network of all your machines that are joined together with Tail Scale uh you can SSH between them um and and it manages like the host names and stuff. So that exists. So I don't know if you walk away with anything from this talk um uh use key pairs uh if you can store your private key in a uh secure area like a uh the Enclave or a TPM um store your your keys in uh one password if you want something that's a little bit more portable. Uh that uh you know the advantage there Wolf is you've got a desktop computer and uh and a laptop computer or a couple of laptop computers and one password is uh storing that for you.

unknown 1:20:03

Right?

Wolf 1:20:04

Yeah we should probably point out that um the thing people consider to be good behavior with SSH key pairs is that one uh combination of machines equals one key pair. So my desktop and GitHub is one key pair and my laptop and GitHub is one key pair. Do I do that? Okay no I don't uh and one password violates this because onepassword is I have a key pair that connects me to GitHub and I get to use that same key pair on every machine because my one password keys are on every machine. So you you're kind of picking do I want to believe more in the way one password works here or in the way people want you know say that is good behavior for SSH. So and I picked the one I liked.

Jim 1:21:12

So I I said I talked to like 15 different machines. So the way you said is good behavior would be I would generate 15 private keys assuming just on my desktop I would create 15 private public key pairs and I would pick the right public key to put on each of those remote machines. They would all be different.

Wolf 1:21:32

15 pairs of keys oh it's worse than that you you'd put on in each one of those the 15 yeah you what you said is right.

Jim 1:21:45

Yeah I was right so I've got a desktop and I also have a laptop so it would be another 15 pairs of keys so now each remote host is going to have two public keys one for my desktop one for my laptop that becomes unmanual you're the only it does not scale you're the only person I've ever heard that has done that I'm perfectly and I don't and I don't do it anymore because it's so hard it's so hard yeah I I'm perfectly happy with on my desktop I have a uh a private key and a public key and I install that public key out there on on the remote machines and then on my laptop I have another private key and a public key and I install that public key on all my machines and I actually have two laptops and I and I do that for each. But total I have three different private keys. In fact to note that if you were Google if you were using certificates this uh one key pair per computer pair thing would be trivial um because you don't install the public key at the destination sure sure sure because then you're using a different certificate every time I think I don't know I don't know anyway whatever the point is the the the common wisdom does not scale better to do it the compromise way yeah I think so I think so I I like the way I do it and now I really like the way I do it because I'm storing it in the in the secure enclave I I feel good about that I feel like I'm finally doing something right more right than I was anyway uh I I don't know what else to say that's a lot of stuff about SSH um actually more than I thought we would talk about but it's all interesting things and people should uh use key pairs uh they should absolutely protect their private keys make sure you get your permissions correct and boy it sure works nice when it's all set up uh I'm just gonna add on to that two things to reinforce words you said earlier um ED25519 yes and get your config right it's worth looking up if you if you want Google to help you if you want ai to help you if you want documentation to help you all of those are fine choices but config is a very important file please do the homework to get it right yeah and it's you know it's like anything once you understand it it's really really simple and I you know trust us understand your config file it's only a couple of lines that you have to worry about and and it makes your life uh much better so or makes it not work at all yeah or or makes it not work at all then you're locked out of the machine forever and then you just go home do you um do you feel like this is it is it time to wrap it up I I yeah let's wrap it up I I don't I I'm sure they're tired by now.

Wolf 1:24:59

Well let me let me put a bow on it then okay um uh a thing that we care about is feedback so uh you can uh reach uh you can see us and our addresses our Mastodon addresses or our email however you want to reach us um at the actual website which is http runtimearguments dot fm we'd love to hear your feedback um I'm not saying we'll always respond directly we always have so far um that would be feedback at runtimearguments dot fm that's straight email we have uh a uh identity on mastodon at runtimearguments at hacyderm.io um talk to us tell us what you want to hear tell us what we said wrong tell us what we said right uh tell us uh that you hate tabs and love white span whatever like whatever you want to tell us we're here for you or maybe you have ideas for things you'd like us to talk about and and that would be best of all yes um I don't think I have anything else uh I'm gonna say goodbye you want to say goodbye Jim I'm all right bye from me Jim and and bye from me thank you for coming I appreciate it all right catch all later