8: I’ve Been Using AI to Code for a Year. Here’s What I Learned

Wolf has a lot to say about AI and coding!

If you’re expecting AI to write entire programs for you while you sit back and watch, we're going to disappoint you right up front—that’s not what these tools do well, and chasing that fantasy will waste your time. But what if we told you there’s a tool that could help you interpret cryptic error messages, navigate tricky syntax in unfamiliar languages, write the documentation you always skip, and spot those needle-in-the-haystack bugs that eat hours of your day—but only if you understand what you’re asking it to do? Today we’re diving into the reality of AI tools for programmers: not the magic bullet some promise, but the practical truth about what actually works and how to avoid the pitfalls that can corrupt your work.

Wolf has been using AI tools in his daily programming work, and he wants to share what he's learned—both the genuine benefits that have made him faster and more effective, and the critical mistakes that can trip you up if you’re not careful. We’ll explore why AI isn’t magic, why being a domain expert matters more than ever, and how these tools are more like really sophisticated auto-complete than the thinking machines some claim them to be. Whether you’re skeptical, curious, or already experimenting, you’ll walk away with concrete strategies for making AI work for you instead of against you.

Take-aways from the episode:

• You must be a domain expert. You must understand the problem, and you must have the knowledge and skill to evaluate the result.  This might be the most important thing I’ve said in this whole episode.
• You must read and understand  everything the AI gives you. This is fundamental. It’s part of your job.  It always has been.
• Working with AI is an iterative collaboration, not just pushing a button on the magic “answer machine”.  You will be doing a lot of work if the problem is hard.  And it’s the same work you’ve been doing, maybe all your life: explaining to the computer exactly what you want.
• You must carefully, fully, and deeply explain to the AI what you want (Prompt Engineering), and even then, you (plural) will iterate over and over and over again before reaching a satisfying answer.
• Current AIs don’t think, they produce the most popular results they can that satisfy your prompt.  They don’t give you the best answer.  They give you the most popular answer.
• AI is great at analyzing and helping you to improve your existing code. It’s less great when you start with nothing.

Hosts:
Jim McQuillan can be reached at jam@RuntimeArguments.fm
Wolf can be reached at wolf@RuntimeArguments.fm

Follow us on Mastodon: @RuntimeArguments@hachyderm.io

If you have feedback for us, please send it to feedback@RuntimeArguments.fm

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky

Transcript

Jim: 0:03

Thanks for tuning in for another episode of Runtime Arguments. I'm Jim McQuillen, and with me is my friend Wolf. Hey everybody. Wolf, how's your week going so far?

Wolf: 0:17

Um well I'm on vacation. I've I've done uh at least two fun things. Uh one is uh I drove down to visit uh my old buddy uh in Ohio, and we went to a Spyrogyra concert that's kind of a jazz fusion band. The lead is a saxophone player named Jay Beckenstein, longtime hero of mine. I played the saxophone for a long time. I had a great time. The other thing I did that I'm feeling pretty good about is I've been using this new tool called Atuan, it's improved history, A-T-U-I-N. Open source, written in Rust, and uh I've been a longtime FC user. F-C stands for fixed command. Um, and the Atuan guys uh accept PRs. So I read all their rules, their contributor guidelines, their how to all that stuff. Um I joined the right Discord and I've started working on a PR. I made an initial stab. I'm working on it. I'm feeling really good about doing some work in Rust. I love this product. I want it to do the things FC did, um, and this is uh a ton of fun. What what have you been up to?

Jim: 1:40

Oh, I've been uh working, gosh, for the past several weeks, integrating uh Square credit card processing into my application for one of my clients. Uh they've they've been doing credit cards the old way and having to key in the information into the system. So now they can they can swipe credit cards and enter credit card information. They can even do payments online, their their their patients can can uh pay their bill online. All because we chose Square to partner with on this. And it's really we we we went into production yesterday, and it's really pretty cool. Find a little a few little issues, but we're we're really happy. The customer's really happy with it. So that's kind of what what I've been up to. Um back to you, Spyrogyra. I remember hearing about them back in the 70s, I think. I don't really know anything about them.

Wolf: 2:28

I I think you're gonna have to uh turn me on to a couple of their albums so I can uh yeah, they uh they have a couple of really famous songs, uh like one is Catching the Sun, uh, but the thing you just said about the 70s, um, that's when they started. Um, and Jay Eckenstein does all the talking during the show, and he said this. I thought it was funny. He said, uh back in the 70s when we started, I was 20. And now we're in the 20s, and I'm 70.

Jim: 3:02

Well, they've got longevity, so you'll have to uh turn me on to some other recordings. Um, you know, this is our I don't know what this is, eighth or ninth episode, something like that. Uh, and we've been getting feedback on past episodes, and we love feedback. That's how we know that we're uh uh doing something right, and sometimes uh maybe we're doing something wrong, and the feedback is a great way to tell us about that. Um we did an episode a couple of weeks ago on uh VPNs, and uh Wolf, you got a chance to actually uh start using Tail Scale. Tell us about that.

Wolf: 3:44

That's right. Um so there's a couple reasons I moved to Tail Scale. First, you were very convincing, and you offered some pretty compelling arguments as to why Tail Scale is pretty good. Um and when you use Tail Scale, you still do the things I've been doing. You SSH between machines, it's just over the tail net. Um, but uh the most important thing is I actually had to move to uh Tail Scale, and here's why. I upgraded from Xfinity, um, who has been nothing but bad experiences for me, um, to a little Indiana uh company called Metronet, and their fiber optics, they're giving me one gig symmetric. Uh but the thing they hooked up in my house is carrier grade NAT. And I didn't know what carrier grade NAT is, but if I s if I break it down, carrier grade NAT means you don't get to open any ports in the in the actual gateway, you don't get to connect to it or control it or anything. So I couldn't do the thing I was doing before, open holes and let SSH through. Um, but tail scale was exactly the answer I needed. I did have to do a little manipulation of how I was using SSH exactly, adding some uh uh forward agent stuff and uh putting some uh on one of my machines I hadn't moved my keys into one password. Um but your advice actually just made everything great. I was super happy to switch to tail scale. Thanks for introducing me to that thought.

Jim: 5:50

I I love hearing that. And if uh if you, the listener, wants to learn more about that, uh look at a previous episode we did on VPNs. Uh I think it might give you some instruction on uh on how you could get going with setting up a VPN. If you need to access your home network from uh outside the house, or maybe you need to access your work network from outside work. Um Tailscale, uh WireGuard, there's uh open VPN, there's all kinds of solutions, and we talk about those in that episode.

Wolf: 6:21

And Tailscale doesn't require you to pay for uh dynamic DNS.

Jim: 6:29

Right. Well, yeah, you don't need dynamic DNS in this situation, so that's good. And there's a nice free tier for Tailscale. Uh and the people at Tailscale are fantastic, they document everything and they're very open about it. So I I just love them. Um another episode we did quite a while ago was on crypto. Um, and one of the things I mentioned during that episode was the guy in Wales who lost his hard drive. That's sort of internet lore at this point. Uh, but James Howells, uh, he lost a hard drive with 8,000 Bitcoin, which is worth uh what's Bitcoin now? Last time I looked, it was like$118,000 per Bitcoin. So he's something north of$800 million on this drive that he can't get at because he lost the drive. Uh his girlfriend inadvertently threw it away in a landfill, and he's been fighting for years trying to get permission to go search the landfill. He was actually going to buy the landfill, uh, but the the the the township that owns it, they're not interested in any of this. So he's sort of uh put it all in the back burner now. He says he hasn't given up, but he's not actively pursuing it. Man, it's close to a billion dollars that he would have had. That's something else. Anyway, that's another episode we did oh two months ago. Uh so go check that one too, uh, the cryptocurrency episode. But let's get on to today, uh, this episode. Um the subject of today's episode is something that Wolf is really passionate about. He's been trying to convince me that I should be using AI. So, Wolf, here's your chance. Convince me.

Wolf: 8:18

Um if you're expecting AI to write entire programs for you while you sit back and watch, um I I'm gonna disappoint you right up front. That that's not what these tools do well. And chasing that fantasy is a waste of time. But what if I told you there's a tool that could help you interpret cryptic error messages, navigate tricky syntax in unfamiliar languages, uh, while the uh write the documentation you always skip and spot those needle-in-the-haystack bugs that eat hours of your day. But only if you understand what you're asking it to do. Uh today we're diving into the reality of AI tools for programmers. It's not the magic bullet some people promise, but the practical truth about what actually works and how to avoid the pitfalls that can corrupt your work. I've been using AI tools in my daily programming work uh for at least a year, and I want to share what I've learned, both the genuine benefits that have made me faster and more effective, and the critical mistakes that can trip you up if you're not careful. We'll explore why AI isn't magic, why being a domain expert matters more than ever, and how these tools are more like really sophisticated autocomplete than the thinking machines some claim them to be. Whether you're skeptical, curious, already experimenting, whatever, you will walk away with concrete strategies for making AI work for you instead of against you. Um I'm a tools guy, and AI has become one of my tools. Um a fundamental one. Uh so before I get to actually talking about it, I want to give three quick disclaimers. Uh the first thing is, oh my god, is AI changing fast? Um the things I say today, they might not be true by next year. They might not even be true by next month. Um AI is a tool. And like many other tools, it can be used for good, for evil, uh, or for everything else. I'm not gonna argue this. It's true of other tools, it's true of AI, we have absolutely seen good, we have absolutely seen evil. Um, and AI answers that happen to be wrong are gonna sound right. Uh the AI is gonna be super confident when it tells you the wrong thing, and the whole time it's gonna be buttering you up to tell you how great you are, um, making you want to believe it. So those are the disclaimers. Um I'm gonna start right in with my opinions. I want you to know what you're getting. Um, I use AI in my work, and I'm I'm telling you that right up front. Um it is helpful to me, and I'm gonna tell you how. So helpful, in fact, that I'm recommending it to you to you, Jim, and to everybody who's listening. Uh I'm careful, and you need to be too, and I'm gonna tell you what to watch out for. If you use it well, it's a great tool. It will increase your speed and the quality of your output. If you use it poorly, it will waste your time and corrupt your work. This is important. Like any tool, AI is a lever. It can help you in the same way and at the same scale that using a really good editor or IDE, or moving to a source-level debugger, or adding source code control, or picking the right languages for your problem, or using the right libraries, or making testing part of your regular process. Perhaps it could help a little more than these things. At the moment, perhaps not. Uh, like the things I mentioned above, you probably won't understand how helpful it can be until you've actually tried it. Remember how skeptical you were uh about source code control?

Jim: 13:11

You had to pull me into you pulled me into using Git. Uh I I didn't understand it, and I don't know how many years ago it was now, but you convinced me it was good, and and I trusted you, and boy, I can't imagine turning back now. Uh, and I kind of think AI might might be like that.

Wolf: 13:32

Um there there is this weird thing going on. I I'm not a hundred percent sure whether it's true or just headlines, but there are places, and God I hope not many, where management is requiring their development staff to use AI, and and usually they have a specific one. Um they measure, they're watching you. I think that's bad all by itself. Um, if you are using AI, and if you don't, they fire you. Um I think that's wrong. I think it's short-sighted, I think it's stupid. I I it might even rise to the level of evil. Um, you don't get fired for using Vim instead of VS Code. The VS Code users are gonna look at you funny. Um if I do this right, you're gonna want to use AI. Um if after you use it it actually proves to be helpful for you. The whole point is it's about what you're building, not how you build it. Um anyway, there are some myths. There are things about AI um that people believe that aren't true, and that affects how they think of it while they're using it. Um the number one thing people think about AI is that AI thinks. AI doesn't think. Um the AI tools we have right now are much more closely related to uh autocompletion um and or a thing called mob programming. It's like pair programming, except instead of one person typing and one person telling them what to do, it's one person typing and five people telling them what to do. Um it's just that the tools that we have in AI right now are so far beyond um what we know of in autocompletion and mob programming uh as to be uh well past what what you might have imagined. I don't know where we're gonna be in the future. Um these things might not be true anymore, but that's how it is right now. It's important to know that AI is trained by looking at lots of input. A lot of input. Um it's really good at leading you towards the average. Uh AI is very much like taking the combined advice of a billion people at once, if they all summarized it down into one very confident suggestion. Uh if what you're looking for is rare, your results will not be as good. If most examples in the wild are wrong, um, for instance, uh the majority of code gets the security handling wrong. And because that's the majority, um, when AI helps you, often AI suggestions are gonna get the security wrong. Um so uh the third uh thing I want to warn you about is that AI doesn't actually invent brand new solutions. I mean m maybe occasionally by accident. Um AI gives you the average. Because AI is trained on samples, not on rules, that means AI is better at examples than it is on calculation. Um when you ask AI uh who's the president, well, that's a fact that it has seen billions of times, it can recite that fact back to you. If you ask AI what two plus two is, you're asking it to do a calculation. Um it doesn't do calculations, it it does uh a sampling. Um so at every query, every time you ask AI something, it doesn't give you the best answer. It gives you um, at least related to your prompt, the most popular answer. Um the I think these are important things.

Jim: 18:43

Uh y you're not doing much to convince me so far that this is good. I uh my limited use of AI so far uh probably is like a lot of people, uh, when I search Google and Gemini gets in the middle of that and offers the the it's usually the first answer, maybe it's always the first answer, uh, is something that Gemini curated for me. And I can't tell you how many times it just hallucinates an answer that just isn't right. Almost daily I'm getting answers out of that that I just can't trust. And I'm I'm not happy about that. And maybe one of the biggest problems I have with it is remember, it used to be when you search Google and it gives you a whole list of answers, and there's always dates on those answers. Um, so I can tell, you know, if I'm searching for something for iOS development, if it gives me an answer from something from you know 2013, I know it's not even gonna be about Swift, it's gonna be about uh uh uh what's what's the old. Objective C. What's that?

Wolf: 19:49

Uh Objective C.

Jim: 19:51

Objective C. Yeah, it's gonna be an Objective C answer, which is useless to me if I'm programming in Swift or now Swift UI. Uh the answer's just Oftentimes are wrong and never have dates, so I don't know how current this information is. Uh, and it's annoying. So I that that's one of the reasons I've been hesitant to go further with AI. And that's uh that's why you're here today to convince me that I should stop worrying about all those things and embrace what what's really out there now.

Wolf: 20:24

Uh there's absolutely uh places where AI falls down more than it succeeds, and browser search, I think, is one of them. Um an interesting property of uh a search with some search engine is that the longer you make your search description, the bigger your query, the narrower uh the response. If you give too many words, uh sometimes there aren't any matches, you just don't find anything. Um so a good search is very cogent. It's a few words that are very important to the thing you're looking for. That's almost exactly the opposite of what you want when you're using an AI in the way I use an AI, in the way you would use an AI, when we're working with code and with uh text and uh when we're trying to make new things or examine old things. A prompt, that's what they call what when you would have done a search in a in a search engine, they call the query. When you're telling the AI uh what you want, they call that a prompt. And building a good prompt, your prompt is going to be long and deep and descriptive, and if you're good at that, they call that prompt engineering. Um so search engines are a place where AI is faced with some challenges I feel like it can't overcome. And of course, uh SEO optimization, I I don't think that helps. There's lots of things in the way uh in that particular case. Um the things I'm gonna talk about, I believe are different and that they will make a difference for you. Um let's start with uh what things AI is good at and what things AI is bad at. Um let's talk about it all with respect to what you and I do, coding. Um, that's our major job. There are things associated with coding. Yeah, we have to write text for documentation uh or um comments or things like that, but a lot of it is about code. Um so this talk is gonna be about using AI to help you code. There are other things to use AI for. I most of them follow the same rules uh I'm gonna talk about here. You will be able to apply. Um so, good and bad, uh a thing you might want to use AI for is coding from scratch. So starting with a blank page, let's write something. Hey AI, write it. Um if you want to build an exceptional API, AI will not take you all the way there. It will give you average answers. Yes, it can help, but exceptional is not something it can do by itself. You have to understand what you need, you have to be able to explain it sufficiently well that the AI can create it, and then you have to understand both what you need and what the AI provided to know if those two things match. And by the way, we we actually have a word for a person with the skill to describe a problem in sufficient detail and correctness to a programmer process such that the correct result is generated. And I think you know this word, Jim. We call that person a programmer. So, that's one of the bad things. Let's talk about one of the good things, a thing I exploit heavily. AIs are great at looking at existing code and analyzing it. Um, they're great at explaining code you don't fully understand yet. Great at, um, as I'm gonna talk about below, pointing out tiny hard-to-spot errors, uh, and great at, since it's good at explaining, again, as I point out below, writing the comments or doc strings. Um there are some specific things that AI does for me that I find super valuable, and a thing to point out is that um a lot of these things that I'm about to say start with a bunch of content from me and the AI works on that. Um the very first thing I said was if you start from a blank page, AI doesn't do as well. If you start from a lot of context, AI does do well. So here are some super valuable things. Um AI can create good names in your program. Maybe you don't need it for one name, but if you have five functions that are all related because either they take the same input or they make the same output, but they do different things with that what you give it, you want related names that are sensible, the right length, are different in the right way, and are the same in the right way. And AI is great at that. Um syntax. This is a thing I use it for all the time. This is especially important for really hard syntax, and I'm I'm looking at Rust here. Um finding libraries in your language that you're working with right now related to the problems you're solving. Um but I I want to caution you about this. I'm gonna talk about why that's not always a good thing uh in just a second. Uh telling me the signature of the function I want to call. Writing documentation for functions or files, modules, specific algorithms, especially for existing code. Um a thing I have a problem with is either my commit message is stupidly short and doesn't say enough or the right thing, or the usual case is my commit message is way too long. The commit message subject goes over 50 characters. Um I'm kind of bad at commit messages. So I really should be better.

Jim: 27:38

How does AI help you with that? Do you upload the the changes? Do you do a git diff and send that to AI?

Wolf: 27:47

Um so in the place where I use uh AI to write commit messages for me is where I'm using PyCharm, which is a JetBrain's editor. All of the JetBrain's editors are built on the same uh internal engine, so they're all alike. Uh it knows about git, it sees what files are gonna be in the commit, um, and it knows what you changed.

Jim: 28:18

Okay, so it's integrated and it and it uh it it it it knows. You don't have to upload a uh a git diff. It it knows what what you're working on and what you've changed.

Wolf: 28:30

Exactly. Wow. Um and and it can write a commit message that is good, that where the subject does fit in 50 characters, where the message body uh actually says what you change, or maybe why you changed it, um completely but briefly. Um so much better than me. Um a thing I love using AIs for is writing tests. I usually use them to write the tests one at a time, and I often know what to test, um, but I can use the AI to help me find the edge cases and make tests that explore those edge cases. Um if I'm doing something, uh I like to use Python a lot. A feature of Python is called a comprehension. Um it's compact and easy to read and explains how to do a particular transformation, but there are also functions, some of them built into Python and some of them in the functools library or wherever, that can do those things instead. Instead of using a comprehension, you can use the map function. Instead of putting an if in your comprehension, you can use the filter function. Instead of nesting a bunch of comprehensions together so that you can iterate over the sum combination of three different lists, and zip is not appropriate for that, you can use the funcTools function product, which will yield back a list of tuples of every combination of the items from each of those lists. For instance, if you're looking at file names in specific languages, um blah blah blah, product can do that. I can have AI show me a comparison. What does my comprehension look like next to the func tools version? And then between those two, I can say, yeah, I'm right. Or, huh, this func tools version is easier to understand. Um that's super helpful to me. Um error messages are a huge thing. Uh when I get an error message, it used to be that the first thing I did was paste it into uh the search field uh of my search engine and hit return.

Jim: 31:24

This is not error messages from AI, this is error messages from your program. Maybe you tried to compile and it spit out a message.

Wolf: 31:32

That's exactly what I'm talking about, yeah. And it's some weird error message. Um or some error message that has a bunch of words in it I don't understand. Um Rust gives you really good and detailed error messages. Um and if you know all of the words it's using, it will lead you right to the right solution. Um, but I don't know some of those words. Uh so I ask AI, I paste the error message into my chatbot, and it gives me a really good explanation of what I did wrong. Um and when uh I'm looking for some needle in the haystack bug uh and I happen to know, oh, it's in this 300 lines of code, um AI can look at that 300 lines of code if I upload the file or paste the 300 lines, or so if I somehow get that code into um uh the AI, uh it'll look and it'll say, Oh, look, you actually modify the uh loop variable inside the for loop, of course you're skipping items. Or or whatever thing it is. It sees that right away. So those are specific things that are very valuable to me. Um and there's some specific things uh that are not usually successful. Um and one of the things that uh a couple of these things have is um they don't start from a first draft or working code or things of that nature. Um if you ask the AI to write an entire program from scratch, uh you're probably gonna be disappointed. Uh if you want to write an entire suite of tests, and you expect that suite to test the right things and or to completely cover the thing you're testing, y you're gonna be disappointed. If you want AI to invent from scratch a complicated, especially multi-step algorithm um with no help, you're probably gonna be disappointed. Um and sometimes uh when I'm asking AI about libraries and APIs and functions, uh, which I said above, it was good at and helped me with, uh sometimes it just hallucinates functions. It hallucinates APIs, it hallucinates libraries. Um you can look uh when it gives it to you and see if it exists or not, or you can just find out when it compiles, or in the case of Python, you can just find out when uh your actual users are beta testing it for you. Um so that's a thing to be careful about. Um there are some specific things you can do to help AI give you the best possible results. Um the first thing to know is that using AI is an iterative collaboration. Both of those words are important. Um if it doesn't do a job for you, um uh what I'm it doesn't do the job for you. You work together, often and most usefully, starting from your existing work, and you during this iterative process constantly supplying new criteria and together, you and the AI, refining the result over and over and over. I have never had AI give me the answer that I wanted on the very first query. That never happens. Um and there's a big difference, as I've already pointed out, between using AI to create something from scratch and from starting on your first draft and using AI to edit, annotate, comment, question, find related thoughts, tell you what could be better, um, tell you where it sees things that are wrong. AI happens to be good at that, whereas creating from scratch, um, as I have used the word before, you're probably gonna be disappointed. Uh the better, deeper, more detailed, more accurate, more precise you can be in describing your problem. This is called your prompt. It's the thing you type into a chatbot. Um, and most of my interaction with AIs is through a chatbot. Uh, I don't have one in my editor or anything like that. Um that's called the prompt, the better your results are gonna be. In fact, this is so important, the ability to do what I just said, to make a prompt that tells the AI the right things, that it has a name. And that name is prompt engineering. I'm probably gonna say that again. Prompt engineering is a skill, it's valuable, it's something some people are good at and some people are bad at. It's like Google search used to be. Remember in the early days when some people really could find what they wanted and other people had no idea what to do? It's like that. Um you're gonna make progress faster on big problems by splitting them into small problems and then approaching in the right order each of them in turn. Uh for instance, if you're providing some service, that's what you want, um, and maybe you've got some code already, maybe you don't, uh, you can you don't want to just ask AI to write the whole thing. You probably want to start with knowing what that service needs to do and figuring out what kind of API you want first, and then implementing functions that satisfy that total API. Um matter what you are using AI for, you absolutely must look at the results to see if they are true or correct and appropriate to the situation. You have to look. And that means you have to be able to tell the difference. You have to understand the problem and the suggested solutions. You must be a domain expert. That might be the most important thing I say when I talk about this whole subject. You've got to know what you're doing. Um, and this means at this moment, vibe coding it i that's not a thing. Uh, if you can't look at the proposed solution and know if it actually solves the problem, if it's a good solution, if it has weaknesses, uh, then fundamentally y your solution is not in the running. Um I know people want to do vibe coding. I know it's exciting. I know it sounds great. I just don't think it's possible. Um, and did I mention you have to look at and understand the proposed answers? Uh it's so important it's worth saying twice. When you're solving a problem with AI, you are almost certain to make multiple, maybe even many iterations. Work with a tool that remembers what you've done and asked so far. Um AI tools don't do that. For instance, search engines are not keeping a context of the questions you've asked them in the way wherein AI answers. But a chatbot in a specific chat, that whole chat it's remembering at once. It doesn't remember any of your other chats or any information from that. But and I'm gonna talk a little bit about this later when I talk about sessions and tokens and things of that nature. Um look like you wanted to say something, Jim.

Jim: 41:12

No, I I was gonna point out this sounds like a session, and and you covered that.

Wolf: 41:17

That is exactly what it is. Um Using a chat interface, uh, and that's my preferred way, I'm gonna talk about which AI tool I use later. But if you do use a chat interface, um, your session, your chat, uh, you need to keep track of how long that is. There are limits. When you decide you're going to attack a new problem, start a new chat. Um that's gonna that's gonna keep it all excuse me, under control. Um but AI uh doesn't always give you good answers. Uh there are times when AI fails you, which for me it does all the frickin' time. And that's the reason you have to understand the problem and have the power to judge the AI's solution. Especially in anything that's uncommon, especially in long chains of iterations where I've asked the AI uh earlier to do some specific thing as part of the solution, and then as we go on, uh it stops doing that thing. Um I have to keep reminding it. Uh that's terrible. Uh especially when the problem to be solved is just too big, complicated, and fragile. Especially where, and I hate this, where the AI finds two opposite solutions, or different anyway, uh, and it keeps flipping back and forth between them. It gives you one solution, you try it, you report that it doesn't work, AI gives you the other solution, you try it, it doesn't work, and then AI gives you the first solution again, and it just wants to go back and forth. That drives me up a wall. Um a thing AIs can do is they can give you code that uses outdated APIs, and there's several reasons for that. One is AIs take a long time to train. They have a window over which they train. Maybe you're using a model that uh has a window that ended two years ago, and there's a new API. Well, the model you're using, the AI, doesn't know that new API. So it's gonna try to write the solution with the old APIs. Um and uh totally hallucinating functions or whole libraries and sticking them in the generated code as though they work. Um and they don't. They're not even there, they they're not real. Um and uh writing security anti-patterns uh because insecure code is the norm, it's the most popular code, it's the most popular answer, um, it's not the exception. AI is often gonna give you insecure code. It's not gonna know what the security problems are.

Jim: 44:39

Uh you're still not convincing me that it's good. I I I I'm looking forward to the part of this episode where you uh actually convince me that I should be using AI.

Wolf: 44:53

Well, um let me tell you how I use AI. Um I'm gonna start by saying AI absolutely helps me. I use it now, I'm gonna continue to use it, I'm recommending it, I'm recommending it to you, Jim. I'm recommending it to everyone listening to this podcast. How you use AI may make you faster or it may make you slower. I use AI to create the things that don't require me, me personally, to create them, but are valuable nonetheless. Comments, commit messages, individual tests, uh to be completely candid, th those are things I kind of skip on when I'm left to my own devices. Uh sometimes those are things that are gonna be very helpful to readers of your code. Uh if you work in a team, if your code is open source, uh those things make a huge difference. Um and they're boring. I don't want to do them. And AI can do them and can do them instantly. Um, and it's terrific. Uh I I want to tell you right away uh that the tool I use the most is Claude. Um I use the Claude chatbot. Um I have used ChatGPT in the past. Um I've used them both enough to have an opinion that for me I like Claude better. ChatGPT is fine, I could live with it. Um, I just prefer Claude. There are a couple of AI powers that ought to be provided by your IDE. You ought to be able to see the calling signature of a function without actually opening up its definition. Um, you know, by hovering over it. It ought to point out what the what the arguments are and what they're what type they're supposed to be and why you would provide a value and what that value might be. Um you ought to be able to see uh the documentation. Uh in Python, when you write a function, uh there is a piece of documentation that comes right under the signature. It's in the form of a string so that it gets compiled into the code and from a REPL you can get at it. It's called a doc string. And in a decent Python IDE, when you hover over some function that's in your code base, you can see that documentation right there. You don't have to go anywhere. And that's helpful if you wrote the documentation, and if um you wrote the documentation in a way that's helpful, um, but no IDE can show you information about a function you don't call. Um and if you aren't using an IDE, uh then maybe figuring out how you could use an IDE is more important than moving straight to AI. Uh IDEs do a lot of things to help you, uh things that are on the way to the things AI does to help you. Um I happen to know, Jim, that you do a great deal of your work in Vim on remote machines. So without a bunch of plugins, which you have problems uh even with the idea of installing, you can't get IDE functionality. Um I don't envy your situation. I feel like you're coding with one hand tied behind your back. Uh I happen to know that you're good and you're getting the job done. I just I just feel like you could be uh going faster and doing better. Um I use Helix and PyCharm. Uh PyCharm is adding AI but does all those things I just said uh without AI at the moment. Uh I'm an expert in Python, but I like Rust. I'm just a neophyte in Rust, and Rust has very high syntax barriers. I use AI all the time to help me write the Rust syntax to say the things I already know I want to say. I use AI to show me alternatives to the thing I've already written in Python, like I said, with funk tools and comprehensions. Um expert or not, there are parts of Python I just don't use all the time. Uh one of the things I don't use in Python is async and await. Um I guess I'm not a Python expert across the board because I'm not an expert in async and await. If I have to use async and await, um an AI can help me do the right thing. Um but I don't just deal with Python in my job. Um if you have to deal with other languages or specific libraries that are so broad you might even call them a language, for instance, the language CSS or the library pandas, I have a good idea what I want to do and a good idea where to look, but AI greatly speeds my work here. Again, starting from what I already know.

Jim: 50:52

Okay, so uh when you want to find out information through AI about CSS or pandas, w what tool do you use for that?

Wolf: 51:02

Um Claude. Claude knows about those things. And uh Claude knows APIs. Um it's modern enough that it uh the window of when it was trained uh is near enough to now that it has a good idea of the APIs for pandas, for instance, or the rules in CSS. Um and not only can I t ask um Claude, and you could do this with whatever AI you want, how to do the thing I already know I want to do, but I can also ask Claude to explain why the thing it just told me works. And sometimes it might be the case that the thing I think I want to do, there might be a better way to do it. Uh not doing that thing, but maybe skipping that thing entirely. And that's a question I can ask. Um using a AI, I can do all of these things. Explain, comment, test, clean up, and not just to my own code, but to legacy code. And uh I've only been in my current job for a little over a year. There is a ton of legacy code. And that code doesn't have tests, and that code doesn't have a lot of comments, and the comments it does have might be out of date. Um, and AI can help me understand that code, to add comments that are useful, to add tests that will make refactoring easy, uh, and just generally improve it in lots of ways. For most programmers, this is where you spend most of your time. Imagine using an AI to get good test coverage on an important legacy module and then refactoring without fear. When I have to solve a hard problem in a new space where I don't know all the needed data structures and algorithms by heart, and the problem is complicated enough to require multiple steps, iterating with AI can help me understand better what I want, um, what data structures and algorithms already exist to help me, uh, and with enough iterations, how to combine those into a solution. For instance, um when I came into this job, uh, and this job is fundamentally about maps and geometry and paths and uh collections of points and lines that are made up of points. Um I didn't know about, I mean, I had geometry in school, and I know a couple of geometry ideas and thoughts, but there are whole libraries about geometry and doing geometry things. Postgres has specific extensions having to do with uh geometry and geography. PostGIS is a thing. Um you can make queries directly about these uh data structures. Um and they're all new to me. I have an idea what goes in and what I need to get out, how I want to proceed, and if I step through it with an AI, I can figure it out. Um that's important to me. Um when I'm looking at a block of code that the compiler has refused, uh I've said this before, where without the AI, I would be Googling, uh, or in my case I use Kagi, not Google. Um AI can find the problem immediately, especially those tiny, almost invisible problems. Um I also use AI to write ordinary non-code text. Email, articles, long-form documentation, essays. In fact, um when I introduced this topic at the top of the show um and talked about what I was going to say, that was a collaboration between me and Claude. Um we went over several iterations. Uh I started with the things I wanted to say. That was input to Claude. Um Claude came back with some improved suggestions. Um I wanted to make sure they were things we weren't missing, so we went over it and over it a couple of times, and finally came to um what I thought was a pretty good outcome. Um that introduction is gonna be uh verbatim in the show notes, so you can read over it. That was a collaboration with an AI. These these other things that I mentioned.

Jim: 56:37

Uh give us an idea how long it took you to generate that uh because it's two paragraphs of pretty interesting information. Well, how long did it take you to do that?

Wolf: 56:48

Um my first draft uh took me about five minutes total. I knew what I wanted, I wrote it down, I uh uh uh uploaded that to Claude and uh typed in a prompt. My prompts are always fairly long, uh, and it gave me a response um which I copied out and put in the notes document that you and I share. That, the thing I just said, that took me about five minutes. The total with iteration and refinement, if I spent 20 minutes total, um that might be a very generous estimate. I I don't think it went as long as 20 minutes.

Jim: 57:35

I thought it was pretty efficient. I was very impressed with the first uh the first one you showed me, and then you and I talked about it for a minute or two, and and you fed it back through, and and it came out with what you see. Uh I was I was quite happy with that.

Wolf: 57:52

Um so the most important thing uh that I'm saying here about using AI for non-code tasks is uh all the same rules apply. Uh it's better to start from your first draft, you're you're going to iterate, you have to know what you want, collaborate. Um of these, the thing I do most is long-form documentation. Um and it's all about analyzing and editing uh what I've written myself. So that's how I use AI. Um, and one of the things I spent a lot of time thinking about is uh is you, Jim. How w if you used AI, how would you use AI? Would it be helpful? And I decided um that there's really two different facets to you in your job. Um you own your company, you wrote uh this uh code from scratch, there's server side and client side, and uh what I'm talking about here is the server-side stuff, which you wrote uh using for the most part Perl and SQL. Two languages you are expert in. Um you don't need help with syntax. Uh you don't need to look up libraries, you don't need explanations of existing code. Um you don't I I think you don't need tests. I don't want to put words in your mouth, um, and I don't know how much you comment things. Um, but this facet um I I don't think I could reasonably recommend AI for you.

Jim: 59:52

U I I I I could see doing the commit messages and the comments. Uh 'cause I struggle with commit messages. Mine are typically too short. And and I I think I could benefit from from introducing AI to that part of my to that half of my work.

Wolf: 1:00:08

Um I also see another facet of you, uh, which is in many ways the opposite. Um in that facet, you are a member of a team, a small team, uh, but a team, and uh the other people on that team write a lot of code. Uh code that will be brand new to you, code in languages that you are not a complete expert in. Code that is using libraries you haven't used before. Um and in fact, you do work with even in your own uh part, in that first facet I talked about, you do work with languages where you're less familiar. JavaScript and JavaScript libraries and HTML. Um these are all places where there are problems you haven't faced before, where there are things that you don't even know you don't know. Um there are things uh where I feel like the way I use AI could be a giant help to you. Um Are you a straight out and out? You're going to be a great candidate for AI? I don't think so. Can AI help you? Can it make you do part of your job better and faster and produce higher quality output? Uh I think the answer to that question is absolutely yes.

Jim: 1:01:45

I'm starting to agree with you. I'm starting to see the the wisdom in what you're talking about. Perhaps uh I I can I can start using some of those things.

Wolf: 1:01:56

Uh and I'm here to help. And I know there are people on your team uh who use AI and they can help too. Um and I think the important thing is to talk about how you can start. Um I think this is important to you, and I think it's important to the people who are listening who haven't dabbled in AI yet. Um the first thing is there's lots of different ways to interact with AI. There's chat, which is my favorite way, uh, there's smart IDEs, um, there's uploading files, uh, there's autocomplete, uh, and some of the tools that you can use include multiple of these capabilities. For instance, I use Claude and I can upload and download files to Claude, and I can make Claude look at my GitHub repositories actually on Git. Um, so let's talk about tools. Um there are at least three that I have experience with and I can talk about. Claude is my favorite, it's what I use all the time. Um ChatGPT is probably the best known chat interface. Um, that's where I started, and I still have a ChatGPT account, and uh a thing at work we call Bing Copilot. I don't know if that's different from just saying the word copilot alone or saying GitHub Copilot, but that's what we use at work. That's the one we're allowed to upload work source to. Um, so I have experience with that. Of all of those, I like Claude the best.

Jim: 1:04:00

Okay, but there's a lot more than that. Uh I I mentioned earlier Gemini. You know, I use it because when I search Google, it it's there. Uh have you used Gemini?

Wolf: 1:04:11

I've experimented with Gemini. I have it on my phone, but I and it might be good. I hear a lot of good things about it, but I haven't experimented enough uh to tell where it falls. Um that's a decision I'm gonna make. Llama.

Jim: 1:04:26

I I see Mark Zuckerberg talking a lot about llama. Uh what do you know about that?

Wolf: 1:04:34

Um well llama 2 was supposedly really, really good, and llama three, apparently, uh a lot of people said it was a step back. I know a lot of meta engineers uh that are smart, kind, good people. I truly respect them. They're great at their jobs, um, but as a company, I'm not a super fan of Meta, so it doesn't occur to me to use Llama.

Jim: 1:05:06

Okay, um Cursor, is that an AI model or just a tool that uses AI?

Wolf: 1:05:13

So Cursor is interesting. Um, it's cross-platform, it is an editor at its heart, and that means you have to be willing to switch editors. It has, or the company has, models of their own, but they also let you uh connect to uh other services like ChatGPT and Claude and whatever and Gemini, I think. Um if you I think it will I don't have a lot of experience with Copilot. I have downloaded it, I have opened up a project, in fact that Attoin project I was talking about earlier, to see what I could use it for, but I just haven't spent enough time to know yet. Um I don't know its powers. Um it is an editor, and I don't want to switch editors, so that's kind of a barrier for me. Um I know uh some of your team members use it and swear by it. There are a lot of people I've heard talk about how great it is. Um that's why I'm looking.

Jim: 1:06:25

So while you're talking about editors, uh some editors I'm hearing a lot about how they have AI built in, like VS Code. I imagine that's probably copilot, right? Built into VS Code.

Wolf: 1:06:38

I so I I don't use VS Code um for a lot of reasons. Uh I believe that its native AI engine model is copilot, but I think, um, and this is kind of a guess, I think, um, because VS Code supports plugins, you can uh have plugins that let you connect to other uh AI models. Um that's my thinking.

Jim: 1:07:12

Uh uh JetBrains, I know you use that. And you said it's it's not using these external models, but it's using its own thing? Or is it not even AI?

Wolf: 1:07:24

Um the version of PyCharm that I'm using right now does not have an external AI. It might not even have what we think of as an AI, but it is, it must have something because it it is doing these commit messages and whatnot in the version I have. I've read and seen YouTube videos that they're going to add full-on AI. I don't know um what models they're going to connect to, if it's gonna be their own or not.

Jim: 1:08:00

Okay, and then there's Xcode, which I use. I do some Swift and Swift UI development using Xcode. And we both saw at uh WWDC back in June that Apple is adding AI into that. Um Apple Intelligence, uh uh I think they're calling it. They're integrating it into the into the Xcode.

Wolf: 1:08:20

Um have you had a chance to look at that, or are we still waiting for the uh the uh uh official release to see uh there's a couple things to say about um Apple, Apple products, Apple privacy uh uh uh position, and um Apple intelligence. And uh one of those things is I don't know that Apple intelligence is yet in a state where it can do all those things I've talked about AI's doing for me. But one great thing about Apple's ideas, Apple's plans, is that um your code doesn't leave your system. It happens right there. And if for some reason it has to leave your system, Apple has a um secure compu I don't I forget exactly the name scheme where they can, yes, run their model in the cloud, but in a way where your data can't be seen by anyone else, including Apple, and then the answers come back down to you. Uh it sounds great. Yeah. If they deliver on the promise of actually doing the things I want them to do.

Jim: 1:09:53

Well, their their track record with privacy is pretty good so far.

Wolf: 1:09:57

It is. Yeah. Um other than Apple, all of the tools that I've that we've both talked about so far, um they do their work in the cloud. That's where their giant models live. Um and that means uh you subscribe and your data leaves your machine and goes to that company. Um consider that. Uh they all provide subscriptions, they all have a free tier, and that's where you should start. Um you often work in sessions, as I've talked about. Your input to the AI is broken up into what is called tokens. Um the AI doesn't learn from your session. Um an AI model spends a long time, uses a lot of input, and burns up a ton of money training. Um, and then once it's trained, it's sealed. It doesn't change after that. You ask it questions. Um The reason it seems to know everything you've asked it so far in a session is not because you're changing the model, it's because with each question you ask, each prompt you give, instead of sending just that prompt, whatever UI you have for that uh AI sends every prompt from that session all together as one thing. So it feels like it you're changing the model, you're making it remember, but you're not. It's just about that session. Um so because of that, if you think you're gonna solve hard problems, you should understand what the limitations are of the particular service you're working with. Um if you have a specific company you prefer, start there. Um, you know, maybe you're a Google person, start with Gemini. Um if you want me to tell you what to use, use Claude. Start at the free tier, just like I said before. I have a subscription. I'm not 100% sure what I pay. I think it's$17 a month. Um start with simple tasks so you understand how to interact. Uh writing good prompts, that's a thing you'll want to learn. How to get your data into and out of the AI you've chosen. Um here's a couple of good initial problems. Uh give it some piece of code, uh, maybe copy and paste, maybe a whole file. If this is legacy code you don't totally understand, ask the AI to explain it. Uh maybe ask how to improve it. Um, have it document individual functions. Like I said before, in Python a public function starts with a special string that explains the uh what, why, and how of the function. Uh we call it a doc string. Um you can reach it from a live Python interpreter, it's introspectable. Have it figure out when you look at a function uh where the edge cases are. Um should it add asserts? Maybe write some tests that understand those conditions. Uh once you've solved smaller problems like these, you're going to be full of ideas. Um, and of course, you'll now have the experience and skill uh to ask harder questions and solve harder problems.

Jim: 1:13:54

Wow. All right. You that's a lot. We we've been talking for uh an hour and 15 minutes. Uh some really interesting stuff there. Can you kind of boil it down into uh a a summary, some takeaways for us?

Wolf: 1:14:12

Yeah. Um there are some very important things uh in that giant pot of uh facts that I provided. But the most important thing, the number one takeaway, the thing you absolutely have to understand is this you must be a domain expert. You must understand the problem, and you must have the knowledge and skill to judge the result. This is the most important thing. You've got to read and understand everything the AI gives you. This is fundamental. It's part of your job. It always has been. Working with AI is an iterative collaboration. It's not just pushing a button on the magic answer machine. You're going to do a lot of work. If the problem is hard, it's more work. Um and it's the same work you've been doing maybe all your life. Explaining to the computer exactly what you want. And that explanation must be made carefully, fully, and deeply. That's your prompt. Uh and even then, you're going to iterate over and over and over again before you reach a satisfying answer. It's really important to know, understand, and remember that current AIs don't think. They produce the most popular results they can that satisfy your prompt. They don't give you the best answer. They don't give you an innovative answer. Uh what they give you is the most popular answer. And AI is fabulous at analyzing and helping you to improve your existing code. It's less great when you start from nothing. I feel like those are my takeaways.

Jim: 1:16:34

Well, one thing I took away from this is uh AI is uh kind of like the old uh Cold War uh uh uh mantra, and that is trust but verify.

Wolf: 1:16:47

Absolutely.

Jim: 1:16:48

Absolutely have to verify what it does. That's pretty good. I I I'm gonna try it. I I I honestly am going to uh create an account on Claude and start using it, see what I can see what I can do. And fortunately I have you to help me out.

Wolf: 1:17:08

Um there's a thing I want to tell you about these chatbots. Um yes, there's a chat, a session, but also there's usually a place to fill in some things that you wanted to always remember about you. Kind of a profile or something.

Jim: 1:17:29

Okay.

Wolf: 1:17:29

Yeah. Um, that's a thing you should do.

Jim: 1:17:32

Okay. Good. I I'll I'll I'll jump in. Uh so thanks everybody for listening. Uh this was a long episode, a lot of information. I I sure hope you got a lot out of it. Um, please send us some feedback. If you liked what you heard, great. Let us know. If there's things that you think we didn't get right, let us know that too. We we want to make things better. We are iterative as well, right? Um, we do have show notes. Uh take a look. Uh however, however, you access this podcast, there should be show notes. Uh, we have a website. You can see the show notes there. The website is runtimearguments.fm. Uh if you do send us feedback, uh our email address for that is feedback at runtimearguments.fm. And uh once again, thank you so much for listening. Tell all your friends about us. Um Wolf, you got anything to say?

Wolf: 1:18:29

Uh, this one was an absolute pleasure for me. AI has been as helpful to me as, you know, all those tools that I mentioned, source level debuggers and source code control systems like Git. I I love tools and I want other people to see the same benefits I'm seeing, but also not fall into the traps.

Jim: 1:18:54

Well, I I think you did a really nice job explaining that, and uh I I'm looking forward to digging in now. So thank you, everybody.