It’s Not What You Learn, But How You Learn - Even If You Are An AI

In this episode of The Agile Software Engineering Deep Dive, Alessandro Guida explores the difference between generating code and learning software engineering.

Modern coding AI has been trained on enormous quantities of publicly available software. Some of that code is excellent, but much of it is also outdated, inefficient, insecure, poorly tested, difficult to maintain, or disconnected from the realities of commercial software development.

The episode examines several recurring concerns in AI-generated software: black-box code that developers may accept without fully understanding, inefficient implementations, error handling that does not integrate with the surrounding product, inconsistent security, and user interfaces created without user research, workflow analysis, prototyping, or human feedback.

It also considers a more systemic risk: what happens when AI-generated code increasingly becomes part of the training material for future AI models? Weak patterns may be copied, normalised, and amplified until commonly generated code is mistaken for well-engineered code.

More agents do not automatically mean better software. Ten coding agents today and fifty tomorrow may simply produce more code, more complexity, and more work that still requires human review.

AI can be an extraordinarily valuable engineering tool, but it does not assume responsibility for the result. The engineer remains accountable for understanding, testing, securing, maintaining, and operating whatever ultimately enters the product.

The episode concludes with The Old Grumpy Engineer’s Trust Test, a humorous examination of whether engineering trust should ever be granted automatically-and a brief, nostalgic appearance from Clippy.

Transcript

SPEAKER_00 0:30

Welcome to the Agile Software Engineering Deep Dive, the podcast where we unpack the ideas shaping modern software engineering. My name is Alessandro Guida, and I've spent most of my career building and leading software engineering teams across several industries. Today we are going to talk about AI-generated code, but perhaps not from the usual angle. The question is no longer whether AI can write code. Clearly, it can. The more interesting question is whether generating code is the same as learning software engineering. There is a reason children do not begin learning English by reading Shakespeare. There is also a reason software engineers study Boolean logic, algorithms, architecture, testing, security, and design principles. We do not simply point them towards the entirety of GitHub and say, read all of this. You are now a software engineer. Yet coding models have learned from enormous amounts of publicly available software. Some of it is excellent. Plenty of it is also outdated, inefficient, insecure, poorly tested, and difficult to maintain. The model learns what code commonly looks like. That is not necessarily the same as learning what good software engineering looks like. In this episode, we will examine some of the problems I have encountered while reviewing AI-generated code. Hidden assumptions, inefficient implementations, disconnected error handling, inconsistent security, and user interfaces designed without interviewing or testing with a single real user. We will also consider what happens when AI-generated code finds its way back into repositories, tutorials, and future training material. The AI learns from human code. Then it begins learning from AI-generated code. Weak patterns may be copied, normalized, and amplified until popularity is mistaken for quality. And adding more agents does not automatically solve the problem. Ten coding agents today and 50 tomorrow may generate more code more quickly. They may also produce more complexity and considerably more software that somebody still needs to understand, test, secure, and maintain. Because AI may be an extraordinarily useful engineering tool, but it remains a tool. It does not attend the incident review, explain the security breach to your customers, or receive the telephone call when production stops working. You do. And yes, before we finish, I will apply the old Grumpy Engineers Trust Test to some of the companies asking us to place enormous confidence in their AI. Clippy may even make a brief return. Let's dive in.

SPEAKER_02 3:17

You sit down at your keyboard, you type like a plain English prompt into your coding assistant, and um, in a matter of seconds, an AI just generates this massive, intricate block of code.

SPEAKER_01 3:28

Right. It feels like absolute magic.

SPEAKER_02 3:30

It really does. You just drop it into your editor, you hit compile, and it runs perfectly on the very first try. I mean, for anyone working in tech today, this scenario is incredibly familiar.

SPEAKER_01 3:40

Aaron Powell Yeah, you get to bypass hours of, you know, tedious syntax wrestling. Yeah. You just jump straight to a working application.

SPEAKER_02 3:47

Aaron Ross Powell Exactly. But uh that distinction, that feeling of magic is kind of the mission of today's deep dive. We're exploring why generating code is, well, fundamentally different from software engineering.

SPEAKER_01 3:58

Aaron Powell Because the immediate gratification is undeniable, right? You're seeing working software just materialize from thin air.

SPEAKER_02 4:03

Yeah.

SPEAKER_01 4:04

But the danger lies in mistaking that localized speed for actual engineering quality.

SPEAKER_02 4:09

Aaron Powell And to unpack that, we're looking at issue number 41 of this uh agile software engineering newsletter. It's titled, It's not what you learn, but how you learn, even if you are an AI. Trevor Burrus, Jr.

SPEAKER_01 4:20

It's a really compelling argument.

SPEAKER_02 4:22

It is. The author basically argues that AI models are nowhere near replacing human engineers. And like the limitation is hardwired into the very method we use to train them.

SPEAKER_01 4:32

Aaron Powell, which is a fascinating angle.

SPEAKER_02 4:34

Yeah. But real quick, before we tear into the actual mechanics of how AI learns to code, a quick reminder for you listening. We are distilling the core insights today, but you should absolutely go read and subscribe to the full original article.

SPEAKER_01 4:48

Oh, definitely. The author includes these excellent illustrations and you know a much more in-depth technical experience that you really don't want to miss.

SPEAKER_02 4:56

Yeah, you need the visuals. Also, while this deep dive is entirely free, it's a massive help to us if you take just a second to press like on whatever platform you're using.

SPEAKER_01 5:05

It really is.

SPEAKER_02 5:06

Subscribe to receive all our future issues and you know, repost and forward this deep dive and the related article to your friends and colleagues. It helps a ton.

SPEAKER_01 5:14

Aaron Powell So jumping in, what I love about this newsletter is that the author avoids all those um typical dramatic predictions about AI taking over the world.

SPEAKER_02 5:23

Aaron Powell Right. No terminator scenarios here.

SPEAKER_01 5:25

Aaron Powell Exactly. Instead, they focus on this quiet, really practical crisis that's unfolding in code bases right now. We are systematically confusing the ability to produce text that compiles with the ability to engineer a resilient system.

SPEAKER_02 5:41

Aaron Powell So to understand why a piece of code can like compile thlawlessly but fail fundamentally as engineering, we have to look at the education of the AI itself, right?

SPEAKER_01 5:52

Aaron Ross Powell We do. We have to compare how humans build competence versus how these large language models were actually trained.

SPEAKER_02 5:59

Aaron Powell Because human competence is rarely just about rote memorization.

SPEAKER_01 6:04

No, not at all. Think about how human engineers learn their craft. They don't just sit there and read millions of lines of finished code.

SPEAKER_02 6:11

Right. They study the foundations.

SPEAKER_01 6:12

Yeah, the constraints of computing. They learn Boolean logic to understand, you know, how decisions are processed at the hardware level. They study data structures from memory allocation. They learn compiler theory.

SPEAKER_02 6:22

They're basically taught how to test assumptions.

SPEAKER_01 6:24

Exactly. How to recognize logical contradictions, and most importantly, how to revise their approach when a system behaves unexpectedly.

SPEAKER_02 6:31

They learn the why before the what. I mean, the author mentions how children learn English. They don't start by analyzing Shakespeare. They start with the alphabet, simple grammar rules, and like continuous correction from teachers. They build a structural foundation.

SPEAKER_01 6:48

But meanwhile, the education of an AI coding assistant completely bypassed all of those foundational principles.

SPEAKER_02 6:55

Oh, entirely.

SPEAKER_01 6:56

These models weren't taught compiler theory. They weren't taught memory constraints. They were just pointed at massive repositories like GitHub and told to ingest billions of lines of existing code.

SPEAKER_02 7:07

Aaron Powell Which is wild when you think about it. It's sort of like trying to train a master chef by um locking them in a room and forcing them to eat 10 million random meals from across the city.

SPEAKER_01 7:18

Aaron Powell That is a perfect analogy.

SPEAKER_02 7:19

Right. You feed them Michelin Star Scallops, sure, but also like thousands of burnt hot dogs.

SPEAKER_01 7:25

Yeah. And you never teach them heat control or flavor profile.

SPEAKER_02 7:28

You never even teach them how to hold a knife. You just tell them to output something that looks like what they ate.

SPEAKER_01 7:33

And what they output is just a statistical average. The AI learned the probabilistic shape of software. The shape of it. It learned what code commonly looks like across millions of repositories. But calculating the mathematical probability of what the next line of code should be, that is a completely different mechanism than reasoning through the architectural logic of why that line must exist in your specific system.

SPEAKER_02 7:59

Aaron Powell Okay, but let me play devil's advocate for a second.

SPEAKER_01 8:02

Go for it.

SPEAKER_02 8:02

If I ask an AI to write a function to sort data and it gives me a function that sorts data, why does it matter if it used probabilistic guessing instead of deterministic reasoning? Like if it works, it works.

SPEAKER_01 8:15

Aaron Powell Well, because it introduces what the author calls the black box problem.

SPEAKER_02 8:18

Aaron Powell The Black Box?

SPEAKER_01 8:20

Yeah. When a human engineer writes a complex function, they understand the hidden assumptions within that code. They know how it's supposed to behave if it receives a null value, or say if the database connection drops for a millisecond. Right. When you accept an AI-generated block of code simply because it compiles and it solves your immediate problem, you are inheriting a black box.

SPEAKER_02 8:45

You know, the inputs and the expected outputs, but nothing else.

SPEAKER_01 8:49

Exactly. You are entirely blind to the hidden assumptions the AI made in between. As the author bluntly states, and I love this quote: accepting code without understanding its internal logic isn't engineering. That is hope.

SPEAKER_02 9:04

Wow. That is hope. I can hear engineers pushing back on this, though.

SPEAKER_01 9:08

That's for sure.

SPEAKER_02 9:09

Let's say I'm staring down a massive sprint deadline, right? I use an AI agent, it generates the component, the code compiles fine, it passes the basic unit tests I wrote. I've just saved myself four hours of work. Isn't that efficiency the entire point of the technology?

SPEAKER_01 9:25

Aaron Powell And that right there is the core trap the newsletter warns about.

SPEAKER_02 9:29

Really?

SPEAKER_01 9:30

Yes. Because you are optimizing for the micro solution while remaining completely blind to the macro environment.

SPEAKER_02 9:35

Aaron Powell Okay. Unpack that for me.

SPEAKER_01 9:37

So an AI model optimizes for the most statistically probable answer based on its training data, right? Right. But it is entirely unaware of the specific architectural constraints of your company's production environment.

SPEAKER_02 9:47

Aaron Powell Walk me through the mechanics of that. Like how does a statistically probable piece of code actually break a real system?

SPEAKER_01 9:54

Aaron Powell Let's use a data filtering function as an example. You ask the AI for code to filter inactive users from a database.

SPEAKER_02 10:02

Sure. Happens all the time.

SPEAKER_01 10:03

It generates a script that uses a nested loop. Now, in your local development environment where you only have, say, 50 test users, this code runs in milliseconds.

SPEAKER_02 10:13

Aaron Powell It passes your unit tests.

SPEAKER_01 10:15

Exactly. It is functionally correct. But a nested loop has what engineers call terrible time complexity.

SPEAKER_02 10:21

Ah, I see where this is going.

SPEAKER_01 10:23

Yeah. If you push that same AI-generated code to a live production server that suddenly has to filter 10 million users, the processing time doesn't just increase proportionally.

SPEAKER_02 10:33

It skyrockets.

SPEAKER_01 10:34

It skyrockets exponentially. The server runs out of memory and the entire application crashes.

SPEAKER_02 10:38

Just because the AI didn't ask you about anticipated user load? I mean, it didn't know if this was a weekend prototype or a global enterprise application.

SPEAKER_01 10:46

Aaron Powell Exactly. The algorithm was mathematically common on GitHub, but architecturally fatal for your specific use case. The AI is completely blind to scale.

SPEAKER_02 10:57

Aaron Powell And this blindness extends beyond just performance inefficiencies, doesn't it?

SPEAKER_01 11:01

Oh, absolutely.

SPEAKER_02 11:02

Yeah.

SPEAKER_01 11:02

The problem compounds significantly when we look at how AI-generated code interacts with the broader operational reality of a mature product.

SPEAKER_02 11:10

Aaron Powell Like when things fail.

SPEAKER_01 11:12

Yes. Specifically what happens when things go wrong.

SPEAKER_02 11:15

Aaron Ross Powell Because that operational reality is a huge void for AI. Let's talk about error handling. Because an AI doesn't know your company's incident response protocols or like who is on call this weekend. Aaron Powell Right.

SPEAKER_01 11:27

When an AI generates a module, it will often include basic local exception handling. No standard try-catch block.

SPEAKER_02 11:33

Trevor Burrus, Jr. Just to prevent the app from crashing immediately.

SPEAKER_01 11:35

Trevor Burrus, Jr. Exactly. But local exception handling is totally disconnected from a mature engineering organization's established telemetry and logging strategy.

SPEAKER_02 11:43

Aaron Powell And by telemetry you mean the systems engineers use to monitor the health of an application in real time.

SPEAKER_01 11:49

Correct. Imagine a user clicks a checkout button on a website. That single request might bounce through five different servers: an inventory server, a payment gateway, a shipping calculator.

SPEAKER_02 12:02

Right, microservices.

SPEAKER_01 12:03

In a mature system, engineers attach a correlation identifier to that request. It's essentially a digital tracking number.

SPEAKER_02 12:11

Okay.

SPEAKER_01 12:12

So if the payment gateway fails, that tracking number allows the engineering team to trace the exact path the user took across all those different servers to find the root cause of the crash.

SPEAKER_02 12:22

So if I use an AI to write the payment module, it doesn't know about my company's specific digital tracking numbers.

SPEAKER_01 12:29

It has no idea they exist. The AI might catch the payment failure, print a generic error message to a local hidden log file, and then just silently drop the user's request.

SPEAKER_02 12:38

Oh wow. So the central monitoring system remains completely blind to the failure.

SPEAKER_01 12:42

Yes. The AI recognized the statistical pattern of an error block, but it completely failed to grasp the operational necessity of translating that error across system boundaries.

SPEAKER_02 12:52

And if we apply that same lack of context to security, I mean the implications go from frustrating to genuinely dangerous.

SPEAKER_01 13:02

Security is entirely dependent on context. Generic security, which is what the AI provides, is not the same as product security.

SPEAKER_02 13:10

Aaron Powell Right. The AI just sees the word authentication and pulls whatever protocol was most popular in its training data.

SPEAKER_01 13:17

Yes. And it has no idea if you are building a casual mobile game to track your workout or a highly regulated hospital database storing sensitive patient records.

SPEAKER_02 13:27

Aaron Powell That's a terrifying distinction.

SPEAKER_01 13:29

Aaron Powell It might implement an authentication library that looks incredibly clean and functional, but completely violates the specific regulatory obligations of the healthcare industry.

SPEAKER_02 13:38

Or it might grab an outdated library.

SPEAKER_01 13:40

Exactly. It might utilize a cryptographic standard that was considered secure when the model scraped GitHub three years ago, but has since been deprecated because someone discovered a vulnerability.

SPEAKER_02 13:50

Because you can't just bolt security onto a product after the fact. It has to be engineered into the architecture based on the specific threat model of the data you're protecting.

SPEAKER_01 13:59

And the AI simply doesn't have the situational awareness to make those threat assessments.

SPEAKER_02 14:04

You know, that lack of situational awareness is also why AI struggles so profoundly with user interfaces, I think.

SPEAKER_01 14:11

Oh yeah. The author is brutal about this in the newsletter.

SPEAKER_02 14:13

Aaron Powell Yeah. They point out that whenever AI generated software has to interact directly with human beings, the resulting UX is usually just terrible.

SPEAKER_01 14:22

Aaron Ross Powell They mentioned seeing AI interfaces that would make a professional UX specialist reconsider their career.

SPEAKER_02 14:28

Trevor Burrus Which is hilarious, but why is the AI so uniquely bad at this?

SPEAKER_01 14:32

Aaron Ross Powell Because good user experience design requires human empathy.

SPEAKER_02 14:36

Empathy, yeah.

SPEAKER_01 14:37

And an understanding of human frustration. When human engineers design an interface, they observe real workflows, they test prototypes, they watch where a stressed, distracted user gets confused or clicks the wrong button.

SPEAKER_02 14:50

Right.

SPEAKER_01 14:51

The AI cannot do this. It has never interviewed a user. Trevor Burrus, Jr.

SPEAKER_02 14:54

It's just assembling digital building blocks based on proximity.

SPEAKER_01 14:58

It places a submit button next to a delete button, not because it makes logical sense for a human workflow, but because statistically those two buttons frequently appear in the same files across its training data.

SPEAKER_02 15:11

So the button technically works, it submits the form, but it totally ignores the ergonomic reality of the human being trying to use it.

SPEAKER_01 15:18

You cannot discover human empathy by reading other people's code.

SPEAKER_02 15:22

That is such a good point. So if we zoom out and look at the landscape we are creating, we have engineers prioritizing speed, accepting code with hidden architectural flaws, generic security, poor user interfaces. Where does all this contextless code ultimately end up?

SPEAKER_01 15:44

Well, it goes into production code bases, it gets pushed to public repositories, and it fills up programming tutorial sites.

SPEAKER_02 15:50

Which brings us to the massive systemic risk the author highlights the feedback loop.

SPEAKER_01 15:55

The feedback loop of Doom, really.

SPEAKER_02 15:57

Yeah, because future AI models have to train on something. And the internet is currently being flooded with the output of current AI models.

SPEAKER_01 16:04

We are building a digital echo chamber. As AI-generated code proliferates, the next generation of large language models will ingest it as training data.

SPEAKER_02 16:14

And the subtle mistakes, the lack of operational context, those questionable architectural patterns, they all get normalized.

SPEAKER_01 16:21

Exactly. A flawed but functioning algorithm gets generated thousands of times by AI agents. The next AI model scans the internet, sees that algorithm everywhere, assumes it is the established industry standard, and begins generating it millions of times.

SPEAKER_02 16:36

Wow. So popularity masquerades as quality.

SPEAKER_01 16:39

Just because a million AI agents were to function a certain way doesn't mean it's the most secure or efficient way to write it.

SPEAKER_02 16:45

No, it just means it's common.

SPEAKER_01 16:46

We risk a scenario where models learn from the synthetic output of other models, slowly drifting further and further away from intentional human-directed engineering.

SPEAKER_02 16:56

Aaron Powell Which actually brings up a critical point the author raises about trust. They admit they might sound like a grumpy old school engineer longing for the days when software was shipped on physical floppy disks.

SPEAKER_01 17:07

Oh, the floppy disk days.

SPEAKER_02 17:09

Right. But they ask a really fundamental question. Do we actually trust the companies building these AI tools with our engineering foundations?

SPEAKER_01 17:18

And to be clear, the author synthesizes a look at the major players here: Meta, XAI, Microsoft, Anthropic, and forces the reader to consider the conflict of interest between corporate incentives and engineering reliability.

SPEAKER_02 17:33

And just to pause here, we are just conveying the author's impartial analysis here for you listening. We aren't endorsing any specific political or corporate stance. We're just looking at the engineering implications.

SPEAKER_01 17:44

Exactly. The author is pointing out that we're being asked to hand over the bedrock of our software generation to companies whose historical incentives don't necessarily align with careful, resilient engineering.

SPEAKER_02 17:54

Aaron Powell Because you have companies whose historical corporate DNA was built around hyper-engagement, right? Moving as fast as possible to capture user attention. Trevor Burrus, Jr.

SPEAKER_01 18:02

Or companies pushing product integration at all costs.

SPEAKER_02 18:05

Yeah, the author specifically calls out the frustration of Microsoft constantly interrupting workflows to push copilot. They literally note that it makes them miss the days of the Clippy Assistant.

SPEAKER_01 18:14

Aaron Powell Which is saying something. But the underlying engineering question is how do those corporate traits impact the reliability of a coding agent?

SPEAKER_02 18:22

Right. Can you separate a tool from the incentives of the organization that built it?

SPEAKER_01 18:27

The author suggests you can't. If a company treats unpredictability as an acceptable product feature, or if they prioritize speed of market adoption over cautious safety testing, that philosophy is going to bleed into the code their AI generates for you.

SPEAKER_02 18:42

You know, the author also mentions anthropic, noting they remain cautious about companies that cry wolf on safety while still balancing heavy business incentives. It all comes back to trust.

SPEAKER_01 18:53

Trusting the output requires trusting the motivations of the creator. And in a landscape driven by intense commercial pressure to dominate the AI market, blind trust is just a dangerous engineering strategy.

SPEAKER_02 19:04

It really is. And this all leads to the ultimate reality check of this deep dive. The author isn't saying we should abandon AI.

SPEAKER_01 19:10

No, not at all. It is an extraordinarily powerful tool.

SPEAKER_02 19:13

Yeah, it can accelerate boilerplate development, assist in rapid refactoring.

SPEAKER_01 19:17

It acts as a tireless pair programmer. But utilizing a tool does not transfer accountability.

SPEAKER_02 19:23

No. The speed of code production might have shifted to the AI, but the burden of engineering remains entirely on the human. Because when a mathematically probable algorithm causes the central database to crash at 3 a.m.

SPEAKER_01 19:36

on a Sunday, the AI doesn't get the pager alert.

SPEAKER_02 19:39

No. The AI doesn't have to join the frantic incident response call. The AI doesn't sit across the table from a compliance regulator to explain why a generic security library led to a massive customer data breach.

SPEAKER_01 19:51

Trevor Burrus, Jr. You do. The engineer remains fundamentally responsible for whatever ends up in the production environment.

SPEAKER_02 19:57

Absolutely.

SPEAKER_01 19:57

If you are using an AI to generate software that manages financial transactions, controls physical machinery, or processes sensitive data, you cannot abdicate your role. Trevor Burrus, Jr.

SPEAKER_02 20:08

You have to review the logic. Trevor Burrus, Jr.

SPEAKER_01 20:09

You have to provide the architectural constraints, you have to supply the operational context that the AI simply does not possess.

SPEAKER_02 20:16

That is such a sobering reminder for anyone working in software development right now. We have covered a lot of vital ground today, unpacking the difference between probabilistic guessing and deterministic reasoning, the hidden dangers of the black box efficiency trap.

SPEAKER_01 20:32

Aaron Powell The glaring void in operational telemetry and security.

SPEAKER_02 20:36

Right. And the looming threat of the AI feedback loop.

SPEAKER_01 20:40

It's a lot to consider.

SPEAKER_02 20:41

It is. Before we wrap up, a final reminder of our focus rule for today. Please hit like on the platform you are listening on right now, subscribe to catch all our future issues, and forward this a deep dive and the related article to your team.

SPEAKER_01 20:55

Yeah, it is especially vital to share this with that one colleague who might be leaning a little too heavily on their new AI coding assistant to meet their deadlines.

SPEAKER_02 21:04

We all know one. It is a critical discussion about accountability that every engineering team needs to have before an automated mistake reaches production.

SPEAKER_01 21:11

Absolutely.

SPEAKER_02 21:12

I do want to leave you with a final provocative thought to mull over, building on the concept of that digital echo chamber we discussed.

SPEAKER_01 21:18

Okay, let's hear it.

SPEAKER_02 21:19

So if future AI models are destined to train almost exclusively on the sterile, probabilistic, contextless output of other AI models, the baseline of internet code is going to become entirely synthetic, right?

SPEAKER_01 21:33

Yeah, that's the trajectory.

SPEAKER_02 21:34

If that happens, will messy, organically human written code eventually become a highly sought-after premium commodity?

SPEAKER_01 21:42

Oh wow.

SPEAKER_02 21:43

Like 10 or 15 years from now, will tech companies run marketing campaigns bragging about having artisanal, organically human engineered software just to prove to their enterprise clients that their systems possess actual real-world operational context?

SPEAKER_01 21:59

That is, I mean, it flips the current narrative completely. The human touch, the ability to understand nuance, failure, and context becoming the ultimate luxury feature in technology.

SPEAKER_02 22:08

Definitely something to think about the next time you drop a prompt into your coding assistant and hit enter. Thanks for joining us today. We'll see you next time.

SPEAKER_00 22:27

Thank you for listening to the Agile Engineering Deep Dive Podcast. If you found this episode valuable, feel free to share it with someone who might benefit. A colleague, your team, or your network. You can access all episodes by subscribing to the podcast and find their written counterparts in the Agile Software Engineering newsletter on LinkedIn. And if you have thoughts, ideas, or stories from your own engineering journey, I'd love to hear from you. Your input helps shape what we explore next. Thanks again for tuning in, and see you in the next episode.