Agile Software Engineering

Before You Add Macros, Learn BNF (Compiler Theory part 1)

Alessandro Season 1 Episode 38

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 23:19

Send us Fan Mail

In this episode of The Agile Software Engineering Deep Dive, Alessandro Guida begins a two-part journey into compiler theory with a look at Backus-Naur Form, or BNF.
For many software engineers who studied computer science in earlier decades, BNF belonged to the world of compilers, formal languages, syntax, parsing, YACC, and the famous Dragon Book. Today, many developers may never encounter it directly.
But the need for it has not disappeared.
Whenever we add formulas, macros, templates, filters, workflow conditions, configuration syntax, rule engines, or domain-specific languages to an application, we are doing language design. Often informally. Often accidentally.
And that is where complexity starts to grow.
This episode explains why syntax is not just documentation, why a small language is still a language, and why grammar matters when applications become programmable.
It is not a call for every developer to become a compiler engineer. It is a reminder that some old computer science disciplines remain deeply relevant, especially when we give users the ability to express logic inside our systems.
Part 1 focuses on BNF, syntax, and grammar.
Part 2 will continue with compiler thinking more broadly: lex, YACC, compilers, interpreters, the Dragon Book, and why the discipline still matters in the age of modern development tools and AI-generated code.

Support the show

This Podcast is an audio version of the written Agile Software Engineering newsletter.  If you want to go deeper, don't forget to subscribe the newsletter too.

SPEAKER_00

Welcome to the Agile Software Engineering Deep Dive, the podcast where we unpack the ideas shaping modern software engineering. My name is Alessandro Guida, and I've spent most of my career building and leading software engineering teams across several industries. And today I want to take another small historical detour. Not to talk about the newest AI model, not to talk about yet another framework, platform, cloud service, low-code tool, or miracle productivity booster that promises to remove all complexity from software development. Today I want to go back to something older, quieter, and perhaps almost forgotten. Bacchus NAR form. BNF. For many of us who studied computer science some decades ago, BNF belonged to the world of compilers, formal languages, syntax, parsing, and the famous Dragon Book. It was one of those things you learned when computer science still spent time teaching you how programming languages were defined, not only how they were used. At first sight, BNF may look like a historical curiosity, a notation from the age of compiler courses, parser generators, and serious-looking textbooks. But it is much more relevant than it may seem, because every time we add a formula language, a rule system, a template syntax, a query filter, a workflow condition, a configuration format, or a macro feature to an application, we are doing language design. Often without calling it that. And that is where trouble begins. A small language is still a language. It needs syntax, it needs grammar. It needs validation, it needs understandable errors. It needs to evolve without becoming an accidental collection of exceptions, shortcuts, hidden precedence rules, and undocumented behavior. This episode is part one of a short two-part journey into compiler theory. Part one is about BNF, syntax, and why grammar still matters when modern applications become programmable. Part two will look more broadly at compiler thinking itself. Lex, YACC, interpreters, compilers, the Dragon Book, and why this almost forgotten discipline still has something important to teach modern software engineering. This is not nostalgia. It is a reminder that when we allow users to express logic inside our systems, we are no longer only adding features, we are designing languages. And languages need grammar. Let's dive in.

SPEAKER_01

Today we're uh we're actually looking at what you might call a lost foundation of computer science.

SPEAKER_02

Yeah, lost is definitely the right word for it.

SPEAKER_01

Right. And we're pulling this from a really fantastic issue of the Agile Software Engineering newsletter. It's titled Before You Add Macros, Learn BNF.

SPEAKER_02

Which is part one of their series on compiler theory.

SPEAKER_01

Exactly. And our mission today is basically to figure out why the, you know, those simple little text boxes and search filters in your favorite apps are actually secretly ticking time bombs.

SPEAKER_02

Yeah, time bombs that could totally bring the whole system down.

SPEAKER_01

Right. Which I mean, it sounds super dramatic.

SPEAKER_02

Oh, it sounds incredibly dramatic, but uh it is a very real and a very expensive problem that is just completely hidden in plain sight.

SPEAKER_01

Yeah. So to understand why, think about well, think about a house being built.

SPEAKER_02

Okay, sure.

SPEAKER_01

Usually, you know, you walk by and you admire the things you can actually see.

SPEAKER_02

Yeah.

SPEAKER_01

Like the framing or the layout of the rooms, maybe the nice finish on the kitchen cabinet.

SPEAKER_02

Right, the stuff that's obvious.

SPEAKER_01

Exactly. Nobody really stands on the sidewalk, points at the ground, and goes, wow, what a beautifully poured concrete foundation.

SPEAKER_02

Yeah, I've definitely never done that.

SPEAKER_01

Right. But if you skip that concrete and you just start building your walls directly on the dirt, I mean it doesn't matter how beautiful the house looks.

SPEAKER_02

No, of course not.

SPEAKER_01

Eventually the dirt just shifts. The structure starts to sink, and the whole thing basically collapses under its own weight.

SPEAKER_02

Aaron Powell Yeah, because the foundation, I mean, it's entirely invisible, but it's the single variable that dictates whether the structure is actually going to survive contact with reality.

SPEAKER_01

Aaron Powell Exactly.

SPEAKER_02

And you know, for anyone studying computer science back in the 80s or the 90s, Bacchusnar form or BNF, that was literally that concrete foundation.

SPEAKER_01

It was huge.

SPEAKER_02

It was. It was an absolute staple of compiler theory. Like you learn very early on that a programming language isn't just this uh this loose collection of commands. It actually requires a really rigid mathematical grammar.

SPEAKER_01

Aaron Powell But if you look at the landscape today, I mean that foundation has kind of vanished from the surface, hasn't it?

SPEAKER_02

Oh, completely.

SPEAKER_01

We operate in this era of really slick APIs, cloud platforms, and you know, those drag and drop low-code environments. Trevor Burrus, Jr.

SPEAKER_02

Yeah, the low-code stuff is everywhere. Trevor Burrus, Jr.

SPEAKER_01

Right. And everything is just so heavily extracted now that it feels like nobody really needs to think about formal grammar anymore.

SPEAKER_02

Aaron Powell Well, we might not think about it, but the need for it certainly hasn't disappeared. Right. And because we've sort of lost sight of that foundational discipline, developers and product teams are routinely falling into this massive hidden trap.

SPEAKER_01

Aaron Powell Which the author of the newsletter refers to as the trap of the accidental language.

SPEAKER_02

Aaron Powell Yes, the accidental language. And it's a huge trap.

SPEAKER_01

Let's unpack that a bit. Yeah. Because the scenario the article describes is just, well, it's painfully relatable for anyone who has ever built a software product.

SPEAKER_02

Oh, absolutely. It always starts with just one innocent little phrase, right?

SPEAKER_01

It's just a small syntax.

SPEAKER_02

Yes. Just a small syntax.

SPEAKER_01

A product owner comes in and they're like, hey, users just want a simple formula field so they can calculate their own local tax rate.

SPEAKER_02

Or, you know, the support team asks for a tiny template feature for automated emails just to swap out a customer's first name, something super basic.

SPEAKER_01

Right. And on day one, it seems totally trivial.

SPEAKER_02

Completely trivial. I mean, you add a little text box, you write a really quick script that looks for, say, a dollar sign or a bracket, swaps out some text, and you call it a day. It takes maybe an afternoon.

SPEAKER_01

But then reality hits.

SPEAKER_02

Yeah, it always does.

SPEAKER_01

The users get a little taste of that power and they immediately demand more. So week two, they're saying, hey, this is great, but uh, can I use a wildcard?

SPEAKER_02

Right. And week three, they ask, well, what if I want to add a custom variable?

SPEAKER_01

Exactly. And by week four, it's can I nest an IF statement inside another IF statement? Oh, and hey, how do I escape a special character if my customer's name actually has a bracket in it?

SPEAKER_02

And before you know it, you are just constantly patching the code.

SPEAKER_01

Aaron Powell Right. Just band-aids everywhere.

SPEAKER_02

Yeah. You add a special condition for one big enterprise customer, you throw in some quirky shortcut for another one.

SPEAKER_01

Aaron Powell And then let's say someone tries to nest a complex wildcard search inside one of those math formulas.

SPEAKER_02

Aaron Powell Oh, and your simple little script was definitely not built to handle that level of depth.

SPEAKER_01

Aaron Powell No, not at all. So the text parser just gets totally confused, maybe gets caught in an infinite loop, and suddenly your entire application just freezes.

SPEAKER_02

Aaron Powell Yeah. Bit by bit, feature by feature, you've accidentally built a sprawling, wildly complex system.

SPEAKER_01

Aaron Powell Well, let me just push back on this for a second, putting myself in the shoes of like a modern developer.

SPEAKER_02

Okay, sure.

SPEAKER_01

Isn't that just being agile? Like we are iterating, we're listening to our users, we're getting real-time feedback, and we're just giving them the power they're actively asking for.

SPEAKER_02

Right. It feels like that.

SPEAKER_01

So why is that organic growth so inherently dangerous?

SPEAKER_02

Aaron Ross Powell Because when you iterate on syntax without a deliberate plan, you aren't just adding features.

SPEAKER_01

What are you doing then?

SPEAKER_02

You are literally designing a programming language on the fly.

SPEAKER_01

Wow. Okay.

SPEAKER_02

And an undesigned language becomes an unmaintainable nightmare. When a language lacks deliberate architecture, I mean the edge cases just multiply exponentially. Oh I can imagine. Yeah. And the rules become wildly inconsistent.

SPEAKER_01

It's kind of like writing a legal contract without using any punctuation.

SPEAKER_02

Yes, exactly.

SPEAKER_01

You end up with that classic grammar joke, you know, let's eat grandma versus let's eat grandma.

SPEAKER_02

Right. The comma saves lives.

SPEAKER_01

It really does. Because if the user doesn't have a strict, predictable grammar to follow, the system might misinterpret the command entirely and will actually end up eating grandma.

SPEAKER_02

Aaron Powell That is a perfect analogy. Without the comma, without that formal structure, the meaning shifts entirely. Right. And in software, this manifests as silent failures. Or you get those completely cryptic error messages. Oh, I hate those. Right. Have you ever used a search filter in a database where you try to combine an A and D with an ORR and the system just throws up a blank screen?

SPEAKER_01

Yes, all the time. Or you look at the documentation and instead of clear rules, it's just this messy list of 10 hyper-specific examples.

SPEAKER_02

Yeah, and you just have to blindly guess how to combine them.

SPEAKER_01

I have absolutely been there. You end up just copying and pasting their exact example because you know that if you change even a single parenthesis, the whole thing just catches fire.

SPEAKER_02

Exactly. And that fragility happens because syntax is not just user documentation, it is a fundamental engineering decision.

SPEAKER_01

Okay, that's a key point right there.

SPEAKER_02

Yeah. Creating small domain-specific languages is incredibly useful, don't get me wrong.

SPEAKER_01

Sure.

SPEAKER_02

But an undesigned small language means the developers themselves will eventually be terrified to touch their own code.

SPEAKER_01

Because they have no idea what's going to break.

SPEAKER_02

Precisely. They have no idea what undocumented edge cases their users are relying on.

SPEAKER_01

Aaron Powell So if building these tiny features organically is what ultimately causes the collapse, how do we force ourselves to be more deliberate? How do we uh pour the concrete, so to speak?

SPEAKER_02

Aaron Ross Powell By treating syntax as math.

SPEAKER_01

Math. Okay.

SPEAKER_02

Which is exactly what Bacchusnore form or BNF was originally invented to do. It is the historical antidote to the accidental language.

SPEAKER_01

Aaron Powell And the newsletter has a really brilliant section where it demonstrates this, right?

SPEAKER_02

Yeah.

SPEAKER_01

It builds a tiny calculator language from scratch using BNF.

SPEAKER_02

Yeah. Just a feature that lets a system add and multiply numbers.

SPEAKER_01

Now, we are deliberately not going to read the actual BNF code to you on this deep dive.

SPEAKER_02

Please don't.

SPEAKER_01

Yeah, because reading lines of recursive brackets, pipes, and hierarchical terms out loud would basically sound like we're reciting robot poetry.

SPEAKER_02

Robot poetry is honestly the best way to describe it. Audibly, it's totally impenetrable.

SPEAKER_01

Right. But visually, visually, it is fascinating.

SPEAKER_02

It really is.

SPEAKER_01

And I strongly urge you, the listener, to pull up the written article in the Agile Software Engineering newsletter to actually see these examples for yourself.

SPEAKER_02

Yes, definitely that.

SPEAKER_01

Because seeing the syntax visually map out the logic, it makes the entire concept click instantly.

SPEAKER_02

Oh, absolutely.

SPEAKER_01

And when you look at it, it's actually kind of funny. How so? Well, it just looks wildly overcomplicated at first glance. It's this rigid, academic, multi-layered notation just to define what a number is or what a simple plus sign does. Right. You look at it and you think, why on earth am I writing five lines of formal recursive logic just to teach a computer what two plus three means?

SPEAKER_02

It does feel like massive overkill, at least until you hit the core aha moment of compiler theory.

SPEAKER_01

Okay. Lay it on me.

SPEAKER_02

Let's consider a slightly longer math equation, say two plus three times four.

SPEAKER_01

Simple enough. Two plus three, four.

SPEAKER_02

Right. Now, if you solve that on paper, what is your actual process?

SPEAKER_01

Aaron Powell Well, I multiply the three and the four first to get twelve, and then I add the two. So the answer is fourteen.

SPEAKER_02

Exactly. And you do that because human beings intuitively learn the order of operations in grade school.

SPEAKER_01

Right, PIMDIS or whatever. Yeah.

SPEAKER_02

We just know that multiplication has a higher precedence than addition. But a computer does not know that.

SPEAKER_01

Really? It doesn't just know math.

SPEAKER_02

No. A computer just sees a flat string of text characters. If it reads naively from left to right, it adds two and three to get five, multiplies that by four and spits out twenty.

SPEAKER_01

Which is completely wrong.

SPEAKER_02

Exactly. So how does BNF actually solve that order of operations problem without the developer just having to write an endless messy string of if-then statements?

SPEAKER_01

Aaron Powell By using structural depth to enforce the rules.

SPEAKER_02

Okay, what does that mean?

SPEAKER_01

In that quirky robot poetry of BNF, you don't just list symbols in a row, you define a strict hierarchy.

SPEAKER_02

Ah, a hierarchy.

SPEAKER_01

Yeah. You separate the concept of a term, which are the elements being multiplied from a factor. By placing the rules for multiplication lower down, deeper in the grammar tree, you literally force it to bind more tightly.

SPEAKER_02

Oh wow.

SPEAKER_01

Yeah. You are explicitly teaching the computer precedence simply by the physical shape of the rules.

SPEAKER_02

Oh, that makes so much sense. You are entirely removing the ambiguity before the computer even attempts to run the math.

SPEAKER_01

Yes. You're forcing the system to stop treating the input as raw text. Instead, it converts that text into a structured logical tree. Right. It knows exactly what is allowed, what takes priority, and exactly where every single piece belongs. That's brilliant. And looking at the modern landscape, the newsletter makes a really bold claim about why this matters more today than ever before.

SPEAKER_02

It does.

SPEAKER_01

It points out that the boundary between users and programmers is rapidly vanishing.

SPEAKER_02

And you really see that if you think about the software we interact with daily now.

SPEAKER_01

Yeah. AI is everywhere.

SPEAKER_02

Exactly. When a marketing manager writes a highly structured AI prompt where they define variables, they set negative constraints, they dictate an expected JSON output format. What are they actually doing?

SPEAKER_01

Aaron Powell I mean, they're programming.

SPEAKER_02

Exactly.

SPEAKER_01

They wouldn't put it on their resume as software engineering, but structurally, that is programming.

SPEAKER_02

And exactly the same thing happens when an HR director sets up a complex workflow condition in a CRM or when someone builds an automated rule for a ticketing system. Right. We are giving millions of everyday users access to these small, highly specialized languages.

SPEAKER_01

And if we are going to hand everyday users the power to program our platforms, we owe them a predictable grammar.

SPEAKER_02

Aaron Ross Powell We owe them clarity and we owe them safety. If a user makes a structural mistake in their workflow rule, they shouldn't just get a silent failure that slowly corrupts their entire database.

SPEAKER_01

That would be a disaster.

SPEAKER_02

They need a system that understands its own grammar well enough to immediately flag it and say, hey, you opened a parenthesis right here, but you never closed it.

SPEAKER_01

Aaron Powell So does that mean every web developer and product manager out there needs to just drop what they're doing and go back to school to become a compiler engineer?

SPEAKER_02

No, no, not at all. And the source is actually very clear about easing that exact anxiety.

SPEAKER_01

Good.

SPEAKER_02

You do not need to write native machine code generators. You don't need to build complex memory optimizers.

SPEAKER_01

Aaron Powell Okay. So what's the actual goal then?

SPEAKER_02

The goal is simply to adopt the compiler theory mindset.

SPEAKER_01

Aaron Powell Ah, the mindset. So it's about the philosophy of how you handle user input.

SPEAKER_02

Exactly.

SPEAKER_01

Like instead of a bouncer at a club just quickly glancing to see if you have a piece of paper that kind of looks like a ticket. Right. It's more like the bouncer checking if the ticket is for the right day, the right seat, and actually valid for this specific venue before letting you anywhere near the door.

SPEAKER_02

That is a brilliant way to frame it. The compiler mindset essentially means separating your syntax from your meaning.

SPEAKER_01

Separating syntax from meaning.

SPEAKER_02

Yeah. You parse the input before you ever try to execute it.

SPEAKER_01

Let's break down that mechanism a bit. What does parsing actually look like under the hood?

SPEAKER_02

So it happens in distinct stages. First is lexing.

SPEAKER_00

Lexing.

SPEAKER_02

Yeah, that's where the system reads the raw text and chops it up into meaningful tokens. So two plus three becomes a number token, a math operator token, and then another number token. Got it. Then comes parsing. Right, step two. Where the system takes those tokens and builds that hierarchical tree we talked about earlier, what we call an abstract syntax tree or an AST.

SPEAKER_01

AST, right?

SPEAKER_02

Yes. And you validate everything structurally before a single action is actually taken.

SPEAKER_01

So you're designing for extension from day one.

SPEAKER_02

Exactly. So when users ask for those inevitable new features we talked about, your language can actually scale without just collapsing in on itself.

SPEAKER_01

Okay. So if I'm a developer listening to this right now, and I suddenly have this cold sweat, realizing I've accidentally built a sprawling, unmaintainable language in our app's search bar.

SPEAKER_02

Which happens to the best of us.

SPEAKER_01

Right. But where do I turn? Like what is the roadmap out of the accident language trap?

SPEAKER_02

Aaron Powell Well, the newsletter actually provides a really phenomenal library of language design. Oh, nice. Yeah. It lists five foundational texts that chart the whole journey from abstract theory all the way down to modern practical application.

SPEAKER_01

Aaron Powell Let's trace that journey then. Where does the theory begin?

SPEAKER_02

It starts with a real historical document, the LGL 60 Report.

SPEAKER_01

Ah, LGL Vixie by Peter Nohr.

SPEAKER_02

Yes, by Peter Neuer and others. And alongside that is John Bacchus's paper on algebraic language. Okay. These two works basically represent the genesis of formal structure because before this, programming languages were mostly defined by writing out these long-winded prose descriptions and just providing a few examples.

SPEAKER_01

Oh wow. That sounds messy.

SPEAKER_02

It was. But these documents brought actual mathematical engineering precision to syntax. They proved you could systematically define a language rather than just describing it.

SPEAKER_01

Aaron Powell It's the moment the industry basically stopped eyeballing the blueprints and finally started using a ruler.

SPEAKER_02

Exactly.

SPEAKER_01

But how did that raw math translate into the complex systems we use today?

SPEAKER_02

Aaron Powell So that bridge is the legendary text called compilers, principles, techniques, and tools.

SPEAKER_01

Oh, the dragon book.

SPEAKER_02

Yes, universally known as the dragon book because of the literal dragon on the cover. Right. It is the symbolic heavyweight of compiler theory. It takes the formal grammar from Bacchus and Nar and places it into the full pipeline.

SPEAKER_01

The pipeline from text to execution.

SPEAKER_02

Right. It explains the entire mechanism of taking raw source text, running it through the lexer and the parser, building the abstract syntax tree, extracting the meaning, and then finally executing it safely.

SPEAKER_01

Okay, so that covers the grand theory and the full pipeline.

SPEAKER_02

Yeah.

SPEAKER_01

But what about the practical side? Like if I need to actually write the software that reads this grammar, what's my toolkit?

SPEAKER_02

For the actual mechanics of reading grammar, you turn to parsing techniques by Dick Grun and Serial Jacobs. Parsing techniques. Yeah. If BNF is the map, parsing is the vehicle. This book tackles the incredibly complex challenge of writing software that can recognize and structure really messy human input.

SPEAKER_01

Messy human input is definitely the hardest part.

SPEAKER_02

Without a doubt. And finally, for the modern developer, you know, the person building domain-specific tools, complex configuration readers, or workflow engines.

SPEAKER_01

Right. The stuff we build today.

SPEAKER_02

Exactly. The ultimate modern playbook for that is Language Implementation Patterns by Terence Parr.

SPEAKER_01

Ah, I've heard of that one. That's the text that skips the deep abstract theory of machine code generation and just focuses straight on the programmable layers we add to everyday apps.

SPEAKER_02

Precisely. It gives you the actual patterns to safely build rules, filters, templates, and formulas for your users without creating a total maintenance nightmare.

SPEAKER_01

You know, what really stands out about this journey is the unbroken lineage of it all.

SPEAKER_02

Yeah, it's pretty remarkable.

SPEAKER_01

Because even if a developer today isn't using raw BNF, maybe they're using JSON schemas to strictly format their data payloads, or peg grammars for modern parsing.

SPEAKER_02

Or even just writing highly complex regular expressions to find patterns in text.

SPEAKER_01

Exactly. They are still standing squarely on that same concrete foundation.

SPEAKER_02

Absolutely.

SPEAKER_01

They're utilizing tools that were built on the fundamental premise that a formal definition is infinitely better than just a vague intention.

SPEAKER_02

Well said. The notation we use day-to-day might evolve, but that core engineering discipline remains completely untouched.

SPEAKER_01

So to summarize the core lesson from today's deep dive, whenever you allow your users to express complex logic inside your application, you are no longer just designing a feature.

SPEAKER_02

No, you are not.

SPEAKER_01

You are designing a language.

SPEAKER_02

And languages need grammar.

SPEAKER_01

They need grammar. Because if you fail to define that grammar up front, the language is just going to define itself organically.

SPEAKER_02

Yeah, through a really messy web of accidents, edge cases, and incredibly frustrated support tickets. And uh I'd actually love to leave you with a final thought to mull over.

SPEAKER_01

Let's hear it.

SPEAKER_02

The next time a user complains about a quote unquote bug in your application search filter or a workflow tool starts behaving unpredictably, ask yourself, is the code actually broken? Or is the invisible language you accidentally created just speaking a dialect that even you don't fully understand?

SPEAKER_01

Oh wow. That is a phenomenal question to end on. If you want to stop building on dirt and start pouring a solid concrete foundation for your own projects, you really need to subscribe to the Agile Software Engineering newsletter where this deep dive originated.

SPEAKER_02

Highly recommend it.

SPEAKER_01

Seriously, go subscribe right now so you don't miss part two of this series on compiler theory.

SPEAKER_02

Yeah.

SPEAKER_01

And make sure you are subscribed to this podcast as well.

SPEAKER_02

Yes, definitely.

SPEAKER_01

But most importantly, share this deep dive and the agile software engineering newsletter with your network. You know, your team, your product managers, your developer friends.

SPEAKER_02

Sharing is the best way to spread the knowledge.

SPEAKER_01

It absolutely is the best way to spread the knowledge of agile software engineering and stop the plague of accidental languages. Thanks for exploring the foundations with us today, and we'll catch you next time.

SPEAKER_00

Thank you for listening to the Agile Engineering Deep Dive Podcast. If you found this episode valuable, feel free to share it with someone who might benefit. A colleague, your team, or your network. You can access all episodes by subscribing to the podcast and find their written counterparts in the Agile Software Engineering newsletter on LinkedIn. And if you have thoughts, ideas, or stories from your own engineering journey, I'd love to hear from you. Your input helps shape what we explore next. Thanks again for tuning in, and see you in the next episode.

Podcasts we love

Check out these other fine podcasts recommended by us, not an algorithm.

Darknet Diaries Artwork

Darknet Diaries

Jack Rhysider