Agile Software Engineering
This podcast explores how craftsmanship, architecture, engineering rigor, and organizational practices come together in modern R&D environments. Each edition refines and deepens my earlier reflections, building a coherent and evolving body of knowledge around Agile Software Engineering
Agile Software Engineering
Before You Add Macros, Learn BNF (Compiler Theory part 1)
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode of The Agile Software Engineering Deep Dive, Alessandro Guida begins a two-part journey into compiler theory with a look at Backus-Naur Form, or BNF.
For many software engineers who studied computer science in earlier decades, BNF belonged to the world of compilers, formal languages, syntax, parsing, YACC, and the famous Dragon Book. Today, many developers may never encounter it directly.
But the need for it has not disappeared.
Whenever we add formulas, macros, templates, filters, workflow conditions, configuration syntax, rule engines, or domain-specific languages to an application, we are doing language design. Often informally. Often accidentally.
And that is where complexity starts to grow.
This episode explains why syntax is not just documentation, why a small language is still a language, and why grammar matters when applications become programmable.
It is not a call for every developer to become a compiler engineer. It is a reminder that some old computer science disciplines remain deeply relevant, especially when we give users the ability to express logic inside our systems.
Part 1 focuses on BNF, syntax, and grammar.
Part 2 will continue with compiler thinking more broadly: lex, YACC, compilers, interpreters, the Dragon Book, and why the discipline still matters in the age of modern development tools and AI-generated code.
This Podcast is an audio version of the written Agile Software Engineering newsletter. If you want to go deeper, don't forget to subscribe the newsletter too.
Welcome to the Agile Software Engineering Deep Dive, the podcast where we unpack the ideas shaping modern software engineering. My name is Alessandro Guida, and I've spent most of my career building and leading software engineering teams across several industries. And today I want to take another small historical detour. Not to talk about the newest AI model, not to talk about yet another framework, platform, cloud service, low-code tool, or miracle productivity booster that promises to remove all complexity from software development. Today I want to go back to something older, quieter, and perhaps almost forgotten. Bacchus NAR form. BNF. For many of us who studied computer science some decades ago, BNF belonged to the world of compilers, formal languages, syntax, parsing, and the famous Dragon Book. It was one of those things you learned when computer science still spent time teaching you how programming languages were defined, not only how they were used. At first sight, BNF may look like a historical curiosity, a notation from the age of compiler courses, parser generators, and serious-looking textbooks. But it is much more relevant than it may seem, because every time we add a formula language, a rule system, a template syntax, a query filter, a workflow condition, a configuration format, or a macro feature to an application, we are doing language design. Often without calling it that. And that is where trouble begins. A small language is still a language. It needs syntax, it needs grammar. It needs validation, it needs understandable errors. It needs to evolve without becoming an accidental collection of exceptions, shortcuts, hidden precedence rules, and undocumented behavior. This episode is part one of a short two-part journey into compiler theory. Part one is about BNF, syntax, and why grammar still matters when modern applications become programmable. Part two will look more broadly at compiler thinking itself. Lex, YACC, interpreters, compilers, the Dragon Book, and why this almost forgotten discipline still has something important to teach modern software engineering. This is not nostalgia. It is a reminder that when we allow users to express logic inside our systems, we are no longer only adding features, we are designing languages. And languages need grammar. Let's dive in.
SPEAKER_01Today we're uh we're actually looking at what you might call a lost foundation of computer science.
SPEAKER_02Yeah, lost is definitely the right word for it.
SPEAKER_01Right. And we're pulling this from a really fantastic issue of the Agile Software Engineering newsletter. It's titled Before You Add Macros, Learn BNF.
SPEAKER_02Which is part one of their series on compiler theory.
SPEAKER_01Exactly. And our mission today is basically to figure out why the, you know, those simple little text boxes and search filters in your favorite apps are actually secretly ticking time bombs.
SPEAKER_02Yeah, time bombs that could totally bring the whole system down.
SPEAKER_01Right. Which I mean, it sounds super dramatic.
SPEAKER_02Oh, it sounds incredibly dramatic, but uh it is a very real and a very expensive problem that is just completely hidden in plain sight.
SPEAKER_01Yeah. So to understand why, think about well, think about a house being built.
SPEAKER_02Okay, sure.
SPEAKER_01Usually, you know, you walk by and you admire the things you can actually see.
SPEAKER_02Yeah.
SPEAKER_01Like the framing or the layout of the rooms, maybe the nice finish on the kitchen cabinet.
SPEAKER_02Right, the stuff that's obvious.
SPEAKER_01Exactly. Nobody really stands on the sidewalk, points at the ground, and goes, wow, what a beautifully poured concrete foundation.
SPEAKER_02Yeah, I've definitely never done that.
SPEAKER_01Right. But if you skip that concrete and you just start building your walls directly on the dirt, I mean it doesn't matter how beautiful the house looks.
SPEAKER_02No, of course not.
SPEAKER_01Eventually the dirt just shifts. The structure starts to sink, and the whole thing basically collapses under its own weight.
SPEAKER_02Aaron Powell Yeah, because the foundation, I mean, it's entirely invisible, but it's the single variable that dictates whether the structure is actually going to survive contact with reality.
SPEAKER_01Aaron Powell Exactly.
SPEAKER_02And you know, for anyone studying computer science back in the 80s or the 90s, Bacchusnar form or BNF, that was literally that concrete foundation.
SPEAKER_01It was huge.
SPEAKER_02It was. It was an absolute staple of compiler theory. Like you learn very early on that a programming language isn't just this uh this loose collection of commands. It actually requires a really rigid mathematical grammar.
SPEAKER_01Aaron Powell But if you look at the landscape today, I mean that foundation has kind of vanished from the surface, hasn't it?
SPEAKER_02Oh, completely.
SPEAKER_01We operate in this era of really slick APIs, cloud platforms, and you know, those drag and drop low-code environments. Trevor Burrus, Jr.
SPEAKER_02Yeah, the low-code stuff is everywhere. Trevor Burrus, Jr.
SPEAKER_01Right. And everything is just so heavily extracted now that it feels like nobody really needs to think about formal grammar anymore.
SPEAKER_02Aaron Powell Well, we might not think about it, but the need for it certainly hasn't disappeared. Right. And because we've sort of lost sight of that foundational discipline, developers and product teams are routinely falling into this massive hidden trap.
SPEAKER_01Aaron Powell Which the author of the newsletter refers to as the trap of the accidental language.
SPEAKER_02Aaron Powell Yes, the accidental language. And it's a huge trap.
SPEAKER_01Let's unpack that a bit. Yeah. Because the scenario the article describes is just, well, it's painfully relatable for anyone who has ever built a software product.
SPEAKER_02Oh, absolutely. It always starts with just one innocent little phrase, right?
SPEAKER_01It's just a small syntax.
SPEAKER_02Yes. Just a small syntax.
SPEAKER_01A product owner comes in and they're like, hey, users just want a simple formula field so they can calculate their own local tax rate.
SPEAKER_02Or, you know, the support team asks for a tiny template feature for automated emails just to swap out a customer's first name, something super basic.
SPEAKER_01Right. And on day one, it seems totally trivial.
SPEAKER_02Completely trivial. I mean, you add a little text box, you write a really quick script that looks for, say, a dollar sign or a bracket, swaps out some text, and you call it a day. It takes maybe an afternoon.
SPEAKER_01But then reality hits.
SPEAKER_02Yeah, it always does.
SPEAKER_01The users get a little taste of that power and they immediately demand more. So week two, they're saying, hey, this is great, but uh, can I use a wildcard?
SPEAKER_02Right. And week three, they ask, well, what if I want to add a custom variable?
SPEAKER_01Exactly. And by week four, it's can I nest an IF statement inside another IF statement? Oh, and hey, how do I escape a special character if my customer's name actually has a bracket in it?
SPEAKER_02And before you know it, you are just constantly patching the code.
SPEAKER_01Aaron Powell Right. Just band-aids everywhere.
SPEAKER_02Yeah. You add a special condition for one big enterprise customer, you throw in some quirky shortcut for another one.
SPEAKER_01Aaron Powell And then let's say someone tries to nest a complex wildcard search inside one of those math formulas.
SPEAKER_02Aaron Powell Oh, and your simple little script was definitely not built to handle that level of depth.
SPEAKER_01Aaron Powell No, not at all. So the text parser just gets totally confused, maybe gets caught in an infinite loop, and suddenly your entire application just freezes.
SPEAKER_02Aaron Powell Yeah. Bit by bit, feature by feature, you've accidentally built a sprawling, wildly complex system.
SPEAKER_01Aaron Powell Well, let me just push back on this for a second, putting myself in the shoes of like a modern developer.
SPEAKER_02Okay, sure.
SPEAKER_01Isn't that just being agile? Like we are iterating, we're listening to our users, we're getting real-time feedback, and we're just giving them the power they're actively asking for.
SPEAKER_02Right. It feels like that.
SPEAKER_01So why is that organic growth so inherently dangerous?
SPEAKER_02Aaron Ross Powell Because when you iterate on syntax without a deliberate plan, you aren't just adding features.
SPEAKER_01What are you doing then?
SPEAKER_02You are literally designing a programming language on the fly.
SPEAKER_01Wow. Okay.
SPEAKER_02And an undesigned language becomes an unmaintainable nightmare. When a language lacks deliberate architecture, I mean the edge cases just multiply exponentially. Oh I can imagine. Yeah. And the rules become wildly inconsistent.
SPEAKER_01It's kind of like writing a legal contract without using any punctuation.
SPEAKER_02Yes, exactly.
SPEAKER_01You end up with that classic grammar joke, you know, let's eat grandma versus let's eat grandma.
SPEAKER_02Right. The comma saves lives.
SPEAKER_01It really does. Because if the user doesn't have a strict, predictable grammar to follow, the system might misinterpret the command entirely and will actually end up eating grandma.
SPEAKER_02Aaron Powell That is a perfect analogy. Without the comma, without that formal structure, the meaning shifts entirely. Right. And in software, this manifests as silent failures. Or you get those completely cryptic error messages. Oh, I hate those. Right. Have you ever used a search filter in a database where you try to combine an A and D with an ORR and the system just throws up a blank screen?
SPEAKER_01Yes, all the time. Or you look at the documentation and instead of clear rules, it's just this messy list of 10 hyper-specific examples.
SPEAKER_02Yeah, and you just have to blindly guess how to combine them.
SPEAKER_01I have absolutely been there. You end up just copying and pasting their exact example because you know that if you change even a single parenthesis, the whole thing just catches fire.
SPEAKER_02Exactly. And that fragility happens because syntax is not just user documentation, it is a fundamental engineering decision.
SPEAKER_01Okay, that's a key point right there.
SPEAKER_02Yeah. Creating small domain-specific languages is incredibly useful, don't get me wrong.
SPEAKER_01Sure.
SPEAKER_02But an undesigned small language means the developers themselves will eventually be terrified to touch their own code.
SPEAKER_01Because they have no idea what's going to break.
SPEAKER_02Precisely. They have no idea what undocumented edge cases their users are relying on.
SPEAKER_01Aaron Powell So if building these tiny features organically is what ultimately causes the collapse, how do we force ourselves to be more deliberate? How do we uh pour the concrete, so to speak?
SPEAKER_02Aaron Ross Powell By treating syntax as math.
SPEAKER_01Math. Okay.
SPEAKER_02Which is exactly what Bacchusnore form or BNF was originally invented to do. It is the historical antidote to the accidental language.
SPEAKER_01Aaron Powell And the newsletter has a really brilliant section where it demonstrates this, right?
SPEAKER_02Yeah.
SPEAKER_01It builds a tiny calculator language from scratch using BNF.
SPEAKER_02Yeah. Just a feature that lets a system add and multiply numbers.
SPEAKER_01Now, we are deliberately not going to read the actual BNF code to you on this deep dive.
SPEAKER_02Please don't.
SPEAKER_01Yeah, because reading lines of recursive brackets, pipes, and hierarchical terms out loud would basically sound like we're reciting robot poetry.
SPEAKER_02Robot poetry is honestly the best way to describe it. Audibly, it's totally impenetrable.
SPEAKER_01Right. But visually, visually, it is fascinating.
SPEAKER_02It really is.
SPEAKER_01And I strongly urge you, the listener, to pull up the written article in the Agile Software Engineering newsletter to actually see these examples for yourself.
SPEAKER_02Yes, definitely that.
SPEAKER_01Because seeing the syntax visually map out the logic, it makes the entire concept click instantly.
SPEAKER_02Oh, absolutely.
SPEAKER_01And when you look at it, it's actually kind of funny. How so? Well, it just looks wildly overcomplicated at first glance. It's this rigid, academic, multi-layered notation just to define what a number is or what a simple plus sign does. Right. You look at it and you think, why on earth am I writing five lines of formal recursive logic just to teach a computer what two plus three means?
SPEAKER_02It does feel like massive overkill, at least until you hit the core aha moment of compiler theory.
SPEAKER_01Okay. Lay it on me.
SPEAKER_02Let's consider a slightly longer math equation, say two plus three times four.
SPEAKER_01Simple enough. Two plus three, four.
SPEAKER_02Right. Now, if you solve that on paper, what is your actual process?
SPEAKER_01Aaron Powell Well, I multiply the three and the four first to get twelve, and then I add the two. So the answer is fourteen.
SPEAKER_02Exactly. And you do that because human beings intuitively learn the order of operations in grade school.
SPEAKER_01Right, PIMDIS or whatever. Yeah.
SPEAKER_02We just know that multiplication has a higher precedence than addition. But a computer does not know that.
SPEAKER_01Really? It doesn't just know math.
SPEAKER_02No. A computer just sees a flat string of text characters. If it reads naively from left to right, it adds two and three to get five, multiplies that by four and spits out twenty.
SPEAKER_01Which is completely wrong.
SPEAKER_02Exactly. So how does BNF actually solve that order of operations problem without the developer just having to write an endless messy string of if-then statements?
SPEAKER_01Aaron Powell By using structural depth to enforce the rules.
SPEAKER_02Okay, what does that mean?
SPEAKER_01In that quirky robot poetry of BNF, you don't just list symbols in a row, you define a strict hierarchy.
SPEAKER_02Ah, a hierarchy.
SPEAKER_01Yeah. You separate the concept of a term, which are the elements being multiplied from a factor. By placing the rules for multiplication lower down, deeper in the grammar tree, you literally force it to bind more tightly.
SPEAKER_02Oh wow.
SPEAKER_01Yeah. You are explicitly teaching the computer precedence simply by the physical shape of the rules.
SPEAKER_02Oh, that makes so much sense. You are entirely removing the ambiguity before the computer even attempts to run the math.
SPEAKER_01Yes. You're forcing the system to stop treating the input as raw text. Instead, it converts that text into a structured logical tree. Right. It knows exactly what is allowed, what takes priority, and exactly where every single piece belongs. That's brilliant. And looking at the modern landscape, the newsletter makes a really bold claim about why this matters more today than ever before.
SPEAKER_02It does.
SPEAKER_01It points out that the boundary between users and programmers is rapidly vanishing.
SPEAKER_02And you really see that if you think about the software we interact with daily now.
SPEAKER_01Yeah. AI is everywhere.
SPEAKER_02Exactly. When a marketing manager writes a highly structured AI prompt where they define variables, they set negative constraints, they dictate an expected JSON output format. What are they actually doing?
SPEAKER_01Aaron Powell I mean, they're programming.
SPEAKER_02Exactly.
SPEAKER_01They wouldn't put it on their resume as software engineering, but structurally, that is programming.
SPEAKER_02And exactly the same thing happens when an HR director sets up a complex workflow condition in a CRM or when someone builds an automated rule for a ticketing system. Right. We are giving millions of everyday users access to these small, highly specialized languages.
SPEAKER_01And if we are going to hand everyday users the power to program our platforms, we owe them a predictable grammar.
SPEAKER_02Aaron Ross Powell We owe them clarity and we owe them safety. If a user makes a structural mistake in their workflow rule, they shouldn't just get a silent failure that slowly corrupts their entire database.
SPEAKER_01That would be a disaster.
SPEAKER_02They need a system that understands its own grammar well enough to immediately flag it and say, hey, you opened a parenthesis right here, but you never closed it.
SPEAKER_01Aaron Powell So does that mean every web developer and product manager out there needs to just drop what they're doing and go back to school to become a compiler engineer?
SPEAKER_02No, no, not at all. And the source is actually very clear about easing that exact anxiety.
SPEAKER_01Good.
SPEAKER_02You do not need to write native machine code generators. You don't need to build complex memory optimizers.
SPEAKER_01Aaron Powell Okay. So what's the actual goal then?
SPEAKER_02The goal is simply to adopt the compiler theory mindset.
SPEAKER_01Aaron Powell Ah, the mindset. So it's about the philosophy of how you handle user input.
SPEAKER_02Exactly.
SPEAKER_01Like instead of a bouncer at a club just quickly glancing to see if you have a piece of paper that kind of looks like a ticket. Right. It's more like the bouncer checking if the ticket is for the right day, the right seat, and actually valid for this specific venue before letting you anywhere near the door.
SPEAKER_02That is a brilliant way to frame it. The compiler mindset essentially means separating your syntax from your meaning.
SPEAKER_01Separating syntax from meaning.
SPEAKER_02Yeah. You parse the input before you ever try to execute it.
SPEAKER_01Let's break down that mechanism a bit. What does parsing actually look like under the hood?
SPEAKER_02So it happens in distinct stages. First is lexing.
SPEAKER_00Lexing.
SPEAKER_02Yeah, that's where the system reads the raw text and chops it up into meaningful tokens. So two plus three becomes a number token, a math operator token, and then another number token. Got it. Then comes parsing. Right, step two. Where the system takes those tokens and builds that hierarchical tree we talked about earlier, what we call an abstract syntax tree or an AST.
SPEAKER_01AST, right?
SPEAKER_02Yes. And you validate everything structurally before a single action is actually taken.
SPEAKER_01So you're designing for extension from day one.
SPEAKER_02Exactly. So when users ask for those inevitable new features we talked about, your language can actually scale without just collapsing in on itself.
SPEAKER_01Okay. So if I'm a developer listening to this right now, and I suddenly have this cold sweat, realizing I've accidentally built a sprawling, unmaintainable language in our app's search bar.
SPEAKER_02Which happens to the best of us.
SPEAKER_01Right. But where do I turn? Like what is the roadmap out of the accident language trap?
SPEAKER_02Aaron Powell Well, the newsletter actually provides a really phenomenal library of language design. Oh, nice. Yeah. It lists five foundational texts that chart the whole journey from abstract theory all the way down to modern practical application.
SPEAKER_01Aaron Powell Let's trace that journey then. Where does the theory begin?
SPEAKER_02It starts with a real historical document, the LGL 60 Report.
SPEAKER_01Ah, LGL Vixie by Peter Nohr.
SPEAKER_02Yes, by Peter Neuer and others. And alongside that is John Bacchus's paper on algebraic language. Okay. These two works basically represent the genesis of formal structure because before this, programming languages were mostly defined by writing out these long-winded prose descriptions and just providing a few examples.
SPEAKER_01Oh wow. That sounds messy.
SPEAKER_02It was. But these documents brought actual mathematical engineering precision to syntax. They proved you could systematically define a language rather than just describing it.
SPEAKER_01Aaron Powell It's the moment the industry basically stopped eyeballing the blueprints and finally started using a ruler.
SPEAKER_02Exactly.
SPEAKER_01But how did that raw math translate into the complex systems we use today?
SPEAKER_02Aaron Powell So that bridge is the legendary text called compilers, principles, techniques, and tools.
SPEAKER_01Oh, the dragon book.
SPEAKER_02Yes, universally known as the dragon book because of the literal dragon on the cover. Right. It is the symbolic heavyweight of compiler theory. It takes the formal grammar from Bacchus and Nar and places it into the full pipeline.
SPEAKER_01The pipeline from text to execution.
SPEAKER_02Right. It explains the entire mechanism of taking raw source text, running it through the lexer and the parser, building the abstract syntax tree, extracting the meaning, and then finally executing it safely.
SPEAKER_01Okay, so that covers the grand theory and the full pipeline.
SPEAKER_02Yeah.
SPEAKER_01But what about the practical side? Like if I need to actually write the software that reads this grammar, what's my toolkit?
SPEAKER_02For the actual mechanics of reading grammar, you turn to parsing techniques by Dick Grun and Serial Jacobs. Parsing techniques. Yeah. If BNF is the map, parsing is the vehicle. This book tackles the incredibly complex challenge of writing software that can recognize and structure really messy human input.
SPEAKER_01Messy human input is definitely the hardest part.
SPEAKER_02Without a doubt. And finally, for the modern developer, you know, the person building domain-specific tools, complex configuration readers, or workflow engines.
SPEAKER_01Right. The stuff we build today.
SPEAKER_02Exactly. The ultimate modern playbook for that is Language Implementation Patterns by Terence Parr.
SPEAKER_01Ah, I've heard of that one. That's the text that skips the deep abstract theory of machine code generation and just focuses straight on the programmable layers we add to everyday apps.
SPEAKER_02Precisely. It gives you the actual patterns to safely build rules, filters, templates, and formulas for your users without creating a total maintenance nightmare.
SPEAKER_01You know, what really stands out about this journey is the unbroken lineage of it all.
SPEAKER_02Yeah, it's pretty remarkable.
SPEAKER_01Because even if a developer today isn't using raw BNF, maybe they're using JSON schemas to strictly format their data payloads, or peg grammars for modern parsing.
SPEAKER_02Or even just writing highly complex regular expressions to find patterns in text.
SPEAKER_01Exactly. They are still standing squarely on that same concrete foundation.
SPEAKER_02Absolutely.
SPEAKER_01They're utilizing tools that were built on the fundamental premise that a formal definition is infinitely better than just a vague intention.
SPEAKER_02Well said. The notation we use day-to-day might evolve, but that core engineering discipline remains completely untouched.
SPEAKER_01So to summarize the core lesson from today's deep dive, whenever you allow your users to express complex logic inside your application, you are no longer just designing a feature.
SPEAKER_02No, you are not.
SPEAKER_01You are designing a language.
SPEAKER_02And languages need grammar.
SPEAKER_01They need grammar. Because if you fail to define that grammar up front, the language is just going to define itself organically.
SPEAKER_02Yeah, through a really messy web of accidents, edge cases, and incredibly frustrated support tickets. And uh I'd actually love to leave you with a final thought to mull over.
SPEAKER_01Let's hear it.
SPEAKER_02The next time a user complains about a quote unquote bug in your application search filter or a workflow tool starts behaving unpredictably, ask yourself, is the code actually broken? Or is the invisible language you accidentally created just speaking a dialect that even you don't fully understand?
SPEAKER_01Oh wow. That is a phenomenal question to end on. If you want to stop building on dirt and start pouring a solid concrete foundation for your own projects, you really need to subscribe to the Agile Software Engineering newsletter where this deep dive originated.
SPEAKER_02Highly recommend it.
SPEAKER_01Seriously, go subscribe right now so you don't miss part two of this series on compiler theory.
SPEAKER_02Yeah.
SPEAKER_01And make sure you are subscribed to this podcast as well.
SPEAKER_02Yes, definitely.
SPEAKER_01But most importantly, share this deep dive and the agile software engineering newsletter with your network. You know, your team, your product managers, your developer friends.
SPEAKER_02Sharing is the best way to spread the knowledge.
SPEAKER_01It absolutely is the best way to spread the knowledge of agile software engineering and stop the plague of accidental languages. Thanks for exploring the foundations with us today, and we'll catch you next time.
SPEAKER_00Thank you for listening to the Agile Engineering Deep Dive Podcast. If you found this episode valuable, feel free to share it with someone who might benefit. A colleague, your team, or your network. You can access all episodes by subscribing to the podcast and find their written counterparts in the Agile Software Engineering newsletter on LinkedIn. And if you have thoughts, ideas, or stories from your own engineering journey, I'd love to hear from you. Your input helps shape what we explore next. Thanks again for tuning in, and see you in the next episode.
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Darknet Diaries
Jack Rhysider