When you just said that about like, I'm not sure why I just I love putting myself together a macro sandwich and uh just taking a big bite with great power comes great responsibility. Welcome to ADSP the podcast, episode 16, recorded on February 20th, 2021. My name is Connor, and today with my co-host Bryce, we wrap up part two of our two-part discussion on MacBooks. Oh man, my head hurts.
BryceUm and then so so then the other the other like more advanced stuff that you get into with uh with the preprocessor is dealing with commas and um uh removing or adding um parentheses. Um one of the things I was showing my uh my coworker how to do is he was writing a macro where um it was basically a macro that was an abstraction around um something like an if statement. And he wanted to be able to pass arbitrary blocks of C code as the second and third argument of this macro. And he was like, but you know, if there's a comma in there, then uh everything breaks. Uh because then if there's a comma, it'll it'll treat that, you know, that argument as being multiple arguments, um, and that'll be a problem. And so I told him, okay, that's fine, just um uh just always put parentheses around that second and third argument to protect the commas. And then there's a trick that you can use to uh uh always strip leading and trailing parentheses from uh uh an argument. So that way you you, you know, if if I wanted to write something that has some commas in it as a single argument, I just put it parentheses around it, making it into uh uh one of these sort of pseudo data structures. And then later I can just remove out all the parentheses to um uh to get the original argument that the user intended. This is a fairly common pattern throughout boost libraries that do advanced uh macro stuff.
ConorThat's actually so as much as this is sort of saddening me uh and confusing me, that's useful uh because I did not know that. And I'm currently, every once in a while, I live stream porting an old uh C code base that is the implementation of the J language, which is the um predecessor or successor, I should say, to APL. And it originally had 10,000 macros in it. And it it isn't even C. Like, so that's the thing is seeing this now, the Boost preprocessor library, has me thinking that there is C and there's also C, and then within those two languages, you can have like you can build your own like macro DSL because it's not really like I'm not really porting C to C. I'm porting this C macro DSL where they don't use for loops, they use a do macro. Um they they have a macro for everything, like their functions are macros, their you know, ifs and uh you know for loops are macros, and a ton of their macros use this uh parentheses trick where it has like a it's a 10-argument macro and every single one has parentheses around it, uh which I always thought was odd, but clearly you just explained why. So my question is is like is like why? Like is this is this kind of stuff not possible with a hygienic macro system or a language that supports that? Or like clearly someone's gone to the trouble of doing this, and it's if people are utilizing it, it's not just like a fun pet project.
BryceIt it probably is possible with other systems, but I'll I'll tell you the reason why people go to macros. Um, and it's that it it's it's both the advantage and the downside of it that um macros are purely textual. Um there is no um knowledge of you know C semantics, um it's just purely textual and that makes it very powerful. Like if if if there's something that you just can't do in the the C ⁇ language, like j just this very simple model of hey, we're pacing pasting together strings, basically. That's all that we're doing. We're pasting together strings. Um that can be very powerful, and like sometimes there's something that just the syntax of C um prevents you from doing something, um, or prevents you from abstracting around something where that macros could let you do. Um and in particular it can be very useful if you're dealing with things like um uh language extensions or like non-portability or like extensions that only some compilers support or compiler bugs, um, then it bec it can become very useful to use uh macros because um uh you you know again, it's just it's just this purely simple textual model um uh that uh is completely ignorant of the underlying language.
ConorSo yeah, I hear you saying that, but like on one hand it's simple, like textual text replacement is simple, yet in the process of using a simple thing, you've now created a DSL which you need to go and learn.
BryceOh yeah, yeah.
ConorUm which is the thing about like when I am working through the J source code, uh you know, we've managed to already delete like I think seven and a half thousand of the ten thousand macros. Um, but we're still left, I think, with like two or three thousand, and you have to go and learn, like, and that's the thing, is they're not even well documented, they're not well named in the famous style of APL and J languages, like a lot of them are one and two characters, like G A, I inferred stood for get array, so it's for constructing an array type. Um, and then there's a bunch of like variants of G A. There's G A0, G A T S, G A, etc. etc., which is very similar to what you know, not not exactly with the tag zero, tag one, but yeah, there's a trade-off because you don't have like standard macros. It's just it's all custom. Um but I guess that is an argument that you that I hear when it's you know, insert programming language plus you know the standard algorithms that come with it, and people are like, well, you know, I could just use a for loop and I don't need to go learn everything, whereas if I go learn the algorithms, like sure, maybe that gets me something, but I have to go and learn them. So there's there's some there's some uh echoing of me saying, oh, you have to go and learn all the macros, but I just what is your defense to, or maybe there is no defense, and you're just saying, like, this is it has a use, and yes, it's easy to shoot yourself in the foot, because like the the most egregious uses of macros that I've come across in multiple different code bases are when you're looking at a local function that has a macro and you see some variable name and you it all fits on one screen, and like you can't see the variable, and you're like, oh, is this using globals? Okay, that's not great practice. You search for the global, it's not a global, and then if you've seen the problem before, you go, oh no, don't tell me this macro is doing this, where the macro declares the local variable off screen. And so like you have context that is like important for reasoning about local functions or local code, and then you have to step into the macro and find out that oh, we're actually declaring that variable x in the macro. And so it's not just that like you need to know the macros, like you need to understand the implementations of them in certain uses, and that just for me is the level of like obfuscation and and increased difficulty in reasoning about that code is is way, way, way too way past the line that's it's also they're also basically impossible to debug, um uh you know, if you get an error, and and and that can make them very brittle.
BryceLike it's hard, it's hard for macros to to give you good errors. Um, and that in and and in many ways it makes it very similar to um uh type-based uh uh temp like type-based template metaprogramming in C ⁇ , where it was very powerful. They're actually very similar. They were both they're both very powerful, um, they're both very functional, and they both have no facility to give you good diagnostics, um, which can make them very hard to debug. Um, I mean, I wouldn't I would say don't use macros on if there's some other solution that uh uh works just as well or even that works, you know, um 90% as well. Um the places where I'm use macros, it tends to be one, if I need to abstract over non-portability. Um so um if there is some extension or something that one compiler um has that another one doesn't, um using a macro to hide that um I think is very powerful, um, sometimes unavoidable. Um and if I'm gonna do that, I'll you know, I'll try to make the macro as clean as possible. If if I have to pick between having if-deft code versus using a macro, I'll use the macro. And let me explain what I mean by that. One way to deal with non-portability is to say, okay, I'm gonna write one version of the code with an if-def, and then I'm gonna write another version of the code that that doesn't use this extension that's in the else of the if def. Um and uh I like to avoid that as much as possible because i I think it either leads to duplicated code, which is a maintenance burden, you know, you've duplicated the entire thing, or you've made the um the if-def very constrained and you've done it in such a way that it becomes very hard to actually like read what the code means. Um so if instead of having two versions of the code, I can have one version of the code and one macro abstraction, I'd prefer the macro abstraction. The other place where I uh have used macros and where macros have historically been used is to um uh deal with or to to to minimize r repetitive code. It's actually very much the same thing. It's that like, you know, instead of hand instead of writing, you know 20 versions of this function, each of which are slightly different, um, I'll just write a macro that parameterizes the difference. And um again, it's you know, having less repetition in your code, I think can be good. It might make you know, using that macro might make it a little bit more confusing, but it means that if I need to update all 20 of those definitions, I can just do it in one place, you know?
ConorYeah, I think I mean I think yeah, my in my career I've written, I want to say one macro that went into production, and that was the use case where I had a single line that was repeated with like a couple just textual differences that was using the double hash. Um, and so I had a single line macro above like three functions that all fit on the same screen that like drastically simplified the repetition. Um, but like the thing was is like the macro was right there. You could see the entire contents of the macro, uh, contents of the macro. And I still felt bad about it, but um I mean it did make locally the code I think easier to reason about because it was less noisy, but yeah. I mean, I have real mixed feelings because of I I feel like a lot of these sort sort of arguments you can make against like C in general, like, oh you know, very powerful, but very easy to shoot yourself in the foot. However, C comes with an ecosystem of like static analyzers and compilers that have differing levels of warnings that you can enable, and all of that leads you in the right direction. Whereas my guess is that for C macro DSLs, like there is none of that.
BryceUm which means Yeah, maybe I I I th I would say macros were probably were a lot more important to the language in the C plus in the pre-C11 era. Um there were a lot of gaps that uh macros had to fill. Um uh so what there's another there's another pre-processor trick. Um uh I would argue the most ad the most um uh uh advanced trick um which is uh file iteration. Um so this is sort of the way that you can do um uh stateful uh macros. So the basic idea is you have some header that um does not have include guards, and um uh essentially it's intended that the header will be included multiple times, and each different time it will be included with a different um uh you know uh with a variable, some iteration variable that is defined in a different way. Um uh you know, something like what I'm showing you right now. And so, like, you know, you write your you write you write this iteration header um where it uses this, you know, some macro like in, and then every time that the header gets included, it has an increasing value of in. Um uh and it can become a very powerful tool for avoiding repetition. One of the major problems that we had pre-C11 was we didn't have any variadic um uh uh uh template parameters. And so to make something like tuple work before C11, you just had to pick a number of arguments to support, and you'd have to, you know, you'd you'd you'd create um uh a tuple where it has, you know, it has 10 template parameters, each one is defaulted, and then you have you know a specialization for each number of argument and each each number of parameters that have been provided. Um uh you know, this this is how things like tuples and any other variadic structures worked pre-C11. And it was very repetitive to uh write that code by hand. To you know, the only way if the only way that you could write variadic um uh templates was to manually you know unroll things, to manually unroll up to the limit that you wanted to support, um uh that would that could be very error prone and a huge pain for um uh uh for maintenance. And so you know, using the preprocessor to make that a little bit simpler was a common technique. There's even so my my old my old boss and mentor, Hart McKaiser, he wrote Boost Wave, which is um boosts um C preprocessor. Um and I once I once asked him why'd you write your own preprocessor? And he said, you know, I wrote it to embarrass all the vendors because I wrote it in a period of time when nobody had a conformant implementation of the C preprocessor, and so I wrote one to show them that it could be done and to embarrass them. I'm not sure whether whether any vendors were ever aware of this. Um maybe Hartmoot was joking, but that's the story I was told. And one of the features that Boost Wave supported was um it could do partial pre-processing. So you could tell it, hey, just expand this set of macros, or just expand these um uh you know headers. And that could be very useful if you wanted, you know, you wanted to write some of this preprocessor code and then you wanted to um expand some of it out so that you would um avoid some of the compilation time costs. Um uh but you didn't want to, you know, pre-process everything. Um I think we used it in HPX, I think it was used in Boost Phoenix at some point, um, which was a uh a metaprogramming uh library uh in the C in the pre-C11 era. Um yeah.
ConorSo there's two things we should start to wrap up, seeing as we're way past our 30-minute uh goal. But um, you know, I was I was I knew we were gonna talk about the C preprocessor slash macros today, and in the back of my head, I was thinking, you know, the title of our podcast is algorithms plus data structures equals programs. You know, it's we talk about whatever we want, but I just felt this is gonna be a little bit outside the scope. No, sorry. We if you take a look at the link, it'll definitely be in the show notes. We've got all your algorithms and all your data structures. Just defined with magic.
BryceIf they can be called that. Um if uh if a if a random collection of parentheses and text can be called data structures, then indeed we have data structures. Yeah. Yeah, we didn't even talk about some of the um the craziest stuff. Like I wanted to I wanted to show you all of this nonsense for how like you detect whether there's a comma in a certain set of variatic arguments, and then all this craziness for how you detect whether a variatic is uh is empty. Um but uh that would take another that would take another hour, and I I fear I might like break your break your mind if we were to do that.
ConorWell, you never know. We can we might split this into part one and part two, and then we'll we'll give our listeners just a break. Give them a chance to go check out the boost preprocessor uh website. But so final question. Um putting your ISO C Library Evolution Working Group hat on, yeah. I do believe one of the goals of the standards C standards committee is to eradicate all of the use cases of uh the defined macro, and then to eventually, I know it's like a it's a what do they call it, a long-term goal, um, is to remove it. Uh and I if I recall, I think your stance, I don't want to put words in your mouth, I think your stance is that that'll never be possible. Right. Um so do you want to comment on it being an overarching goal slash whether or not you feel that is actually possible?
BryceI think that it is it will never happen because I I think there will always be a use case for um uh for a uh a textual macro system like this. That there's always going to be a use case for doing some sort of simple um uh like textual um uh generation of C ⁇ code. And if we if we eliminated the C ⁇ preprocessor, people would just go, you know, they would go use they would go write Perl scripts or bash scripts to do it, right? They would just go do it with some other tool. Um and and one of the one of the big benefits of the C ⁇ preprocessor, and one of the reasons why it's often used for these things, is because it's like a core part of the language. So Fortran doesn't always so you can write Fortran where it expands, where it has macros, it uses the C preprocessor, but that's not like the default Fortran mode. Um I'm and in fact I'm not even sure whether it's officially supported by the Fortran language. But I'll tell you this, it's definitely a thing that uh uh uh every Fortran compiler supports, where usually it's with a different file extension. So if you just do like if you just compile a.f90 file, it won't support C style macros. But if it's a.fpp file, that will be Fortran with the preprocessor running. Um uh and and that just like signals to me that even if we were to gonna get rid of Um uh the preprocessor, I'm sure that there would still be a mode and that there would still be use cases where people would want C with the preprocessor, as opposed to C without the preprocessor. I would argue that maybe the mistake that C made was it picked the wrong default. Maybe we should have gone the route of Fortran where by default there was no preprocessor. And if you needed the preprocessor, it was something that you explicitly opted into.
ConorI mean, that's a familiar story in C.
BryceBut uh yeah, of us getting the defaults wrong. But I I I would really argue that, you know, I I don't think that we'll ever fully eliminate the use case for it, even with something like const eval, even with something like, you know, hygienic macros, even with something that, you know, with some sort of lazy evaluation mechanism, um uh even with reflection, um uh which static reflection will address a lot of the use cases, as will just the constant uh the expansion of what constExpert can do. Um means that there's a lot of things that you used to have to do in macros that today you can do in just the native language. I think a lot of that will reduce what the use cases for macros have been historically, but um I think that there will continue to be uses and that we'll find that there will be new uses. And again, I I a lot of language features um uh have been or a lot a lot of facilities that are now important to C were first prototyped with macros. You know, Boost had a library that that emulated move semantics um and relied on macros. Um uh the same can be said for a bunch of other C11 facilities. Um so I don't or concept is another good one. A lot of um uh before we had concepts in language, um uh a lot of libraries used um some macro-based concept framework. Um I I think that 10 years from now we'll be using macros perhaps in slightly different ways than we're using them today, maybe to prototype, you know, whatever the new hotness is 10 years from now, what whatever the next upcoming C ⁇ feature is, but I don't think they're going away.
ConorHow do I I I mean I'm not I'm not a compiler hacker yet. Uh so I guess I can't I can't speak to this, but like I've I feel like just because a language feature is prototyped using macros isn't like other languages don't have textual replacement macros, yet they still manage to iterate, and I would feel like that's just because they work on the compiler, like they just make the change in the compiler or the interpreter.
BryceLike that's it's not an unfair point. But but I would argue that a lot of I would argue that a lot of the the most powerful languages in the world do have um some sort of uh macro system, like a lot of Lisp-style languages, and a lot of them have textual um uh mechanisms too. Um uh you know, some languages don't, but a lot of there's a lot of very powerful languages out there, a lot of functional languages that do have facilities very similar to this.
ConorYeah. Well, you definitely well, you've definitely taught me at least one thing. And um maybe one of these days, well, I'll probably I'll wait till I I give a talk that summarizes all the all the crazy stuff that I've I've had to port in the existing code base. But like just in the the last live stream that I did, we I went from like you know a C function with variables declared at the top, and then this do macro for loop, and so like I got rid of the macro, looked at the for loop, and then was like, oh, that's just a std mismatch, and then replaced replaced the for loop with an algorithm. And so yeah, I just my feeling is that a lot of these cases we're getting tools or the tools already exist um to like use something that's not a macro, that's better.
BryceBut yeah, you you may be right. You may be right.
ConorI mean that maybe that's just because I haven't fallen down this macro rabbit hole that clearly you have.
BryceI I don't know why I derive great joy in uh in writing these little macro facilities, but I really do.
ConorUm it actually that just made me think of it's a quote that is not you don't want to be associated with like some people program to be happy, and uh some people are happy by being clever, something like that. Where uh like that's not a quote you want, like the the quote that that came from was a Ruby talk, and it was the person that was uh making the comment was just saying that you know in in Ruby land we really want to generate nice code that's pleasant to read, and and like your programmer happiness is like an important thing. Unfortunately, what makes some people happy is to write incredibly clever code and then a comment that says, Look how clever I am. And uh when you just said that about like I'm not sure why, I just I love putting myself together a macro sandwich and uh just taking a big bite. Sounded a little bit like that, but um, I will have to spend more time. Spend more time with these macros, and uh we're gonna have to come back and revisit this topic. Maybe we'll get it.
BryceOne of the reasons why I might like it is because um uh doing stuff in macros, like doing even very simple things in macros can often be um uh like challenging to do. Like, you know, making something like uh incrementing an integer work. Like, you know, that's not something that it supports natively, but you know, there's that trick that I showed you that you can do to implement it. And I think I just like I I enjoy making those like simple things work, like because the the thing that I'm looking to implement is not something that's very complex, it's very simple, it's very clean, um, and uh there's not a lot of complexity or baggage there. Whereas typically I'm writing concurrent software where um the the thing you're trying to implement is very complex, there's a lot of trade-offs, there's a lot of performance um uh uh you know issues and guarantees and expectations to worry about. And uh, you know, if I'm just having to write a macro that does, you know, the the the increments and integer, like by comparison, that's just like a very um it's a it's it there the problem space is more constrained in a way that's very enjoyable to me.
ConorAlright, so what I'm gonna do is I'm gonna I'm gonna get two different clips for our intro. One is with great power comes great responsibility from Spider Man, and then I'm gonna play the song I've got the power, uh and somehow splice your voice into it, talking about the power of macros.
BryceUm and that'll be I like it. I like it.
ConorThanks for listening, have a great day, and we'll see you in the next episode.