Bryce

I think I'm like the only committee member that's like just thrilled about macros and the pre-processor. Like, I love macros.

Conor

I'm confused on why we're not recording this on like April 1st. Welcome to ABSP the podcast episode 15, recorded on February 20th, 2021. My name is Connor, and today my co-host Bryce tries to convince me why macros are on.

Bryce

Yeah, so I want to talk about the C preprocessor. I'm I think I'm like the only committee member that's like just thrilled about macros and the pre-processor. Like I love macros. I I, Bryce Loback, am a member of the C committee, and I am telling you right here and right now, I love macros. Macros are great. I mean, let me put it to you this way. Let me make you the case. Think of a language feature, like a major language feature from the C11 era. Every one of them at some point was prototyped in some like library-only form and boost, and it would not have been possible without macros. See, like macros are amazing. They're so powerful. They they let you do things that like they let you do syntax hacks that are just not possible uh uh in the regular language. And even a hygienic macro facility like some of the ones that we're looking at having right now, like some could argue that constival functions, which consteval functions in C, it's a it's something that's similar to constexpr, but constexpr says this function may be evaluated um at compile time as a constant expression. You know, evaluated in in the the equivalent of being inlined fully at compile time. Um uh so it's a little bit stronger than constexpr and it's an error if you use it in a way where it would need to be evaluated at runtime. Um but even that it doesn't give you the same power of macros, um, because macros are inherently textual. And so because they're textual, they let you do all sorts of of hacks. Now, of course, um as somebody who has had some involvement in modules in the past, um macro I'll also tell you that you know, macros are terrible, um, and the the whole textual inclusion model for C is a large part of the uh the compile time woes that we have. Um uh and uh while I agree with that, you know, they do have their uses. And uh I wanted to do this episode because recently um recently one of the people in my team, I needed them to create a new fancy macro abstraction for something. Um a portability layer around some new uh thing that we're putting into uh one of our compilers. And uh like as he got started with it, I started teaching him some of the like dark arts of of uh macro hacking in C. And uh and I introduced him to Boost Preprocessor, and just as I was doing that, I just remembered my love and passion for um uh macro programming. Um and I don't know why I love it so much, um, because it's kind of horrendous, but I think that that's part of why um that's part of that's part of the appeal. Is it's like, oh cool, like you know, I you would never think that you could do this thing in the preprocessor, but you can. Did you know the preprocessor is turning complete?

Conor

I did not know that. No.

Bryce

Yeah, it's turning complete. Have you have you ever heard of Boost Preprocessor?

Conor

Uh I've heard of it, but I don't know what it is. I mean, I assume it's adjacent to the C preprocessor, like doing a bunch of stuff with that.

Bryce

It is a pre-processor library. So Boost Preprocessor is um a uh I I would argue it's like the only generic, like general purpose library for pre-processor programming. Um and I think it's it it's it's it's probably gotta be my favorite boost library. Um I I will always have a special affinity for Boost Spirit, which you know I worked on very early in my career, but Boost Preprocessor is the one boost library that like I still use today. Um and it's it's a little bit crazy, all the stuff that you can do with it. Um first of all, let's talk about um uh data types. Um you've probably never thought about preprocessor data types, right?

Conor

No. Okay, so here's so the Just before we get into this, this is gonna be good because you you have you you have a real hard sell here. Because I started my career working for a company that I I I don't want to use the word um I don't want to use that word, but let's just say that they used, they heavily relied, our code base heavily relied on the preprocessor for different like modules that our software system supported. And it led to it led to just crazy things. Like picture the same function with like four different preprocessor contexts around the function uh declaration, like the parameter lists, and they're all lined up for so like you know, module A has you know five of the ten possible uh function parameters or arguments, module B has six out of the ten, and then they line them all up and they do it to the extent that there's like four hundred character function declarations with like six different contexts. So like I've been on the bad side of preprocessors, so I'm you know, good luck, Bryce. Good luck on trying to get me excited about um uh boost preprocessor. Show me the magic here.

Bryce

Alright, alright. So the first thing that you have to understand about um uh preprocessor programming is that parentheses and commas are special. Um so uh because macros are always textual, if you have something, you know, if you call a macro like um if I have a macro called A and I call it with um like I pass in a uh a template type um where I've got uh more than one argument, more than one template parameter to it. So something like, you know, a parens x um y uh x open bracket y comma z. Um uh you might think that this passes one argument to the macro A, and that argument would be x brackets y, comma z close brackets. Uh but that's not actually how it works because it's textual. It doesn't it doesn't recognize uh the angle brackets um as having any special meaning. So this will pass two arguments to A, and those arguments will be X less than Y, comma Z uh greater than symbol. Um follow me so far. But um but parentheses are uh uh special in uh the preprocessor. They they do have a uh uh a special meaning. And so if I was to take that same argument and just surround it in parentheses, then it would be passed through as one argument. Um uh so if if I took that same expression, I just wrapped it in parentheses, then it would be just you know just a single argument to that macro A. And so this becomes the basis of all of the uh pre-processor-based data structures uh that uh uh Boost preprocessor supports. So the first one um is called an array. Um and uh uh it is well actually we we we should we should start with tuples. Um don't you mean? So a tuple are are we gonna are we gonna do this? You want me to call it tuples?

Conor

Well, I don't I mean that's the correct pronunciation of it. Okay.

Bryce

All right, tuples. All right, so a preprocessor tuple is um uh just a comma separated list of elements that are surrounded by parentheses. Um so like you know, opening parentheses one, comma, two, comma, three, closing parentheses. Um and this is like the most fundamental and basic um uh preprocessor data structure. Um and it's one of the more useful ones because it's what you'd um uh it's also what you would use as the argument to a function like macro, right?

Conor

Okay.

Bryce

Okay, all right. So now let's talk about the next one, um, the array. So the array is a two-element tuple where the first element is the number of elements in the array, and the second element is another tuple um of the actual members of the array. So for example, you know, open parens2, comma um parens x comma y is an array with two elements. The first is x and the second is y. Um I actually never really used the uh the array uh preprocessor data structure that much. Um usually I use tuples or uh the next one um uh sequences. So um a sequence is um just like a group of uh things that are in parentheses. So like uh opening parentheses one, closing parentheses, opening parentheses two, closing parentheses, opening parentheses three, closing parentheses. So it's just like a a group of um of uh like parentheses that are right right next to each other. Make sense?

Conor

I'm following. That's uh you could say that I'm following, yes? Just for the listeners, in my head, currently I'm going I'm confused on why we're not recording this on like April 1st or uh if there's like an Omega Lol troll coming. Like I'm just I'm confused because I'm thinking like what? But keep going. This is I'm sure you're gonna get to you're a you're a you know a top-tier marketer, so I'm sure you're gonna have a you know, you're gonna wrap this up with a bow and I'll see some value at some point.

Bryce

Um next let's talk about a list, and uh this one uh should actually be pretty you know familiar to you. Um Lisp list. Yeah, it's like a lisp list. So it's uh it's a it's like a uh a con style list um uh with a head and a tail. So um if you're familiar with like lists and lisp, it works just like that. Um if I had a list of like three elements, um uh the way it would work is e each like the the list is always a tuple of two elements where the first element is the head of the list and the second element is another list that contains all the rest of the list. Um so if I had list the three elements, it would be like opening parens one, comma, opening parens, two, comma, opening parens, three, comma, then you have this special terminator that in boost preprocessor is called boost pp nil, and then you'd have three closing uh parens. Um so that this is a data structure that should be familiar to you, right?

Conor

Yep, I mean still following.

Bryce

All right, so so now you've got some data structures. Um uh and boost uh the boost preprocessor library gives you all sorts of useful facilities to working with these data structures and doing things like you know, apply some macro, apply, apply every element of one of these data structures to some macro, um uh, you know, expand them all in a certain way, access certain elements. Um, and so then you can start seeing how you can do uh, you know, some programming. And and of course, you don't really have um uh state uh uh in this programming model. Um so it's it's sort of a purely functional programming model. Now there is a way that you can do um sort of stateful programming. Um what we'll talk about that a little bit later. It's I'd argue the the craziest thing. Um but you know the idea is that like you pass around everything by value um uh and uh any any modifications that you do is is you know you you add to something, then you you know call another macro with it. Um there's there's no mutable state that you can work with. Um okay, so then the next thing to talk about, which um uh I'd argue is the most useful boost preprocessor function, is a uh a function called hang on, let me go find it in the in the reference, called um expand. And a lot of people's code base have some version of this. Um so what expand does is it's a function like macro that takes a single argument and it does double macro expansion on its argument. Um so this is useful when you want to um uh delay expansion of some macro um uh uh until like you know until a later time when it's when it's rescanned. So for example, if you want to um uh programmatically construct an invocation of some macro, like you you know the name of the macro like function that you want to call, um, but then you want to like construct the arguments to it and then expand it out, and then like like call that macro, that's what you'd use expand for. Or like if you wanted to do some text pasting, um, because one that's one of the things you can do in the C preprocessor is you can um uh you can paste together tokens. So if I wanted to go and like dispatch to a macro with a certain name um based on like you know basically catting together some other strings, um, you would use expand to uh uh expand to expand out the name uh of the thing that you're gonna go and then invoke. So this is like the most frequent thing that I would use. Um uh does that kind of kind of make sense?

Conor

It does make sense, yeah.

Bryce

Yeah. So um once you have like expand in these data structures, you can start doing like a lot of really cool stuff because you can start um uh you can start like, for example, building up yeah, programmatically building up lists of arguments, um, or programmatically iterating through some data. And then you can also um uh do this sort of like REPL type. That's that that's the perfect analogy for for expand. Expand is like REPL. So expand is like REPL or like I forget what it's called in Python, like uh maybe it's like eval. You know, it's like a function in a in a in a dynamic programming language where you say, hey, give me some text of this programming language and then I'll evaluate it um uh dynamically. Sorry, pause.

Conor

What does REPL mean to you? For me, it means read eval print loop.

Bryce

Um yeah. It that's the same meaning meaning for me.

Conor

I'm confused then. It's like REPL is like uh it's like a you know way of developing in Python or Haskell where you are just building up expressions slowly and you execute expressions and then you get the value of it back.

Bryce

R right, right. But it's like it's like it's like that idea in uh yes, I we're talking about the same thing. Um uh I'm sort of using it as an analogy here. Um the the way that you often use boost pp expand is when you are like you're dynamically if well dynamically at compile time, you're building up this expression that you then want to evaluate. That's what boost pre expand is useful for. It's like I'm I am building up this macro expression that I want to evaluate. And then I use boost pp expand to uh uh do that evaluation.

Conor

I see. This seems, yeah, so I think it's similar, more similar to like eval in bash.

Bryce

Um right, right. It's more similar to some something like val and bash. I forget what it's called in Python, but uh I think maybe it's called eval in Python.

Conor

I think it might be a val in Python too, yeah.

Bryce

Yeah, yeah, but eval and bash is a is a really good example because bash and preprocessor and the C preprocessor have a lot um in common. Um and of course, uh you also have um all of your uh you have the analogs of all of the Control Flow statements. So there's BoostPP if, um which is uh uh you know just an if condition. Um uh there's a few different forms of that. Um there is boost pp4, um uh which is just like a for loop. And then uh there's a whole bunch of uh operations for doing like all of the various like built-in math and and working with like built-in data types like bools and ints and whatnot. Um uh because of course you need that too in any sort of programming framework. Now keep in mind in something like a um a preprocessor pound if um you can just use um uh regular C style operations, like if you do you know pound if, you know, a um less than you know zero, like that'll just work. But if you're just in the macro um uh evaluation, it you know, that won't work. If in a macro evaluation, if you have a less than zero, that's not gonna evaluate uh uh special because the macro evaluation is just purely working with text. Um so if you want to do something like less than, you need some special magic for that. And now, Connor, I'm going to show you um what one of these, what some of these things look like, and um we're gonna see how how uh uh how terrifying you find it. So I'm gonna show you the the increment function. So this is what the increment function looks like.

Conor

Oh goodness, how many lines is this file?

Bryce

Um well so so this one supports incrementing uh um uh 250 up to 256 data types. So a lot of the operations in um the boost preprocessor library um uh will only work for up to like 256 of something. And you might wonder why. That's like a weird limit. And I'll explain and I'll also explain why uh uh uh Connor just laughed. So boostPP ink is a function that takes some value and it adds one to it. And here's how it does it there are um 256 macros that are called boost pp ink impl. So there's like tag zero, tag one, tag two, tag three, tag four, tag five, tag six, all the way up to 256. And for tag zero, the value expands to one. For tag one, the value expands to z to two. For tag uh two, the value expands to three, etc. up to uh to the last one. And so then what the the boost pp ink macro does is it takes the argument that you gave it, which is a number, and it pastes that number to the string thrust pp ink impl. So if I call it with You know, five, what it does is it takes that five and it pastes it to the end of a string thrust pp ink impl, and then it expands that's that macro. Um, and so that would go in dispatch to the macro that's called, you know, underbar tag five, and the value of that macro is six. So then the result of that macro expansion expansion is six, which is the the result that you'd expect if you incremented five by one. This is how all of the basic uh uh uh math and logic operations work in Boost Preprocessor. I'll show you the funnier one, Connor, um, which is so there's decrement that we're scrolling through, but okay, so here's bool. Um uh thrust PP bool. And this is um it's supposed to be a bool data type, just sort of like you know, um C's bool, where you you give it some ex some you know uh value and it returns either one or zero. And it does the same thing, just in the case of bool, the tag values are for tag zero, the value is zero, and for all 255 other tags, the value is one.

Conor

There's no better way to do that?

Bryce

There is no better way to do this. There is no better way to do this because you don't have any you don't have any basic operations. Like the the in macro evaluation you basically can do like the only real primitive operation that you have is text pacing. So there's the the um uh I mentioned this a few times. Um uh there's the uh the double hash operator. Yeah so if you do a like if you do a double hash B within some function like macro, that will paste the token the expansion of A to the expansion of B. You have that, and you have the ability to do like uh delayed expansion, and that's it. So if you want to build up any more complex operations, you have to build them up based on um uh uh uh concatenating together strings and then um expanding them out. So every operation has to be implemented table based like this. But this actually gets to what I think is the the um my favorite um macro trick. Um uh and it's the one that like I I use most frequently. Um and uh I I it's the thing that I think is it's just one of the more natural and approachable things. Like you don't have to be doing anything anything sort of out there like I would do with the preprocessor to to appreciate this. Um and it is this pattern for variadic um uh or not variadic for I well I call it macro overloading. Um uh uh and it it's a it's a style of variadic macro where you can write a macro in a way where you can have one implementation if it takes you know one argument, one implementation if it takes two arguments, one implementation if it takes three arguments, etc. So it's sort of like overloading. Um so here I'm showing you what the pattern looks like. So um the idea is like let's say that I wanted to have um some macro um uh called well the example here is plus, um, where I wanted it to support um taking an arbitrary number of things and then like adding it together. Well, if I use this uh this um magic thing that uh in our library we call uh thrust pp dispatch, um uh what you can do is you you write your main function, the name of your interface, in this case plus, and it and plus is gonna be a variadic macro. Um, but it's not gonna actually contain any of the implementation logic. It's just gonna have this call to thrust pp dispatch. And um that call it'll take as its first argument the first part of um uh of of a name. So usually you make it the the name of the function itself. Um and then it'll take um all the variadic arguments. Um and then you write um a series of implementation functions. So I write a function called like plus or a macro called plus zero um that takes zero arguments, a macro called plus one that takes one argument, a macro called plus two that takes two arguments, a macro called plus three that takes three arguments, and thrust PP dispatch um will when you call it, it'll count the number of arguments that it was called with, the number of variadic arguments that you passed in, and then it will paste that number to the name that you gave it as your first argument, and then it will invoke the um the concatenated together name with those arguments. So if I call plus my variadic macro with uh two arguments, it will call plus two with two arguments. Um so I I find this to be an incredibly useful pattern for working with variadic um uh arguments because you can do things like optional parameters very easily here. Um uh write that if if I want to if I want to have a macro that can take you know up to three arguments and um uh you know the last argument can always be optional. Um uh it's very easy and simple to do with this, and uh I can do it all under one name. And that's really the nice part of this is that um just like in C, I can have sort of like an overload set uh uh under the same name.

Conor

Oh man, my head hurts. That's where we'll end part one of this conversation. Tune in next week for part two of Macros Almighty.