Conor

They didn't teach you Roman numerals at your fancy high school with the telescope?

Bryce

Um, no, they did not. No, they did not.

Conor

Seems like something they should have done.

Bryce

Um, I I I mean, it doesn't really come up that often.

Conor

Welcome to ABSP the podcast, episode 110, recorded on December 22nd, 2022. My name is Connor, and today with my co-host Bryce, we talk about compiler diagnostics, printf debugging, and more. And we're moving straight into our New Year's Eve episode, and we're now gonna let Bryce talk about compiler diagnostics.

Bryce

So, some of the compilers I work with don't have great diagnostics. And um so so what is a compiler diagnostic? What do I mean by compiler dial diagnostic? What I mean is when what there's one of two things that leads toward compiler diagnostic. Thing number one is you, the user, did something wrong. You wrote some code that was incorrect. And the compiler has to communicate that to you in some way. And the second thing is the compiler did something wrong. Like, you know, the compiler like you wrote valid code, you know, maybe it was very maybe it was very obscure code, maybe it was code you shouldn't have written, but you wrote valid code and the compiler like sees it and is like, uh like I give up, I blow up, I run out of memory, something. Um, and then the compiler has to in some way tell you, like, uh I like you gave me something that I choked on, like, sorry, like, you know, here's what happened. Um and d you know diagnostics, especially in C are an important part of your tool chain experience. Um there's this thing, this sort of amorphous thing that I talk about, which is um uh uh like the robustness of a compiler. There's like a question of like, you know, how many how many of the language and library features do you support, right? Like how many of the boxes do you check off? But then there's also the question of like how robust is your support? Um and you know, that's like partially a question of like how well have you tested it, but also there's only so much testing that you can do outside of like compiling real world code. Um and when you do compile like real world code, like some amount of it might be quirky or weird and might lead to you know um uh problems where the things that your compiler can't handle where even though it should have, or things where maybe it's a little bit unclear whether the code is correct or your compiler is correct, um, but still like it's something that that um uh you know it'd be nice if uh if the compiler could handle even if it's maybe you know undefined behavior or you know an edge case or something that's like not officially supported. Um and sort of the the degree to which your compiler can handle you know real world code is is what I like to call like robustness. It's it's uh sort of like the quality of the features that you support. And I I am faced with this perpetual problem where it is um it's hard to get people to care about robustness and diagnostics. Um in compile in the compiler space, people primarily care about performance. So the code that you can compile correctly, like how well does that code perform at runtime? And also how well does it perform at like how fast how long does it take to compile? Um and like increasingly in the past few years we've cared more and more about um uh tooling and like things like safety and and static analysis and and stuff like that. Um uh but I uh I think it's it's still been hard, at least in some organizations, to um to get people to care about um diagnostics. Um and they're so important, especially if you have a less robust compiler. And I think one of the reasons why it's hard to get people to care about diagnostics as opposed to things like performance is that it's hard to quantify, right? Like, how do you quantify how good your diagnostics are? And you know, some compilers do um test their diagnostics. They're like, okay, well, if you give me this ill-formed code, I'm ex I'm supposed to produce this type of um you know of error message. And sometimes like, you know, Kling and LLVM have a um uh uh have testing tools that make it pretty easy to test whether you get a particular um error message um or a particular kind of error message from uh a particular piece of ill-formed code. Um and so like that that's part of like what um uh the uh the testing is. Uh uh, but I don't think that can uh a lot of compilers do this and certainly maybe they don't do it as as well as they um as they should. Um so I've been thinking about like how could we do a better job of quantifying um uh uh diagnostics. And um, you know, I I've over the past few years I've been accumulating um a collection of what I consider diagnostic failures of like real world um uh cases where somebody hit a bug and they were like, I I got this error message, I don't know what it means. Um and then like eventually we figured it out and was like, oh well that was just an unclear diagnostic. Um and you know, there's some there are some attributes of a diagnostic, you know, does it point you to the correct location in the code? Where the like does it point does it uh correctly point you to the origin of the issue? Um uh does it correctly like describe um uh uh problem that's being encountered. Um you know, we could probably go and make a list of like those those attributes. Um but one thing that I've been thinking about a way to like quantify this would be just take a corpus of like bad of diagnostic that are maybe questionable, or maybe it's just like include good diagnostics too. But just like take like a hundred test cases of like some ill-formed code, or maybe some some well-formed code that some compilers choke on, and then generate those diagnostics that that that code produces for all the major compilers, and then like create a survey where each person is shown the diagnostic, um, maybe like without the code, and then they're like asked, like, hey, what's the problem? Like, like, or how helpful is this diagnostic? Um, and then you know, they they like they pick a score, and then like in the next screen, they're shown, well, okay, here was the source code, and here was the actual problem, and then they're asked, like, now that you've seen the actual problem, um, like how helpful do you believe the diagnostic was? Was it like that you just misread the diagnostic, or like was the diagnostic actually not that helpful? Or so some so some sort of survey like that, which would then give me some actual concrete data on um uh how good are different compilers' diagnostics. Um and maybe that would also help me learn more about how to improve diagnostics. Um but uh but yeah, this this is um this is something that I'm I'm quite interested in in um in trying to to to learn how we can quantify diagnostics because I feel like if I can quantify it, then it makes it easier for me to um to uh uh to convince people that it's an important um uh that it's something that's important, that it's something that we should strive to improve.

Conor

Alright, folks. We're gonna here's what we're gonna do.

Bryce

Oh, there's two Connors.

Conor

Have you seen Rust diagnostics? Is my question.

Bryce

Um not as much as I would like to, but I hear that they're very good.

Conor

Which code base do I want to work in? Because I have three different Rust ones open right now. Let's do this one, but let's just go down to a test. And uh so, like for instance, let's see if I actually have this turned on right now. I have a couple local variables. So we I always say we're not gonna live code, but just while Bryce was talking about um diagnostics, I just I consider I haven't worked with Elm, but I heard Elm was the language that coined the term compiler-driven development. And Rust has really, I think it's like the one of the best languages.

Bryce

Well, what what what is compiler-driven development?

Conor

Compiler-driven development is this idea that you should design your compiler errors and diagnostics to be so helpful that it actually guides you while you're programming. Oh, versus it's kind of the opposite of what you were just saying. Like, you know, your diagnostics being so unhelpful that it's just confusing and it's gonna waste time. Whereas, like, there are literally extensions in uh Rust that not only can the are the diagnostics so good it tells you what's wrong, you can like set up your editors to just like I know Gonzalo, who's an individual we we both work with at NVIDIA. I don't know what editor he used, but he says he just hits tab and that does cargo fix, which just automatically like fixes sort of this, you know, the equivalent of clang tidy fixes in C.

Bryce

So wait, wait, then what does he do if he needs to indent code?

Conor

I don't know. I don't I didn't I didn't ask, uh but uh so I just added mute, the mute keyword, which is basically the uh analog of const, except they do the opposite things. And you can see here that I have a VS Code extension called error lens or something like that. What is it? Yeah, error lens. And it inlines the errors that I get. And so this is technically a warning, but it's saying that this variable does not need to be mutable. So it immediately, you know, does that. And if I do something else uh like this, well, so that that's not the error I was trying to get. So I just took a variable that I'd already used and then used it again, but I just typed it and put semicolon and it says path statement drops value.

Bryce

Okay, okay, but but but let's so so go back to that previous diagnostic because I have some questions about that.

Conor

So This is basically saying like it's it's uh this is the equivalent of your no discard.

Bryce

So it says path statement drops value um and then in uh back ticks warn path statements um on by default. So is the diagnostic just the first part? Path statement drops value? So actually let's just so that because this is not actually the this is a modified version of what you get when you the reason I'm asking is because the way that that's phrased, I think, is a bad diagnostic because it it it it doesn't have the context. Like if it says something says like path statement drops value, it should say like what's the value that was dropped. You know, it should like quote it in the diagnostics.

Conor

So this is the actual diagnostic, which they are.

Bryce

Okay, so so in the actual diagnostic here, it does give you carrots. Well, but but okay. Um what what is a path statement?

Conor

Uh I'm just I think it means the statement in this current path. I've actually never seen this error before, but my guess is that they're telling you literally right here that like if you are not going to use the value that's being returned from this expression or statement, put drop parentheses around expected.

Bryce

Is it possible for us to construct a statement like this that drops two value? Or that has or sorry, is it possible for us to construct a statement like this that has like this statement that that is just you know the name of the variable semicolon? So it's pretty clear here. But I want to see like what happens if that statement was like more complex and it like it named multiple variables, because if it named multiple variables, then that diagnostic path statement drops value might make it un maybe unclear which thing it drops.

Conor

I don't know. Does Rust have a semicolon operator?

Bryce

Comma operator, I think is what you're getting.

Conor

Oh, yeah, sorry, comma operator?

Bryce

My guess is no.

Conor

It does not, no. Um so uh off the top of my head, I don't I don't know.

Bryce

Yeah. But but but see, like for me, like I I would want that diagnostic to to to tell me in the warning message part, like what is the thing that was dropped.

Conor

Expected, it's pointing right at it.

Bryce

Well in the in the in the the carrot part, yes.

Conor

It it's pointing at at the at the um uh at it, but um and I mean i if if it's if it's always gonna point out it, I still think it's in more complicated expressions, like if we go back to adding the unnecessary mute and compile, you get like you get like multiple colors. Yeah. You're you're putting mute on input, and then it points directly at mute and says, remove remove this mute.

Bryce

And then I'm pretty sure if you go Yeah, but but even though it shows the carrot where it points you towards that particular thing, um, I don't know that I I still think I would prefer for the textual warning to name the variable that does not need to be um mutable.

Conor

It's pointing right at it. What are you talking about?

Bryce

Right, but in in the in the tooling example that you just showed, um go back to the IDE. So um uh is there a way to declare multiple variables in a single statement in the Rust?

Conor

Uh yes. Okay.

Bryce

But you would have to use like destructuring to do it and my my my point is that the carrot like the carrot syntax is great, but the textual diagnostics are obviously like used in in um in tooling. And sometimes people don't always look at the carrot statements. Sometimes like the carrot statement, like if you're on a on on a um uh an environment where the where the carrot like the line is so long that it wraps around, sometimes the carrots can be like hard to read. And by carrots I mean like like uh that it reproduces the source code and that it puts like you know um uh uh the little carrot uh uh symbol and the line below the source code to show you like, oh, this is the thing in that line of source that causes the problem.

Conor

Interesting. So when I used uh when I destructured a tuple into two variables a and b and then put mute in front of a, it doesn't say anything, so you don't get a warning for that. Although I'm not even sure, like I assume this is valid code, it's compiling. I mean, let's try and do this a plus equal one. Alright, because we have to call it underbar a, which actually at this point, no, we don't need this to be underbar.

Bryce

See, like these diagnostics are like better. It's instead of saying like variable, you know, can be, you know, it doesn't need to be immutable, it says variable a is assigned to but never used. That's a better diagnostics to me. It like it talks about what its subject is. I don't want diagnostics to ever say anything that's like like uh ambiguous about like what it what it means. And like these are simple examples that we're looking at, but uh it when you and you when you get to complex code, like the the i i it may become not unintuitive, like what is meant. Like if it if it just says like variable, you know, uh uh doesn't need to be mutable, like in this case it's clear what variable is meant. But my point is that like in the wild, and and when you're developing diagnostics like this, all of the test cases that you write uh like this is gonna make sense. But then like in the wild in a big production code base, um, you know, maybe there's some some some complex case where somebody gets this diagnostic and they're like, What I I don't like what which variable are you referring to? Like it doesn't cost anything for the diagnostic to explicitly say, like, hey, this is the thing, this is the name of the thing that that uh I'm talking about you.

Conor

I mean, I think in this case, like there's only one variable on the line. So in the case where you're using error lens, it's pretty clear. And here there's only one variable, and there's five carats underneath the underneath the five characters of that variable name. So like asking this, asking you're so you're basically saying that you want this warning colon variable does not need to be mutable. It doesn't cost anything to put the name of the variable there.

Bryce

My point is more like as a as a as a matter of policy, when a diagnostic talks about like a thing, like a variable or an object or something, it should say which thing it's talking about, always. Because you never you never know um uh uh how that diagnostic may that diagnostic may come up in more complex, it may be emitted in more complex scenarios than you originally mattered. And in the scenario that uh that the then that you originally envisioned, I mean, in the scenario that you like in the cases that you envisioned this diagnostic being emitted for, it may be perfectly clear what variable is being talked about. But there may be cases where the that diagnostic is produced and it's unclear, or there may be cases where you know the the user reads it and I think this is often what happens where somebody like we like the diagnostic does tell you what the problem is, but you've um uh like you read it quickly and you assume that it's something else. You assume that it's talking about some other variable or something. Um uh and and you know, and then you go and spend 30 minutes trying to figure out, oh, this other thing that I assumed it was talking about, you know, that's the that's where the problem is. Um instead of if it had just been clear, if it had just told you specifically which thing it was talking about, I think things would be better. Like I'm I'm picking on this one particular case here because I think this is exactly the sort like the source of a lot of C's diagnostic problems. If you look at a lot of like poor quality C diagnostics, um it's it's because of like diagnostic messages like this. Now, a lot of those are from the era before we before diagnostic messages reproduced the source line and before diagnostic messages did things like the carrot diagnostics where it shows you where the problem is, and like that has made things better. But I still think that like this is one of the root causes of of uh of poor diagnostics, is when when the error message does not um uh like tell you uh uh what things it was dealing with. And or like think about um uh uh you know assertions. Um in sort of the early days and I think still um common lot of code bases, when you have like assertions in C, um I you know, the simplest form of assertion just like a it checks the thing and then um it if it if it fails, it um you know it just says assertion failed. Um a lot of testing frameworks a lot that are like assertion based or that have like some macro for you know testing this A equal B or you know something like that, they uh those macros these days are constructed in such a way where if you want to do some something like you know, assert or test that A is equal to B, uh if that fails, it will print out to you what was the value of A and what was the value of B. Um because you know that's useful information. If it just tells you, hey, like, hey, this assertion that A equals B fails, then you're like, okay, well, like, well then what was the value of A in this case and what was the value of B? Um like I like that's the information that I would need to know to be able to diagnose to diagnose what the problem is. Or or I think diagnostics that don't provide um uh uh diagnostics that aren't format strings, I think are often bad diagnostics. Diagnostics that don't like substitute in um uh context from the source of the error are often bad diagnostics.

Conor

F strings for the win. You know that in Python they have this beautiful feature where with an F string, if you just have an F string, you know, F quote brace name of your variable n brace end quote, it prints it out. But if you add an equal sign at the end of that variable, it'll turn the value of that variable into the name of the variable equals this.

Bryce

So it's like I'm not sure I understand.

Conor

So like the initial f string is f quote brace name of your variable, we'll call it x, end brace, end quote, and that's just gonna print out whatever the value of x is. But if you add an equal sign right after the x, it's then gonna instead of printing out the just the value of x, it prints out x equals and then the value so that anytime you're doing printf debugging, which you know admittedly is not the always the best way to debug, but you know, we've all been there, and sometimes you just need to do something quick and dirty. Putting a bunch of those statements all over the place, you just need to add equal sign to each of your variables, and then and then like how often are you creating a little f string that is you know the name of the variable equals or colon uh and then the actual value. Uh it's just a very nice thing that I found out a couple weeks ago and was like, wow, that's such a nice small addition.

Bryce

People give printout debugging a lot of uh uh it gets a bad rap, but um at least in uh my career and especially, I you know, I mostly Work on concurrent systems. Um in a lot of cases, um uh firing up a debugger um will uh you know run running the the code under the debugger will mean that the error won't reproduce.

Conor

Um and so you know isn't that isn't that a problem with the debugger though?

Bryce

And not like Notion, like the it is almost always the case that running that building just may don't forget even forget even like running it in a debugger. Building the like the bug may not reproduce in debug versus in release mode. And you know, it it may be that that opening up in the debugger is the the way depending on how you build the code and the platforms that you're on, um, but often in the platforms that I'm on, the like the production version of the code doesn't just build in release mode, it like builds in release mode and like explicitly like turns off like a lot of additional debugging things in like stack frames and like information that like without which the debugger can do absolutely nothing. Um and the the difference between that release build and the debug build is sufficiently large that um you're you're just you're you've almost got like a completely different code that you've compiled. Um and the performance characteristics of it are sufficiently different that something like a race condition may not appear in the the debug version at all. Um and and and so you know that's why I've often had to lean on uh printf debugging. And sometimes I can't even do the printf debugging because the the output the IO itself will cause the race condition to go away. And so instead of actually printing out what I want, I will instead you know store um uh I'll sort of buffer somewhere the information that I need to print out later. Like I'll have some variable that that will save like like the value that I wanted in print out, and then afterwards I'll like only print it out at the very end. So I I I think people I think people, you know, um people who are very anti-printf debugging have perhaps worked in an environment where they've only had to deal with really one sort of bug, and that's sort of been like um logical um uh uh programming bugs. Like, oh, like I wrote the wrong code as opposed to something that's um a little bit more nuanced, like some sort of concurrency bug, or um uh something that's um uh you know some sort of um some sort of performance bug or something that only that only only occurs when you're building and running um sort of at speed of light.

Conor

So what should all the front-end, back end compiler engineers uh that are listening to this take away? That there's some survey coming out that's gonna help them in the future, or that they should just do better?

Bryce

I don't know yet because I haven't started down this road of trying to like come up with a way of quantifying diagnostics. If anybody has done that or has thought about that, like or I I I guess I'm I'm sort of looking for two things. One, I'm definitely looking for case studies of of poor diagnostics. So if anybody has code that leads to the like C code, like like that's ill-formed, where they got a diagnostic message and they spent like a good amount of time trying to understand it, and like the problem was just that the the compiler just provided a vague or unclear diagnostic. I'd love to see, I'd love to get a lot of test cases of that. So send me those test cases. Um anything where there was uh code that was ill-formed or or code that was well formed, but just like any compiler, poor compiler diagnostic um test cases that you've come across, um, I'd love to see those. But I also um I'm I'm trying to understand what are like the attributes of a good diagnostic, what makes a good diagnostic. Um there's you know what we talked about before about that I want to see diagnostics that provide the the relevant contextual information um that point you to the correct location in the source that sort of describe the the the issue in the right ways. But uh I I'm sure that there's some way of quantifying um uh what good diagnostics are and like like like the the sort of the attributes, the the the key properties. Um I just haven't quite figured out yet. Um or I I guess I haven't really started the exercise of sitting down and thinking about what are those attributes. But if people have ideas, I'd love to hear that. Or if there's prior work in this area, we'd love to hear that.

Conor

I know we I think we already have a couple guests that we said we were gonna bring on, and the next two episodes are gonna probably be the retro and uh forward-looking podcasts that we do on 2022 slash 2023, but then we should, because on our short list of Rust people to bring on is JT, uh Jason Turner's cousin, and they are I know they've done a lot of work on the Rust um compiler diagnostics. So um, I mean, they were recommended to us by Jane to, you know, come on just in general and talk about Rust, etc. But we could also have them on to talk about uh compiler diagnostics as well.

Bryce

Here's a question. Who invented caret diagnostics? Like not just like for not in like Rust or in C, but just in general, who invented Carrot Diagnostics?

Conor

And also I know that a lot of this stuff though was invented in like in in Rust land. I don't know if the carrot stuff examples.

Bryce

Well, I I I think uh maybe uh some uh a listener will correct us if we're wrong, but I believe Clang was the first C compiler to have carrot style diagnostics. I I think I think that even before that, GCC maybe had diagnostics that pointed you to the line of code, but um I don't know, maybe we should have somebody like Richard Smith on, because maybe he'll he'll be able to tell us more about the history. Because I really think Clang was the first C compiler to revolutionize diagnostics and everybody else sort of followed suit. So maybe we can have Richard Smith on sometime and he can um tell us a bit about the philosophy of diagnostics in Clang.

Conor

Yeah. I mean, well, he's one of the main three folks at the moment on the Carbon teams.

Bryce

Yeah, and maybe maybe then he could tell us a bit about what they have in mind for for diagnostics and um and carbon.

Conor

Yeah. I mean, we've been ever since CPP North back in July, which is almost half a year ago by now, we were supposed to have uh Chandler on um on the very last day, but then it ended just being too hectic and we we didn't get a recording with him. And ever since we've been it's been on our back burner. I think we have like a you know, on the backburner, like 20 different folks that we're supposed to bring on. But 2023, we'll make it happen. All right, happy I mean, technically I said it's the New Year's Eve, New Year's Eve episode, but it's actually December 30th today. Potentially you're watching, listening to this on Saturday, though. So if it is Saturday, December 31st, happy New Year's. Uh well, I mean, for those of you that celebrate the New Year's that corresponds to the, what is it, Gregorian, Julian? I don't even know what calendar the uh Western world follows. One of them.

Bryce

Well, so my my my thinking on this is it's pretty safe to say happy holidays because pretty much everybody across the planet gets time off around this time of year. So, you know, that's a holiday.

Conor

That was the last episode, though.

Bryce

Whatever.

Conor

What about Happy New Year's, though? Is that fine?

Bryce

Um, I mean, I think What is the calendar?

Conor

Do you know? Is it Gregorian?

Bryce

Um I mean if if Howard Hinton is listening to this, he's uh Yeah, yes, I believe it's the Gregorian calendar, is the one that's used in most of the world, yeah.

Conor

Alright, yeah. Gregorian, happy new years. Yes.

Bryce

Introduced in October 1582 by Pope Gregory, the I can't read Roman numerals, um, as a modification of and replacement for the Julian calendar. Yeah, it's X I I I. I don't know what that is.

Conor

X I I I? Come on, really? Take a guess.

Bryce

I don't know. 13?

Conor

Take a guess. Yeah, that's correct.

Bryce

I guess I can.

Conor

They didn't teach you Roman numerals at your fancy high school with the telescopes?

Bryce

Um no, they did not. No, they did not. Seems like something they should have done. Um I I I mean, does it really come up that often? They probably tried to teach it to me in Montessori school, but I uh I very much rejected the Montessori educational program. It was very structured. Like you like in Montessori school, they give you like these these um like wooden blocks, and they want you to like stack them from like smallest, from largest to smallest to build a tower. And I would I would be like, no, I want to go like do something creative, like build some other structure. And they'd be like, no, creativity is not allowed here, like you're supposed to like do this thing. And I'm like, no, I'm a creative soul. I want to go go do this other thing instead. It was not a great fit for me. It's a very i Montessori School was very structured um and uh and I am a very chaotic and uh a creative person.

Conor

All right, on that note, folks. Twenty twenty two, that's a wrap. We will talk to you in twenty twenty three. Yes, we will. Thanks for listening. We hope you enjoyed and have a happy new year.