Julien Bet His Career on Replacing PHP at Facebook | The Hack Story

Julien Verlaguet didn't just write code at Facebook — he tried to change the language 4 billion people's apps were built on.

In this episode of Computers, Coffee, & Beer, Julien tells Keith Adams the full inside story of creating Hack: a gradually typed dialect of PHP that brought static analysis, type safety, and sub-50ms error feedback to one of the largest codebases on Earth.

It started as a security project. It became an "adventure" — one that involved pitched battles with the HHVM team, developers who associated type systems with traumatic C++ compile times, and the realization that Facebook's "spaghetti noodle" dependency graph made strict typing nearly impossible.

If you've ever wondered what it takes to introduce a new programming paradigm inside a company running at Facebook's scale — the politics, the engineering, the human friction — this is that story.

🔗 Follow Computers, Coffee, & Beer for more war stories from the engineers who built the modern internet.

Chapters:
0:00 - Intro: Meet Julien Verlaguet
2:15 - Why Hack existed: PHP's security problem
7:17 - The pitch to the HHVM team (Keith's "okay kiddo" moment)
15:30 - Gradual vs. optional typing explained
28:00 - Facebook's spaghetti dependency graph problem
45:00 - Building a sub-50ms type checker
1:09:09 - The nullable salt vulnerability (why weak typing is dangerous)
1:15:26 - "TypeScript for PHP" — Hack's legacy
1:51:46 - The C++ developer's scar: winning over type system skeptics
1:54:53 - The George Lucas / Steven Spielberg analogy for innovation

🎧 Full episodes on YouTube + your favorite podcast app.
 @ComputersCoffeeandBeer  

📩 For Business Inquiries: don.broida@gmail.com 


=============================

Channel About: 

Computers, Coffee, & Beer is hosted by Keith Adams and Julien Verlaguet — two engineers who helped shape the modern internet and have the war stories to prove it. 

Keith built the HHVM JIT compiler at Facebook, helped virtualize the world at VMware (employee ~80), and went on to Slack before founding Pebblebed, an early-stage VC firm for technically ambitious founders. 

Julien created the Hack programming language, spent years writing safety-critical compiler software for Airbus and nuclear plants, and now runs Skip Labs. 

Together they dig into the systems, languages, and decisions behind the technology that actually runs the world — told by two people who were in the room when it happened.

=============================

#Hack #PHP #Facebook #Programming #TypeSystems #HHVM #SoftwareEngineering #DevTools #ComputersCoffeeBeer

Transcript

SPEAKER_00 0:00

Welcome to Coffee, Computers and Beer, your daily podcast ritual.

SPEAKER_01 0:04

I don't know about daily. I don't know about daily.

SPEAKER_00 0:08

I'm Keith Adams. I'm here with my old friend Julian Verliguet. We're here to talk through uh through Coffee, Computers, and Beer with you. You're Keith Adams, tell us more about who you are. I'm Keith. I'm I'm one of these computer people we have a lot of in San Francisco, California. Um I had a long career as a technologist. I was one of the I was the 20th engineer at VMware uh in the early aughts. Uh did a bunch of stuff with the hardware software interface there, then was relatively early to Facebook where I did a bunch of back-end stuff, and also started Facebook AI research about 13 years ago. I'm part of the generation of people that got fired up about AI because of what was going on in computer vision back then. Um then was chief architect at Slack for a bit. And these days, uh eventually came to feel there that your investors should understand what you're building and started a venture firm with a bunch of other remarkable people uh called DoubleBeb. Awesome. Um maybe I want to talk about myself a little bit too, since you're not asking.

SPEAKER_01 1:02

I'm gonna just go ahead and do it.

SPEAKER_00 1:04

Julian, and yeah, who the hell are you actually? Why why are you how did you end up on this thing? Who's this guy?

SPEAKER_01 1:09

I mean, I saw some lights, I thought there might be some free food. I was like, all right, let's let's try this. No, yeah, most seriously, my background is in programming languages and uh static analysis and this sort of stuff. I worked first on critical embedded software in a different life where I was working on programs that could not fail. I mean, if they failed, people were gonna die. And that was interesting work. Then I moved to Meta, Facebook, what it was called back then. And I worked with you, Keith. Uh back when you were working on HHVM, I was working on hack. So I created this language, which is still in use today at Facebook, is um gradually typed dialect of of PHP. And I guess that's what we you are going to be talking about today.

SPEAKER_00 2:01

I mean you you've already there's already a little bit of a definition that's probably worth unpacking here, which is you already said gradually typed. Yeah. Um people these days might have experience with gradual typing, but this used to be an exotic thing to explain. But what do we mean?

SPEAKER_01 2:16

So it's actually funny because um I think very few people know the exact distinctions between all these things. And what is interesting is when I got started on this, it was super esoteric. And now, because of TypeScript, it seems like normal, right? Like I when I got started on a hack, the idea I I remember this vividly. Like I was in the security team back then and in the security team at Facebook, and we were trying to find security holes within the code base, right? And so build static analysis that would show vulnerability and stuff like that. And I remember that we were running into you know problems because we just didn't have access to the types, and without the types, there's very little you can say about a program statically, right? And so this idea came of building a statically typed version of uh PHP, which back then seemed coconut, right? And the the first idea was we're going to just write the stuff that is very critical to Facebook that we need to get right, and we're going to write it in that subset, which back then was called strict mode. And I remember my my manager back then was like, the HHVM team will never go for this. Never. Never. They are like these people who live in a cave, they barely talk to people. If you talk to them, they grunt, and you're never gonna get them. And so the HHVM team back then was you, Drew Prosky, and and Jason Evans. And so I'm I'm preparing my stuff, you know. I am trying because the thing is, what I needed from you guys back then was I needed to change the syntax. I needed to add syntax for the types, right? And without your approval, this was just never gonna work because I would have had to add a conversion layer, which was being slow and horrible.

SPEAKER_00 4:20

And you know, it'd have been like coffee script or whatever, one of these horrible things. And by the way, uh if I can say from our side of things here too, like the the content that's actually required in a language runtime is just to throw this stuff away. We just need to be able to parse it and ignore it, right? Like you're building an out-of-band tool that's gonna care about this stuff. It's not even like you needed us to make some sort of deep change or whatever. You just like didn't want us to be shocked when you go messing around with the grammar.

SPEAKER_01 4:43

Well, the way I was seeing it is if you are back then the HHVM team was really in charge of the language because there wasn't a language team. And so that means that every time somebody has a wacky idea about what to add to the language, I think the right instinct should be no. You know?

SPEAKER_00 5:01

And I think the reason partly the reason we had a reputation for being grunting, cave-dwelling uh you know, ogres was like, first of all, partly we were. But secondly, uh, you know, you were not the first person to propose a static typing system for PHP to us.

SPEAKER_01 5:14

I remember.

SPEAKER_00 5:15

Um But anyway, well, it sounds like I'm I'm jumping ahead here.

SPEAKER_01 5:18

So No, no, I remember. And there was, I would say, naive solutions that were proposed before and by people who were well-meaning but just didn't realize that there are there are many, many problems in computer science that you can solve by reinventing the solutions and rediscovering what people have done before you. And type systems is not one of them. It's just not. Like if you go and walk in and try to think from first principles and rebuild, it's not gonna work. You you it's not. You need to learn the theory to get this stuff right. So there are well-meaning people who are trying things who didn't realize that it was never gonna work this way. So I remember one who you know was explaining to me, I hope he won't recognize himself or feel offended by that. But you know, he's telling me, yeah, and now I'm type checking variables, I'm type checking integers because you know, if I see an integer, I know it's an int. And you're like, okay, what happens when it's a variable? Well, when it's a variable, I look back in the source code, and if I find the definition, then I know that and and was explaining all of this, and you're like, dude, I know what you're doing. This is not how you're gonna get this to work. And and again, this is not a dumb person, this is not um uh you know a lazy person, this is just somebody who just didn't know that this is not how you get this stuff done.

SPEAKER_00 6:45

Anyway, there's just like decades of stuff that like you're not gonna that yeah, that that's my that you're not gonna find from scratch or take your whole life to find from scratch. Yeah, exactly.

SPEAKER_01 6:55

So here I am thinking the HHVM team is probably gonna shut down any new idea regarding the language. So I'm preparing myself for weeks. You know, I have I have something that's that thick, like I have every angle. So I heard that Napoleon was, you know, when he was going into battle, he would prepare all the possible outcomes so that he would know how to react quickly. So he had a decision tree of the things that could happen and what he would do then. That was exactly me. So I have, you know, Keith, Jason, and Drew, you you're both all sitting in this room, and I'm at the whiteboard and I'm like presenting my stuff, you know, this grandiose thing. And and it goes on for a little bit of time, you know, like it goes on for I'd say 10 minutes. I'm explaining what I want to do, why it matters for security. And at some point, Keith gets irritated, you get irritated, and you say, What are you trying to do?

SPEAKER_02 7:56

Like, what is this? Well, why are we here? What is this thing? And then I kind of stop and say, I want to build a statically typed version of PHP. And I remember your faces, you know, all of them look at each other, like, okay, five seconds of silence. And then in your tone, and I don't know if I got it wrong or right or wrong. I you you'll tell me now if I was wrong. The tone that you said, you were like basically you said yes, and you were like, okay, kiddo. You wanna you you wanna fuck yourself? You wanna hit a wall? All right, no problem. I'm gonna watch you hit that wall. It won't be my responsibility. As long I mean, you want to run and and smash your head against the wall, go for it. Yeah, yeah.

SPEAKER_01 8:43

And and and I was very happy with that because all I wanted was nobody to stop me doing what I was gonna do. And that meeting was important where I just needed the HHVM team on board with the idea of modifying the syntax and throwing away whatever I was gonna add to the language. And so that worked, but I have to say it was a stressful period.

SPEAKER_00 9:08

Oh, I'm sure. I'd like and my recollection of that conversation, by the way, too, is and this is a while ago, right? So it's possible that I'm an unreliable narrator here. My recollection of that was that for me, that was like fifth or something in a series of people having this idea. Do you know what I mean? So there was there was a lot of um, it felt like throat clearing when you were sort of saying, like, oh, static types would be good, and they let you do this, and they let you do that. I was like, yeah, no, no, no kidding, they're good. So what's next? And and what happened uh in every other instance of it was you know, the people I was talking to were sort of folk language designers, right? They were they were, you know, people that sort of assumed, hey, I know a lot about PHP, I spent a lot of time programming, I took a compiler course, I can I can get something together here. And like their proposal would basically fall apart under kind of a little bit of scrutiny, right? You'd sort of be like, well, what do you want to do about arrays? And they'd have some story about arrays and be like, well, what about the other kind of array? What about the third kind of array? What about the fourth kind of array or whatever? And uh array in in uh PHP is this kind of weirdly flexible data structure that captures what Python people would call a list and a dict and a typical. It's an interesting way of thinking. Right, right. There are these weird things that you kind of use everything for. And um an array is not a very informative type and doesn't tell you anything about what you pull out of arrays, so like you need to have a story here. And you're basically the first person that like didn't kind of completely fold up their tent and be like, let me get back to you on that with that one. So I mean our attitude was a little bit um you're right that like we were a little bit uh protective of the language, but it was sort of an environment where a lot of people had a lot of ideas, not all of which kind of stood up to a half second of scrutiny. And I was actually really relieved to finally talk to somebody and and come out of that one of those meetings and be like, this might actually work. Like it seems like you actually know what you're talking about and you're actually thinking about it.

SPEAKER_01 10:52

Cool. Well that that that's very so it's good. It wasn't as bad as I thought then. So you go back to your original question, which was what is a gradually typed language or gradually typed dialect, which is what hack is. So it's a dialect of PHP because it's neither a subset or a superset. And that's also why we didn't call it uh PHP extensions or uh PHP whatever, because calling it PHP when you're neither a superset or a subset is I think that was my take back in the days. I got a lot of put back pushback from that from management. I thought it was disrespectful to the PHP community. They have a language, they have made choices. If you're going to make different choices, things that you don't want to support um or things that you want to add, and if you want to do both, you are a dialect. Like you most things are gonna work, but there will some things that you'll have to adapt. So it's a dialect of PHP, and for gradually typing, so the distinction is actually a little bit subtle. It has to do with so in fact, hack is not strictly gradually typed. Uh it's um so there's a distinction between optionally typed and gradually typed. So optionally typed means there is no check whatsoever at runtime, right? So I'm calling into a function that is annotated with a say an annotation that says int, um, if I check this annotation at runtime, that's gradually typing, right? And then now past the point where I made this check, all the body of the function can be trusted with a type system, right? So that's gradually typed. Optionally typed is you completely drop the type at runtime, you don't check anything. So that's TypeScript, right? For example. And the problem with H H VM, I mean the problem for academics. So, you know, whenever I was presenting that work to academics, we didn't, you know, fit any, we didn't tick the right boxes because we were neither the way I was presenting it back then was gradually type when we can, optionally type if we must. And so the typical two cases where gradual typing becomes problematic is when you have uh deeply nested data structure. Uh so imagine you have an array of something, right? And you want to check if it's an array of it, right? So there are some trick you can place when it's it. Because when it's int, maybe at insertion time you can remember that you actually insert an integer, and then it's it's relatively cheap to maintain at runtime that in this thing there's only in, right? But things get really trickier if what you insert in there is more complex, right? Let's say you insert a closure, let's say you insert a class, let's say you insert, you know, maintaining at runtime, you know, what is the common interface between those things is just not feasible, right? There's gonna be way too expensive. So whenever there's a boundary between something that isn't type check and simp something that is supposed to type check, what HHVM and Hack did was, well, if it's cheap to check, we'll check it.

SPEAKER_02 14:22

And if it's expensive, we're gonna go like, well, you know your own, right? Um the other thing.

SPEAKER_00 14:28

And by the way, to some extent, if I remember right, and and you should, I might have the chronology wrong here a little bit too, but like we were also at this time trying to stay compatible with open source Zend PHP. I mean HVM was open source, but with with Zend PHP, the reference implementation we were using, and they had a notion of what they call type hints, which is like not a hint at all, yeah, but was a type assertion, basically, right? So you'd have parameters, and if you mismatch the parameter at call time, it would, I believe, throw or just abort. It would throw, yeah. It would throw and so we were trying to match that pers you know, we were being a little persnickety about saying, oh, the unit tests from Zend run exactly the right way and things like that at the time. Yeah. Um, which felt important to the project, because otherwise people assumed it was some wacky hack that worked on some, you know, no pun intended, that worked on some subset of stuff that Facebook uses. So this is exactly right.

SPEAKER_01 15:18

So what happened is basically what Zend had as type hints that were enforced at runtime, those were gradually typed because you would enforce them at runtime. And the all the stuff we added, the generics, the closures, etc. etc. This was although there was callable in I think in Zen type. So we would check that.

SPEAKER_00 15:46

Yeah, go ahead.

SPEAKER_01 15:47

Exactly. So callable didn't tell you anything about the type the parameters and whatnot. So we just checked that it was a closure, but we didn't check the rest. So we basically checked at runtime what we what was checked by Zen, and then everything that we added was just dropped, so it was optionally typed. So technically, hack is not gradually typed, and that was you know uh a long explanation for something relatively simple.

SPEAKER_00 16:12

Would it be would it be like uh more precise to say that hack was a at least at this time that hack was an optional typing system for runtime that did partially overlapping subset gradual typing, maybe?

SPEAKER_01 16:26

That could be a way of putting it. That could be a way of putting it. Although I think they did more complex things after, where they were trying to maintain the type of elements in an array or things, things of the sort. So I don't know what the HHVM team was up to post-2015, something like that. So you know, it's possible they they went in other directions.

SPEAKER_00 16:50

Between like TypeScript and uh and hack and analogous systems these days, I feel like kids today don't always like know what we're talking about here exactly, but it was like the kinds of bugs that you know you were worried about in the security team were these bugs where you know somebody passed a string somewhere that an object belonged. And because PHP sort of struggles to complete requests no matter what, like struggles to find some meaningful interpretation of the values it's messing around with, it can kind of run on for a while. Like you'd think you'd just crash right away, and it doesn't necessarily, and sometimes it has horrible side effects from running on for a while. And uh that was extremely common in a big complicated code base like this.

SPEAKER_01 17:29

And so that's also so this has a name, and I think you actually uh encouraged me to write a quote answer about it. So you know the difference between weakly typed and strongly typed. I think there are there's a lot of people who are confused about what it means. They think that strong means really strong and weak means not so much, but really the the the difference between weak and strong is not dynamic versus static. So for example, Python is strongly typed, but it's a dynamic language, C is uh is uh strong is uh statically typed but weakly typed. And so the difference between the two is basically in a weekly typed system in in a nutshell, the system will make conversions for you. Right. So in a weekly type system, whether it's statically or dynamically, it will say, Oh, you passed me null and this I was expecting a string. Surely you meant the empty string and just keep on going, you know. And that was the problem with with PHP, right? That PHP is weakly typed, but PHP for that from a security perspective is is very, very difficult because you have uh people who write code and let me give you a concrete example. Imagine you have a string that is very important, like it's the salt for a cryptographic library, right? Something like that, right? And now you expose it to all sorts of users, and those users have very, very complicated code paths that can that can get into this library, right? And now you have one of those code paths where actually somebody is passing null, right? And it's it's not like one time in a thousand, by the way, right?

SPEAKER_00 19:26

It's like one every once in a great while this thing turns out it was null.

SPEAKER_01 19:30

It's more than once in a great while because probably there's like 20,000 different gatekeepers in the middle, so that you know, depending on how we A B testing this, depending on which path you are, so it's way less than one in a thousand. You will have this path where you can hit this thing with assault that is null, and that means empty string, and that means horrible things are gonna happen. Well, for an attacker, this is gold, right? For an attacker who who, you know, so let's say an attacker had access to our source code, that's exactly the kind of thing they would be after, because then they can escalate, and this is this is this is gold for them, right? So this is exactly what we were trying to close with hack at the hack v1. V1, v1, v1 doesn't even had um implicit subtyping. So that's how strict it was. Okay, okay. V1 was really like Hindle Milner, and that's was it literally Hindle Milner?

SPEAKER_00 20:33

Was it literally uh unification algorithm?

SPEAKER_01 20:35

Yeah, it was a unification-based algorithm, it wasn't exactly Hindle Milner, but it was without implicit subtyping. So it was really very, very strict, and what I wanted was something really by the book, so that I could, you know, go with a straight face to anybody who was using it, uh, to um and say, well, you can actually trust the types that are coming out of it.

SPEAKER_00 21:00

And when you say strict here, by the way, Julian, what is the relationship between that adjective strict and sound?

SPEAKER_01 21:06

Okay, so I apologize if this is a if this is a digression here.

SPEAKER_00 21:11

This is one of the things I was curious about in the early design of this.

SPEAKER_01 21:14

So sound means that you can trust the types, right? Uh strict is not a term that is really academically defined. This was just how we were defining things. But in our world, what it meant. So the problem with type systems is that we talk a lot about soundness and you talk a lot with a bunch of uh people, a bunch of programmers who actually manipulate type systems all the time. And because they've been exposed to a very particular breed of type systems, they don't realize that they're using certain terms um that in fact don't exactly mean what they think it means. So I'm going to talk about the term sound. And what sound means in the mind of the typical developer is they think that it means that the types are going to be used by the compiler to produce more efficient code. And the reason why they think that is because that's pretty much what every statically typed language does, right? Sound statically typed language does. You enforced soundness, you proved that you know um something is an int 64, an int of 64.

SPEAKER_00 22:32

I know it's a float, I can put it in a floating point register, I can add it with floating point instructions. Let's go.

SPEAKER_01 22:37

Exactly. And that's goal to anybody who's writing a compiler, right? That's what they're after. Now the thing is, type systems, there is a very wide range of type systems. And I would say they go from, on the one hand, type systems that are completely stupid, like uh Bob Harper always likes to joke about it. He says there's no dynamic language, there's only it's a dynamic language is a is a statically typed language with a very basic type system. And the type system and the type system says everything is of type value, PHP value.

SPEAKER_02 23:15

PHP value, and then and then you know, you can do what whatever you want with it, right?

SPEAKER_01 23:20

That's like having uh, if for those who don't know programming languages, that's like having a logic system where everything is true or true implies true, and that's all the things that you can build in this logic, you know, it's true or true implies true.

SPEAKER_02 23:39

That's a lot of nice theorems, right?

SPEAKER_01 23:43

And so, of course, with the problem with this this logic or the problem with this with this type system is that yeah, you'll be able to prove theorems very easily or enforce types very easily, but the problem is that you don't get very interesting properties out of this thing. So it's very easy for me to define a type system over pretty much any language that is sound, but this type system doesn't have interesting properties for a compiler, right? And then you have so sound does not mean strict, uh strict in the non-academic sense, right? I can have a sound type system that is very, very loose, right?

SPEAKER_00 24:24

It's just not very informative. It's uh big, big chunky sets of values in each type.

SPEAKER_01 24:28

Exactly. On the other hand, you have type systems that are very, very rich. In fact, so rich that they contain all the mathematics. And again, at this other end of the spectrum, most programmers have not been exposed to those kind of programming languages, right?

SPEAKER_00 24:46

Although it's 2026. You know, I'm talking in spring of 2026, and and the kids today want to talk to me about lean all the time. Like it's amazing.

SPEAKER_01 24:54

Uh it's this stuff is cool. This stuff is very cool.

SPEAKER_00 24:58

Well, that's nothing to do with hack, unfortunately.

SPEAKER_01 25:01

No, but what's cool about those systems is that they let you have all of maths in your type system. So I can now say this is a function that returns um, you know, prime numbers. I can say that. And that's cool. Now, before these kids get too excited, maybe maybe send them my way, you know, the kids who get too excited.

SPEAKER_00 25:23

I think this would be an entertaining uh horrible.

SPEAKER_01 25:27

I was super excited about when I learned about uh cock, which now is called Rocket, I believe.

SPEAKER_00 25:34

They renamed it?

SPEAKER_01 25:35

Yeah, they renamed it.

SPEAKER_00 25:36

They finally gave in to pressure, huh?

SPEAKER_01 25:37

Yeah, because it was called COQ pronounced.

SPEAKER_00 25:40

Why why why did the French programming language researchers name it C O Q? This French word for a male chicken.

SPEAKER_01 25:48

It means hen. Normally it's the symbol of uh France for many, many things, but it's also a very unfortunate translation in English. Not translation, uh, it's it's an uh very unfortunate pronunciation in English because COQ sounds like another word in in in you know to English speakers.

SPEAKER_00 26:08

What few tricksters got to watch like awkward PL researchers uh stumble over in conferences for an entire decade.

SPEAKER_01 26:15

Exactly. I don't know if that was the intent, but let's just say they I think got past the joke and now happy side effect.

SPEAKER_00 26:23

Okay, all right.

SPEAKER_01 26:23

Yeah, they are renaming it. So back then I was learning about it, and I was excited about type systems because the first time I I really used OCaml, the what got me excited about it is I wrote a bunch of OCaml, a lot of it, and I fought with a type checker sum, and then it just worked. And this was the first time in my life this happened.

SPEAKER_00 26:48

I had this experience with Haskell in '98, by the way. And seriously, just like we're describing. Like you write 3,000 lines of code and it runs the first time, and you're like, oh my god, what is this magic?

SPEAKER_01 26:57

Exactly. And so I had, you know, experience with C, with Java, with all sorts of programs, a lot of Lisp, uh, but mostly C, C, Java, um, some Pascal, you know, a lot of experience with different kinds of languages. But my favorite language back then was C. And my experience with it was I write a lot of C, a bunch of C code, and then I run my program, but I run it in GDB because I know I know it's gonna cycle. I'm not even trying to run it without GDB. I'm like, I know this is gonna crash, you know. Uh even if I'm writing something completely trivial. And the first time I get this experience where I'm like, well, wait a minute, I just wrote 2,000 lines of code, and yes, I had to fight a little bit with a type checker, but it just worked. This is amazing, you know? And so the next logical steps. I just found out about that back then. I was a student in Paris, and somebody comes along and says, you know what? Those types that you use in OCaml, it doesn't have to stop to that. You know, we have this other system where you can get all of mathematics in it.

SPEAKER_00 28:14

You can describe whatever you want in those types, and it's like while we're here, like when I was like a work-a-day C programmer, I would occasionally hear people talking about this, even after it like learned a little bit of Haskell. And I'm not sure I understood quite what they're getting at. The abstraction here is basically types are just sets of values. And you could imagine an enormously more flexible language for describing sets of values and enormously more capable. So, for example, like I can't say in in a C program, maybe I can in C26 or something, I should check. But like, it's hard to say at least this returns even integers or this returns odd integers, for instance. But that's kind of a work-a-day thing in these more uh highbrow languages.

SPEAKER_01 28:55

Yeah, exactly. And so when I first learned about it, I stormed into the room of my professor uh back then, uh, which wasn't very polite, but he was used to it.

SPEAKER_02 29:06

And I said, It's amazing, we'll prove everything. And so he was used to this kind of you know outburst for me. So he turns around and saying, What do you mean by everything, Julian? And like, I just learned about cock, it's the best thing ever. We'll be able to prove everything, there will be no more bugs in programs, this is a revolution, and he was like, Okay. Again, what is everything?

SPEAKER_01 29:37

So he got me thinking, but I I didn't really quite get it. And it took me a long time to actually get it. So I I spent maybe a year writing a lot of you know proof-carrying code and this kind of stuff. And and it took me a while to realize that of all the programs that we write, some of them have a clear spec, right? Like, so for example, if you're gonna sort numbers, yeah, I can write the mathematical function that specifies exactly what the output should look like. So this is a really good candidate for formal methods, because once I've written it once and proven that it worked this way, I it it always works for this particular property. And there are some programs that are like that. So, for example, compilers, especially when they target well-defined languages, is another good target where we know exactly what the meaning of the input is, what the meaning of the output is. So, yeah, formalizing it makes sense, and having a program that proves that the pro the the code that we're writing is actually correct makes sense. So that's great. And don't get me wrong, I still love this stuff for this kind of stuff. But in general, most of the programs we write, uh what it's supposed to do is ill-defined. Like it's it's going to be refined over time. It's based on, depending on what humans prefer, what the business prefers, what so if I look at the ocean of code that's out there, a lot of it is ill-defined. Like, what shall this thing actually do? The ultimate answer is probably in the hands of a human being at some point who's gonna say, yeah, I should do this or I should do that, right? And when that's the case, those formal methods they hurt you. Because, you know, when you write a program in this kind of environment, the work, the amount of energy it takes to write something is huge. Because, so for example, every single loop, you have to prove that the loop terminates. Sometimes it's a huge amount of work because sometimes it terminates for non-trivial reasons, right? And now you're imagine you were writing some piece of business logic, and your boss comes around and says, Oh, yeah, yeah, yeah, I don't want to calculate the price this way, but this way. And it completely invalidates your proof of how the program terminates. Well, all this energy that you put into these things, you know, proving this property or that property, proving it terminates, proving is all energy that will be wasted if the program needs to be modified and the program needs to be refined, right? Right, right. So the the so again to go back to our initial conversations, which was you have to do that.

SPEAKER_00 32:34

Sorry to keep dragging us out of the weeds here. Let's we've got a story to tell.

SPEAKER_01 32:38

No, that's fine. So we have dynamically type type systems, which are really not saying anything interesting other than this is a value and you can compose it with other values. Good for you, you know. Uh they can be sound. You have uh extremely strict type systems which have their own, you know, uh types that are very expressive, and I think are useful for some things, which the things I think they're useful at is for programs whose output is well defined and is not going to change over time. And then you have what everybody is used to. And what everybody is used to is C, Java, you know, you name it. And when you say statically typed, that's what they think of. They think of Java, you know? And I actually suffered of that with uh you know some people within the company at Facebook back then, and I'm not gonna name names, but one of the problems I faced was the what we were proposing with Hack, you know, if the dumb type system that is fully dynamic is here, and if the super strict, you know, uh numbers.

SPEAKER_00 33:57

Yeah, you can prove like number theory or whatever.

SPEAKER_01 33:59

Let's say C is here, what we were offering with Hack was here, right? So we were between a fully statically typed solution and uh you know the dumb type system where you really cannot trust anything. And the reason was relatively simple. The reason was the million dollar question you want to ask yourself when you're adding a type system to an existing dynamic language is if I have an array of int, shall this be compatible with an array of mixed, you know, where mixed can be anything? And if the answer is no, you're gonna break a lot of code because they'll there'll be a lot of code out there that was written by people who didn't not have types in mind, and they will expect to be able to have this operation working. If it's not working, you're gonna break a lot of things. If the answer is yes, and you want this to be efficient, you don't have much of a choice other than to represent the integer as a boxed integer. I mean, boxed, it doesn't have to be allocated, but you need something that, you know, says what the type is and something with a value. Because if you don't, then you don't know how this array is going to be used later down the road. And playing that game lazily, meaning you have a more compact representation, but then you only expand it if necessary, has its other kind of forms, kind of problems.

SPEAKER_00 35:34

Wait, we've tried to play these games in the runtime too, by the way, and and didn't ever quite get them to pay off while I was there. I'm told that they paid off eventually, but we had a design for something called 7 pack that was like gonna segregate all the type tags in the front of you know you know in head matter for the array, and then like have sort of unboxed values that are tightly compacted in like a dynamically allocated segment afterwards and blah blah blah. Um and it's like you you you'd come out ahead on some like memory hierarchy things. You'd come out ahead on the fact that like your your heap footprint got smaller was the hope. But it was tough to kind of come out ahead in terms of how much more compute there was with kind of packing and unpacking these little bit packed tags and stuff. Um I think everybody ends up playing with this once they get aggressive with with runtime implementations for languages like this.

SPEAKER_01 36:18

Yeah, I mean it reminds me a lot of the kind of problems you have with persistent data structures, right? Where they typically have an amortized cost that is okay because they have this one operation that runs once in a while. So typically, you know, the the well-known example is implementing a queue in a in a purely functional setup consists in having two lists, right? And then once you you one of the lists gets empty, you invert the other and and whatnot. And the problem, the trick is it seems like a good idea until you know a bunch of people ends up holding a reference to the version of the list that hasn't been inverted yet.

SPEAKER_04 37:00

Right, right.

SPEAKER_01 37:00

And then they all call that thing again and again and again. And these are the kind of problems you can have in in these kind of setup, right? Where you think that you're gonna pay that cost just once, but no, because multiple people hold references to the old fashioned. Hack was never going to be a statically type langu uh a statically typed language in the sense of C sharp. And that I think was misunderstood by a lot of people in the company and created, made the company, you know, go in a bunch of I think wasteful directions. Not bad directions, but wasteful, in the sense that a lot of people were completely convinced that they were going to compile hack the way you compile C, uh, if we kept on going at this typing thing, you know. Right, right. And and you're like, no, this is this is never gonna happen. This is not what this type system is. And I think a lot of it is because of a misunderstanding of this spectrum of type system that I explained to you. Had they been this understanding that there is type systems that are super precise, so precise that you can get all of maths in it, you have type systems that are completely dumb. You know, the only thing you can say is this is a value. Um, and then you have all sorts of things in between, that that would have helped a lot.

SPEAKER_04 38:34

Yep, yeah.

SPEAKER_01 38:36

But uh, so how did we get there? I think we started from gradually typing, gradual what gradual typing means, and then how how did we end up in this direction? I don't remember.

SPEAKER_00 38:46

Uh I mean it's probably my fault. I like I asked you, I know I asked you about the difference between sort of sound and strict at some point, uh, because you're sort of talking about strict mode, which is kind of where it I think that was the first design point you hit, was kind of what is now hack strict mode.

SPEAKER_01 38:59

Yeah. And so hack strict mode, no, what hack strict mode is today was not what hack strict was. Hack strict was much, much stricter than hack strict mode today. So hack strict was really Hindler Milner on top of PHP. And the original design was to get um security experts to convert the most important parts of our code base to uh a really strict version of PHP so that we could control it.

SPEAKER_00 39:30

From your perspective at this time, Julian actually out of curiosity, just because you were actually, you know what you were thinking at the time. Like if that was all hack did, right? If you just sort of made the strict mode thing, yeah, it's a little bit unwieldy in some ways, but it's okay because only like 10, you know, a tenth of a percent of the code needs to be converted to this. And it gets rid of this class of security vulnerabilities, would you have like considered that a success? Or would you have did you always kind of have the ambition to incept the whole code base? No, no. We're talking about, as far as I know, the largest PHP or or PHP adjacent code base that's been created. At this time it's sort of tens of millions of lines of code, uh maybe low tens. No. It will make it into the hundreds.

SPEAKER_01 40:16

Yeah, it was already big, yeah.

SPEAKER_00 40:18

Yeah, yeah. So very, very large programs, sort of comparable to Microsoft Office or Windows or something. It's a really, really unwieldy software project.

SPEAKER_01 40:26

Yeah, exactly. And so so back then, I think I would have been okay with uh having a niche, uh, because the the problem is the original impact I was hoping to have was not really working. Because what was happening is that we struggle to find pieces of code base that was really isolated. Um and so what was going on is so you know, in in most programming languages, um what you what you have is you have a dependency system that is relatively script strict. And this is really nothing to do with dynamic versus static. So for example, if you write JavaScript, the JavaScript folks are very, very uh careful about the size of their dependencies because typically this is going to affect the time to load in a browser, at least when they're working client side, you know? And the same is going to be true about a lot of other languages where because it's going to affect compilation time. So if I pull on a bunch of dependencies I don't need, well, now all of a sudden it takes forever to compile something simple. And so what we have a tendency to do is to split up the work into sub pieces that are independent, and we try that to keep the dependency graph as a DAG as much as possible. We try to keep it as a tree or a DAG because you can have some sharing in in the middle. But the problem at Facebook is that we had this thing called autoloading. And so with autoloading, what was happening is whenever you used a class or a function, you know, HHVM just figured out which class of function you were talking about, and it was just gonna load that.

unknown 42:19

Yep, yep.

SPEAKER_00 42:20

It was actually a PHP thing, by the way. The H H VM implementation of auto loading was the PHP, you know. We we had the same idea, yet the same weird magic methods and everything. But sorry, go ahead.

SPEAKER_01 42:31

No, but I mean it seems like a good thing because it lets the developer move faster. Uh, but the problem with it is that it's um it's how do I put this? It's it doesn't uh it doesn't force the programmer to structure their code more, right? And to keep the dependencies in a way that makes sense. And so the way Facebook ended up was a bag of dependencies. Everything was spaghetti noodles, everything was depending on everything, and you really could not pull it, pull out a class without pulling out millions of lines of code, right?

SPEAKER_00 43:09

By the way, like a non-obvious consequence of this thing that we're just gonna load stuff for runtime is basically figuring out the dependency of a module is like on the wrong side of the halting problem, right? The only way to figure out what this thing depends on is to compute it forward and see all the symbols it touched. And if it happens to be all the symbols in the code base, great, its dependency is everything. Um if it's some subset, awesome, it's some other subset, but it's gonna be a dynamically determined subset. So it wasn't like you could even it's not just that it was like intertangled and and you know not DAG-like and mutually nasty. So it wasn't even like you could say statically, oh, look at how nasty this dependency graph is.

SPEAKER_01 43:41

Exactly. Yeah, it's complicated. And so because of that, basically it was very difficult to isolate subparts of the code base where security engineers could work on. Where you could say, all right, this is a very important piece of program, make sure that this. works because it's basically impossible to identify leaves. You don't have real leaves. Like everything depends on everything and everything is pulling in everything. And so because of that, what the original intent was to support the teams, the security teams that were working with hack and get them to work more comfortably. And so we started relaxing things, right? And so we went from okay this is going to be Hindler Milner, this is going to be very, very strict and we'll be able to trust the types to okay we're going to let you know a lot of things you know go in. And eventually I don't want to say it was a best effort kind of thing because it was never like TypeScript. You know TypeScript they had a lot they made a lot of design decisions where it was unsound or out of convenience. And so we never did that except for a couple of things that were very specific. So for example arrays when you access an array um typically it could be undefined if we cannot prove that you are out of bound. Right. Now the problem is if we make you check that something is uh defined or not every single time you access an array because we cannot prove that it's not undefined, basically people are going to be throwing food at your face. You know, developers are never going to go for it. So this was one where like okay it's unsound but we'll have to swallow that one and it is what it is. And then the difference between strict and normal mode became do we tolerate the absence of type annotations? So we considered that a file was strict if everywhere where we expected type annotations was actually annotated and um a file was partial if there was some annotations missing. And this was a drift from the original meaning of strict the original meaning of strict was everything is annotated and all its transitive dependencies is also annotated and everything provably works but was not usable in practice. So to answer your original question was would I been satisfied with you know stopping there yes had we had you know better success with uh but I realized that to have to be more useful to those security teams we needed to relax yeah the current system and the moment we relax the current system it's attracted other teams who are like well wait a minute I hear you have autocomplete and things of the sort and well I miss my you know Java days.

SPEAKER_00 47:02

I like I'd like to be able to rename a method sometime this century for instance. Yeah I'd like to be able to do normal programming stuff.

SPEAKER_01 47:08

I remember diff. This was my go-to diff to explain to people why we had to they had to adopt hack. So that was years later while hack was being mass converted in the code base. And it was a diff with all the prominent engineer of Facebook 2010 something like that and they're trying to rename a method but it was a name that was very common maybe it was render or something like that you know a name that appears everywhere. And it it it it sparked a debate that lasted for a week and in the end they decided not to do it.

SPEAKER_00 47:51

Look this is proof that you know if you need one week of debate of your top engineers to decide not to rename a method you have a problem so yeah that was uh interesting curiosity too would you have actually if you knew with perfect certainty before doing any of this that the that in order to succeed you're basically going to have to like convert all of this you know tens of millions of lines of code, convince all of these people you've never met, uh deal with all these sort of, you know, deal with years of people criticizing your baby on sort of scant information or one bad experience. Like do you actually think you'd have set down this road if you knew it was going to that the the mission was actually to capture the all of dub dub dub to capture the whole code base?

SPEAKER_01 48:39

I don't think so I don't think so. But I think you know this is the famous quote of uh uh JFK applied to computer scientists maybe we mentioned it in the last episode I don't remember but you know there's this quote that says we don't do things because they're easy normally the quote from JFK is we do them because they're hard right but in the computer science world it goes like we do them because we thought they were going to be easy exactly there's there's some truth to that in you know the the the hack project I think what I underestimated and and for good reasons. What I underestimated was how hard it was going to be to scale and it it was due to this dependency problem. Because when I attacked the problem at first I was convinced I was just going to do what every other language does. You know split it up into libraries have a build system you know have a a DAG of dependencies and build it this way. And it took me a while to realize so there was Yuan Padiolo back then who was trying to start an effort within the company to add you know um to add restrictions to how things could depend on one another in w but clearly it was going nowhere and Loing looking you know I looked at what he was doing I think it was very good. I think it was called overlay back then. So his idea was there's no way we're going to bring structure to w. So what we should do is bring our own level of structure on top of the mess by using symbolic link. So the idea was we're going to fake this hierarchy that we need by with a bunch of symbolic links. And so that was called overlay. And I thought the approach was interesting but I thought there was so little buy-in in the company that I thought this is never going to work. So the stuff that I underestimated was when I started this project was well here's a code base where everything depends on everything and you have a bunch of PHP developers who are never going to wait for 20 minutes for their type checker to say yes you can try this change that you made now. But I mean I I learned a lot in in this process and especially I learned a lot about systems system programming distributed systems I learned it a bunch but I threw that had I start so that's true that's one dimension the one dimension that I did not foresee was how hard it was going to be in terms of system programming. The second dimension was I did not I I did not realize how much human friction that would uh that would that would get you know and a lot of it was fighting with people who have emotional reactions to this kind of stuff and and you know some of them I understand their point I think they're misguided but it doesn't mean that I don't understand where they're coming from like I remember one guy vividly who you know came from a company where it was all C. And he absolutely hated it so I actually sat down with him and got him to explain what kind of code base this was. So this was pre C11. So imagine C with very very oh C. Everything is a virtual method you know everything is new delete whatnot and he was telling me and and templates everywhere. And he was telling me I could not modify anything without waiting half an hour for the compiler and whenever it didn't work the the error messages on those templates were just completely unreadable. I just didn't know what what any of this meant. And now I arrive at Facebook and I'm productive again. I get to write stuff it actually works. I enjoy myself I have a good time and here you come again with those generics and all this shit that was making my life miserable and so the reason is from his perspective he associated type systems to C because that was his experience of type systems right and so my job was to convince somebody like that that you know type systems are not all equal and some of them actually can make you more productive and and so this was some kind of arguments that I had to deal with. I would say people who were very emotional about you know hack and bringing that into into their workflow. Some others were more interesting. So some others human interactions were I would call it plain bad faith you know like people who are just against everything for no reason at all. They just don't like the idea.

SPEAKER_00 53:51

So I had this guy's the thing at the butt there's like a broad phenomenon inside of tech companies is like the vast majority of things end up not working. The vast majority of things like don't pan out or if they do pan out it's sort of pretty eye of the beholder about whether they succeeded or not. And like you can kind of um a perception of smartness by like just bitching about everything. If everything that comes down the pipe you're just like that's stupid and you're stupid that's also stupid and you're also stupid. Like you can sort of look smart for quite a while and and some people do kind of play that game unfortunately.

SPEAKER_01 54:24

Well I mean the problem is if you play that game you're going to be right 95% 95% of the time. And so now you could say well I'm right most of the time and that's true. You know but all what you're doing is shooting down everything you see. And so that's why for me in in my book the people who matter in a company or in any endeavor are the people who stick their neck out to say something positive about something that everybody else is shooting down. Those are the most valuable people whatever you're doing whatever you're building so I just read a a comic book recently about uh George Lucas and how you know he was making Star Wars and there's this scene there's this moment where I mean I didn't know that but while he was making Star Wars basically everybody involved thought of it as a joke. You know like the actors didn't like it. Yeah yeah yeah it was a B movie like everybody thought this was this is cheap stupid junk everybody who was working on this project you know uh didn't believe in it and then he shows it to a bunch of people who are you know prominent name in the industry and pretty much all of them say it's crap and you know it's never gonna work and you have Steven Spielberg who's tells him some I I paraphrase because I don't remember it's something like this is going to be one of the biggest successes in in this the the history of cinema you know and so that guy is fucking gold because that's what you need you need the guy who's gonna stick his neck out and who's gonna say this is the shit this is what we need to do because saying something is crap guess what nobody remembers the name of all these people who said that this movie was crap right nobody remembers their name right this is cheap this is easy everybody can do it but the people who say this is gonna work that is and so for me the people who did that for me with hack so it's not the same level as Star Wars obviously I mean you know I may hold that you know history is still out we'll see what history says we'll see we'll see but uh you know there was Joel Poba Joel Poba really really stick his neck out for me and he was like back then you know he's like I really believe in this he really pushed in that direction obviously Alak Mengrajini who was part of the project but there was also you thank you motherfucker well I tried you did put uh stick your neck out to say well this stuff you know we need to give it a chance and we need to and that was precious at that time so to go back to your original question which was would you have done it knowing what was ahead of you probably not say for the the I underestimated dramatically how difficult things were going to be technically especially on the scaling side not so much on the language side because this this part I knew well but then and then the second dimension that I did underestimate a lot was how a pain in the ass humans can be when it comes to programming languages.

SPEAKER_02 57:55

I've I've had people you know who try to wrong feelings indeed who you know try to convince me that life was not worth being live lived without this or that feature.

SPEAKER_00 58:13

I'm doing just fine I have friends I have yeah exactly I have hobbies like I people like me. I mean the thing that that struck me about the product the whole hacklang episode from the outside the whole adventure and I think this this fulfills like my favorite technical definition of adventure which is something you are glad you did that you would not have started if you had any clear idea of what was involved when you started it right I think like one of the things that jumped out at me from how you prosecuted it that I thought was really impressive was it sort of started with this um basically type theory idea. It started this sort of like set of ideas that you need a very particular background that you possessed to be able to make to exploit. And I think most people would say like well okay like I I had all the good ideas like why are these bozos opposing me? And you took there were at least two areas that were really obvious where you took the ergonomic complaints seriously. One was like as a systems programmer right that basically you needed to have what we would now call an LSP or whatever, right? That we needed to have this kind of long-lived server that would kind of pull modifications to the file system and incrementally update a model of the program instead of some big bang run end to end over the whole code base C compiler style thing because as you said nobody's going to wait around for that.

SPEAKER_01 59:30

And that's related to dependencies, right? Because everything depends on everything, you have to do it this way.

SPEAKER_00 59:36

And you mentioned the problem but I don't think you quite got to the solution but like the the um I I I did this too by the way at the I was a C programmer. I associated generics with like inscrutable error messages right that like if you have a generic system and I have a type error now and I'm using somebody else's generic it means there's going to be 17 files I've never seen before in my life and the error messages are going to be about like I couldn't unify this thing in this file and then I couldn't unify that thing in this other file because this thing in this third file you've never heard of and so on. And then it was going to be this this whole big mess. And uh my impression is that you sort of took that criticism really that the the you took that concern enough to heart to actually um I think it's was it four lines is four lines the max for a hack error message? Did it ever get to five?

SPEAKER_01 1:00:22

I did get to five but yes I started going with this.

SPEAKER_00 1:00:25

But yeah like it usually was sort of what you wished you know a sane human being would say which would be like I can't make this work because you told me this was a vector of foo and you just stuck a bar into it and I don't have any way to turn this bar into a foo. You know and it was like perfectly understandable as opposed to you know 14 C header files you know that that are that came with the SCL implementation.

SPEAKER_01 1:00:49

I think we need to give credit where it's due and and C templates came before and are doing something a little bit different. So first of all it came before and it came as you know something on top of C and C adds a lot of constraints to what you can do. And because it is um a system programming language in a system programming language you typically have to specialize all the types because you don't want to so in a in a non-system programming language what you end up doing is you tend to allocate for everything. So everything is a pointer and then everything has the same size and it makes everything easier right that's what Java does what C does. And then the few cases where they don't do that, it's called a value class and it's you know very well identified in the programs right with C what they have to do because it's a system programming language they have to specialize everything because they don't know what the size of different things are going to be. So the moment you have a generic if you want to be able to emit the code that it's going to you know you need to specialize the code. And this specialization process you know added to the fact that it was added as a hack on top of headers and how headers work in C, made it such that it was going to be very difficult to build good error messages on top of that, right? Like I start from the header system and then I just start unfolding everything and then you know you get to the end of that and yeah it's it's making sense of what happened there is going to be difficult.

SPEAKER_00 1:02:33

So let's just say that C generics is solving a different problem and I don't I most of my professional life was like building C right like I'm not I'm not some hater or anything.

SPEAKER_01 1:02:45

I'm just saying like I know what the what the guy you were talking to who was afraid of your error messages was complaining about that's all yeah I mean and and I understood how where he was coming from too so I was I was not complaining about him. There are other people who were acting in bad faith who I'll get to later and I have some funny anecdotes there. But I don't think he was acting in bad faith. I think he was truly trying to help the company even though I I disagreed with him. So the thing with error messages that we try to do with hack for every single type we keep a witness right so whenever you know we believe that something is an int, we're going to keep a place in the source code that say that shows to a human being this is an int and you cannot argue with the fact that this is an int, right? And so what ends up happening when you have a subtyping relationship uh you let's say you want a to be subtype of b. So let's say you want to return something you want to return a value for that return to be correct you need the value that you're returning to be subtype of the annotation that was added by by the developer right right and so the error message has three parts one is why did I start the process of uh you know checking that this thing was subtype of this so this is going to point at the return right it says well you are telling me good sir you want to return this thing right yeah yeah um and then then what happens is the subtyping process starts and we go check a bunch of things and we will hit an incompati incompatibility right and that's where instead of just dumping pre-printing the return the thing you're trying to return and the thing that was annotating we do something else. What we do so if you pre print both types there will be many cases where those both types are actually very complex and it will be frustrating experience. Right right so what we did instead is we remember all the witnesses of why types are incom uh why types are what we think they are and oftentimes it's going to be annotations and we point it to the user and we say well on this hand you have an int and on this end you have a string. And so really what we're telling to the user is we're letting him make a choice out of three options right we say okay you want to return this thing three options for you number one you don't return this thing. If you don't return this thing the problem goes away right number two this thing over there that's an int, you need to change it. It cannot be an int because it's or this other thing over there that's a string, you need to change that right and what you're telling him is your choice buddy it buddy is going to be one of those three right we're not it's we're not gonna let or a completely different change if you want but But you'll have to do that. And what's nice when you do that is that the refactorings and all the tools that we were able to do in terms of refactorings, in terms of refactoring, were actually really good and really fast. Because what you can do now is you want to change a type, you want to change something, you first change the type, and you let the type system go point you to all the places where you have to change it. So that was the witness system. And I tried to sell it to this C guy, and it was difficult because the scars of C were deep, very, very deep. And the other scar was the speed. So a lot of people complain uh from that world that you know compilation times are slow. And we built uh an incremental type checker where the feedback was sub 50 milliseconds for pretty much any any change you were making. So all in all, we converted people around, but it took time. And and so some of them were.

SPEAKER_00 1:07:08

And I think the that was one of the things, by the way, Julian, that I thought was just like an unusual experience uh being and partly it's that we're in the same shop as your users, right? Like you are an employee of the company that is you know has the user of your language there. But I you know you actually sort of cared about this as a product and an experience and went back and changed things and invented things and had ideas that were driven by that. And and from the outside perspective, it seemed like 95% of the work was that, right? It was sort of there was this initial spark of insight and sort of set up sets of ideas around, you know, how you're gonna impose some order on this this type chaos of PHP programs that turned out to work and that was hard, and not many other, you know, you sort of succeeded there where other people didn't fail, had failed, but you didn't sort of declare victory then and say, like, oh, okay, well, here's some tools, go use them. Um you actually sort of sat there and watched how people were struggling with it and and changed the tool in response. That sounds like an obvious thing, but it sort of wasn't always the case, especially in the PL world. I think there was there even in even these C people, right? Like there are many, many, you know, engineers who work on static languages who just, you know, the only the only program in their language they really work with is their compiler, right? And that's probably par for the course for a lot of these folks.

SPEAKER_01 1:08:27

Yeah, I mean, the thing is I never saw myself as a language guy. I always saw myself as a dev tool guy. And so what makes my eyes sparkle is to make developers more productive. Um and so for me, if I'm writing a tool that, you know, is really, really cool theoretically, has a lot of good properties, but is not used by developers, then I failed my mission. I mean, it's not that I failed my mission, it's that I don't care. You know, I could be building the best stuff in the world, but nobody cares, nobody uses it. It doesn't interest me. And you're right that in the academic world, that is not the norm. Although there are exceptions. So for example, Jan Vitek, um, who you you also know, um, you know, he clearly deeply cares about this stuff, and he really wants to see whatever he builds to be used, and so that's something we have in common. Uh, another one is Greg Morissette, who you know I talked to a bunch, and he he's very happy. I mean, I haven't seen him in a couple of years, but back then he was telling me, you know, the success of Rust. A lot of ideas from Rust came from him with C clone, so it was academic ideas, he didn't participate in the writing of Rust or anything, but a lot of the ideas in Rust, you know, were were started in in his work with C clone and whatnot. And he's very happy about that. He's very happy about the fact that his ideas led to things that and you have other people in the PL world who unfortunately I think don't care at all about it. So I understand where they're coming from. I think the argument that comes back all the time is you know, look at uh foundational maths, and there was a bunch of stuff that we thought was not useful and then turns out to be revolutionary one century later. And I hear those arguments, but the problem is a programming language is something that's meant to be used and manipulated by human beings. And if if it's not, it's not a programming language, it's something something else, it's an internal form, it's it's whatever you want to call it. But if you're gonna work on programming languages and you don't care about how you know people interact with those languages, that that's strange to me. The other thing is sometimes you'll have a form of snobbism. That is like we write those languages that are you know for better people, who you know, and and all those all those who don't understand it are just too dumb to understand the the good stuff that we do, you know. So there can be also a form of snobbism going on. But in all honesty, I think I really and I spent a lot of time thinking about these things. I really don't know what drives the adoption of a programming language at the end of the day. You know, if I look back, it's it feels a bit like music, you know, like you've got some records that just you know, boom, they just explode.

SPEAKER_02 1:11:43

And you ask the artist, and he thinks, this is my worst song. I don't understand what's going on, you know?

SPEAKER_00 1:11:49

Yeah, I would have I'd have made the third verse make sense if I realized I was gonna sing this every night of my life for the rest of the for the next 20 years. Yeah, yeah.

SPEAKER_01 1:11:56

Exactly. And and I feel like there's some of that going on with programming languages where you know you you've got some programming languages that are super successful, super popular, and you look at it and you're like, that thing? You know, so for example, JavaScript is a good example, right? Like I or yeah, JavaScript is a really good example. Like it's now becoming the most popular programming language with TypeScript. Now, I understand now, I mean I understand it was imposed through the browser, and then people had to use it in a browser, but then who thought it would be a good idea to use that server side? And the whole argument was, oh yeah, you'll get to use the same language and the same libraries, client side, and server side. But you don't. You don't, you never do. So now I'm using an inefficient language server side too. And now, don't get me wrong, I think today it's actually a reasonable ecosystem. Don't get me wrong, I think writing you know your backend in TypeScript is totally fine. It's just that the evolution of all that, if we had to rewrite it from scratch, yeah, I don't think we would go down that route, right? And so if I had to yeah, I I I have a hard time and and in some other cases I totally get it. So for example, Rust, um I get it. I understand why people are excited about it. It is really, you know, filling a need, you know, like we system programmers have been battling with all those ideas, especially since C11. They have a bunch of concepts and C only enforces them partially, and they want a way to program that makes all of that safe. And Rust comes in and says, I have a solution for you guys, and that's gold. It's exactly why you want programming languages. This is awesome. Sometimes programming languages are successful for good reasons that I understand, but most of the time it beats me.

SPEAKER_00 1:14:01

Um yeah, I think there's LARP there's network effects and weather and like just yeah, hit pit-driven phenomena, and somebody that's smart that you respect tells you about it, and so you take it more seriously or whatever. It's the mysteries of influence. Well, since we hit on TypeScript here briefly, I want to share a little bit. So I I actually ended up Johnny Apple seeding hack into Slack, my job after Facebook. Also a big PHP monolith for the for the main application, having some some of the problems that initially motivated Hack as well. And you know, I of course said, well, you know, there's this thing, and and a few people started playing with it. And you know, a similar a version of the flame war uh that that erupted at Facebook sort of ensued over the next couple of years, and over time it ended up moving to to being a hack monolith as well. Uh and and I found in that time, right? So this is now 2016 to 2020. Um I found in that time frame, if I'm explaining hack from scratch to people, I had my routine that I sort of picked up from Facebook, which is like, well, when you want to do this, you do this, and when you want to express that, express that. And I found more and more of that was me saying that, and then then being like, oh yeah, like in TypeScript. Okay, right, right. So TypeScript, got it, got it. Okay, so it's TypeScript for PHP, cool. And eventually I started shortening this too. It's like TypeScript for PHP, which I feel I'm doing you a disservice every time I do this because I know that was not the chronology and was not the order of influence here. Um I'm curious like how that lands for you. Do you know what I mean? Like because it's probably like the most popular language I would imagine that people are reaching for by default now on in server project. TypeScript.

SPEAKER_01 1:15:40

I'm very happy that TypeScript is successful. I do not build things because I want to put my name on things and get people to say I was the first to do this. I don't give a crap about that. So I'm very, very happy that TypeScript is more and more popular. I'm happy that types are more popular among um developers. And I'm happy that a lot of those ideas that were not started by me. Like there was, you know, academics who were talking about this stuff. But I would say hack was the first project at this scale, right? Like trying to get a type system on top of an existing code base that was that large, that was the first time you know this was attempted. And I would like to believe, I don't know, I wasn't there. I'd like to believe that that have inspired at least comforted, you know, the people at Microsoft with the idea that they could pull this off with JavaScript, right? So from my perspective, the success of TypeScript is great. I'm very, very happy with it, and you're totally fine explaining that hack is like TypeScript for PHP because that's what people are familiar with. I'm not offended whatsoever. In fact, I'm I'm I'm a little bit proud of it. You know, it's yeah, yeah, it's kind of cool. Everybody knows TypeScript. So funny enough, I just wrote uh with my team at Skip Labs a re-implementation of TypeScript uh that is both sound and incremental. Uh that hasn't been released yet. So that's maybe for another episode. But I have a lot of things to say about TypeScript.

SPEAKER_00 1:17:22

Is it literally is it literally TypeScript? It's not a TypeScript like system.

SPEAKER_01 1:17:25

It is it's literally TypeScript, uh at least all of TypeScript syntax are a few things. Uh because we want it to be sound, there are some constructions that we cannot support. So for example, if you have eval or if you have uh dynamic imports, like you're importing something based on some value or things of this nature, this we cannot we cannot do. But it's a very large subset of TypeScript. Yes. And the syntax is exactly the same. And it was a lot of work because there's a lot of there's a lot of things in the in the type system. I think what's nice about TypeScript is that they built a type system that is very expressive. So the language lets you express a lot of things. What I think is a little bit unfortunate, and I don't like to say negative things about um this kind of stuff without having the con the whole context. I wasn't there when those decisions were made. But I think one a little bit unfortunate thing is that you sometimes have a hard time understanding what is checked and what isn't. And that was that was something that I think was a little bit more obvious in hack where you could actually see what were the parts that were checked and what wasn't checked.

SPEAKER_04 1:18:45

Yeah, yeah.

SPEAKER_01 1:18:45

Um here in TypeScript, they let you write really complex types with you know very powerful constructions. But then when you manipulate them.

SPEAKER_00 1:18:55

Yeah, like all these operations on sets, and yeah.

SPEAKER_01 1:18:58

When you manipulate them, you realize that there is a bunch of cases where it's not checking anything. It just tells you, oh yeah, if you tell me that this is it, then this is it, you know?

SPEAKER_00 1:19:09

Yeah, let's go.

SPEAKER_01 1:19:11

Exactly. But I think from so this is secondhand information because again, I wasn't at Microsoft when this was built, but my understanding is how TypeScript came to be. The idea was more to enable the kind of tooling you would find in Visual Studio, you know, like enable autocomplete and these kind of things. And I think they saw type checking, like the checking part as more like a bonus. And if that's what you're optimizing for, I think what they produced makes sense. If you have a complicated type and you clearly cannot prove that it's the right type, but at the same time you cannot really show why it's incorrect, then maybe just keeping the type around to get the autocomplete to keep on working is maybe the best path forward. Right, right. So they were they were optimizing for something else than what we wear at Hack and with Hack, and so it it kind of makes sense. Way it learned.

SPEAKER_00 1:20:12

Yeah, yeah, that makes no sense. And Julian, you're a very humble guy, so I probably should have like rung this bell a little bit harder earlier, but most programming language designers are not successful, like even very professional, even professional programming language designers. The vast majority of people design a programming language where the most complicated program ever written in it is the compiler for that language, where there are never any users of the language who are sort of outside of the community of people who are interested in it for some academic reason. Um so like you've you designed a programming language that had some novelty to it, that solved a real problem and achieved industrial adoption. And I think for everybody who is around and and in the house at the time, obviously created a lot of value, and that's pretty rare. And so we're kind of describing a pretty weird arc, a really weird career arc that Julian was able to walk here.

SPEAKER_01 1:21:01

Thanks. I mean, yeah, um it I don't need more praises. I'm very happy with uh where I'm at. I'm I'm very, very flattered that you think that this was uh uh uh a good uh great achievement. I I feel very good about it myself. Part of it is luck, you know, so it's also being in the right at the right place at the right time and meeting the right people. I mean, you know, this conversation I had with you, I think had I had it with the HPHP team, you know, because had I come, you know, a couple of years earlier, the HPHP team when I talked about this idea with hyping, and I think it was uh uh Williams who was working on it. What was his first name?

SPEAKER_00 1:21:46

Mark Williams.

SPEAKER_01 1:21:47

Mark Williams.

SPEAKER_00 1:21:48

Yeah, yeah.

SPEAKER_01 1:21:49

They weren't so keen on on this idea, so this could have been killed, you know, from the get-go. So there's also the people who I met with in the company who helped me. I mean, I mean, I mentioned Polar, but there was others, Alloc, Mengrajini played a big role, yeah. Uh and and others. So there's a little bit of luck. There's um a lot of good timing, you know, meeting the right people in the right place, etc. Uh and that's also something Facebook needed at this time, so it was a good time to go after something like that. Uh and yes, a lot of hard work, which I'm very proud of. And uh that's that doesn't change that. The other thing is building hack is the reason why I build Skip. And it's because it was so hard to scale Hack, because Hack was a fundamentally incremental system, which led me to build Skip because the way Hack was working was like uh like a language server, but you start the server, it takes in all the files, then it watches the file system, and whenever the file system changes, it takes those changes into account. And that is a fundamentally incremental system. When a file changes, you don't want to recompute everything, right? And so I built this runtime, which was basically I wiped out the OCaml garbage collector completely and replaced it with our own system where we had a heap that was a persistent heap that would persist persist data long term, and this heap was uh reference counted, if my memory serves me well. And on top of that, we had an OCaml runtime that was working with this thing. And the problem is back then we were maintaining the dependencies by hand, and so we had basically to do the cache invalidation by hand. And so to stabilize the system, because what what kept on happening was a change comes in and not everything is updated perfectly, but it still works. So the user keeps on working with it, and then two days later they have an error, this method doesn't exist. When you go look at the file, clearly the method is defined, and you you don't know how you ended up in this state. Right? So what we built back then was a thing where you take the code base and you dump what we called the signature. So the signature was basically the entire state of everything we thought we knew about the code base. All the types, all the definitions, everything, everything we thought we knew was in the state. And then for weeks we would have um uh daemons that would check out revisions in Git. So back then we were still using Git and not Mercurial, that were pretty far away. And so the that was actually a ton of work for the server because Git changes a ton of things and move directories and do all sorts of work, right? And then once we hit a n a new revision, another revision, we would dump the state again. And we would go around like that until one once in a while we would go back to a state that we knew and we would dump the state again and compare the two states and make sure that everything was the same. And that lasted for weeks, for months. And there were still bugs. There was one bug, it took us a year to find it, and we didn't even find it using this this DAC. We had to add a bunch of because it was all concurrent with a lot of you know race conditions in there, that and and very subject. There was no Watchman involved. We built our own stuff that was actually working. And there was no need to build uh there was no need to build a Watchman, but uh I put a tool out there that was exactly like Watchman and was actually working. And people said, we don't want to use this because it's written in O'Camel and O'Camel kills babies. And so that's how they ended up writing The Watchman. And uh it seems like I'm bitter. I'm not at all. My my impression back then was look, I have this thing, it's working, you want it, go use it. You don't want it, go write your own thing. But there was no issue with the Watchmen because we were not using that. And uh but the yeah, we run into a bunch of race conditions, a bunch of subtleties, and that's what led me to the desire to go build Skip. Uh, because I thought there is a lot of tooling that we want to write today that would be awesome if it was incremental. I mean imagine you're a C developer. And imagine if you had a fully incremental front end or a fully incremental linker. You know, how would it be? But I think they're still using because they're not using Skip, they're still using very coarse-grained dependencies and it's still pretty slow. Right? While it could be much, much more efficient. So for example, if you have a template and you all the logic that unfolds the templates is written in Skip, all those dependencies attract for free. So of course, if you're not careful, you end up recomputing a lot uh for for not because you change one thing that happened to be used everywhere. So this is not magic. You have to work on that. But at least the parts that keeps track of the dependencies and efficiently only recompute the parts that are necessary, that is handled for you, right? And so that was how you know banging my head against the wall with hack for years led to the creation of skip, which I think uh would was worth mentioning because it was it was very, very painful, very painful and very interesting but painful years.

SPEAKER_00 1:27:59

And that does open the the door for the next for whatever we get to talk about skips on too. Well, thank you for telling us about the history of the hack programming language, Julian. And uh, you know, if you enjoyed this, remember to smash, like, and subscribe on your favorite podcasting platform. Uh, this has been Coffee Computers and Beer. See you next time.