We Rebuilt PHP From the Inside | The HHVM Story

Two veteran Facebook engineers reveal how they saved the company from its own code — by building a compiler that turned the world's most mocked programming language into the backbone of a platform used by billions.

Keith Adams (who built the HHVM JIT) and Julien Verlaguet (who created Hack) go deep on the inside story no one's told: how a "bunch of clowns pushing PHP" faced down Facebook's spiraling server costs and won.

In this episode:
🔥 Why Julian thought Facebook was "not serious people" — and what changed his mind
💣 The hyper shell endpoint: a production root shell anyone could hit with a URL parameter
🖥️ How HPHP (ahead-of-time compiler) worked — and why it failed spectacularly
⚡ The JIT compiler pivot: why "there are many ways to be slow and few ways to be fast"
🧠 The "repurposed JIT phenomenon" — why you can't just drop PHP into the JVM or V8
📉 The dopamine shift: learning to celebrate removing complexity instead of adding optimizations
🏆 How HHVM saved Facebook billions in server costs and reshaped engineering culture forever

⏱️ Timestamps:
0:00 — Julian's first impression: "Facebook was just clowns pushing PHP"
3:20 — Keith's VMware origin story (employee ~80)
7:00 — The PHP monolith: "git blame zuck" and select star from users
13:00 — The hyper shell endpoint — Facebook's most insane production hack
21:25 — What HHVM actually is (and why it's not just a faster PHP)
33:50 — The HPHP era: compiling PHP to C++, and why it broke
42:10 — The mental model shift: from interpreter to JIT compiler
52:40 — "There are many ways to be slow and few ways to be fast"
57:20 — The dopamine hit of removing notes vs. making things faster
1:05:21 — HHVM's legacy and what it taught us about language design

🎙️ Computers, Coffee, & Beer is hosted by Keith Adams and Julien Verlaguet — two engineers who helped shape the modern internet and have the war stories to prove it.

🎧 Full episodes on YouTube + your favorite podcast app.
@ComputersCoffeeandBeer

📧 For Business Inquiries: don.broida@gmail.com


=============================

Channel About:

Computers, Coffee, & Beer is hosted by Keith Adams and Julien Verlaguet — two engineers who helped shape the modern internet and have the war stories to prove it.

Keith built the HHVM JIT compiler at Facebook, helped virtualize the world at VMware (employee ~80), and went on to Slack before founding Pebblebed, an early-stage VC firm for technically ambitious founders.

Julien created the Hack programming language, spent years writing safety-critical compiler software for Airbus and nuclear plants, and now runs Skip Labs.

Together they dig into the systems, languages, and decisions behind the technology that actually runs the world — told by two people who were in the room when it happened.

=============================

#hhvm  #php  #facebook  #compilerdesign  #softwareengineering  #programming  #techhistory  #JustInTimeCompilation #hack  #ComputersCoffeeBeer

Transcript

SPEAKER_00 0:00

Today we're gonna be talking about HHVM and how HHVM came to be at Facebook. So HHVM is a JIT compiler developed by Keith, um Keith Adams, right here, Jason Evans, and Drew Poroski. And this is the HHVM story. I'm Julian Verdeguet, I'm the CEO of uh Skip Labs.

SPEAKER_02 0:20

So I'm Keith Adams, sometimes known as KMA for my login at various companies. I'm founder and managing partner at PebbleBed, an early stage VC firm that works with technically ambitious founders.

SPEAKER_00 0:30

You managed to do pre-IPO in pretty cool companies like three times, right? Like, like, you know, among engineers, you're like, that guy, you know, wherever wherever Keith is going next, you know, that's where we need to go, you know? Because VMware, you were what employee number, number what, 80, something like that?

SPEAKER_02 0:52

Yeah, yeah, like just under 100, I think. Um, about 20 engineers. So, you know, this wasn't like two guys and their dog dreaming about virtual machines or something, right? There was like a real direction set and a product and and customers and things. But honestly, like VMware was just kind of a lucky hit, right? Like that was I was graduating from college in 2000. I went to Brown University, a lot of really bright people from the computer science class as at Brown ended up at Sun Microsystems, especially people who were excited about OS's and CPUs and the things that I was excited about. I had an offer from Sun Microsystems actually, which I was incredibly excited about. But uh there was also this crazy VMware place. And when uh when I went to see a talk by a guy named Ole Agason, who I had the privilege of working with for nine years at VMware, he came to Brown University and gave like a pizza talk about how this thing worked. I thought it was amazing and thought he was amazing. I thought, wow, I gotta, I've gotta, I've gotta expose myself to this person any way I can uh while I still while it's still an easy decision to make. And if this VMware thing blows up, hey, at least Sun Microsystems will always be there. Which of course, like, you know, ended up being the opposite of how this all worked out. You know, Pachay and my friends at Sun who uh went on to do amazing work in the 2000s. It feels like a ton of luck. It was an incredible place, really brilliant people, really uh in hindsight, like really high standards of sort of technical excellence, really high standards of just like what it meant to do a good job and what it meant to ship something quality to customers and how good things really needed to be before they were sort of ready for other people to consume, which I'm glad I've had those standards kind of uh imbued in me relatively early. After the 2008 financial crisis, um VMware had already been acquired by EMC at this point. EMC had uh just like decapitated VMware, just gotten rid of Diane Green, who was the founding CEO of VMware, um, and had sort of absorbed started absorbing VMware more into EMC proper. There was a lot of kind of mis, you know, general bad vibes, I'd say, uh, among the rank and file at VMware. Certainly for the first time, I'd always assumed VMware was going to be my entire professional identity, right? I'd I was 30 years old at this point and had been there since I left college. Um and for the first time I thought, like, well, if Diane's disposable, I guess I'm disposable too, and what does that mean? And there's a recruiter at Facebook named Sarah Currien who like got the VMware uh like got the phone book, basically. But this is back when people had landlines on their desks at work, right? So this is the little Cisco phone that you pick up. And she war dialed the VMware directory for engineers, and like anybody who picked up, uh, she was like, Hey, my name's Sarah, I'm a recruiter from Facebook. Like, can I get coffee with you? And 99 times out of a hundred, I'd have probably said no, but like, you know, of course this was the week to do it, and that was why she was doing it. Um and kind of one thing led to another.

SPEAKER_00 3:29

So that's actually funny because my uh the way I was recruited at Facebook originally I didn't want to join Facebook at all. I wanted to join Google and for me, um Facebook was not serious people. So you know, for me Facebook was just a bunch of clown pushing PHP, don't know what they're doing whatsoever. And uh I was working in a startup back then in in Mountain View, and uh a recruiter from Google um contacted me and uh my roommate was working at Facebook back then. I don't know if you remember pad. Yeah, yeah, and and pad was like, Well, you should definitely try Facebook or refer you and whatnot. So I I I tried a loop at Facebook, but really my eye was set on I mean my my mind was set on on Google. And uh it was the interview process, and you were actually part of it. Like the the crowd that interviewed me, my my screen interview was with Jason Evans. Let's start with that. You know, the the screen, right? And so I I have him on the phone. I mean he's he's he's great. I love him. Um I have him on the phone, and uh he says his name, but uh you know I don't I don't recall, I'm I'm like I I don't pay attention. I'm like, okay, well let's do the interview.

SPEAKER_02 4:49

By the way, Jason Evans of still of J.E. Mallock fame, I would say, right? Exactly. Probably wrote wrote the memory allocator you're using to listen to this, for instance. Sorry, go ahead.

SPEAKER_00 4:59

And um so that's actually what what I said. I was like, well, wait a minute, you're Jess Jason Evans from J.E. Mallock? And he's like, yeah, like I know your stuff. And I started talking about his code. I mean, it's pretty cool. You have cool data structures in there, and that that was amazing. So then I wrote some code, which I'm sure was wrong, and I think at the end he said that I'm pretty sure that's wrong, but I wouldn't be able to tell you why. And so and so that's how I I passed that interview. And then the the the crowd was great. I had um uh Ben Matthews who interviewed me. I had uh I had you. You you was a pretty good one. It was like uh, well, how do you write a garbage collector? What's a garbage collector? What are the trade-offs of a garbage collector? And and we started you know, ranting about garbage collectors. And yeah, I wonder.

SPEAKER_02 5:53

There's something loopy in it because you were writing OCaml and you sort of like wrote it a little bit like it was C. Like you wrote it with like a bunch of loops constructs in it. And I was like, let's be OCaml programmers. Like go ahead. Like, I get it. Like you know I'm a C programmer, but you know, take me do this idiomatically for my benefit, please. So actually, I was dumbing it down for you, Keith.

SPEAKER_00 6:12

I I was hoping that you know the C programmer you are would not understand the OCaml beauty. So no, but uh in all seriousness, it was it was a really cool interview. And I walked out of it thinking, wow, I really want to work for for Facebook. They have really cool people there.

SPEAKER_02 6:29

I think that's in hindsight, by the way, Julian. I probably had a similar set of preconceptions, right? So uh I guess a little bit of of backstory here too, by the way. Like when I was interviewing in 2008, they actually had been in Silicon Valley for very long, a recent transplant from Boston, right? When it was a you know Harvard startup. And the kind of perception of them among locals, which like mostly didn't know a lot of people who worked there, was like, okay, there's this weird thing, it's a big hit on college campuses or something. And the assumption was that these are people that like maybe had some product sense, but that there was no sort of serious technology going on. And so I wasn't really sure what I was getting myself into when I showed up for an interview. Uh Jeff Rothschild was the first person that that gave me the interview there, and I'd like had some contact with people from Veritas uh in the process of of working at VMware because Veritas kind of was deep in the DNA of of System Z Silicon Valley at the time. Um and he actually kind of nagged me a little bit about VMware in the interview. He's kind of like, yeah, this stuff's kind of good for test and dev, I guess, but like nobody's ever gonna run that garbage in production. And I was like, well, actually, sir, I mean, I kind of do use it and you know, uh so you know it kind of put me on my back foot a little bit. I wasn't expecting to have to defend the validity of what I'd been doing for the last nine years. Like I thought I'd be teaching these PHP bozos a thing or two about how computers worked, and it was kind of it was a little bit the other way around. And Bobby Johnson also um other really bright people. Yeah. You know, Facebook at some level was like uh in terms of its impact on the economy, it was a pump that turned venture capital dollars into top line for network appliance, for NetApp, right? It was it was storing photos in these big NetApp filers, many, many, you know, uncountably many petabytes of this stuff, um, because it had a huge hit on its hands for photos. It was like the first time people in a mass consumer way were really sharing photographs with each other online. There were things like Flickr, but they were kind of for prosumers who were you know hobbyist photographers. It wasn't like normal people snapping photos with their cell phones and things until right then. And so uh those were extremely expensive ways to store those photos on the one hand, but they needed kind of fast, reliable storage of all these things.

SPEAKER_00 8:26

Uh when I joined one of my goals, and in fact it was, you know, during the the onboarding, there was one day where they asked you, where do you think you could have impact for Facebook, da-da-da. And there's a piece of paper where you wrote it down, and I wrote down that I was gonna make PHP better. That this was shit. This this was not, you know, this is not gonna I I I wasn't gonna leave it at that. I mean, I spent a bunch of years working on, you know, um critical embedded software. Uh so you know, compilers for things that end up in Airbus, in trains, in planes, in nuclear plants, and I mean kind of things where if the software fails, people are gonna die, kind of thing, right? Hence the name, uh safety critical software. And when we were writing code at this first company, the goal was zero bug. Right? If you had a bug, you had to email all your customers, it was very long and complicated process. So the goal was really to get it down to zero bug, no matter how long it takes. And from that to dub dub dub, like the PHP code base at Facebook, it's like, whoa, what this is so and and all in all fairness, it was actually a pretty good code base. Like if I compare to other companies that I worked for afterwards, especially for being like a paleo PHP code base, right?

SPEAKER_02 9:54

Especially for being a code base that never got rotor tiled from Zuck hacking away in his dorm room in 2004 or whatever. Yeah.

SPEAKER_00 10:01

I mean, the one thing that I would say was detrimental to the quality of the code was that they would never finish transitions. So migrations. So they would start migrations to new things, and then somebody would come along with a new idea, and they they hadn't finished off the migration before.

SPEAKER_02 10:20

And so what ended up one of my sort of party tricks I would do in bootcamp as late as 2015, right? When I when people when new people would join the company, uh, was actually doing a little tour of places in the code where things still get blamed to Zuck. And you could still find like SQL string constants that are like select star from users where ID equals blah. That's like git blamed zuck or whatever. And is that code even live? Probably not. Actually, there's probably like no way to actually get to that code in the code base anymore, at least not with any URLs that are live. Um sorry, I didn't mean to interrupt your story there. No, I I but it's the transitions went all the way down to the the unfinished transitions go all the way down to select stars and users where ID equals.

SPEAKER_00 10:59

Yeah, exactly. I mean, that was also the first the first uh stuff I was working on, right? The first stuff I was working on at Facebook was static analysis to uh find security holes. And in fact, I was very surprised how well the the the the technique was working because normally you know in a dynamic language you really don't get very far. And the the reason is relatively simple, which is that if you don't have the types, you lose the control very quickly. Like the moment you call a method, if the method has a somewhat generic name, like render, if you're rendering HTML, uh you really don't know what you're calling, and therefore you really don't know what to analyze, and basically your analysis and is not going very far. But I was still able to build an analysis that found a lot of security holes, uh, because there was a lot of old style PHP, you know, where everything is a top-level, the kind of stuff that you're describing, like SQL queries, where clearly it wasn't escaped the way it was supposed to be escaped at all. And so so we were we were able to found a lot of uh a lot of interesting uh bugs this way uh back in the code base. I think it's cool for us to say that now. It has been what 15 years? Because you know.

SPEAKER_02 12:20

I assume it's all like outer space, laser beam, sci-fi now. And if it isn't, some reachability analysis has been performed or whatever.

SPEAKER_00 12:28

Because 15 years ago, I would have been wary, you know, saying that there was security holes within Facebook. Uh, because you know that was my job, and I didn't want too many people to know outside of Facebook that this was the case. But uh there was this funny story of a guy who built an endpoint. So I was in the security team, that was my first team before joining HHVM Hack. And there was a guy who built an endpoint, and the endpoint was literally take the parameter and run it in hyper shell. So hyper shell being the shell that runs this command as root on all machines in production.

SPEAKER_02 13:07

Oh, yeah, yeah. The root production command. Yeah, yeah, the the destroy all of the.

SPEAKER_00 13:12

So he had a little web page, a little internal web page where you typed any text you liked, then you press run, and the thing was root in production everywhere. So that I have I have this static analysis that immediately goes like alert, alert, what is this? I can see the veins of Erling, who was working on the team, like his veins were pumping, you know, like what the fuck is this? And he went to see this guy, and and the guy was like, Yeah, but it's really useful when I'm on holiday, you know, I can push stuff in production. And I saw the moment where Erling was gonna struggle him. Like, shut up. And I mean, it's just to give you an idea of how how far things were back then. We went from, you know, cowboy cow cowboy land to something actually pretty serious. Like if you wanna if you wanna hack into Facebook these days, hmm, you bet that's a pretty major project, I think, right?

SPEAKER_02 14:22

That's uh there's some research professionals of hard. Yeah, we're talking about like the mid-2000, you know, the early 2010s, roughly, right? Like late, late aughts to early 2010s. Like the people that started to sort of come into Facebook at that time might have had backgrounds that were more like yours and mine, where lives are on the line or billions of dollars were on the line, or you know, there's sort of an expectation of fitness to purpose of any of the software you're making that wasn't necessarily there for internet products, you know, that were just sort of consumery before. I think like there was a time with Facebook where you could kind of be like, well, we're like, we're not negotiating with aliens for the future of Earth, right? These are just people posting pictures of their cats. Like it's more important that this be fun and easy to iterate on than it is that like, you know, we're not lives aren't on the line broadly. But past a certain point, like there's sort of so much value and sensitivity and private data and everything else involved that like lives essentially are on the line. I was always pretty impressed how seriously people took that responsibility. It's just it's impossible to do it perfectly, if that makes any sense, right? So um, for example, like there was a conflict happening in a region, right? And uh like a civil war, basically. And it's a civil war where like nobody in the Menlo Park office speaks the local languages. Okay, this is before you know AI, right? We can't just translate it and we can't just read the messages. And it comes to our attention that people are using Facebook Messenger basically to organize roving gangs, right? To organize like groups of people that are going out and killing other people. Now, we have no context about the civil war. Are these the good guys? Are these the bad guys? Does the concept of good guys and bad guys have any relevance? We have we have no bloody idea, right? Uh do you want to turn messenger off in that region? Is that the right thing to do? Uh again, AI doesn't work. There's not like some neural net you can snap on the front of it to try to classify messages as being like potentially about violence or something like that. And I mean, there this was a pretty big emergency that ran for a while, and there was no kind of obvious commercial impact. I just was literally lives on the line. People were using our communication product to do stuff of extreme consequence. And it reminded me, like it was a strange problem. It sort of reminded me a little bit of a capitalist version of the social the socialist calculation problem. One of the reasons communism ends up being really hard to do in practice is that economies are chaotic, right? If I'm trying to make a plan for the economy of the Soviet Union and I'm trying to figure out in three years or five years, uh, one of the things I need to figure out is like how much yellow paint are we going to need in the city of Kiev three years from now. And that's like almost impossible. Information doesn't really exist. Like you need things like price signals and you know companies getting created and destroyed to kind of actually meet that demand. And this was like a weird consumer internet version of that, if that makes any sense. It was like there's sort of so much power concentrated in this one platform that there was almost no way to wisely wield that power. And I don't know what to do about this this observation, by the way. I don't think there's any sort of super clear um sort of civilization-wide question like what you should do about this.

SPEAKER_00 17:17

I remember I remember some of those uh some of those problems too. And there was no clear solution. And I also there was a lot of suspicion um in the I mean towards the company back then. I remember especially when you know there was the bug, uh where apparently Facebook was publishing all your private messages on your timelines. Do you remember that? Back then I was working on Facebook, you were two. And I had a bunch of friends who just wouldn't believe me. You know, they they thought it was a c cover up because the real story was before, you know, at the beginning of Facebook, there was no messenger. And there was barely anyone on Facebook. You know, it was your wall, was you and your five friends. And so people were talking to each other. By the way, right? There was no news. There was no news feed either. Yeah.

SPEAKER_02 18:11

Airsatz distribution of content, right? It was like you'd go to pro from profile to profile. I'd go to Facebook to like look at your profile.

SPEAKER_00 18:18

Exactly. And so what people were doing is they were messaging each other on the wall, on their wall directly. And so once Timeline was, you know, was launched, basically you would take all these old messages that were on the wall that have always been public, right? But now they're in a form that looks like private messages are actually on your on your wall or were brought brought into your wall, except that they're they've always been there, right? So all all the media in France went nuts, including media that I really trusted, you know, like like um newspapers I'd been reading forever, and they published this thing that there was this bug at Facebook, da-da-da.

SPEAKER_02 19:05

Same thing up in the States, by the way, the New York Times is all over this. Like name your favorite. I assume Le Monde is one of the ones you're thinking about.

SPEAKER_00 19:10

Le Monde is the is the one I was thinking about. Yeah, yeah. And what really surprised me is that no journalists, no team, no newspaper, nothing actually corrected themselves. Because the fact that they made a mistake, that was one thing, you know, that happens. But there was, you know, in Le Monde, which was a newspaper I read for many, many, many years, all we had in there was page 34. There was a little, a little a little article that said, Oh yeah, Facebook maybe didn't really have a bug, but that doesn't mean anything because there was all the these other bugs that were there before, and those were weird. Exactly.

SPEAKER_02 19:52

Don't worry. You should still think they suck.

SPEAKER_00 19:54

Okay. You're like, okay, so truth doesn't really matter, right? Like it doesn't matter that there was no bug, that this is all blown out of proportion for nothing at all. But it it yeah, I mean, it it changed my perspective on many things. Because I mean, if you know, today I was asked to actually run a social media platform, I I think I would struggle. You know, like like how getting that.

SPEAKER_02 20:26

Yeah, yeah. You'd be sort of hemmed in. Yeah, I know what you mean.

SPEAKER_00 20:29

You're like, you know, either you make it too restrictive and then your network is never gonna grow. Nobody is gonna because they're gonna feel like imprisoned, like they cannot say the stuff that they care about. Or you open the gates, but then you know, you end up in this cesspool hate. I mean not to point at some specific, you know, uh social network that has done that recently, like open the floodgates and see what happens. Uh but you know, it's I mean, uh don't get me wrong, I'm I'm in favor of um uh freedom of uh of speech, and I really believe in freedom of speech. Uh but yeah, running a social social media platform is is rough. But tell me more about uh I want to hear the story about HHVM. And okay, yeah.

SPEAKER_02 21:18

So we know how we say this back and forth to each other, and like normal people don't know or care what the hell this thing is, by the way.

SPEAKER_00 21:24

So we should of course everybody knows what HHVM is, you know. They're Taylor Swift, the kids today, Michael Jordan, yeah, yeah. HHVM, you know, of course.

SPEAKER_02 21:36

Yeah, so uh HHVM, which stood for the hip hop virtual machine, uh a little bit of a long story on the name, uh was uh basically a JIT compiler, a just in time compiler for PHP that me, Jason Evans, who we talked about a second ago, and Drew Pirofsky started working on in uh 2010 and eventually delivered to production in in early 2013, late 2012. And uh it was a reaction to kind of economic pressures, actually, that Facebook faced in certain ways. So, you know, you hear a lot about compute costs in the context of AI right now. In the microcosm of Facebook, we were spending a lot of money on compute per user at the time as well. It was one of the things that actually kind of threatened the economic model of Facebook, in part because the ads model hadn't really taken off yet at the time we started this. And uh the amount of CPU we actually had to spend per user to run the site was really, really enormous. Um and a big chunk of that we felt, and we had some reason to believe was because of the PHP language, which is a it's a developer productivity language, right? It's interpreted language, it's you know really easy to change, it's really easy to get started with, but it's really, really uh uh, you know, really free with its use of CPU resources and memory resources. And so you have to buy a lot of computer if you're if you're gonna try to run a mass market site with it, and we were trying to do something to tamp those resources down. This is the second major effort that Facebook undertook to get this problem under control. And the first one was called hip hop, uh, or it originally was called HPHP, because Hai Ping Zhao, uh, an engineer at Facebook before I started there, uh got it started. And then it turns out that PHP is a trademark that's owned by like a company in Israel, and they're a little bit litigious. Some words were exchanged. And so if you sort of grep for HP, HP in user-dicked words, you'll find like hip-hop and not that many other words. So hip-hop was the name of that thing. And it was an ahead-of-time compiler. So it worked, you know, the way your C compiler works, or your Go compiler works, or your Rust compiler works these days, um, where it like takes this gigantic code base that was, you know, dub dub dub, which we've referred to before, and it spits out a big an elf binary. It spits out a thing that listens on port 80, um, you know, the same way that a static language compiler would. We should also mention, you know, not only is Facebook kind of a PHP application at this time, it's a PHP monolith. It's a it's one big PHP application.

SPEAKER_00 23:57

Um so that's then it's like 80% of the Facebook code is PHP, right? Something like that.

SPEAKER_02 24:03

Exactly. Something like that. At the time, a gigantic amount of the intellectual property and the thing that makes Facebook go is this one repo in in PHP. And it was a sort of understanding across the engineering organization that there was a you know big, hairy, audacious goal would be make www run fast. Um and one approach to do that is like take it all, turn it into C, run it all through GCC. And that was what uh what hip hop did, what HPHP did. And do you want to let me know?

SPEAKER_00 24:33

The original goal was not that, right? The original goal was actually to transpile it to C and then get all the PHP developers to switch to C to develop the the web backend, right? That that was.

SPEAKER_02 24:45

This is uh this is a slightly, you're right, this is a little bit of a side quest here, but but hyping you know was a C programmer, like a lot of people who were coming into Facebook at the time, and um, you know, kind of squinted at a PHP from across the room, said, Hey, there's curly braces, there's semicolons, like what's the problem here? This is basically C anyway. Um if we just kind of turn this lame, slow PHP code base into a beautiful, fast C code base, then we can really start cooking here. And so the idea was really initially that like you'd run it one day, check in all the crazy C that you generated, and then move forward into a glorious C based future. Um this proved to not be the way that things could work. It turned out there were reasons people liked PHP. Some of those reasons included the kind of interpreted workflow, right? That I like save one file and can you know see what happened.

SPEAKER_01 25:34

Yeah.

SPEAKER_02 25:35

Um whereas if you know you've got at this time it's already like tens of millions of lines of code, it's already a huge application. Where if you've got some huge compile cycle in the middle of that, that really kind of messes that flow up. So that would have faced enormous like people who just quit their jobs who worked on the product side of things.

SPEAKER_00 25:49

I mean, imagine the compilation time. Right? Imagine you're working on www in C. Yeah. And in a in you know, in a code base that hasn't been thought out for it. Imagine the compilation time of you know, making getting anything done in W. You would wait hours to compile it.

SPEAKER_02 26:09

PHP has no concept of a header file, right? So there's no there there were there would be a lot of work getting it into the shape where it's sort of a project that you can actually work on in an incremental way uh in a way.

SPEAKER_00 26:18

I don't think it was possible. So I looked into that back in the days because I wanted to, in fact, Pad was the first, so Yoan Padiolo was the first to go on this crusade to try to um split up the code base into you know clean submodules and get the dependency layout figured out properly. Um and and he he he went into this whole visualization tool that was called overlay, something like that, that would show you the code base and the different regions of the code base and what depends on what. The problem is that there was no incentive to go after that, but to give you an idea, there was chunks of code with cyclic dependencies that would pull in more than eight million lines of code.

SPEAKER_02 27:04

Yep, exactly. This is the problem is you end up with like some big cycle somewhere that like where the bottom calls the top and now you're done.

SPEAKER_00 27:11

Because of autoload, so the fact that you know whenever you use a class, it's just automatically loaded by legit, same with functions. There's no reason to pay attention to dependencies, right? If something works, you're just going to use it and it works, right? And if you don't pay at all attention to dependencies, you end up in really interesting situations where that we had the logger that depended on ant and stuff like that, because you know there was some some useful method in there.

SPEAKER_02 27:40

And if I could observe something too, by the way, this is like a you know very broad, very abstract claim that like you should not please don't take this at please don't bet your company on this advice, okay? But broadly, like there are clean engineers and dirty engineers. Like this is something I've observed. And that's nothing to do with that's not a quality judgment. I actually like dirty engineers, there are great dirty engineers. They're dirty engineers that like can do incredible stuff. Julian, I think of you as a very much a clean person. Like anybody that can do something new with a type system, you're probably a clean temperamentally. Um clean people find me dirty. Clean people like find me hard to work with because I'm so because I'm making these layering violations all over the place. I'm the only time really dirty engineers find me kind of persnickety and like I'm somewhere in the middle, basically, right?

SPEAKER_00 28:21

I mean it's funny because I see myself as in the middle too. Like what you're just saying is I mean, in my world, building a type system for PHP is the dirtiest thing you can do. Like I mean I mean, imagine you go to a type system conference and you're the PHP guy. I'm sorry. It was hard.

SPEAKER_02 28:45

So a type is a set of values. Some values are in the type, but other values are not. Right.

SPEAKER_00 28:56

Yeah, no, I mean it's it's uh it was harder than you thought. And and and so what's interesting is I see myself in the middle too. So I agree that temperament has a lot to do with it. Um, but it's not the only thing. The the other thing that can matter is where are we at in the in the life of a project. Right. So if you're starting a project and you don't know if it has value yet, I think you should be as dirty as you can be to get to to that information. Uh while if you are, you know, later down the lifetime of the project, then you should pay more attention. Another distinction too between clean and dirty engineers is another dimension, sorry, is people who like to start things from scratch. And people who like to maintain or optimize what's there. Yep. And you need both. And what's interesting is that both classes of engineers tend to despise each other.

SPEAKER_02 29:55

So you've got, you know, the people who decide as like not solving any of the real hard problems. Exactly.

SPEAKER_00 30:01

So you've got the people who are maintaining the existing system and optimizing for it, like, oh yeah, all these clowns, you know, building this pie in the sky project that's not never gonna go in production, like this is all bullshit while we're doing the real work. And then then the other people, you know, who are building the new system, they're like, oh yeah, all these clowns maintaining this old ass code base going nowhere when the new shit's gonna solve all of that. And so what's interesting is that you need both, and sometimes they are both right and sometimes they're both wrong. I've seen people cling on to old shit that was gonna retire. Like I can remember some people still optimizing HPHP a couple of days before HHVM was entering production, and at some point you're like, dude, let it go. Um, and and at the same time, I've also seen the other the the the other way around, like people going off building new shit, like rewriting everything in Rust. That is, let's rewrite this in Rust. It's like, why? It will be faster. Do we have performance problems? No, but it will be faster and greater and better.

SPEAKER_02 31:15

But and memory safe, yeah.

SPEAKER_00 31:17

And and what problem are you solving?

SPEAKER_02 31:20

Rust, by the way, is a fascinating of talking about this like dirty clean thing. Like, Rust is like the clean's attempt to remake the world, right?

SPEAKER_00 31:26

It's their attempt to sort of be like we remake the C well, not the entire one.

SPEAKER_02 31:30

Sure. Sure, sure, sure. But like what if we kind of wipe the earth of like these nasty things in C. And I find it fascinating. This is not a novel observation, but I do find it fascinating that like we've got a whole bunch of Rust platforms at this point, and not so many Rust products that anybody finds compelling, right? Like there are far more Rust game engines than there are Rust games you want to play. Yeah. And and there's at least one person who's who suggested that um that that might be because like layering violations are what where fun and surprise kind of come from, right? That like that you need to be able to try weird shit in a game. Like both as a player and as a developer, you need to be able to kind of be like, well, what if this happens or whatever? I mean, I want to talk shit about Rust for a second.

SPEAKER_00 32:12

So, first of all, before I talk shit about Rust, um I was a huge believer in linear types, unique types, call them whatever you want. Um that you know, unique types were going to solve the problem of garbage collection, and that we were we would be able to, you know, do system programming in uh memory-safe languages. In fact, I believed in it so much that I spent three years of my life developing a compiler which was called Linear ML, which was a version of ML with linear types, so very much like Rust's before its time with an ML syntax and without um mutation. I used to believe, I used to believe that you know this stuff was gonna solve everything. And now I don't know. So part of me really wants Rust to succeed. I mean, succeed, it has succeeded in some some way, but like kind of overtake the world of system programming. And the main reason is I dislike C so much that I really would like to see it go away. Um, but I mean I I have a hard time believing that you know Rust is is entirely going to conquer the the realm of system programming. And and here's why. So I'd say if you really care about performance, right? If you really want to squeeze out everything you can out of the machine, I feel like a language that is an ocean of undefined behavior is always gonna have an edge because the compiler has just so many degrees of freedom that it will be able to do more things than you know a safe language. So some people tell me, yeah, but they're unsafe in Rust. That's true, but but given that it's not the default, um there isn't there's I mean, unless all your code is unsafe everywhere, like to get the same level of freedom for the compiler, I feel like it's it's going to be a tough one to to to to uh to succeed in. And then the one thing that I'm happy with though, with Rust, is that they are um taking over what I call the C Bros. So I don't know if I ever told you this concept, but the C Bros. So there are people who do Cross.

SPEAKER_02 34:39

Am I one of them before we get into this?

SPEAKER_00 34:40

You're not one of them. Let me explain to you what I call the C pro. So you've got people who do C for good reasons, like they want to actually, they they need performance, they need to do low-level stuff, like building a JIT, fine, completely legit. Um uh, so I'm not talking about that. I'm talking about people where you go, you you go, you talk to them, you're like, hey Bob, why why are you writing this in C? This is a very, you know, it's it's a very simple service that you could run in Java and go. And and the answer is like, because it's fast, bro. And so and so you go in, you look at the code, and you're like, that's not fast. You're copying everything everywhere. Oh, wait, there's undefined behavior here, there, there, there, and you're like, well, and and so for these people who are using C because you know GC is slow and because they're fast, and you know, because they're they're the shit, if they could all switch to Rust, great, right? Right, right. But I think that the serious hardcore um uh system programmers, I think they will always need to do enough dirty tricks that I don't know that you know Rust can take over. Now, don't get me wrong, I hope they will. I hope Rust destroys C because I hate C so much.

SPEAKER_02 36:11

And we host like uh we host a weekly Rust learners meetup uh Wednesday evenings at Pebble Bud, by the way. So we have like, you know, people who are and the the target audience is basically like, I don't know any Rust. Um help me to get like the Rust tool chain on my laptop and just teach me what's going on. And watching that population cycle through, the thing I'm sort of grateful to the Rust community for is just like getting kids excited about systems programming and getting people excited about the potential of programming languages in a way. Because uh, for a lot of people who don't think of themselves as systems programmers or don't think of themselves as PL people, they are having this experience where they're like, oh, this notation is making a difference, right? This this this different way of writing a program is making it easier to be safe in ways I care about. And it is like making my program faster and sort of putting things in reach in terms of machine orientation that like weren't in reach before. And so I think those are kind of positive things broadly. I think that demystifies some of the low-level machine stuff. If I had to rebuild civilization from scratch, like I probably would still start with a C compiler, honestly. Um, because like that's where that's where the productives are gonna come from.

SPEAKER_00 37:16

Like tide defs and you know, syntax for function function signatures. A couple path dependent things aside. I get what you're saying, yeah.

SPEAKER_02 37:25

Yeah, a couple path-dependent things aside, right? There's there's unforced errors in C and C inherited them all, unfortunately. And with when you look at sort of some of these projects like Zig and Rust that sort of feel like these huge breaths of fresh air, there just used to be this um assumption that you you know, one of the things that constrained Street Strip's views of what C had to be was that it had to literally compile C source code.

SPEAKER_00 37:46

But it doesn't. Most of the time. I mean it does in the basis.

SPEAKER_02 37:50

But these days it doesn't. Yeah, these days it doesn't. At the time it had to for a while. Yeah. At least it had to be able to include the headers with the you know, with the right if-def garbage. Anyway. Um we're we're off in a in a weird programming language rut what when we meant to be talking about H H VM for a second.

SPEAKER_00 38:06

So let's go back to HHVM. Which is C. So tell us for a second why you chose C because I know Jason is a C guy. You are a C guy as well. I was a C guy. So was it true? Uh why do you choose C?

SPEAKER_02 38:19

A little bit multifactorial. First of all, Facebook's a C shop at this time, right? So the FB code code base, which is a sort of back end static language, you know, has a build step code base to it, uh, was primarily C code base. Uh at this point, I'd been working in C for about a year and a half. So I my first job at Facebook was working on the search backend, and because it was a C shop, I've been doing this.

SPEAKER_00 38:40

That's where you met Jordan, right?

SPEAKER_02 38:42

Yeah, that was where Jordan and I first crossed paths. Jordan DeLong, by the way, is one of our ex-colleagues on HHPM. The the other kind of complication here was a big chunk of the HPHP effort was moving all of the PHP extensions into the code base of hip hop, right? So PHP is not just a core language, it's also sort of a thing that lets you output PostScript and a thing that lets you parse PDFs and a thing that lets you parse XML and a thing that lets you do read JSON and blah, blah, blah, blah, all these you know, regular expressions, all the stuff that you're sort of used to pip installing in Python land or whatever.

SPEAKER_00 39:16

Isn't it why there has never been a JIT for Python? It's the same reason, right? Like there's there's so many extensions that make assumptions about you know what's the internal.

SPEAKER_02 39:26

I mean, there have been several JITs for Python, as a matter of fact.

SPEAKER_00 39:28

No, but JITs for Python that actually took off. Yeah, they take over the world. Yeah. Yeah, take over the world.

SPEAKER_02 39:33

That's a big chunk of it, is that like the whatever the runtime you have for that Python JIT has to look basically exactly like the like CPython's runtime or else. Um we should try. I don't know if you ever talked to Sam Gross about the stuff he's doing about trying to get rid of the lock.

SPEAKER_00 39:47

Yeah, he's trying to get rid well, he did get rid of it, the global locks, right?

SPEAKER_02 39:51

Yeah, yeah. It's a it's a going concern. You can if def it out now. The core of what became HHVM, right, which is the JIT compiler that that we were ultimately building, in some ways is kind of like a C program, right? It's organized like around structs and is pretty procedural and is you know passing around references to fairly flat data types and things like that. It's not you know deeply, deeply C in the in the ways that were fashionable at the time. Um it got sort of more metaprogrammed, especially post-C11. Um, but honestly, we're doing a lot of the metaprogramming stuff with C preprocessor in the early days of this, just because that was what I was more comfortable with. But anyway, we're getting into implementation details here. Why even do this, right? Like why why agit? Like why would we do better than a ahead of time compiler while that's I know, I know.

SPEAKER_00 40:37

It's because the interpreter was getting too slow. So I was taking well, first two things. First, the interpreter was getting too slow. So the developers, whenever they're trying to make a change on their dev because so so that people know, uh on the HPHP setup, there was the uh the compiler that was taking the PHP source code and producing the C ⁇ , transpiling to C and then compiling that to native. This is not what people were running on their sandboxes, because that process was super long and um super super expensive. So there was also an interpreter. And the problem with the interpreter is that it was really getting too slow. Uh and so both so there was both the interpreter that was getting too slow, and the compiler also that was running into some kind of hard limits. Was it the linker? Was it the binary size? What was it?

SPEAKER_02 41:36

There well, so the hard limits were were gonna be fixable. So among the hard limits it was running into was that the uh it was outputting a binary that was greater than two gigabytes. And we know we were among the first people to try to do that with the GNU linker chain toolchain because we were getting into sort of signed 32-bit integer errors dealing with uh you know text segments basically in the toolchain. Um these were just basically mysterious like crashes or even worse, mysterious corruptions of the output binary when you get this big static binary out of this link process. We worked through them one at a time. That wasn't the thing that sort of was the death knell for HPHP, but it was an example where it's like, okay, we're really starting to abuse this toolchain from the perspective of what was engineered for. Um and then on HPHP i, this thing was just sort of supposed to be there for these goofball PHP programmers who, for some reason, can't be bothered to compile things. We don't understand why. They're they must not be good at computers. But they get this crummy interpreter, and the crummy interpreter was an AST traversing interpreter, right? Basically had a big syntax tree in memory of the entire code base. Um it would incrementally update that syntax tree when you change stuff on disk. And you know, the if you can imagine uh the difference between sort of a bytecode interpreted, you know, the it's usually kind of an order of magnitude implementation improvement going from I traverse an AST to I you know run around switching on bytecodes, and then typically maybe another order of magnitude to get from that bytecode interpreter to native code.

SPEAKER_00 43:03

And the first PHP interpreter was reading the file character by character, right? Oh, yeah, fun fact. That's another order of magnitude, and it was implementing loop with L seq. Oh, I didn't know that. Yes. Wow. It was actually So is this a file pointer? Like it just was a file pointer? Yeah, it was a file pointer that was moving back. And I mean, as far as I know, I I I I hope I'm not making stuff up here, but that's somebody who told me this story.

SPEAKER_02 43:32

We should uh we should try and get Rasmus Leredorf on here sometime and get all the way up. Rasmus Leredorf, the father of PHP, by the way, which originally was like a very modest little project. Uh PHP stood for personal homepage. The idea was like it was mostly HTML, but you wanted to like have a little integer in it that you read out of a database or something to be like you're not. It wasn't when it was like you're the 1003rd visitor to this page. People just wanted to be able to do that, or they just wanted to be able To like do one little database populated field in the middle of a mostly static page or whatever. And that was the problem PHP was meant to solve, not some big complicated dynamic web 2.0 thing like that.

SPEAKER_00 44:11

Although for arrays and references, I really would like to know what happened there. Because arrays and especially the way they interact with references and how you know it's looking at the reference count to determine if it's going to copy or not.

SPEAKER_02 44:26

That's that's for me is just there's some odd stuff there that like especially because the syntax is pearl-like and because like the culture of doing web stuff was very Perl-derived at the time. I suspect, and I don't know if anybody's coped to this exactly. I suspect there's a misunderstanding of Perl semantics.

SPEAKER_00 44:47

I have a theory regarding references. Please. I think the theory goes like this like they introduced value semantics, which by the way is great. I like value semantics. That's not the problem.

SPEAKER_02 44:58

Well copying things, yeah, one another.

SPEAKER_00 45:00

So so the idea was you pass around an array, if somebody modifies this array, you're actually modifying a copy so that you don't affect the original array, which in it's is very functional in nature, so I I have no problem with that. Except so good. Except that, well, when you make your copy, what happens when you hit a cycle? Right? And the moment you hit a cycle, the easiest way to handle this cycle is just to look at the reference count and say, you know what, this could be a cycle, I'm just not gonna copy this. And and that's my theory, that you know, managing cycles would have been too hard and too expensive. And so the easiest way to get a copy and write that you know worked at reasonably well quickly was to just look at the reference count.

SPEAKER_02 45:50

Okay, yeah, we're definitely like in in PHP nerd corner here, just super briefly. It is a the way that PHP implements automatic memory management is with reference counting. Like when the count on an object goes to zero, it reclaims it. That's not such a terrible strategy. Lots of perfectly respectable languages do this.

SPEAKER_00 46:06

It's part of the semantics in PHP, so that's a bit different.

SPEAKER_02 46:09

The thing, the the thing we're referring to here is that it's programmer visible, that that's how it implements it uh in the way that Julian's referring to. And that's uh that is unusual. And it does, and when when you get down to trying to do a high performance implementation like we were doing in HHVM, it is a pretty heavy constraint to work around. Um and is one of the reasons that it'll probably never be competitive with you know Java or something like that, no matter how clever. Like arrays.

SPEAKER_00 46:34

I mean, for me, arrays was the the the harder stuff for the type checker, for all the problems were with arrays and references. A lot of it is actually is actually okay. Like, you know, the it's object model there are a few weirdness in it, like protected does not apply to your children, but it applies to the whole tree because you know there's no object in PHP. And so normally protected means it's something that anybody uh underneath you can, you know, redefine. That's not the case in PHP, right? In PHP protected means underneath and on the side, because in fact you don't have a tree, you have a forest because there is no object type. And so you are so this is weird, but and and also checking for visibility at runtime is weird. You know, that's that's so they had some weird things, but frankly nothing too unreasonable. But arrays and references, which is the bread and butter of you know the language, that was like just very, very difficult to to type check and get that that get that in a good place, unfortunately.

SPEAKER_02 47:50

One reason for why we thought a JIT was necessary was development environment. I think another uh you know, another implicit desire here was unified the two implementations, right? There was sort of a uh class of bugs which were the interpreter behaves subtly differently from the compiler. Um some of this was, for example, degrees of freedom that C has that you didn't have in the interpreter. So uh if we're relying on the C toolchain to do things like uh evaluating arguments, the order that you that you evaluate arguments to a C function call, that's actually up to the compiler how it does it, which people don't expect, by the way. Like most C programmers implicitly.

SPEAKER_00 48:25

You guys were running in sandboxes for a while, right? Um before going to production. Yeah.

SPEAKER_02 48:30

That's true. Yeah, we got to be in the development environments for a while before we hit prod, probably almost a year, actually, now that I think about it, before we hit prod. Um and so we wanted to make it so that you'd run into the same bugs running in production that you're running in your sandbox. That also influenced a little bit of sort of the at least the early days philosophy of HHVM, which was like instead of having, you know, one of the ways to sort of get a promotion at Google is to like make a new gear for V8. Right? Like so V8's got you know 27 modes now, right? Because everybody who makes a new mode for V8 gets promoted to senior staff engineer. But it means that the landscape of sort of V8's behavior is actually kind of jagged, right? And if you're an expert JavaScript programmer and you care a lot about how your code performs, it can be difficult to reason about how it's going to handle it. One of the things that C programmers, C pre C programmers, and and good enough C programmers like about their tool chain is they feel like they're able to reason about the compiler's behavior.

SPEAKER_00 49:24

I mean, C developers come on. There's only a handful of them who are capable of that. Like even at Facebook, even you know, in in those kind of shops with top notch notch engineers, the people who actually know what C does, a few.

SPEAKER_02 49:40

You don't think they wouldn't be able to kind of like hand expand all the templates and everything and kind of be like here's what this loop that reversed a string really would look like. You don't think they could do that?

SPEAKER_00 49:50

I don't know. Maybe that's an interesting question.

SPEAKER_02 49:54

I think it's possible. And maybe maybe this is like a sign of the times. Like I came up in an era in the 90s, like they still made you take courses where you built little toy compilers and where you wrote a machine code and stuff like that.

SPEAKER_00 50:05

But no, they do that now, but they just vibe code the compiler is completely possible.

SPEAKER_02 50:09

Well, yeah, duche.

SPEAKER_00 50:11

It just works.

SPEAKER_02 50:12

Yeah, yeah. The other thing that this is the reason to sort of HHVM is is what you know JIT people would call a baseline JIT. It didn't have it basically was like just trying to do an okay job turning your PHP into machine code. It wasn't 15 different layers of optimizers on top of that that were going to you know optimize, then de-optimize, then re-optimize, then inline again, then re-optimize a third time, and blah, blah, blah.

SPEAKER_00 50:33

And you had to fight a lot of people who wanted to build things like that. Right. I remember back then there was competing projects. There was a team that was attempting to compile to the JVM, if I remember well. And you also had people within the company who wanted to for you to target the JavaScript V8, because V8 was all the rage back then. Now there's you know a little bit more dynamic.

SPEAKER_02 50:58

The actual hackathon project I worked on that was sort of the the dawn of HHVM with Jason Evans and Drew Poroski was actually V8 targeted. Like we were we were jitting to JavaScript. We were sort of translating PHP to JavaScript and then running it through v8 and kind of got the the programming language shootout benchmarks or some subset of them working a little bit and said, ah, look how blazing fast this is. But yeah, v8 was an inspiration at the time. I think I also was coming from from VMware, which was a machine-level JIT, right? The the software-only version of VMware took unsafe x86, spit out safe x86, and that was how we were able to run Windows on Linux and Linux on Linux and FreeBSD on Linux and so on. We didn't try to make the code faster, it just tried to sort of preserve reasonably linear slowdowns. And where possible, zero slowdowns, when there had to be a slowdown, make it a constant factor. Um and as long as we did that, you know, we knew we'd be at least able to sort of boot and run Windows in an acceptable fashion, and that was what our customers needed.

SPEAKER_00 51:54

Dude, I remember once, sorry to interrupt, but we were at Epic Cafe, we were having lunch, and um you were ranting about um, you know, people who wanted to run some kind of optimization on HHVM, and your whole approach was we don't need to, we just have to translate the bytecode into you know native code, and it's gonna be good enough. And and I'm with my bagel in my mouth, and you're like, Julian, there's no amount of money you can give to a compiler person not to implement those optimizations. There's just no no amount of money I can give them to say no GCE, no people, it's not happening. You're not doing this.

SPEAKER_02 52:41

And so it's an unnatural act for them, right? Like it's like you know, they're they've got this enormous bag of like awesome tricks to make to make code fast. Like, why would you just not make the code fast when you can make the code fast? The problem with running PHP wasn't, you know, that we failed to strength reduce multiplying by five into shift two and add a copy, right? The problem with like your PHP not being fast was that like it didn't even know it was a freaking integer. You know what I mean? The problem with not being able to run PHP fast was like you're sitting there being like, is it a string? Is it an array? Around this time we were lucky to have a little bit of literature support here too. There was uh the the term the repurpose JIT phenomenon kind of entered the programming language lexicon with a publication around then, which is basically when you are building a just-in-time compiler for a language, you're implicitly making lots of decisions about what to make fast and what to make slow. And those decisions are going to be basically tuned to the characteristics of the language that you're supporting. So, for example, we talked before about the fact that PHP uses reference counting and that it exposes that to the programmer. If you want to piggyback on something like the Java virtual machine, well, the Java virtual machine's got a tracing collector, and it's you know, incredible thousands and thousands of man hours have been poured into making that tracing collector amazing. But no matter how amazing it is, you're gonna be basically paying to garbage collect twice by running reference counting on top of this tracing collector. So it's really, really hard to kind of make up a gigantic headwind like that out of the gate, no matter how smart your compiler is. So trying to recycle another JIT, we thought was going to be a loser. We thought PHP deserved its own first class thing, the same way JavaScript had V8, uh, the same way VMware had to make its own thing for the x86, because the x86 doesn't look like anything else. I thought PHP didn't look like anything else either. And if you've ever worked on one of these systems, by the way, like this was true in the early days of VMware as well. There's this terrifying thing where for sort of, you know, 18 months to two years or something, just nothing works. And your professional life is full of people saying, KMA is an idiot. This thing he's saying is foolhardy, it's a big, uh, you know, it's a big hole in the ground that we're pouring money into, and it won't work. And they have a lot of evidence on their side because they basically have six quarters in a row of me saying lots of stuff that sounds good, and what works? Nothing. All right, the site doesn't work, people aren't able to use it, and so on. So we were really happy when we got to the point where the site runs, uh, because at least we can get a little bit of win here, where at least it's faster than the old school interpreter, and at least we're getting a little bit of benefit from it. Um, however, one thing that was concerning is that the the day that we first were able to run the site, remember, the whole point of this project is that we want it to run fast. And when you compared the JITS version of the site to the ahead of time compiled version of the site, you know, the approach that we thought was inferior and that we thought we'd be able to blow the doors off of, we were something like 10x behind. It was more than that, I think, but like 12, 14x, but like a discouraging distance, like an ocean to swim across in terms of performance. And you know, you find a couple of low-hanging fruits, right? You profile a little bit, you find a few things right out of the gate, and you get that down pretty quick. But it's like we got it down 2x. So now it's 7x behind, which is still really, really far. And the profile looks really flat, right? There's not one place where it's all going. There was like a week or two where I was basically just like, well, shit. You know, I may have made a really major mistake with my career and the career of the people who bet on me here. Um, but what we eventually ended up doing was uh was just kind of taking a, in hindsight, what I'd now think of as sort of a machine learning approach, which is like, no, the fundamentals are still on our side, right? The truth is out there somewhere. There is some way better JIT to build than this one. We just don't know quite how to get there, and it's not going to be one step to get there from here. So uh we got the team together, got some pizza, got some beer, and spent a whole bunch of hours just kind of covering a wall with post-it notes of optimization ideas. And at this point, the thing was kind of mature enough that most optimization ideas actually didn't pan out. And it wasn't because they were bad ideas, it was just because like computers are frustrating and hard because performance is elusive. There are many ways to be slow and few ways to be fast. Um and what the the mental transformation I've learned you have to make in these environments where you're past the easy part of doing performance work is you have to view the work as just pulling the post-it notes off the wall. Right? The work is just I took this thing and I tried it. And maybe it worked. Awesome. We're closer to the goal, but maybe it didn't work. Who cares? There's few there's fewer things on the wall and we know more now, even if all we know is is the things that won't work. Um so the mental shift of like I now get the dopamine hit when uh I take a note off the wall versus I get the dopamine hit when the thing goes faster is a really important transformation to go through in one of these things, right? If you start to view it as like, this is just a search process, it's out there somewhere. We will eventually find the right veins to mine, even though I don't know what they are right now. And, you know, it can be a great day at the office if we didn't make anything faster, as long as we pulled 10 notes off the wall today. That's still a lot that we know today that we didn't know yesterday, even if all we know is about things that didn't work instead of things that do work. That's still gonna mean we put better posted notes up tomorrow. That's still gonna mean that the ones that we haven't taken down yet are more likely to be where the juice is. And then once, you know, once you started getting rolling on this process, it kind of took on a life of its own, right? Like it kind of you get into this rhythm where you just show up every day and that's what you do, and it starts to be kind of habituating, right? Once we got through from 7X, uh, you know, Jordan DeLong actually found something really early on that was a gigantic win, like a weird 14% win. Um, it changed the way that we thought about this because there was nothing in the profile that was 14%, right? The the part of the system that Jordan optimized wasn't taking 14%. So how did it make it 14% faster? Well, it turns out caches explain a lot of weird mysteries that Andal's law will hide from you if you're thinking in terms of, oh, we have to spend all the time on the thing that's the top of the profile. Caches are this, you know, mysterious medium that can cause this part of your program to slow down that part of your program. And it was a really important clue. And we started sort of digging, you know, pulling on those threads more actively once we once we had that clue. Um and you know, once we got through parity, once we got to, you know, 100% of the performance of the of the production system, we still had a ton of momentum. And then the week after that, it was sort of we were 10% faster. The week after that, we were 20% faster. And pretty soon, uh, you know, Zuck was in our pod ringing the gong to turn it on in production.

SPEAKER_00 59:12

I remember that. So I don't know if you remember this story, but this was maybe uh a few weeks before you guys were going into production. And back then, I was in your team meetings because hack had been merged with HHVM. So Hack was before a team of its own, and then uh Pobar was actually the one who pulled it in. So you know, Pobar became the manager of Hack, and that's what Joel Pobar, by the way. Joel Pobar, who's now I mean, uh he's back at Facebook now, right? Like he's been brought back to run AI stuff, and I mean, great guy. He wasn't like, exactly. I don't I don't know if it was that briefly, but I mean I think he's back at Meta now. I remember you guys are churning to get to get to poverty, right? And from the last meeting, I know that there's just 8% missing to get to poverty. And I show up at your pods, and there's you, there's Jason, there's there's a whole crew. I'm like, hey Keith, so you know, I was working on a task on dub dub dub recently, I think, you know, some minor stuff, and you know, I I I I noticed something strange that you know I feel like my code could be a little bit faster, like eight percent faster. Do you think you could do that for me? The whole team was like looking at me like like shut the fuck up, frame of smoke, yeah, help on us, right? With smoke coming out of the yes, really, you're doing this, and and again, I mean, uh HP HP was still optimized back then, right?

SPEAKER_02 1:00:49

There was still growing in there drag racing uh moving target at that time.

SPEAKER_00 1:00:54

Exactly. So it was 8%, but it could have been could have been 12 the next day if there was because I I mean you kept it fair the whole time. You never there was never any push to prevent people from optimizing HPHP. And I think that's great because this was fair game the whole time. You never said, all right, you people who are still optimizing the old system, stop doing what you're doing. It's not your system.

SPEAKER_02 1:01:25

Right. So it got a lot, lot harder to compete with.

SPEAKER_00 1:01:28

Yeah, yeah, exactly. And so so then came the gong, uh, you know, when you know you were you uh Facebook um H H VM got got into production. And then what happens after that? Then you then you you're kind of knighted by by Zuck, and then and then people just bow as they see you, you know, going through going through the the corridors. What's what's going on after that?

SPEAKER_02 1:01:55

I mean you you heave a big sigh of relief, right? Uh me and Joel you know went shopping for a cake. We we uh had a little party for the team, you know, because everybody uh did pull pretty long hours to get this over the line. And uh then after that was kind of like back to work, right? So there was still, you know, there's still a bunch of post-it notes on the wall after that, actually, right? So you said, well, okay, we still got all this roadmap. And he kept kind of just burning that roadmap because you know, every 10% of CPU you save is billions of dollars of of you know essentially profit margin for the entity Facebook at that time. To be clear, there's like a bunch of tooling that we relied on that other people built that was amazing, right? I'm thinking right now Perf Lab, um, for instance, which like had a bunch of really interesting tricks and a lot of pretty sophisticated statistics to make the there's a lot of cool stuff that came out of Facebook.

SPEAKER_00 1:02:41

I mean, React came out of Facebook at this time, GraphQL came out of Facebook at this time. It feels like those early 2010s for me were like the best you know engineering time of my life. Like it seemed like a bunch of really smart people who just wanted to get shit done and who were moving super fast and building building a ton of interesting stuff. And wherever I go now, I'm trying to rebuild that vibe, you know, re-re-get this vibe of Facebook 2010, Facebook 2011, 2012, around there. That's really the time that for me was the peak of you know, interesting people all over the place. You wanted to know anything about the kernel, you had a bunch of kernel experts, the best in the world. You wanted to know something about programming languages, you had great we we hired Simon Marlowe, we hired, you know, and and there was interesting. The Haskell community left and right, yeah, the Haskell compiler guy. Um and yeah, it was it was a great time.

SPEAKER_02 1:03:49

There was something sort of in the air in hindsight where there was a lot of fearlessness, a lot of willingness to try new things, a lot of conviction that um you could kind of blow up the way that we've been doing things and and do it in a massively better way, and a lot of positive examples of people actually achieving that. Yeah, exactly.

SPEAKER_00 1:04:05

I think it's a rare thing when you work in a company where people are convinced that they can build uh the next big thing. You know? Yeah. Uh in most companies I work for uh before Facebook, um, they always believe that, you know, whatever the other guys have been building is better. Right. And and Facebook was not like that. They thought, no, we we can we can build something better, and if we just put enough time and enough energy into it, we can do it. All right, everyone, thank you for listening. It was it was really great to spend this time with you guys. Thanks, Keith.

SPEAKER_02 1:04:42

Likewise, William. Thank you.