Code with Jason
Code with Jason
244 - Jeff Dwyer, Founder & CEO at Prefab
This episode explores how Prefab enhances deployment workflows by integrating feature flags with Java microservices and Ruby on Rails, drawing on Jeff's experiences at HubSpot and EasyCater. We discuss strategies for minimizing deployment risks, improving PR reviews, and mentoring junior developers through clear objectives and constructive feedback. Real-world examples and practical advice offer insights into building efficient development systems and fostering growth in engineering teams.
Links:
- Prefab
- Jeff Dwyer on LinkedIn
Life hasn't been the same since the pandemic. Instead of working at an office around people all day, like we used to, most of us now work remotely from home, in isolation and solitude. We sit and we stare at the cold blue light of our computer screens toiling away at our meaningless work, hour after hour, day after day, month after month, year after year. Sometimes you wonder how you made it this far without blowing your fucking brains out. If only there were something happening out there, something in the real world that you could be a part of, something fun, something exciting, something borderline, illegal, something that gives you a sense of belonging and companionship, something that helps you get back that zest for life that you have forgotten how to feel because you haven't felt it in so long. Well, ladies and gentlemen, hold on to your fucking asses. What I'm about to share with you is going to change your life. So listen up.
Speaker 1:I, jason Sweat, host of the Code with Jason podcast. I'm putting on a very special event. What makes this event special? Perhaps the most special thing about this event is its small size. It's a tiny conference, strictly limited to 100 attendees, including speakers. This means you'll have a chance to meet pretty much all the other attendees at the conference, including the speakers. The other special thing about this conference is that it's held in Las Vegas. This year it's going to be at the MGM Grand, and you'll be right in the middle of everything on the Las Vegas strip.
Speaker 1:You got bars, restaurants, guys dressed up like Michael Jackson. What other conference do you think you can go to, dear listener? Or you can waltz into a fancy restaurant wearing shorts and a t-shirt, order a quadruple cheeseburger and a strawberry daiquiri at 7 30 am and light up a cigarette right at your table. Well, good luck, because there isn't one Now. As if all that isn't enough, the last thing I want to share with you is the speakers. And remember, dear listener, at this conference, you won't just see the speakers up on the stage, you'll be in the same room with them, breathing the same air. Here's who's coming.
Speaker 3:Irina Nazarova oh yeah, here's who's coming For Freedom Dumlao Prathmasiva, fido Von Zastrow, ellen Rydal Hoover and me.
Speaker 1:There you have it, dear listener, to get tickets to Sin City Ruby 2025, which takes place April 10th and 11th at the MGM Grand in Las Vegas. Go to sincityrubycom. Now on to the episode. Hey, today I'm here with Jeff Dwyer Jeff welcome.
Speaker 2:Thank you for having me on, Jason. It's a pleasure to be here.
Speaker 1:Yeah, thanks for being here. You were commenting just a second ago on the piano that's sitting behind me here and sounds like we're both piano dabblers, something like that, or maybe you're more than a dabbler, I don't know don't know, uh, a dabbler with dreams.
Speaker 2:I think in another life I would be a, uh, piano, jazz player in a really bad like bar outside the airport or something and loving, loving it. But uh, that hasn't been my life path so far yeah, same here.
Speaker 1:I would love to spend more time on piano and I go through these like phases where I'll like get into it and practice every day for like five days and then uh life gets busy and then one day I realized like, oh, I haven't touched my piano in two months.
Speaker 2:I learned to do a little improv blues and that has been the greatest, because now I can sit down and have a little bit of fun, no matter what, and it's easy and it's feels like I I can make it up but I don't have to remember anything.
Speaker 1:I think remember actually remembering songs was not going great as I get older um, my, my go-to is every time I get in front of a piano, I play the entertainer, because I know it's, it's kind of challenging and, um, I you know I take such big breaks that I actually get worse, and so I never get to the point where I've mastered it.
Speaker 1:So that's always something I can um anyway, turning our attention to more computery topics. Um you, you and I met because I tweeted that I was going to Boston. And then you reached out because I was like hey, does anybody want to meet while I'm there? And I'm very glad that you did, and we ended up having a nice lunch and a conversation about prefab, just because I was curious what it was and stuff like that. And it's actually a pretty interesting idea. Can you tell us about that a little bit?
Speaker 2:Yeah, so at Prefab we make tools that help you ship faster, more resilient systems. The core of what we have right now is dynamic configuration and so this sort of by being able to dynamically configure your systems, you can get faster cycle time and faster diagnosis. There's basically a really core and simple tool, but I think there has a ton of applications. The story of how I got to doing a dynamic configuration startup was I was at a Ruby on Rails shop and that was great. I then moved to a Java shop at HubSpot which was going through the roof and going to the moon, and that was great and I learned a ton of really interesting things about how to operate a bigger system.
Speaker 2:But then when I came back to Ruby on Rails and sort of wanted the best of both worlds, it was like how do I bring all of the things that this massive, very large Java microservice architecture have? How do I harvest the bits out of that that I thought were amazing and really missing from Ruby on Rails and then bring them into a Ruby on Rails architecture, which I think can actually be even better with a lot of these systems? So I found that that was sort of missing in the space and then basically did that in situ for a while at a place called EasyCater, which was a Ruby on Rails shop, and that was great. But then I don't want to have to build those core systems every place I go, like I was, like I always, I always I'm going to want these, I'm convinced. So I'm doing it as a startup now.
Speaker 1:And it what I'm going to want. These, uh, I'm convinced, so I'm doing it as a startup now. Um, and it took me a little bit to kind of grasp, um the prefab product. Um, not because you didn't do a good job of explaining it, but maybe just because I can be slow sometimes. But I want to kind of um walk through again the the path that I took to gain an understanding, because maybe that'll be helpful for other people too.
Speaker 1:Um, so, as I recall we were talking, we talked about feature flags and like we made a step from feature flags to configuration and first of all, I found that like a lot of these things that are just like standard practice practice for some people are not universally practiced, and so I don't want to assume that everybody even knows what a feature flag is, and not for like releasing features and stuff like that, and I'm like, wow, that's really interesting, that never even would have occurred to me. And frankly, I don't think that's the way to go with feature flags. But can you tell us a little bit about just like what is a feature flag? Why would you use it? But can you tell us a little bit about just like what is a feature flag?
Speaker 2:Why would you use it? Yeah, so a feature flag is when it's a common core pattern, but it's extraordinarily useful. Basically you can. The idea is that you can write some code, but that the best way to not break production is to be able to ship the code with that, to ship things with that code turned off, and then you can. Essentially you can separate deployment from release and that becomes it's a simple idea, but extraordinarily powerful.
Speaker 2:A lot happens at deployment time. You're shutting all the servers down, you're bringing them all up, you're changing memory. A lot sort of happens. And if you're also shipping new code as part of that, you can have you sort of it's like your defenses are down and then something blows up and you've got to roll it back. It's like it's a terrible time to really ship new code. A deployment is a terrible time to ship new code, which is kind of a funny thing to say, right? So instead, instead, let's get the new code out there and like, okay, check, all our bases are good, everything is fine, and now let's have a very, let's have a lot of control over how we launch it. And so we use this first hubspot and it just reduced the stress and challenges around around releasing new code.
Speaker 2:So often you know you're just sort of you'd have it out in production and it's not even on, and then you could turn it on for just my user in production and see if it works. You're like, yep, I'm confident there. I can turn it on for a beta group and let things bake for a little while. I can slowly turn it on to more and more people maybe a gradual rollout and then turn it on At any point. If I'm checking once, I'm looking at all the numbers, you know, uh, if I want to turn it off, I can just turn it off and it happens immediately. It's, it's a much more civilized way to launch code.
Speaker 1:Um, yeah, yeah, and I want to go even deeper into this and I apologize if I drag us off onto a total tangent um, but I feel like a lot of developers and teams and such don't they haven't yet calibrated their appreciation of how risky a deployment is, um, and I see things like just recently I saw somebody's pull request and there was there were code changes and there were database schema changes and just as a matter of habit, even without like really considering what the particular change is, just my default is definitely to make any schema changes, first deploy that and then make the code changes that depend on that change and deploy those separately.
Speaker 2:Um, I'm curious if you, if you, practice something similar uh, I think that makes a ton of sense and, yes, I think that's a. Uh, that sounds like a great practice. I so a lot of this came in from. I was at this Ruby on Rails place. We were doing cucumber tests. We were like testing everything, we were writing good code, we were the most talented people I've ever worked with and we would also QA stuff manually for hours and like get everything together and we still had to roll back and it was quite stressful all the time where I was at patients like me and it was frustrating and I didn't know a better way.
Speaker 2:We got to hubspot and I think, in particular, like the scale uh, the scale of everything was just so much was hundreds of times, hundreds or thousands of times larger than what we had been doing, and the it just drove home the criticality of and there's so many businesses running their business on top of HubSpot. The idea that you would cavalierly ship a database change and a configuration, a database change and some code and just bring down hundreds or thousands of businesses. You don't want that and that would be very stressful. So in order to not have that happen, you have to figure out some sort of other way to do it and I think the odd irony was that it was so much more critical and important, the things we were shipping at HubSpot. The blast radius was so much higher and yet my chill level was much greater because it was just much more pleasant. Right, you could kind of ship it and then you could go get your coffee, you bring up your graphs, you sort of like have a little peace and quiet, you know just when your thing's up and you turn the button and you're like whoop, it's all broken. You can turn it right back. It was much more so like as far as like I think some teams feel like perhaps that management is telling engineers to do this, management is telling engineers to do this. It was like no, this is for you, this is a quality of life thing for you as an engineer to separate these things. Similarly, what you said around database migrations, and that I think was very important.
Speaker 2:We actually sort of ran on a shared staging instance. All HubSpot engineers were using a shared staging instance. Most people I tell that think that's bananas and crazy and could never possibly work. What if you break staging? You would break all of the other engineers. There was a dawning moment I was like, yeah, but practice, not breaking everyone the idea that we're just going to ship stuff willy-nilly to staging and then we'll be careful once we ship the prod. It's like now you don't have a net, now you haven't even practiced that skill. So thinking like, whoa, how are we going to do this? And I really think what happens when services are running with the old database schema and the new one. It's like, oh yeah, we have build that all into our the way of thinking about it and you need good tools to be able to do that yeah, that's a good point about practice.
Speaker 1:Um, like, I think subconsciously part of the reason why I gravitate toward doing database migrations and their own deployment is is not because I necessarily think every single change that I do that way needs it. In a lot of cases I'm sure it's more expensive than what's justifiable to do it that way, like I'm just adding some tiny column and there's like basically no risk and it makes it a lot more expensive to do that separately. But I think there's something to be said for having a habit, something you always do the same way, so that when it really matters you don't have to think, oh wait, this one actually matters, I have to do it this different way, instead of the way for things that don't matter, you just always do it the good way, yeah, and then let's take the cost of doing it the good way down to zero.
Speaker 2:When the cost is high, we're going to One. It will either go slow or we'll cheat and go around the system anyway. So let's make doing the good thing as easy as possible, but in general, yes, making that the default. Less thinking is always better than more thinking. If it gets kind of complicated and scary, then, like, let's break it into component parts. There's enough challenges and emergent things and things we didn't think about.
Speaker 1:So, like, let's make simple things I tried to follow when I was a manager and I wish more people would which is make it easy to do the right thing. You know a lot of things. A lot of times it's like the path of least resistance is to do the wrong thing, and in order to do the right thing, you have to swim upstream, and obviously it's so much better when it's the other way around.
Speaker 2:Well, and I think this is going to run into, so take your change. There You're saying, okay, all database migrations, even if it's just adding a column, need to be a separate thing. Well, if your deployment process involves getting a PR review from someone else for every single change and going through the merge queue and getting multiple sets of eyes on every single thing, it's going to be slow. If you have to get that PR all the way through before you can get another PR, you have really slowed things down. But if you adopt some practices, like if you say, by having some of these good practices you can take some of those guardrails off and then you can just let people merge to main and go and use good judgment on some of these things, and then you can actually go a lot faster.
Speaker 2:Hubspot was deploying hundreds and hundreds of times a day and the idea and PRs there were in general like do you think you need a PR review? Then please ask for one. But it was not mandated or not required and it forced everyone to use good judgment. But the systems we put in place to make things, to have more thinking on those things and have more eyes on them, sometimes increase the burden a whole bunch. So then I think they can force you to do bad things Like well, if it's going to take six hours for someone to for a pr to get turned around, then I'm going to jam a bunch into the pr so that I don't right um. But if I'm allowed to as a senior engineer with 10 years experience, or I'm allowed to just do stuff when I think it needs to be done, then I can. Things can go a lot faster there.
Speaker 1:Yeah, and and to be brutally honest, like I think most PR review processes are kind of a joke. Um, it's like people, people reviewed the PR and they're either giving like a rubber stamp or they're giving some kind of like really superficial critique. And the worst is when they give some suggestion where it's like not actually a very good suggestion and I have to just think of a way to tactfully say no or just be like okay, I'll throw him a bone and do it because it doesn't really matter, but now I have to take this extra time just to appease somebody and stuff like that, and it really doesn't add any value but it creates that bottleneck.
Speaker 2:And the asynchronous communication, the risk of offending somebody intentionally or unintentionally, the meeting that they have with their manager about how Jason overrode their peers. There's a lot of repercussions from those things that can go under the radar and pull the velocity of everybody way down.
Speaker 1:Yeah, but at the same time, I think a lot of people don't know how to get rid of that. They're like yeah, we don't like this either, but we don't know how to get rid of it. And obviously, the things that you mentioned are ingredients. I have a few ideas of my own. Some of these are untested, um, but okay, and, and a lot of these things are like easy for me to say, but everything is connected to everything else, and so I think some ways to make it more practicable to do this, these things, is um, do less stuff, like make your product simpler and smaller.
Speaker 1:Um, because if you're like a lot of places just say yes to everything and they toss in all this stuff and it's not necessarily the sheer volume of stuff that causes the problem, it's that all this volume creates logical inconsistencies in the system, or at least it creates a lot of complexity to keep track of and and like often very inelegantly modeled complexity it doesn't make maybe there aren't logical inconsistencies, but maybe it doesn't make as much straightforward sense as it could. Yeah, um, and, and that's a big symptom of taking on too much um, and, and I don't mean like, have less throughput, I just mean make the product simpler, keep, keep doing the same amount of work. But if you're putting that work toward a simpler product, then you can like go in the directions that you want to, rather than just be like being like inching forward but, going broader and broader all the time, which is what most places do in my experience.
Speaker 1:Um, I have more thoughts, but I'll pause there in case you have any thoughts uh, who we?
Speaker 2:yeah, I mean we could go all day for, but, um, like, I guess, uh, certainly agree with that. I mean it's hard. It's harder than certainly to just simplify across the board, I think from the lens of PR review. I think figuring out what the job to be done of a PR is and having different types of PRs, I think is very worthwhile To me.
Speaker 2:The question I ask when I look at a regular PR for something going in I'm coming from, if someone like wants my, wants, my eyes on something, I try to answer the question is this going to horribly break something? And is this going to break something? Are they going to be able to measure whether or not this worked? I'm not asking whether they built the right thing or whether they did the right thing, or whether I like it or whether other sorts of things, and that is a powerfully different sort of thing. So it's like I would like to help this person iterate quickly. So my comment will be looks like you're not going to be able. There's no logging. Looks to me like you're not going to be able to tell whether this thing is even working. Looks to me like how are you going to measure this? How are you going to do this? How are you going to think, like, honestly, if there's a feature flag and then a couple lines line and there's like a bug or something wrong, that's just that I think is going to blow up, I might not even mention can figure that out. I would trust the practices that I don't the idea that someone else is going to catch my bugs in a PR. It seems like a bad thing. It's not going to work. You should like I don't, I don't, I was like is this something that can a safe way to iterate? And so really, if it's wrapped in a flag correctly, I'm like I to go.
Speaker 2:I think there are other types of prs which are like I would rather see where there's like they're not done at all, they're unfinished, it's like to do here, to do to do here. We want to have a strategy conversation about this whole pr, like let's put as little time into the code as possible, don't run any tests, don't run anything, just say like I'm thinking of connecting here to here, to here, does this all make sense? And like you know, sort of pseudo code or whatever. Let's have more PR comments on pseudo code PRs. I love those. And then there's one third type of PR which is like I would like to get better at the craft of software engineering.
Speaker 2:Here's my PR. I think it's pretty good at it. Rip it to shreds, give me every single piece, you get you like your best, most punctual. You know, you the the best coder out there, and they just destroy it or something. But you've like asked for it, or it's a rite of passage or something and you're like whoa, if you can get a make a perfect pr. Like that's a useful exercise to do sometimes. But most prs are kind of a combo of all three and it's just a mess and we don't really agree.
Speaker 1:Yeah, I totally agree, um, and, like most of the time too, like, by the time it gets to the PR review stage, it's too late, Like you can't like, in order to even explain the the suggestions that you might have, it's like it would require all kinds of background explanation, and it's like it would require all kinds of background explanation, and it's just, it's just too late, and so that's another reason why that that whole thing is problematic.
Speaker 1:Something that we haven't explicitly called out is, like the quality of the programmers on staff, because I think a lot of this pull request review process other people have said this that, like, a lot of it has to do with a a lack of trust, and I think that's sometimes true and sometimes not true, um, but if, like, if, if an organization just hires every warm body that they can and they want to staff up just for the sake of it, they're not going to get very good devs and they shouldn't trust those devs to just merge to master or whatever, because they're going to do low quality work, and so it does need to be reviewed before it's it's accepted.
Speaker 2:How do we, how do we train our juniors to get to be better? And I think having the flow of like junior rights code, somebody fixes is and helps it, and then they release it, is not necessarily good. The thing I want the junior to learn is a sense of fear, a sense of oh God, a sense of something. Ideally they're causing some production outages and you got to build that sense of like whoa, this code? Here I'm in the belly. This is important, I could really screw things up here or something I want.
Speaker 2:I need to develop that knack and sometimes you only kind of learn by failing. So, okay, let's get some like bumper pool, get some bumper lanes on here so they can't take down maybe everything. Or let's even evaluate for our company, maybe it's fine if they take down everything. Like maybe that's the learning thing. But like let's have a process to let's figure out where those, where those good places are that they can do stuff, where they can learn and fail and figure stuff out. But like keeping their iteration loop tight and actually having them learn, versus being in a place where we're always expect, where seniors are always expected, to go in and fix stuff.
Speaker 1:Um, yeah, well, there's two kinds of bad developers. There are bad developers who will eventually be good and there are bad developers who will never ever be good, and sadly, I think the latter category makes up quite a large portion of the industry.
Speaker 2:Yeah, I certainly am speaking from a position of trying to work at places where we are like we're going to hire good developers, we're going to be a proper software shop, we are going to get good at this and we are going to fire people that aren't going to get to that line, and we're, we're, we're, we're, we're only trying to keep the people that are headed that way, and that is a certain type of software shop.
Speaker 2:There are other types of shops where you're going to throw bodies at it and and have systems in place to try to prevent that being a catastrophic failure, and I have less familiarity with those places. So I think there might be other practices that need to happen to sort of to wrangle that. I'm no expert in that. I feel like there are a number of orgs. That what I'm, what I think happens is there are orgs that could be the first category but have adopted the practices, some of the practices from the second category. Now they're just going. They've got good people, but they're just going slow, uh, and so try to like pick a, pick a lane here and like let's figure out whether we're headed this way and we're going to trust our developers or we're going to like not trust our developers and go this way.
Speaker 1:but let's not just have good developers who we don't trust yeah, yeah, um, I worked somewhere once where my boss was of the opinion that we could hire mediocre people but if we had good enough systems we could get good output from them.
Speaker 1:Um, but I don't think it works like that and eventually he he came around to my way of seeing it after he had some experiences working with some mediocre people, and then we went a different direction after that. But it's not like you know, if you've read that book, the E-Myth Revisited, where he uses McDonald's for an example, mcdonald's can kind of hire anybody because they have a system and you just plug the people into the system and it works. But programming isn't like that, because it just requires so much discretion, I guess, is the word. Like when you work at McDonald's, there's not a lot of discretion. Discretion, I guess, is the word. Like when you work at McDonald's there's not a lot of discretion, but when you're a programmer, there's a huge amount of discretion, and so you can't have, you can't afford to have mediocre people, because it goes off the rails really quickly.
Speaker 2:Yeah, yeah, there's a sweet spot of. There is a sweet spot that exists of expecting more from people and they will deliver. It's not that I've just had continually hired fresh out of college folks who I don't think were any there wasn't anything particularly special about them or anything but then put them on a team with a couple of great individuals and expected a bunch from them. And you know, it was just astonishing to see, after the end of a nine month long or a six month long co-op or something, they're just like real experts on HBase and Java and other sorts of things, like they're cause they've just been in there and working with it and they don't know any better. That they're not.
Speaker 2:You know they do. They have quite the breadth of all of these things, but they have. They're pretty darn competent in something, and then understanding the limits of their knowledge is useful too, so that there's sort of like yep, I think I get this bit of it, but mostly, if you can set people up with the right sized problems, I think they can. I don't think what we do is so fantastically complicated that it's like that it's impossible to learn it, that it's reduced to a special few that can do all this.
Speaker 1:Like it is pretty broad. That brings up a an interesting, interesting aspect of it, which is like the, the, the value created in a team or or with each individual employee employer relationship, um is is a product of the employee and the system that they're put into. So you can have a really smart person, but put them into a dysfunctional system and the value created is going to be very low. Or take somebody really dumb and put them into an excellent system, but they just aren't going to be capable of producing something of great value. There. There has to be both. There has to be, um, good smart people and then a good enough system that allows them to to thrive. And there needs to be direction.
Speaker 1:Like I use this metaphor a lot which is like if you're a chef and you're opening a restaurant, you wouldn't just hire a bunch of cooks and show them the kitchen and say, have at it. You would give them some direction, like is this an Italian restaurant? Is it a Chinese restaurant? What kind of restaurant is this? What's going to be on the menu? Give them some direction, and people want that. Like people want and need that kind of direction in order to do cohesive work.
Speaker 2:Yeah, and and um, and then it's trying to figure out what yeah, trying to figure out what you're going to celebrate and what you, uh, what good looks like there, and the more that is customer value and understanding the problem and getting focused on those things, like I think I think sometimes I think juniors can come in. We don't think they're ready for it, so they just end up kind of on some not so important thing on the side or something. And then if you think about, like, what incentives they're seeing or what they think good looks like, it's just like so far away from where you, where you want them to go, because you didn't trust them to to put them in the critical path of something. So now they're off on a backwater and now they're like, oh, we build backwaters around here. And then you're like, oh surprise, they've built like this thing that delivers no customer value or something, but it's the only thing they were exposed to here.
Speaker 1:Yeah, yeah, that's so sad to see when companies hire a junior and then they put them off in some corner and say, like here's a project filing your tps reports, let me know in a few months when you're done, or something like that, and they're just like, um, okay, I guess, but then they have no direction, no education, anything like that I think yeah. So I'm like totally in favor of hiring junior developers. I think people should recognize that it is an investment and you either have to like do it all the way or not. Do it Like you can't just hire a junior developer and and let them be and like expect good results. You, you have to spend some time with them and be like this is how we do things around here, this is what we expect, and teach them and guide them and actually have somebody spend some time with them so that they can, that they can level up.
Speaker 1:And I think it's a long-term thing, like you might hire a junior and it's not like two or three years, until it's like even net positive, and then it can be great if you can hang on to them for a while. But I think that's why a lot of companies don't hire juniors is because they recognize that it's a big roll of the dice Because, again, there's bad programmers who will be good eventually. There's bad programmers who will never get good, and it's often hard to tell when they're early what kind they're ultimately going to be yeah, it's a, yeah it's a.
Speaker 2:It's a tricky balance across a lot of, a lot of different, a lot of different things there, certainly from an engineering management perspective it is. It is tough to tough to nail it, but yeah, um, so getting back to to feature flags which help yeah yeah, they solve it all.
Speaker 1:So you know there's. There's the poor man's version of feature flags where you just have like an environment variable or something like that and a conditional, and that can be just fine. But there's more sophisticated ways of doing it where you have an actual tool and you can turn the feature flags on and off from some kind of admin panel or something like that. Some kind of admin panel or something like that. But then from there you kind of took that idea even further. As I understand it, it's like hang on a second, if we can dynamically manipulate feature flags, why not dynamically manipulate all kinds of configuration?
Speaker 2:Absolutely yeah. So there's a core of a dynamic configuration system that's just saying like, let's have all of our deployed computers agree on, basically on a key value, pairs with rules, and that's kind of dynamic configuration. So like, is this feature on? Well, we're going to go through a rule list. Is this feature on? Well, we're going to go through a rule list? So, given a set of context given.
Speaker 2:So sometimes you think of feature flags as being user-specific. So like are we? Feature flags are something you can turn on for a user, but why can't you turn it on for a team? Why can't you turn it on for everyone on the pro plan or everyone on the enterprise plan? Why can't you turn it on for just a availability zone, right? So these are feature flags that are focused on actually doing a rollout.
Speaker 2:We want to switch over from this Redis to this other Redis thing and we want to do it for 60%, or we want to start at 10% of US East 1A and then do it across that. Then we'll do all of B or something like that. That's a feature flag too. So when you come with a bag of context which is usually like the user, the team, the deployment context, whatever you have there and then you're going to say I want this bag of key values, like here's all my configurations, and then each of them is going to have some rules that evaluate against the context and pop out the value. That is dynamic configuration. Feature flags is a particular set of that, like really sort of like let's encapsulate this piece of code but becomes pretty interesting to figure out what other things you can do once all of your configuration is configurable like this yeah, and I think you brought up some examples when we were talking.
Speaker 1:I can't recall. Now I wonder if you can recall any examples. It was like I don't know something that's like a numeric value or something like that, and I was like, oh wow, it never occurred to me that you could do something like that.
Speaker 2:Yeah, sure. So one sort of broad category is doing things like flow control, I think. Basically, once you start looking at this, anytime you see an integer in hard code or an environment variable, it's like huh, what would happen if we put something in there? So let's take our Faraday timeouts and retries. Like I don't know what should the timeout be? Should it be 30 seconds? Should the retries be three? You don't really know. There's not. It's hard to figure out that answer ahead of time.
Speaker 2:But if you are looking in your APM and you're seeing your services are all slow, they're timing out to this one thing. Being able to, without deploying, just change the HTTP timeouts from this service to this other service can often fix a critical situation, can nip that in the bud. You can change the Just, so that would be sort of an example of dynamic flow control. Similarly, when you set up Sidekiq how many workers, how many queues, what are the queue priorities? These are things that you can kind of make a guess at. But if you have to commit code, wait for, build pipeline, ship it out and then look at your numbers, it's a lot easier if that slows down our ability to experiment with that and then also to mitigate failures or on the fly when something is actually broken.
Speaker 1:Yeah, yeah, and like even now I'm just thinking of like use cases for that, like I don't know pagination, for example. You might have some sort of guess as to what would make sense to put on the page page, but then it might perform differently in production and you can say, okay, we need to tweak that a little bit so we we can handle a few more, or we have to cut it back to a few less, something like that.
Speaker 2:Um, yeah, absolutely and yeah, it's a, it's a great example. It's sort of you're in there like I don't know, does this really need dynamic configuration? Like would you buy a dynamic configuration system just to change the pagination? Like no, but there's this like multiplying effect, where once the system's already there, of course you throw it in there because it's convenient and it's useful and you can change it and over time there's just like more and more stuff in there and then you you end up with this application that eventually feels very different, because there are lots of ways in which you can configure it and change it on the fly or different controls you can give.
Speaker 1:Yeah, okay, remind me. I believe, if I remember correctly, you have some people using prefab now.
Speaker 2:Yeah, yeah, absolutely. So we've been doing it, for we started in 2023, I guess, and so we've been doing it for a while. We have a number of good customers. We scaled up 10x overnight actually last week, so there's actually quite a bit happening on the servers right now, which is delightful.
Speaker 1:Wait, how did that happen 10x overnight?
Speaker 2:Somebody just didn't mention that they were turning it on to an enormous deployment, which is great. Got it, yeah, and so, yeah, we have got a number of great customers. What we're focused on right now is particularly making a feature flag tool. This whole dynamic configuration thing is amazing and wonderful and I think, but it's, as we discussed, like feature flags. Not even everyone is using feature flags today. So people know they need that, they know there's a system, so we have found real sort of go to market.
Speaker 2:There are people that are unhappy with the incumbent solutions today that can be quite expensive and they're they're often seat based and that just doesn't work for a lot of teams the number of teams I talked to that are not paying for all of the seats, either by having just a small number of seats and then everyone has to ask in Slack for someone else to do something, or they're doing like create people do the darndest things just to get around pricing. That feels really, really unfair. So we're having a bunch of success with people darnedest things just to get around pricing. That feels really, really unfair. So we're having a bunch of success with people that are like let's save 70 or 80% on our feature flag bill. Let's get clients that work in Ruby and in JavaScript and in Python and in Java and just sort of like for all the different things.
Speaker 2:And then let's have the ability to experiment into dynamic configuration. Or they see the dynamic logging which we haven't talked a ton about, but being able to. One other aspect of dynamic configuration is huh, I wish I had more debug logging info and being able to turn on the debug logging for a particular class down to debug instantly and then you start seeing stuff in your logs. Uh, can be pretty compelling, um, but those are sort of people typically right now are buying feature flags and then adding those things on is like, oh my gosh, I didn't realize I could do this as well yeah, like I've found that people like it's easier to sell things that people are used to buying than something that's totally novel.
Speaker 1:To what extent are you finding that people know what feature flags are? That is not novel to the world. Products like that exist, it's a category. But the dynamic configuration I imagine people are less familiar with that. Do you find that people are maybe not so enthusiastic about that, initially because they don't completely get it, and then it's not until they try it that they really get it? Or do people kind of grasp it up front?
Speaker 2:Yeah, I think dynamic configuration exists at almost every large organization. They all have their system. You know Shopify or a DoorDash or any of these companies. They all have a dynamic configuration system internally. They call it like Kumbaya or Gummy Bear or something Right, and that's the thing that allows these big organizations to work. So I think there are a lot of people that are looking for that. In general, the sort of criticality of having dynamic configuration increases exponentially as you get large and then eventually people like rebuild it in-house.
Speaker 2:So the people that like get it out of the box are people someone's got 40,000 Kafka consumers, right, and they've got 40,000 Kafka consumers. They're reading all these events. This is sort of this massive data flow project and they are and some of the wrong answers are spitting out the back end of this Kafka queue. They're like well, what is happening? And you can't turn on debug logging for all of them, right, you would be making petabytes of of data. Uh, so what they would like to be able to do is like, well, it's actually just this one subset of the data that's going through is causing this problem, but we don't know it. So how do we do dynamic logging for through this massive Kafka cluster for just this uh, particular transaction ID or something, and that makes a lot of sense at scale. You're like, oh my gosh. So your question was about how much people get it, and sometimes it's really a function of the amount of pain that they're currently suffering.
Speaker 1:Got it okay? Yeah, yeah, for me that sounds very compelling. My go-to when I think about that is the product that I'm working on. Did I tell you about my continuous integration platform that I'm working on? Maybe I didn't, yeah, um it's.
Speaker 1:It's a very, very early stage project in terms of how much I've built so far. It's been in the works for like a year, but I'm just very slowly chipping away at it because I've decided not to make it a high priority. Um, because it's partly just for fun and to have like an educational. When I do my live coding videos, I need something to code on, and so that's its primary purpose and then hopefully it can turn into a business one day.
Speaker 1:But I can't afford to prioritize doing that because I got other things anyway, um, yeah, I'm thinking about it to apply to that, but I don't think that's. That's quite at the at the stage yet.
Speaker 1:But even the stage that it's at, like, when you run your builds, they run in parallel and right now the default parallelism is two, but I could see it being very convenient just to be like well, I want the default parallelism to be four and I don't want to have to change that in the code and then go through all that rigmarole and deploy it and stuff like that yeah, and I think that's.
Speaker 2:There's just no harm in putting the system in a little early, um, and there's only a benefit, right? So maybe. So, yeah, prefab has a free tier and I'm sure you'd be well within it. But, like, because I want people to put it in before you need it, the more thing. Just, you come across the thing, you use the vs code plugin and you just start it. You say you know prefabget parallelization of the CI and you create a new one. You set two Great, maybe you never change it in your life, but maybe you productize this and then come along and you want to change that to.
Speaker 2:You sell tiers, right, you sell entitlement, or you sell tiers and people in pro have a higher thing and people have an upgrade on top of that to get their level of parallelization. The code would never need to change. All it cares about I think that's when it's really working is like the code. This is a configurable sort of think of it as an injection point, right, it's like there is, we need a number here and we want to, like, declare this as a named injection point, and what happens later on top of that is left for the configuration system to decide, and it can be simple or complex, but that code should never need to change if you just have these named injection points.
Speaker 1:Yeah, yeah. Well, when you mentioned that idea to me when we had that lunch in Boston, that was kind of a new idea to me and it was an eye opener. So I hope for some people listening that's a new idea to them and it opens their eyes too. We don't have to go too deep into this if you don't want to, but I'm curious and we talked about this a bit before too how are you going about finding customers for the product? Is it kind of people you know, word of mouth kind of thing? Are you doing things to actively get out there and do marketing? Obviously, doing this podcast right now, that's one thing, but I'm curious about that.
Speaker 2:Yeah. So on the broad swath of products and building developer tools, I mean a system like this really needs to work. Like it? Just there are a lot of the sort of startup advice you hear is like build it really scrappy and go find some customers and do a thing. I don't think that's the space we're in. No one needs a half thought out dynamic configuration system. No one needs a half-thought-out dynamic configuration system. No one needs a sort of very scrappy feature flag thing.
Speaker 2:Feature flags is a mature market and dynamic configuration is such a core need for your systems that it didn't. There wasn't sort of like, yeah, just sort of scrappy, just sort of do stuff there. I think we're on a bit of a limb trusting ourselves. But the thing is we have seen this and used this in internal companies. So there's a funny thing that you get with developer tools where, like, actually building for yourself is an okay way to build things Because you know that there is really a customer there and you can be your own best customer.
Speaker 2:So we're leaning into that and so we've built it reliably and sort of for scale, I think, because the pain is most pronounced when organizations get larger the per seat costs of feature flags and those sorts of things really hurt sort of 20, 50, 100 engineers and those organizations do have more requirements and needs around other sorts of things. So we have been focusing on building attribute based access control and this month is reliability month, making sort of. We have been having almost an entire separate pipeline. So we really have like belt and suspenders so that pretty much any piece of the system can go down In fact we can whether a Fastly outage or a Google Cloud storage or a Kubernetes outage, we have like a full belt and suspender system here which is I won't say overkill, but it's a lot of kill, but we feel really good about that.
Speaker 1:So Wait so I've never heard belt and suspenders before. Is that just? Is the idea of redundancy?
Speaker 2:The idea of being, yeah, redundancy and sort of two different systems, two different failure modes, I guess, so that you could not just if you have two systems but they're kind of the same thing, then one of them could fail, Then you might have a failure that affects both of them.
Speaker 2:If they both rely on the same piece of technology, then that thing goes down, then you'd have an issue. So we'll have two fully separate ones, Java APIs, and then Go running in Fly and so they'll be like pretty excited about that level of reliability there. So, with all that said, yeah, we're really focused on getting great customers now and it's been enough to do that sort of within our network or sort of tangential to it, and we've been trying to tell people about what we're doing along the way blog, YouTube videos, we do pair programming, other sorts of things and I think the real space we're in right now is like pulling together the testimonials and the case studies of the people that are using us so can get over that uh, sort of display more trust on the website so that it becomes a clearer conversion for these people that are converting gotcha um, yeah, similar thing with with my ci product.
Speaker 1:Um, my entire value proposition is that the ui is better and more pleasant to use and stuff like that, and so I have to fulfill that promise or else the entire endeavor is pointless, and so it's not a matter of just throwing together whatever I can throw together with the least amount of thought and effort and stuff like that. I have to. I have to actually make it good, and it's really interesting to build something with that mindset instead of the just build something that somehow meets the bare minimum functional requirements.
Speaker 2:Yeah, I think you know. Yeah, sometimes we take a month and just sort of focus on our own internal use of the thing, Like, are we happy? Are we happy? And I think it was maybe in like March of this year where basically it's like hey look, our configuration story is not good. It was like our stuff is messy, like because we had dynamic configuration, but when you actually got into the weeds of it, we also had environment variables and those were getting set from Terraform in this one way or they were sort of coming in from this other way.
Speaker 2:Our local development story was not good and you had secrets and people didn't know how to set the secrets and so it was like wait a second, this is like a cobbler's children have no shoes, sort of situation.
Speaker 2:We're selling, it're selling, could I mean configuration, but like our systems are, it's confusing how our systems are configured. So we took a step back and made sure that our system now also supports uh has a good story around supporting environment variables, at support secrets as well, and that was really us trying to make sure we could fully uh, we don't call it dog fooding, we call it cat fooding because we have cats but that we could really we're happy with our own system and I think continuing to do that over and over again it feels like if you just keep doing that, you're going to build something that is great and you love it, and that is going to be much more important than the focusing on marketing too early when it's not really there. Like you just can't fool developers about whether it's good or not Like it's they ain't got time for that and they're going to try it out.
Speaker 1:They're going to know like how much discretion is is possible in choosing where to where to save expense, because you have like a limited budget of time um to to build this thing, and so it's like, okay, we can't afford to do everything we could possibly want to do, so how do we go in the direction we want to go most economically? And there's like infinite different ways that you could exercise discretion over that in order to move forward somehow.
Speaker 1:And you know, something that really really frustrates me is when I see people giving advice to people building a new product to basically what they're saying in different words is be sloppy, throw something together hastily and I'm talking like on the technical side, on the code, because you just got to get to market and then, once you get traction, then you can like get mature and clean everything up and stuff like that.
Speaker 1:Um, but for one, taking a sloppily built system and trying to to fix it, so it's, so it's tidy and easy to understand and work with and stuff like that is 10 times harder, a hundred times harder than just starting with something that was a sound from the first, in the first place, and the way I would put it is like, in the beginning, what you want is you don't want to be sloppy and careless, because that doesn't get you what you, what you want, um, you want to just build as economically as possible, and really that's always what everybody would want, is to always be building as economically as possible. But I think there's a short term, long term confusion there where people think that that by sacrificing those things in the short term you can get to a better place in the long term. But based on all the experiences I've had, I'm pretty confident that that's not really the case yeah, I, it's difficult because I think you can fail in every direction here.
Speaker 2:You can fail in every direction, right, you can fail by overbuilding and underbuilding. You can fail by never, never building something people want but it's nice, right, so and there's a ton of survivorship bias to from out, you know, the places we end up working are the ones that have some crappy code and so because, but they're still alive, right? You don't know, whenever you know not, a lot of people work at the orgs that have great code and no one ever used it, right. But I think one pattern, my maybe like have your cake and eat it too thing, but I like trying to demarcate those zones and have. So I have this, I think I stole it from somebody, but like pioneers and settlers and city planners, right, and there are and there's like and those are three sort of archetypes of where the pioneers are going crazy. There are no rules, they're just sort of like hunting through the forest, they're not building anything of any permanence, and that sort of pioneer Settlers are like starting to try to build houses but the floods are still coming, and city planners are building something really for the long haul and I think, at any given point, an organization.
Speaker 2:I think we would all do a lot better if we could use those words and decide what mode we're in and decide what the right one is for the situation, for that particular code base and that particular company at that particular time. I think a lot of the pain and suffering happens when someone is trying to be the city planner and a couple people are trying to be the pioneers and they're working in the same code base like it's a recipe for disaster and both going too, both going too slow. It's like, uh, both going too slow and not doing a good enough job and I think we end up. We end up there. So I like trying to find zones where we can go like hog wild on a sacrificial architecture that can happen over here and then figure out when we're we're in different. Sometimes you need different people for that, sometimes you just need to like throw the switch. But the lack of words, I think, is really part of it.
Speaker 1:Yeah, yeah, yeah, oh, what was I going to say? Yeah, yeah, oh, what was I going?
Speaker 1:to say I don't remember, but yeah, I totally lost it. Oh, yeah, yeah, okay, I remember. You said the thing about being able to fail in every direction, and I think that's such a great point, because you really can. You can be too sloppy and fail. Because you really can. You can be too sloppy and fail. You can overbuild, like you said, and fail, and it's really hard to attribute cause and effect. It's like did it fail because of this or in spite of this? Did it succeed because of this or in spite of this? It's like did we succeed because we wrote crappy code and moved quickly and stuff like that? Or did we succeed because we wrote crappy code and moved quickly and stuff like that? Or did we succeed in spite of that? And I think we'll never know, because it's like a production system with good code has never been created.
Speaker 2:I'm being a little tongue in cheek there, but I would say at this moment in history, a lot of the easy pickings are the ideas. Startups have been generally picked over or something like there is not a ton of room for a CI system. No one wants Jason's like hackily put together a CI system that isn't sort of novel or good or polished Right, and yet there are. It's possible. Like, right, you watch figma.
Speaker 2:Figma is like years and years and years of this very opinionated kind of thing coming in linear, like it's been a joke for years that like, oh, what the world needs is another bug system or something, and yet someone actually built something that was better than all of the other ones after hundreds and hundreds of dead ones, and I think they had a real strong, clear opinion with good design and a ton of polish on some specifics there and that was able to enable things to win. So maybe LLMs and AI or Gen AI notwithstanding, like the general landscape of things, having some opinion and some polish seems pretty important at this stage of things, sort of, unless there, until there's, like, the next massive technological shift that changes things and you can decide for yourself whether or not that's gen ai.
Speaker 1:but yeah, yeah, man. I have so many more thoughts and I feel like we could talk for hours and hours more about this, but it's probably about time to start wrapping up. Where can people go to learn more about Prefab and you and whatever else?
Speaker 2:channel to reach people. I've been making a bunch of videos and putting them on LinkedIn, so give me a follow on LinkedIn. It's prefabcloud to check out prefab, and I think those are probably the two best spots to find me right now.
Speaker 1:Okay, we'll put that in the show notes and Jeff, thanks so much for coming on the show.
Speaker 2:Absolutely. Thank you, Jason, Always a pleasure, Happy to do. You know episodes two through 10 or here or something.
Speaker 1:Oh, definitely We'll have you back and do another one soon. Sounds good, all right? No-transcript.