Code with Jason
Code with Jason
313 - David Santoro, CTO of Carwow
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode I talk with David Santoro, CTO of Carwow, about his journey from startup co-founder to leading a large engineering team. We discuss carwow's evolution, their focus on scalability, and how they tackled growth challenges across different countries. David also shares strategic product and technical decisions.
Links:
Snail Mail Newsletter Pitch
SPEAKER_00Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Email newsletters are a really nice way to keep on top of what's going on in the programming world. Except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more. That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right. I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like? You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming. The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says, thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary. I'm sure that's what you were going for, so just wanted to let you know that you succeeded. Looking forward to more. Drew Bragg from Philadelphia says, Nonsense Monthly is the only newsletter I deliberately set aside time to read. I read a lot of great newsletters, but there's just something about receiving a piece of mail, physically opening it, and sitting down to read it on paper that is just so awesome. Feels like a lost luxury. Chris Sonnier from Dickinson, Texas says, just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it. Something about holding a physical piece of paper that just feels good. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsensemonthly dot com. That's nonsensemonthly dot com. I'll say it one more time. NonsenseMonthly dot com. And now without further ado, here is today's episode. David, welcome.
SPEAKER_02Hi, Jason. How are you?
SPEAKER_00I'm good, and I have an important question for you. Um have you ever used the Carwow stick of truth?
SPEAKER_01The Carwow stick of truth? Not personally. Not personally.
SPEAKER_00Okay. It's a very useful invention.
SPEAKER_01Yes, we should probably mention what it is.
SPEAKER_00Okay, so first of all, maybe maybe if somebody's not familiar with with Carwow, can you explain what Carwow is?
SPEAKER_02Absolutely. Carwow is uh a platform website um to buy, sell, and and figure out what car uh to buy. Um we are based in UK, Germany, and Spain. But the Secret Truth comes from another thing we do, which is our YouTube channel. I believe we have the one of the biggest, if not the biggest, car-related English-speaking YouTube channel in the world. Um the UK one has over ten million subscribers. It's pretty pretty good. We have an interesting presenter that uh yes has this stick to test if um uh car exhaust is real or just a fake.
Not A Car Guy And User-Centric Vision
SPEAKER_00Yeah, yeah, because a lot of those cars have this this nice looking uh presentation uh of the of the exhaust pipe, but it's maybe not the real thing. Yeah. I I've enjoyed those videos. Um I've I've watched those for years before you and I ever met the other day. Excellent. Yeah, yeah. Um and I'm just curious, are you like a car guy yourself or or not especially?
SPEAKER_02Um I'm not. Um I'm far from an expert. And uh um I think that was the idea. When we started CarWAW, we wanted to make a website for the people that didn't know about cars. Uh so we would be your online expert. Um so we simplified a lot of the jargon, made it easier to find a car, buy a car, deal with the uh avoiding haggling with a dealer, that type of stuff. Uh we we hired experts.
SPEAKER_00And and how did you connect with the people you you built carwell with? And I'm curious about the timeline and stuff like that too.
SPEAKER_02Yeah. Um that was uh early 2011. Uh I left my full-time job at the time because I wanted to do my own startup. I had a couple of ideas, uh and then I received an email called email from this guy, James, that was looking for a co-founder. Um he got my email from I believe the the first person I worked for when I moved to London. And uh yeah, he James was the other founder of CarWow. He built a website at the time was called Car Buzz and was aggregating car reviews, and but he realized he needed someone technical to go to the next stage, which was building a platform to configure a car and getting quotes directly from dealers. So we met over coffee. Uh and this crazily I decided to work for free with him on this idea for about two years before we got investments.
Founding Story And Early Partnership
SPEAKER_00And I'm curious about your thought process behind that and how you decided to say yes, because I've been in a situation a lot of times, and a lot of developers have too, where you meet up with like an idea guy. It's like, hey, I got this idea, you build the thing, and then like we'll get rich together. I'll contribute some vague thing, I'm not sure exactly what, but let's do it. And most of the time it's a joke, and you should not say yes. Um and I'm guessing it wasn't just like blind luck that you said, like, yeah, why not? I'll I'll give it a shot and it worked. I'm guessing that you like thought about it and there was something to this that made you think it was worthwhile.
SPEAKER_02Yes, I think it's probably I picked on future compatibility to an extent. Um at the time uh I mentioned I started building my own ideas. Uh I'm kind of an introvert, so part of the job in a startup is often often you have to sell. Um and I was definitely weaker at that. And I was impressed by James' um approach. He was call-calling CTOs in London and asking for advice. And that was great. And he also demonstrated it he could uh apart from selling, was able to put an idea together and he spent some of his savings with an agency to build a prototype. Um so it was far from just an idea. He put some money in and he showed that he could bring something to the table. His knowledge of cars, his ability to sell. Good business sense. Uh and last of all least the idea was very uh I would say grounded. At the time everyone was trying to make a uh alternative social network or things like that, which I thought was idiotic. Uh so when someone came and said, Hey, we're gonna sell cars online, yeah. This is an industry stuck in the past, I could see, yeah, this could work.
SPEAKER_00Yeah, and I think that's really important. Um, the the sales part of it, like sales and marketing. Like, if the other person is not technical, like that's pretty much the part that they need to bring something to the table for.
SPEAKER_02Totally. Yeah.
SPEAKER_00Yeah. Um, okay, and then you guys worked on it for some time. Um, like, can you can you tell me uh about about how you got your first customers and and the timeline of that and stuff like that?
Rewriting To Rails And SEO Wins
SPEAKER_02Yeah. Um I think I remember we spent the first three or four months rebuilding what that um external agency b uh built. Um it wasn't wasn't a real ideal environment for me, it was building.net. Uh I wanted to build it in Rails, so I had a good reason. Plus, um at the time we had we learned something from the product, so we introduced some changes. Um after that we started building the key flow uh of new car buying, which involved importing lots of data from um relating to price of cars, configuration options, all the stuff, and building a configurator, uh, and then connecting to a dealer finally.
SPEAKER_00Um was that an easy decision to make to rewrite it in in Rails instead of.NET? Was it an easy case to make uh to your co-founder?
SPEAKER_02And in that case, yes, uh, because I was able to I I did I didn't automatically say yes to the I investigated, I looked at the code base, and um there were some significant changes required anyway from to help with the SEO optimization and the design we wanted to change, so I felt that it was faster for me to rebuild it in Rails uh than modified existing one, which was true at the end. I think in the couple of months we we had an improved version of it, and we got the benefits from an SEO point of view straight away. I think a month later we we received a lot more visitors to the website, so it was a right idea.
SPEAKER_00Okay. Um and then you you started working with dealers. How how did that come into the picture? Like what was the product doing before dealers were involved, and then like can can you help me understand how that part went?
From Reviews To Dealer Quotes
SPEAKER_02So before the kind of the the supply side of the market was included, it was a website uh about car reviews, so how to find a car. And the approach that James and Alexander used was to uh almost be like metacritic before cars. So they would aggregate other people's opinion. We didn't have opinions on cars at the at the start. We were a neutral party. So we aggregate the reviews, the scores, and give you the view of the market for that car. Um there was a blog as well. So James was with bright blogs, uh, guides and things like that. That was the key product they built. And our custom our customers decided that we should then the next stage they're screaming for hey, this is great, I found my car, but now I need to help finding the right dealer.
SPEAKER_00Oh yeah. And was that hard to like take uh content from other sources and put it on the car while site? Like there were no problems with uh like anybody perceiving, like, hey, you're taking my hard work that I did to do these reviews, and you're just like stealing my content and using it for your own benefit. Like, was there any trouble with that?
SPEAKER_02No, not at all, because we would pass the l we wouldn't use the whole content, it would just quotes from uh from uh review, which it's it's allowed, and then we would write our own content to summarize the opinion. So we'd say, I don't know, the auto express editor thinks it's a great car, but on the Guardian they mentioned problems with the tires, you know, it would be things like that. So we still write uh quite a lot of original content, it's just that the opinion was a summary of other people's opinion.
SPEAKER_00I see. And how much were you involved with like the SEO part? Like, how did you guys get traffic to the site and stuff like that? Because that that's obviously a different thing than like you can do sales where you like go and cold call people and stuff like that, but then just like getting traffic online is a whole different story. How did you guys accomplish that?
Ethical Aggregation And Content Strategy
SPEAKER_02Uh I guess that's uh involved a lot of learning for the founders. So uh I was spending time understanding SEO, especially the technical side of it, and James was studied a lot of PPC, and he was the marketing person for um first probably the first three or four years actually. Uh even later, it was so close to the marketing team of Carwa because he he ran the whole campaign in the early days. And that's how we grew the the the second product, so the configurator. Um we would buy very s high highly intent customers. So for example, someone was looking for a Volkswagen golf uh deal, so something very uh high intent language. And and we the main product was a landing page, the configurator, and then you receive offers. And even the offers was at the at the start manually entered by James. So we would get a couple of customers a day. Uh James would call dealers in the in the location where this uh customer came from, looking for dealers to say, hey, I have a customer looking to buy a car. Would you like to leave a quote? Uh I'm I'm I'm a believer uh in automating later and trying as much as possible uh to build with things manually or you know Excel or whatever.
PPC, High-Intent Landing Pages, Manual Ops
SPEAKER_00Yeah, there's a couple things to that you might be familiar with. Paul Graham's do things that don't scale advice, which I think makes a lot of sense. And I've I've heard that referred to as flintstoning, where you do it manually first and then automate it later because the flintstones powering their car just with their feet. Um and and I think that makes a lot of sense uh because otherwise it's too much of a risk, like making that big investment to automate something before you know if it's gonna pay off.
SPEAKER_02Absolutely. And uh but also um I have an example actually of something that we automated way later, uh, which was the definition of a discount. So uh at the start we would ask dealers to give us a quote for a specific car. Uh as as we were starting to scale, uh the dealers would say, hey, by the way, I can I can give this much discount on this model and this much discount on this model by excluding the option. So we started building spreadsheets with the rules that the dealership would follow to build the pricing. Um but before uh we had omitted that part properly and built a discount editor and stuff like that, we onboarded a lot more manufacturers, so we get we got a view of all the possible combinations before committing to code, which is slower to change. So yeah, and and we still use that approach a lot these days.
SPEAKER_00Yeah, yeah, I think it makes a lot of sense uh for the same reason you don't want to make that big investment before it's a a couple things, I guess. You you don't want to make a big investment before before you're sure that it'll pay off, and you don't want to make that big investment before you you you are reasonably sure what you want to invest in, you know, what exactly you want to build, because you could build the wrong thing, then you're gonna have to change it, and you have this carrying cost of this technical thing that you built, better to to push that out sometimes sometimes quite far.
SPEAKER_02Yeah, I think the the concept of options, you know, if you can delay the committing to a decision, that's uh the approach we're trying to follow.
Do Things That Don’t Scale And Discount Rules
SPEAKER_00Yeah, yeah. Okay. Um and at what point uh did did it start to feel like this is really gonna work, like this is becoming something real, or or was there ever a point like that, or was it more just gradual uh from the start?
SPEAKER_02Um I think it took us a long time to get to that first sale. So being paid I believe at the time we would charge 250 pounds for a sale from a dealer. Um it took us a long time, uh lots of iterations, lots of landing page changes, message changes.
SPEAKER_00Um and what's a long time, like six months, three years?
SPEAKER_02Uh I think since we launched the configurator product, it took us nine months to get to a product that actually was working. Um then they started accelerating the sale, you know. Um and and again we we were ultra focused. So we only uh were advertising for Volkswagen. We had three dealers in a particular area and we we were laser focused, so we wouldn't spend much money, but we had enough to prove the concept up to revenue. And that's what finally helped us unlock in some venture capitalist uh investment or sit round.
SPEAKER_00Interesting. Can you talk more about that? That choice to focus only on one maker.
First Revenue And Laser Focus On Volkswagen
SPEAKER_02Um yeah, um we were um we didn't have money, so we didn't have any funding, so uh all the money came from our own savings and uh um just from extra side side side gigs we we would have. Um so we were constrained by that. Um so for us it was always about w what's the minimum we can spend, what's the minimum we have to prove. Um because the traffic was coming from those specific keywords like Volkswagen Golf deals, uh we really were able to pinpoint the type of audience we wanted uh and uh and and what they had to configure. Um so again that the slice of the product was enough to to prove yes, if that if we can convince a Volkswagen uh customer to buy from a Volkswagen dealer and get him paid, yes, we can uh um believe that we can do the same for other makes.
SPEAKER_00That was yeah, it probably makes the whole problem a bit more tractable and less overwhelming.
SPEAKER_02Absolutely, absolutely. Uh yeah. And that was our scaling uh even after we got investment, that was our scaling strategy. We would add the make at the time, uh learn the subtlety of advertising with the make, what the dealers of the make wanted. It it wasn't quite as simple as we expected adding a make. Um you know we realized for example that our audience was uh slightly more wealthy than the general market, so um budget manufacturers uh weren't converting as well on our website where maybe higher uh hand cars were converting better. Uh so surprises are that. But uh we only learned that by adding it one at a time. A fun fact about the approach, we we when we launched in Germany, we tried uh hey, we got investment, let's try and put money, let's scale all at once, and it's a total failure. For six months we realized we we we thought we would have to close Germany and we reset everything, started from scratch, had one mega at the time, allowed the people that were working there to learn how to sell, how to manage it, how to account manage, and that approach worked again for scaling another country.
SPEAKER_00Interesting. Um so can you help me understand that a little bit better? Like, what was the Difference between the the two approaches? Like when you first tried to go into Germany, like how how how did you approach it the first time?
Capital Efficiency And Narrow Slices
SPEAKER_02Well we we thought that because we knew already what to do in uh UK, we would just launch with all the manufacturers that we had all at once, trying to sell to everybody. Um But what we didn't consider is that you have to still have to learn. Each country is different, behaves differently. Um the people we hired were new, so they needed to learn. And scaling gradually, one meg at a time, make sure that it's converting well and so forth, uh was the right approach for us.
SPEAKER_00Okay. Yeah, I guess that's something that American companies don't have to deal with quite as much because all the states are you know, there's there's minor differences, and there's unique state laws and stuff like that, but it's all one country, and there's there's a lot of US, you know, and so most people might never need to scale outside of the US. Yeah, yeah. But if you're in Europe, uh things can be quite different from country to country.
SPEAKER_02Even the cars are different, yeah. So the cars can be different.
Adding Makes One By One
SPEAKER_00Okay, wow. Um okay, so you took that that approach of starting narrow and then broadening. By the way, that's that's what I'm doing with Saturn CI too. I try to work in a little contrived pitch for Saturn CI in all my episodes. Um but I'm starting with just Ruby on Rails, and not only that, but it only works for R spec. Um and honestly, a big part of that motivation is just for technical simplicity to make the whole thing tractable. Because if I try to make it work for any platform, it's just a boil the ocean thing, and that's that's too much to do. So that makes it easier.
SPEAKER_02Absolutely, because um adding platform effectively just widens your uh customer base, but at the start you're better off finding an a niche and really understanding that, uh really understanding what the product needs to provide. And hopefully you will learn faster what also other uh niche will require, because some of the needs will be similar. So absolutely you should totally focus on one one one niche, one group.
SPEAKER_00Yeah, think of it like starting a fire, like you're not gonna hold a lighter up to a log and expect to start a fire. You have to start with some kindling and some little twigs and then bigger twigs and sticks and then small logs and then bigger logs.
SPEAKER_02Yes, I I like that knowledge. Yeah.
Germany Misstep And Reset Strategy
SPEAKER_00Yeah. Um okay, and and one more thing I'll say about that is like you know you know where to go, you know who to go after and stuff like that. Like with the Volkswagen stuff, it wasn't like, oh, what what car do we do next? Uh it was the decision was kind of already made. That space was narrowed down. So I imagine that made it easy to make decisions like that. Was that the case?
SPEAKER_02It was, because you um it made it easier to yeah, to manage fewer c marketing campaigns, uh you had to look a fewer path in a product, you had to sell to fewer people, because again, you just needed uh a couple of dealers, and so also the time it takes to manage supply and things like that. Uh so it really strip allowed us to do a lot with three people that we wouldn't be able to do effectively with uh uh if we tried to launch the whole market.
SPEAKER_00Yeah, and how long did it stay just the three of you? When did you hire your first additional person?
SPEAKER_02So we we we spent two years together with with investment, and then in 2013 we managed to uh raise a little bit of funds, hiring a full-time salesperson and our first developer. So we were uh that year we got to about seven or eight people at the end of the year. Um and that's where we added more makes and scaled it a lot faster. I see.
Niches, Kindling, And Decision Simplicity
SPEAKER_00I had a conversation with um Mike Parham, the sidekick guy, some years ago. Uh it might have been on this podcast, um, and we're talking about hiring. And you know, famously Mike runs sidekick as a solo person. Yeah, he uh doesn't employ other developers and he wants to keep it that way. Um, and he does very well with with that model. Um and he said something like when you scale, you don't want to hire just like one developer. You want to wait until you are ready to hire like four developers, because whether you hire one developer or four, you're gonna have roughly the same amount of overhead. It's kind of like the difference between um going from having no kids to one kid is such a bigger difference than going from one kid to two kids. Um and I'm curious, you your thoughts on that. Um, when when you hired, how quickly did you go from zero to one to more than that? And just any thoughts on that part?
SPEAKER_02I was lucky because the first person we hired was a person I worked with in the past that uh was very experienced. You know, I think at the time he had eight or nine years of experience as an engineer. Uh I was keen, he also was keen to the startup. Um didn't need a lot of managing. Uh it was a person uh we we worked well together, we had side projects, uh I was very experienced. Um I think we got managed to be work together uh as two for another year before we hired another person we worked together before. So that that foundation of uh having uh two other very senior engineers apart from me was so helpful. Um the when we s we started, I guess, expanding further, you're right, is when we hired a first ranger in a sense. But the point we were ended up being four, four or five, yeah. So maybe matches what uh you were saying. Okay. But no not everybody has the luck. I think I'm still the they're still with us as first two engineers, and they have tons of knowledge. So I've been very lucky.
Early Hiring And Seniority Matters
SPEAKER_00Well, you say luck, but also it could say that that was just a smart choice on your part uh in in who to hire. Um and you know, you you had to do some sort of preparation in the years leading up to that. Uh like maybe you're the kind of person who um well clearly you must be the kind of person who forms relationships in the industry that you maintain over a long period of time if you if you knew this guy for a long time before, um, and then you're able to persuade this guy. Well, I don't know if it was if it required persuasion. Uh-huh.
SPEAKER_01Yeah, of course, yeah. Hey, by the way, would you like to come and work for this company, be paid less, and you know, who knows? It's always uh some persuasion.
SPEAKER_00Yeah, yeah. So maybe some element of luck to all that, but I would also say just good choices.
SPEAKER_02Hope so. Um I mean yeah, relationships as any any other industry, I think, uh really important. Yeah. Technology.
SPEAKER_00Yeah, and I think those first Steve Jobs said this, you know, the the first few people you hire is a really important choice. Uh if if you hire really good people or or worse people, like that's gonna make a really big difference.
SPEAKER_02Totally. Um and I see a lot of startup hiring juniors really early uh to to save, but uh uh it's really difficult. I mean I I'm sure it works well for some companies, but I wouldn't have worked for Carva again because I wanted inde fairly independent people so that that we could double up rather than add in management. So you're right, if you add in four, great. Uh but if you only add in one, then one needs to be really good.
SPEAKER_00Yeah, it's a false economy. Um people who think that way are making a mistake uh uh uh about uh investments and returns, I think.
SPEAKER_02Yeah.
False Economy Of Cheap Hiring
SPEAKER_00Like they're only looking at the cost side. You know, there's this really interesting view, which is perplexingly common, that like all developers are kind of interchangeable. And it's like, oh, I can I can pay this guy a hundred bucks an hour or this other guy sixty bucks an hour. Uh, why would I pay a hundred bucks an hour when I can pay sixty bucks an hour? I'll I'll get as many of those guys as I can instead of the hundred bucks an hour guys, but it's like you know, they're not all interchangeable, you're not getting the same payoff for all those different investments.
SPEAKER_02Who has this idea? Technical people, awfully not.
SPEAKER_00Yeah, it's crazy. Yeah, yeah.
SPEAKER_02Uh I always I I can see the difference. I mean uh e even with the the current team, you know, there is a variety of skill sets and you you need bit of a combination, but uh I know the really the they used to talk about a 10x engineer, but uh not that much, but I can see the difference between a really good tech lead and a mid-level engineer, you know. That tech lead can ship three, four times faster and making better decisions.
SPEAKER_00It's way more than 10x, I think.
SPEAKER_02Potentially returnable investment, yeah.
SPEAKER_00Because like imagine instead of having hired you, uh or instead of having partnered up with you, uh your co-founder James partnered up with some dumbass, he never would have gotten it off the ground at all. So like that's not a 10x difference, that's a difference between like Yeah.
Sense Of Proportion And One-Way Doors
SPEAKER_02Uh yeah, I mean uh but it's hard to know uh who to bring in and to balance that experience and right attitude. Because again, it's not just experience, is the the type of person who can work in that type of environment, like a scale up or start up, uh where you can't be a perfectionist, you need to think about the ROI constantly, uh be comfortable with change. Uh so I think the personality also plays a role.
SPEAKER_00Uh yeah. Yeah, there's so many there's so many qualities that are needed. Like just just in our brief conversations, I've I've picked up on some things about you. Like I and and you know, I'm I'm making guesses, uh, so tell me if I'm off, but like I feel like you have a sense of proportion. Like you can make a big picture plan for something, but then you also don't discount the details when the details matter. And I sense that you can tell the difference between important details and insignificant details, and what calls for time and thought to be spent on them, and which ones don't. That that's something that's not uh universal among programmers, you know. Uh I'm I'm sure you see this. Uh people get really focused on details that don't call for that much focus, and then ignore the bigger things that do call for a lot of attention, that that kind of uh sense of proportion.
Naming, Modeling, And The “Offers” Trap
SPEAKER_02As you were mentioning that I keep thinking about the early days when there would be debates, hard debates on uh should we use the curly braces on the one line or second line or c code rules and things like that. Uh I hated that. It's it's a waste of time, just pick one and move on. You know, we're really happy with things like rubber takes the decision for you. Uh but it's true, I think uh I always had this desire to to ship and see change and see impact on my work that whenever I I get the feeling that we are wasting time on things that doesn't don't matter doesn't matter in the short term, then uh I get quite frustrated. Um but you're right, certain things you should take the time because they have consequences or um I I call them kind of one-way decisions where it's difficult to come back from it. Those decisions you need to spend the time. If you can make a decision and change it the day after, then why we are spending more than 15 minutes deciding it.
SPEAKER_00Yeah, yeah, yeah.
SPEAKER_02Say live.
SPEAKER_00Yeah, that's a really good point. Um and something that is worth spending a lot of time on is what a lot of people refer to as naming, um, but I don't think naming is exactly it. It's more like um the conceptual Yeah, modeling. Yeah, yeah, yeah. Um I'm sure that the answer is yes, but have you ever did you ever create a model in Carwow, or did anybody ever create a model in CarWow that some point later on you really regretted and you're like, oh man, either that was the wrong choice from the start and we just didn't know it, or reality changed and now this thing that made sense in the past doesn't quite make sense today. Did you have any cases like that?
SPEAKER_02Uh many times, unfortunately. We still have uh probably one of the oldest classes in the code base called offers, the offers table, uh that originally actually included offers and then became just a link table, and we never renamed it because of tons of reports, the username uh uh and and it uh annoyingly it causes so so much confusion for new starter because they they see that oh the table is called offer, but there's no price in it. Why is that? Uh I I wish I didn't name it like that.
SPEAKER_00So what does the offers table really represent?
Aliasing Models And Gradual Refactors
SPEAKER_02Uh now it just represents the a link between uh customer and a dealer. It's just uh yeah, that relationship. Interesting and m and you can have multiple offers from the same dealer. Again, this is partly because uh talking about the very very basic product we built at the start, uh we made it so you can only configure one car and receive offers on one car from one dealer. Um at the very beginning, then obviously uh we implemented that. Um so yeah, the model was wrong. Um but yeah.
SPEAKER_00Yeah, and then naturally it gets baked into a million places, and the the the cost of changing it becomes so much that it's you can't even do it.
SPEAKER_02Yeah, uh it became like that. Uh I'm a bit ashamed that we haven't changed it yet in 15 years.
Merging Codebases And Production Risk
SPEAKER_00Well, you're in good company because every single code base I've seen in my life has at least one, usually a lot more than one, um fundamental, important model that has a very confusing name. Yes. Yeah, and it's hard because it's not just a technical decision, it's also like uh an organizational, psychological, social. There's those factors in it too, and you have to make a campaign to make the change. It's not something that one person can just decide to do one day, um, because it takes an organized effort over a long period of time, and again, it's a great cost. And it's not really in any one individual developer's interest to go and and and take on that project. I mean, they couldn't even if they wanted to, because they would need like permission to do it, they couldn't just go rogue and then go do that. Uh not because they would get in trouble only, but because like again, uh everybody has to kind of be of the same understanding of what's gonna happen and stuff like that, I think.
SPEAKER_02Yeah, absolutely. Uh and uh again, the the challenge is often that these models are used in reports, maybe built by other teams on top of those tables, and they name the table like that the same in the uh reporting database. So yeah, the the ramification some of the changes.
SPEAKER_00Have you ever done the thing where you like take one of those models that is is used in too many places to change and you like create an alias for it or something like that?
SPEAKER_02Uh we we we had done that, we used that approach, yeah. Um and to be fair, the re reason why the table hasn't changed yet because there is also a transition, architectural transition we are doing. So I think at some point it will happen. Uh but other other uh building blocks are happening first.
Migrations, Downtime, And Tradeoffs
SPEAKER_00Right, right. Yeah, the whole thing is really tough. That's that's why it's so common. Um but yeah, just in case, just in case anybody's listening and they're not familiar with that technique, I've had good success with that where I um I I take something, create an alias for it. I this happened to me very early on in Saturn CI. Um the model that I now call test suite run, I originally called it build, and then I realized that build was totally the wrong name. It's not a build, it's a test suite run. But it's like the fundamental model in the application. Um and so I aliased it from build to test suite run. Um and then bit by bit, over the course of months, actually years, because I still have instances of build in the code base that I haven't converted over yet, um, just chipped away at it for for many uh for you know two and a half years at this point. That's how long I've been working at the at the project, and I'm just gradually getting those bad names out of there. Um so that can work, because a big part of it is um risk. Uh touching these things not only introduces cost but also risk.
Feature Flags, No Staging, Ship Often
SPEAKER_02Absolutely. Um we've been working on a project recently of converging two separate code bases into one, which is probably the other way around, some companies do. Um but it was fairly complex, and we definitely had quite a few production errors because of that. Um so any these changes are complex, you have to consider that will have an impact on your customers. Um so again, how to justify uh times.
SPEAKER_00And I imagine as Carwell has grown, you've had to give more thought to not just how do I make a certain change, but how do I roll out this change to production in a way that mitigates risk as much as possible. Is is that something you had you've had to think about a lot?
Backward-Compatible Deploys And CI Guardrails
SPEAKER_02Yeah, totally. Uh we had the uh we had um conversation with uh tech leads recently about uh a change we've done. It was a migration on a big table, so it required uh proper technique. Um But it raised a question a right question around the uh risk versus or cost of error versus cost of delay. Um so this change uh ended up taking two or three days of thinking, development, and running it. Uh but I wanna article say we could have had like five minutes downtime do it immediately. Would have been the better decision in that case, you know? Should we do the proper way where we do the migration with zero downtime but it costs three days of development, maybe one or two people or we we take a hit for five minutes, maybe run it at night. Uh i it really sparked some nice conversations because again the right thing is not always the correct the right decision for the company.
SPEAKER_00So yeah, yeah, and that raises a really interesting point. Like I I I I used to think about programming decisions in terms of like the right thing to do or wrong thing to do or whatever, but I think right and wrong is not the right framing for hardly anything. It's it's all about costs and benefits. If if you do it this way, it'll have this cost and these benefits. If you do it this this other way, it'll have these other costs and these other benefits.
SPEAKER_02And maybe some of the costs uh you're gonna add in the future rather than now. So uh that's more technical depth. Yeah, yeah.
SPEAKER_00Um and are are there any general strategies that you've used to like get things into production safely? Like for example, instead of putting a big change all at once, uh rolling out small chunks in a certain order and stuff like that. Have you had to teach these techniques to any of your developers and that kind of thing?
Safer Migrations, AI Reviews, And Systems
SPEAKER_02Yes. Um we uh something that shocks a lot of people whenever they join Carwow. Yeah, decent sized team now, it's sixty-five people. Uh but we purposely don't have a staging environment, you know. Um we we really encourage people to release often. So small PR, small changes, uh and we use feature flags so that we don't have to release a feature to the public, but you can definitely deploy part of it to production often. Um which means that combined with good monitoring and fast uh reverting capabilities give us a good way to reduce that risk. Uh it works really well for us, but every time I had to resell the approach.
SPEAKER_00Yeah, something that I uh always do, even at this early stage of building Saturn CI, is if there's a change that requires a database migration, I don't deploy the migration and the code change in the same deploy.
SPEAKER_02Yeah. Um funnily enough, we have uh now used one of those AI models to uh spot this problem uh for us automatically, which is nice. Because in the past it required someone reviewing and noticing it. Uh now we were able to uh spot when there is a migration and a bit code change and all that. So hey, if you consider this. It's part of our checklist, yeah.
Squads, Managers, And Reporting Lines
SPEAKER_00Um and for anybody who's not familiar, like why do you do that? Why do you deploy the migration on its own before the code change? What's wrong what's wrong with doing them together?
SPEAKER_02Um I I don't know in general, but for example, on our platform we use ZeroPoo, and the way the deployment works is that uh your new code will be released, and there is a possibility that you add the old code running on uh uh some dinos or web instances uh and the new code running on the uh some other dinos, which means they both the old code and the new code need to be able to work with the same database table without giving errors. If you do a migration you can uh uh uh you can break that so that you get an error for a few minutes and and problems with your users. Uh and that's true also for API changes. We use the same approach for API changes, the the break.
Tech Lead Versus People Manager
SPEAKER_00Yeah, because there's kind of three states. Uh there's like the original state and the end state, but then there's this intermediate state. Um like the intermediate the intermediate state where the code change has been applied. Okay, look, if if you do it the wrong way and deploy it all at once, there can be this intermediate state where the code is the new version of the code, but the database is still the old version of the database, and that's not that's not always gonna work. Um yeah, so if you deploy the database change first, and of course, take care to do the database change in a backward compatible way so that it'll work both with the old code and the new code. Do that first, and then also gives you like a checkpoint where you can do that and pause and say, okay, we just deployed that migration. Is everything still working now? Did anything go wrong? And if anything did go wrong, you know that it can be attributed to that migration. Whereas if you deploy everything together, you can't be sure always whether to attribute the the problem to the migration or the code change. The code change, yeah. Yeah. Yeah, yeah. So there's all these techniques for for mitigating the risk. Yeah.
SPEAKER_02Um on top of that, I think we use a gem, the uh I think it's called Safer Migrations or something like that, that uh is aware of some of the procedure for more complex migrations, for example, renaming of columns on large tables and how to do it properly. Uh we we found that really useful. Because again, yes, they more experienced developer carwal at some point the bro they messed up a deployment like that, but every time you have a new person, you we need to find a way to teach them as well. Um both the combination of the safer migration gem plus uh the automatic review from AI has been really useful to avoid problems.
SPEAKER_00Yeah, I love that idea of adding uh technical safeguards to protect yourself against human error. Because I've always found, I imagine you agree, like whenever there's an issue caused and somebody determines that the way to make sure that never happens again is just like be more careful, just like don't do that anymore.
SPEAKER_02Yes, it works only for that person, maybe.
Conflict, Performance, And Retention
SPEAKER_00Right, that's a non-solution, but if you can put something in place to if you can improve the system to make the mistake physically impossible, that's a much better solution.
SPEAKER_02Absolutely. And uh I have a similar uh opinion on certain documentation, like ah, we've we've read it there, yeah, but you need to know about that. Uh always like to have warnings in the code itself somehow that points you to a documentation to how to do properly, but you need the you need proper failure states in your in your code that then point you to the right solution.
SPEAKER_00Um and I'm curious about the team structure of Carwow and stuff like that. Is you know, CarWA's been around around for many years, so I'm sure there's been room for changes to have occurred over the years and stuff like that. But how does like the reporting structure and stuff like that work? Is it like you, the CTO, and then there's teams who report to you, managers of those teams? Like, how's all that structured?
Two Career Paths And Time Tradeoffs
SPEAKER_02Um I guess we have uh uh matrix organization where um pro engineering team are organizing squads of um normally a tech lead and four developers, a product manager, and often a designer as well. Um the the reporting line are to so engineers report to engineer managers that focus on team performance and they look after two or three teams, two or three squads. Uh and engineer managers report to me either directly or indirectly, depending on how many people we have at the moment. Yeah, okay. Um fairly standard ish, although I think there's new trends at the moment to go back to more technical engineer managers where my engineer managers are more people focused, and the tech leads can be still uh leading the project and coaching, but it's also contributing individually.
SPEAKER_00Okay, got it. Yeah, it's something I'm always curious about because I'm currently in a tech lead role at at my job. Um I've worked there for nine months uh or so as of this recording. Um it's my first time being a tech lead. I I've served in a manager real role before, um, but I've never been a tech lead, and so it's a little bit new to me. And I always ask people when I get a chance, you know, what what does that tech lead role mean to you? So I'm curious, you must have an opinion about this. What what does tech lead mean to you?
Wrap-Up And Where To Find David
SPEAKER_02So the tech lead for me is the uh most experienced uh engineering that team, first of all. Um but they probably the change is they become multiplier of their team. So my expectation is that uh rather than do a lot of individual coding, they would code with the developers, help them break them down, working on bigger problems, uh helping helping the product margin also with the the the design and what to build. Um and but I try to remove the line management responsibilities from the tech leads, mostly based on my experience, because I noticed that people management is a different job, you know. Tech lead is a good evolution for someone who's kind of a good product-oriented engineer, uh but people require a different type of passion and uh and skill set. And I had terrible managers so conscious of that, I try to hire people that really enjoy their part and are really good at that.
SPEAKER_00Uh yeah, you mean that you've worked for you you've you've worked for many.
SPEAKER_02For manager, yeah.
SPEAKER_00Yeah. What are some of the differences you've we just have a couple minutes left, but what are some of the differences you've you've noticed between the weaker managers and stronger ones?
SPEAKER_02Um there's people managers uh more often I see it in hybrid roles. So the I had tech leads that also had people responsibility that weren't really able to have deal with conflicts in their team effectively, or uh salary conversations or um tougher performance conversations, things like that. And and and it at scale the these problems are big, it happens every week. You need to know how to handle it. And they're a big factor in uh retaining people and keeping people happy, the right people. Um so I I I s I've seen in companies where they use a hybrid role where the call a lead developer, there's also manager, where their role changes a lot depending on individuals preferences. So there are people that are really good at the people management, but then they forget to to be technical leaders or people just technical leaders that disregard the the people side. Occasionally find someone who does a bit of buff, but they often sacrifice their coding time because there's you know just there's often enough work for these two individuals. Um it depends, I guess. But for for for us we prefer these uh these two path career uh at Karwa.
SPEAKER_00That yeah, yeah, it's hard to find your time it's hard to find all those skills and abilities in one person.
SPEAKER_02Yes. And and the time, it takes time as well to do this stuff well. Yeah. Um so something is will give, you know. Right.
SPEAKER_00Um okay. Well, this has been a really enjoyable conversation. I think we've touched on some really uh i interesting and entertaining and educational uh topics here. So so thank you very much. Um where can people go if they want to learn more about you or carwau or anything that you want to share?
SPEAKER_02Uh uh from me you on LinkedIn, you can add me, you can find me there as David Santoro. Um about Carwow. Uh we still have uh a medium blog of an engineering team. I don't know if we had the match recently. Let's see. Um let me get engineering at carwow medium. Um it's a bit long the URL. I'll uh maybe put put in a link by the medium.com slash carwau dash product dash engineering. There's a few nice nice articles, uh quite a recent one as well. So take a look.
SPEAKER_00Okay, well uh David, thanks so much for coming on the show.
SPEAKER_02Thank you for having me, Jason. Thanks.