Code with Jason
Code with Jason
310 - Brad Taylor, Fractional CTO
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode I talk with Brad Taylor about his journey as a fractional CTO and advisor for various companies. We discuss his experience with open source medical records, the importance of business objects, and navigating complexity in software projects. We also touch on storytelling in code and managing technical debt.
Links:
A Newsletter You Can Hold
SPEAKER_01Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Email newsletters are a really nice way to keep on top of what's going on in the programming world. Except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more. That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right. I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like? You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming. The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary. I'm sure that's what you were going for, so just wanted to let you know that you succeeded. Looking forward to more. Drew Bragg from Philadelphia says Nonsense Monthly is the only newsletter I deliberately set aside time to read. I read a lot of great newsletters, but there's just something about receiving a piece of mail, physically opening it, and sitting down to read it on paper that is just so awesome. Feels like a lost luxury. Chris Sonnier from Dickinson, Texas, says just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it. Something about holding a physical piece of paper that just feels good. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsense monthly dot com. That's nonsensemonthly.com. I'll say it one more time. Nonsense monthly dot com. And now without further ado, here is today's episode. Thanks for having me. Um has anything bad happened to you today?
SPEAKER_00A couple of things, uh, but hopefully we're on the upswing. A little bit of uh accent with the microphone and some water spilled on my desk, but we're okay.
Open Source EMR And VA’s Mumps
SPEAKER_01Minor, minor disasters. That's right. Yeah. Um, anyway, you were one of the people who reached out to me after I posted on Reddit that I'm looking for founders and CTOs and that type person. Uh, so thank you very much. Um and and tell us a little bit about yourself. You do fractional CTO type work.
SPEAKER_00That's right. I'm a fractional CTO and advisor to a variety of companies in the Andreessen and Index portfolios, but I've been a long time tech leader going back to the early 2000s, uh, working in medical record software, uh, and then did work in uh every industry from pet tech to um computer vision-powered woodworking tools to payments and uh and health tech.
SPEAKER_01Oh wow. Um I think we have a lot in common. You mentioned a lot of things just now that like uh made made some neurons fire on my end. Um so let me let me start with the electronic medical record stuff. Yeah. Um and and I might ultimately want to focus more on your like um fractional CTO business and stuff like that. Um, but but for starters, the electronic medical record stuff, um, that's something I did for a number of years. And so there's probably some overlap in what we did. But you worked on like an open source medical record system, is that right? That's right.
Commercializing Vista And Sales Reality
SPEAKER_00I believe it was the world's first uh when I was at MedSphere. So we um were aware of the veterans administration, the U.S. had created their own medical records system back in the 60s and 70s, written by doctors and nurses themselves who decided to learn this esoteric language called Mumps to write the code to automate the work they did. Um it really started from a great place. And as taxpayers, we can request the source code from uh the VA. And so we did. So the founders, Scott and Steve, requested the source code. It first came printed in millions of pages, which was a huge problem, as you can imagine, and eventually we convinced the VA to give us a CD with a few million lines of Mumps code, then went through the arduous process of commercializing that. And my team was the UI around that um interface. So the uh the backend was written in Mumps, Mumps is a language and a database, super interesting, but kind of a topic for another podcast, I'm sure.
SPEAKER_01Yeah, and by the way, what a fucking name for a language for medical software.
SPEAKER_00Totally. Uh very odd. But uh actually Mumps is still in use today uh across the world a lot in fintech. Um, but in some companies like Epic, that is a massive uh medical record software vendor, uh, they use Mumps quite extensively.
SPEAKER_01Okay. Uh I I actually use another programming language called AIDS. Um not really. That's just like the only thing worse that I can think of. I pretty much, yeah. Yeah. Um okay. So what was the like motivation and all that behind doing this? It obviously sounds very hard, open source, doesn't sound very lucrative. So, like, how does that all add up?
SPEAKER_00Yeah, the idea was basically to have a kind of a jumping off point as an early stage company that meant we've got this very advanced uh medical software that could store millions of records, it could store um you know uh procedures and diagnoses and surgery, all the stuff that's really complex to store. Uh, and we could have a jump start and do it, and then build the UI on top. And the idea was we would charge money for services in kind of a red hat model in the US and be able to give it away for free for those countries that uh couldn't afford those that services model, and they can self-support. Uh and so the hope was basically to have you know thousands of clinics out there running, you know, MedSphere software, which is the VA code underneath, plus this new UI we had written. Um unfortunately VISIN didn't didn't get realized, but it was a fun uh adventure regardless.
Jason’s EMR Project Lessons
SPEAKER_01Yeah, um I I wonder if there were any lessons for you there um that that are similar to some lessons that I've learned in my career regarding the difference between building a good product and making sales for that good product. Because um I I haven't done I haven't done sales in the medical industry myself, uh, but I've heard about it. And and it sounds like the path from I have a product to somebody's buying it is by no means an easy one at all. So tell me about and to the extent that you were part of of that side of it, I'm curious about your experience.
SPEAKER_00Yeah, certainly in that role, I was very adjacent to the sales motion. But over the years, I worked with a variety of health tech companies and do continue in my fresh consultancy. And really, I think we had the challenge of we had built this really cool tech that we thought everyone just should just understand immediately and then be able to implement. Um but it turns out that in health tech, you really need to think about business model first and tech second. Um, and so the challenges we had was we were going head to head with the epics and the McKessons of the world. Um we were a small fry, they were big. And so if you're a CIO for a large hospital system, it's pretty hard for you to say yes to not just going for IBM, not going for the big trusted player and go for someone who's more scrappy. Um, you know, it's a long sales cycle, uh, you know, one to two to three years. That's hard to do on a venture scale when you're thinking about the next 18 to 24 months as your runway before you raise the next round. And in order to achieve that Series A, Series B, Series C next step, you need to have a certain number of customer side, you need to have a certain number of deals implemented. And doing all that with that long sales focal was very challenging and one that the company really struggled, unfortunately, to manage.
SPEAKER_01What was the timeline? Like how long did that startup exist? And were you aware of when it started to become clear that it wasn't going to work out and all that?
Domain Modeling And Conceptual Coherence
SPEAKER_00Yeah, so the casual company is still technically around. They've pivoted and changed kind of business models and been acquired, acquired by companies, and so it's it's still around in some form. I was with the company for about three and a half, four years of that journey. Um, so really in my mind, I my lens was looking at through this open source project, through the commercialization of the VA software. And that experiment, like I said, ran around three and a half, four years before this the new CEO at the time decided to kind of take a different tact.
One Mind Architecture And Scale
SPEAKER_01Okay, interesting. Yeah, I worked on my EMR for about four years. Um, we built one from scratch. One day I got a call out of the blue, I got an email out of the blue from a doctor, and he was like, Hey, I found your blog post. I had written a blog post on how to get uh a website stood up with AWS, elastic beanstalk, and Ruby on Rails. Um, and I guess he was impressed with how like thorough and clear and and whatnot my guide was. Um, and and he reached out and he's like, Hey, this is exactly what I'm trying to do, what your blog post spelled out. Can you like hold my hand and guide me through it or um just do it for me? I I think that's actually um he he wanted me to do that for him and then help him become a programmer. Um and I and I did that. It was just like a weekend project thing to get a production environment stood up for him on Elastic Beanstalk, which was a a technical choice that I later deeply regretted, but that's another story. And and we got away from that. Um But then after I did some work for him, um his exact words to me were, hey, you're actually not a fucking idiot. And I'm like, thanks. Um and and so he's like, how about how about you just do this this whole project? Instead of me learning how to code, um, you just do it for me. And so a weekend project turned into like a few weeks, uh, which turned into a few months. He was under the impression that it was gonna be a finite thing where we would build it, put it in place, and then it's it's finished, you know. And of course, I know that that's not how things go. Any if it's finished, that means it didn't work. Um you know, it's it's only uh anything that's successful is never gonna be finished. There's always gonna be more stuff that you're gonna want. Anyway, a weekend turned into a few weeks, turned into a few months, turned into a few years, and then after about four years, I ended up leaving. But that was a pretty successful project. Um we ended up switching from his uh he had like nine disparate computer systems, and we ended up condensing into a smaller number. There was some amount of just necessary complexity. Uh, like they're not gonna this web-based system was not gonna replace their uh OCT scan machines that takes the scan of your retina and stuff like that. There's just no way around that. But to the extent that was possible, we consolidated the um clinical, billing, administrative, and uh scheduling all into this piece of software. So that was one of the more um gratifying projects that I've worked on.
SPEAKER_00That's really wonderful. Yeah, and and there's just so much complexity to bear when it comes to health tech systems, especially when you think about billing and you think about kind of the provider interaction. Um, it's a lot of work, but really gratifying.
Product Design Sets Code Quality Ceiling
SPEAKER_01Yeah, and a big part of this, and and this is like a broader programming topic too, but this is like really rich soil for um for examples, um, is is like domain modeling. You can call it domain modeling, you could also just call it programming. Um because to me, um whenever whenever I step into a new code base, um the average code base is frankly not very good, as as basically we all know. Um, but but usually the problem is is not merely that the code is messy or something like that. Um it's that the code base is what you might call uh conceptually incoherent, at least to a degree. Um it's you know, in addition to the code not being super well organized, um the domain modeling is just off. Like it doesn't match reality or it's uh not self-consistent, that kind of thing. Um I assume that that's been your experience also. Um I'm I'm curious um to what extent you resonate with that sentiment. Um and also I'm curious is as far as the medical field goes, you have to take these things that go on in medical clinics and hospitals and stuff like that, and figure out how you're gonna model it and represent it in your software. Um that was a lot that I threw at you just now. Feel free to go in any direction you want.
SPEAKER_00Yeah, I mean, to start off with, uh I think you know, the strongest code bases I've seen have one brain that kind of manages the architecture. And not to say that that person designs all of it, but that person ultimately is consulted when it comes to what's the right structures for this? Uh, what's the right ways to um you know name an object, a phrase, um, you know, to connect objects. Um and when a code piece gets larger than one brain, I think it's really that's when you kind of should think about separating it either into discrete modules or into other applications.
Defining Shared Business Objects
SPEAKER_01Uh sorry, I want I want to pause you right there because what you just said is I I've never really heard it come up in conversation before. Um, but I have encountered it in the Mythical Man month, um, where he said, you know, a a system should be it should like come from one mind, um, which I don't think he necessarily meant like super literally, but like it needs to be cohesive and self-consistent and all that. Um and so to hear you say, to hear you say that is really interesting, and I think even a lot of people might find it controversial.
Planning Before Building
SPEAKER_00Um do you think so? Perhaps, yeah. I mean, I think, you know, and that's not to say that only one person can contribute. I think, you know, but it's really about establishing that shared mind, that shared vision, and having a person that that controls that or a couple people who control it. I think about my days at MedSphere, we had this gentleman, George Timpson, who's, I believe sadly, passed away, but George was uh I mean, it was called often the the mother of the the uh Vista codebase. And this is a gentleman who started in his 20s at the VA working in this code base, and essentially was responsible for a lot of the main architecture, and it was all in his brain. We were lucky to work with him uh as a consultant to try to figure out how we could approach building a commercial billing system in a code base that had never had that, right? The VA doesn't bill. Um, and so to do that, you had to have some pretty precise surgery on that code base. And it was so helpful to have George contributing to the code base that he helped write, and the his taste in here's add this field here, change this domain, you know, alter this. And so I think that's an example of where I've seen it done really, really well. Um I'll speak to other examples, not by name, but by structure. I mean, another uh company I worked for had a multimillion dollar, multimillion line code base that didn't fit into one single brain, and it was a bear. And in fact, that code base, unfortunately, is still a boat anchor for that company, and they still struggle because they have yet to be able to find that one human that can maintain multiple millions of lines of source code, you know, the concept of that in their brain. So I think that's really where, you know, and I'm not a huge microservices solve all problems person, but this is where, you know, at least breaking down a monolith into a couple of component parts makes sense at certain scales. That is like probably two or three or four hundred thousand lines, is my rough estimate on that.
Programming As Storytelling
SPEAKER_01Um talking about this stuff is is such a breath of fresh air because uh it's something that goes um unconsidered uh so much, which I find totally nuts because it's it's so important, um, and and so many other things are downstream of it. Um and I always remark on the fact that everything is connected to everything else, um and and your product design at the highest level kind of sets the ceiling for the quality of everything downstream of it, including the code. Um like the the product sets the ceiling on the quality of the code. And what I mean by that is if the product is designed in a way that's conceptually incoherent, then then the code will be conceptually incoherent, incoherent by necessity. And that can be really frustrating when that happens, because there's nothing you can do to there's nothing you can do to make the incoherence of the product design irrelevant. Um you can kind of maybe put a stop to it at some point and say, okay, at this level in the code, we're gonna depart from the product concepts and try to do something that makes more sense, but I've never really seen that that happen. Um it's uh you you're you're just dealing with the concepts in the product. Um but unfortunately the um developers don't often have a lot of influence over the design of the product, and the people designing the product um kind of they they they make a mental separation between product design and programming, as if they are two very separate disciplines and concerns and stuff like that, when really they are one unified thing.
SPEAKER_00Yeah, a hundred percent. And I think the best teams, and I'll construe a team as you know, a designer or multiple designers, a product manager, multiple product managers, and an engineer, multiple engineers, will will first, when they're thinking about a feature or an application, a product, they will come together on what the business object is. And that business object is not necessarily just a data model, but it is also representation and visual means. A business object has constraints in a product.
SPEAKER_01What do you mean by business object?
SPEAKER_00Yeah. Okay, I'll give you a great example. Um, you know, so I was lucky to work on I I believe the world's first um GPS tracker for dogs and cats uh when I was working at Whistle. Wait, wait, sorry, the the world's first GPS tracker. GPS tracker for dogs and cats.
SPEAKER_01For dogs and cats, okay.
Complexity: Essential vs Incidental
SPEAKER_00Yeah, yeah, yeah. So um lucky to work with a really amazing team um here in San Francisco. And whistle, we first start off by building activity monitors like a Fitbit for your dogs, and had a really great team. And the work that we did early on was okay, so we have this tracker. How do we represent a day's activity to the owner? Because we got a bunch of accelerometer data, we got some, you know, some uh orientation readings, but we want to make that meaningful for that owner. So the business object we came to is a daily, uh representation of that day activity for that dog and everything around that day that happens. So that daily had a number of activity minutes, it had a goal, that day had a bunch of activities uh structured in there. And so that discussion then came into well, how do we represent that and show that? What constraints does that daily have? And how does that map into the the you know object uh uh of the code, right? How do I what's the the model? What's the view structure on that, et cetera? And so anyway, coming into the the exploration of that business object that started off with some sketches, what's the end result going to look like, and then how do we model that and what are the constraints on that? And so ultimately we came into this shared understanding of what that business object was. And that's what about that business object because it's not just one team's responsibility, it's a shared responsibility to create that object.
Rethinking Technical Debt
SPEAKER_01That is so interesting. I've I've never heard of that concept of a business object before. Um, and it speaks to something else which is so tragically rare, which is uh I'm just gonna say it the cynical negative way that I always put it, which is um deciding what to do before you do it. Um and I preach that so hard to so many people, and and sadly, it's like it's so hard to get that across. It's like, hey, before we just start doing stuff, we should like take a second and invest some hours um so that we don't waste months. Uh people are so impatient to to get going or or to do what they see as getting going. Um it's like, hey, let's not uh let's not let ourselves be held up by incomplete information and blah blah blah. It's like no no. Like uh your way we waste like three months doing something that may or may not even be a good direction to go in. Um this way we invest like I don't know, four hours in just like figuring out what we want to do with the next few months. Um and and this planning work is work. It's not like we're just uh just because we're not typing doesn't mean that we're not doing valuable work. In fact, this is much more valuable work because this determines the direction we're gonna go in in the next um few months. And I also want to comment that the software industry, as I'm sure you've observed, has had this big backlash against waterfall and big design up front. Um, but the pendulum has perhaps swung too far uh and and and now it's more like no design up front. But just because you know there there's there's a lot in between big design up front and no design up front, uh it doesn't mean no planning at all. It's it's still smart to do planning. That doesn't mean you have to plan out every class that you're gonna have in your code base on day one. That's not the idea. But having, like you said, this idea of a business object and deciding what you're gonna do and spending some careful thought planning that, that is an investment that can have a huge payoff.
SPEAKER_00Yeah, no doubt. And I think engineers shortchange themselves by saying, Oh, just tell me the requirements and I'll go build it. Well, you know, requirements come in a lot of different flavors. I mean, as you said, in Waterfall, there's a you know 25-page requirements document that gets rubber stamped and approved and has version tracking in it. Uh, and I've certainly worked in environments like that. That's not a great way to do it. Um, I've also seen uh environments where requirements are a single line of text that doesn't really describe at all what we want. What does good look like? Uh what will success be when this thing is launched? And so I think there's uh uh an important thing for engineers to do, which is you know help the business, whether it's the product manager, the stakeholder, um, understand what you're actually asking for. Um and in many ways, it's a shared understanding of the outcome of the constraints. And that can get described in a lot of ways, but ultimately it's a lot more detailed than a single line in a ticket, and it's a lot less detailed than a 25-page requirements document.
Entropy, Blades, And Maintenance
SPEAKER_01Yeah, indeed. Um I I'm just trying to think. Um there's there's some interesting questions in here. I'm gonna zoom out really far and ask you a really big question. Uh apologies for putting you on the spot, but I I think you'll do fine with the question. Um, what is programming?
SPEAKER_00Uh I'll give you a similar uh quick response. Uh, it's storytelling. And I'm gonna expand on that for a second. Um, you know, look, as humans, we are communicators. We communicate verbally, nonverbally, we communicate by our actions. Engineers are trying to tell a story to a computer uh of what it should, you know, what an outcome should be. And in order to be a storyteller, you have to communicate in a variety of forms. Obviously, code is one of them, but documentation is another. Hey, communicate with your product manager, with your stakeholder. Um, and storytelling requires some understanding up front. So, how do I go to my stakeholder, understand their problems? It's not just a product responsibility, it's an engineering responsibility too. Because a product manager is thinking about a big picture, about an outcome or an end result. But um, if you don't, as an engineer, understand the details, the why behind, when you make a small decision that you wouldn't necessarily go to a product for, um, that small decision also needs to be rooted in what the customer needs. What's the outcome, what's going to make everyone happy. And so I think as good storytellers, we tell a story that um should be captivating, it should have an outcome that is positive or you know, moves some objective. Um, and and and you know, and we also communicate to our future selves that we're reading this code. So it's important that that code is you know well written and commented in the right places and and thoughtfully designed.
Risk Profiles And Smarter Bets
Safe Bets vs Shiny Patterns
SPEAKER_01Wow. Okay. Man, this conversation is is really something because you're touching on a lot of things that like really speak to my heart, but haven't haven't come up in many or even any conversations that I've ever had in the past, including the like 300 plus conversations on this podcast. Um storytelling, yeah. I've so I've I've thought about this. People make metaphors so much about uh construction and software. We're building the house, we're gonna remodel the bathroom, whatever. And that analogy can work. Um there are certain things that are in fact analogous, and there's other things that are that are very much not analogous. Um and and it kind of does harm to the endeavor um if if you try to use these unfitting analogies where they don't really work. Um because then you get a distorted picture of reality. Um what to me, what it's more like is building a uh a product uh and a software code base and all that is um it's more like writing a novel. Um because like the whole thing has to be I I have been using this word so much, but coherent and consistent and stuff like that. Like, for example, if you take uh Harry Potter and like in chapter one he's in an accident and his and his legs get chopped off. Well, now you have to go and update a whole bunch of parts of the novel or else it's not gonna make sense anymore. Any part where he's running or anything like that, you're gonna have to change that. Um and it's it's not so much like that in a house, because like if you're building a house and you like want to some things are interdependent, obviously, but like I don't know, you can like remodel certain rooms without touching anything else, or you can put an addition on the house and stuff like that, and it's not really the same. And and so I'm glad you said storytelling because I think that is a lot more fitting, and uh there there's the aspect of needing to read it and understand it. Um it's hard to paint a clear picture of how that happens with like a building. You do need to like read and understand buildings, that is true, but maybe not in such a self-evident way as a story.
SPEAKER_00Well, first metaphors are hard. And oftentimes storytelling involves the creation of metaphors and the the abandoning of them as well. Um, I think I like the storytelling aspect because I think a lot of engineers see the code they write as secondary to the outcome, but I think it's just as important as the outcome. Again, it's like, is this code getting maintained into the future? Do you need able to read it in six months and know what the heck you were thinking? You know, uh, is everyone on on the team telling the same story? Or do you have some folks in the corner telling a story with tabs and some folks in the corner telling the tab, a story with spaces? That's a silly example, but you know, using different terminology and metaphors that make it hard, right? I think I told an early engineer years ago that the best testament of their work was that I didn't know it was their work. And not to say that I, you know, that that it was a shame that they didn't see their fingerprints on the code base, but that the code base continued to be cohesive with their contributions. We're all telling the same story, right? Um, you know, if it's written by, you know, Michael Crichton, I don't really want to see some pages from Stephen King in that same novel. That's weird, you know.
Hedging Risk And When To Stop
SPEAKER_01Right. That makes sense. Um, and and I think maybe that speaks to the idea of a code base being the product of one mind. Um and man, there's so many things. And again, everything is connected to everything else, and like business choices and technical choices aren't two separate things, they're the same thing. Um like uh the the the problem with so many code bases, it's because of the business choices that were made, and it's like, well, how do you avoid these technical problems? Well, just make different business choices. Like I I think um the 37 Signals people really get this because they've made choices to keep their products small and as simple as possible and stuff like that. And I get the sense that they've made these choices just as much uh as just as much out of a concern for technical simplicity as business simplicity and stuff like that. Because once you get complicated and you have to start delegating things and you have a bunch of different product managers and stuff like that, uh things the quality gets diluted in a hurry.
How A Fractional CTO Finds Clients
SPEAKER_00Well, I would say that complexity is either a sign that it's something you shouldn't be doing, or a sign that that's where the business value is, right? Like, you know, we take out complex things um because we can can make a business out of it. But sometimes if that's not the core ethos or the main problem to solve for the company, maybe that's a sign that you're taking the wrong approach. I think you know, Rails does this well. Um, that, you know, when you are going too far off piste outside of a common Rails pattern, maybe you should reevaluate and see if there's a pattern you can pull from that uh makes it easier, more straightforward, cleaner code. You know, I think we talk a lot about code smells. I think complexity can be a code smell, unless it's part of the core business in which you say, okay, we're going to accept this particular code smell.
Where To Learn More
SPEAKER_01Yeah. Um, I think a lot about incidental complexity and essential complexity. Um, the complexity that exists in the world, because the world is a complex place and there's irreducible complexity there. But then every code base has a whole lot of incidental complexity, complexity that does not need to be there. Um, but it's there because um I don't know, you know, our our first attempts at anything are gonna be suboptimal. And it's it's often the case that our first attempts at something um are more complex than they need to be, and we only realize in hindset, in hindsight, a more simple way that that could have been. Um and there's just uh over-engineering. Yes. Um there's there's this like psychological uh disease that people have where for some reason they want to make things more complicated than they need to be. And it's so frustrating. Um and I'll mention one more way that complexity sneaks in there. Um this is one of my this this is one of my pet peeves, is uh speculative coding, where there's there's something that's not really a requirement, um, but but code gets put in there just in case. Um AI loves to do this. Oh yeah. I'll I'll say write me a method that that does such and such, and it'll like set the default value of a parameter to nil. And it's like, why did you do that? Like the the parameter will literally never be nil. Um and and so that just adds another path through the code. It adds cost for no benefit. Um, and and it is just a rampant mistake. I see it constantly.
SPEAKER_00Yeah. And I think you know, there's a couple of things I'll say there. Um the first thing that I was thinking about when we were talking is that I think we we talk about technical debt almost like it's something that we always have to pay down and something that's that that was the engineer's fault. And I think we don't do short short trip to ourselves in both things. One, technical debt may be intentional, right? We took on some debt because we're going to do this really quickly so we can learn, in which case, uh hardening it is a feature, not debt. Um, it can often be because uh we didn't understand the scale that we needed to build the system to or a module to, in which case it's a learning that we should actually ask the stakeholder and write it down, okay, how many users do you expect to be hitting the site in this period of time? You know, what kind of traffic do we expect, what kind of traffic patterns? And then if we grow outside of those, that's also not technical debt. That's just a feature. Hey, feature is scale from 1,000 users to 2,000 users. And so if we can think about communicating that differently, we can show a different type of value externally to the rest of the company, to our stakeholders. Hey, this isn't debt, this isn't a bunch of engineering mistakes. This is because our site crew. That's awesome. This is because we decided intentionally to do this one quickly so that we could learn, and then we were gonna invest in it later. So I think I talk a lot about like how to reframe technical debt by how we speak about it and the value with which we identify the rest of the organization.
SPEAKER_01Man, I have so many strong opinions about that whole topic. Um, but I just realized something. Um okay, so I've been thinking a lot lately about entropy and how it relates to software. So, and I think I talked about this in the last few episodes also, but entropy is the principle that the level of disorder in a system never decreases over time. Um and some people, I mean, a lot of people, myself included, um, think of entropy as a negative thing. You know, I have to mow the lawn and snowblow and uh trim trees. Like like entropy is constant I I have to clean my office because it gets dirty. Um there's all this maintenance work that I have to do just to stave off entropy, and I'd rather spend my time doing something more productive. Um, but I read this thing from Sean Carroll some years ago. Um, and he said that you know, from the from the Big Bang to the heat death of the universe, there is a wave of entropy. And this wave of entropy, um, from the you know, you have the disorder at the start. Well, okay, he used the metaphor of a cup of coffee. Um, you have some coffee at one layer and then a layer of cream on top, and at the beginning you have this uh certain configuration where it's like orderly in a way, and then it all mixed together, mixes together, and there's all these swirls and stuff like that, and then at the end you just have this light brown liquid, it's all uniform, and nothing interesting is happening anymore. And so that's what's gonna happen to the univer universe. At the end is just gonna be this uh kind of uniform stillness and darkness, and nothing interesting will ever happen again. Um but in between there's a wave of entropy, and that's where things happen. And I thought about it, like entropy is the only reason there's only there there's there's ever any uh work to be done. Um if people never had to eat or fight or find mates and reproduce and stuff like that, we could all just like sit around and we wouldn't need to do anything. Uh there would be no point in life or meaning in life or anything. There'd be nothing to do. Um anyway, all this led me to think about um software and entropy. And software a a software system is like a bubble of order in a sea of disorder, and uh the the system is constantly being bombarded by disorder, and we have to do work, like an organism has to do work uh in order to maintain the order of its own body. Um we take in uh food and we excrete waste and heat and all this stuff. We export the disorder to outside our bodies so that we can maintain the order inside our bodies. And a software system is the same exact way. We have to do all this really hard work constantly just to stave off entropy. Um, all that is to say the thing I just realized is I think technical debt is entropy, uh or it comes from entropy. Because it's it's I really uh am uncomfortable with the technical debt metaphor because um unlike it's it's one of these ill-fitting metaphors, I think. Because real debt, financial debt, you can just decide not to have it. Um but technical debt, you can't just say, you know what? No, let's let's not have technical debt on this project. Like it it happens no matter what. And I think it happens because of entropy, and you you have to keep you you you have to keep running in place and and paying it down just in order to um to stay at the same level. Anyway, I have so much more to say, but I'm gonna stop there because that was like a big monologue.
SPEAKER_00No, that's fascinating. And I think you know the the bits that that really resonated for me is you know, not only is the code base trying to find order with its inputs and outputs, right? Taking an unstructured bit of data, turning it into business object, running some workflow on that object, exporting that data somewhere else, but the code base itself is managing that as well. The entropy of new updates and version dependencies, uh security vulnerabilities, platform changes, the team changes, that this is the the I mean, this may be getting too idealistic, but the code base is the place where there is harmony in the universe and everywhere else is chaos.
SPEAKER_01Yeah. Yeah, I mean, at least in a relative way. Um just like an organism, you know, organisms aren't perfect, um, and they often break down, and the ultimate fate of every organism is to eventually cease to function. Um, just like with every software project, that's the ultimate fate of everything. Um, so it's it's not perfect, but it's at least more orderly on the inside than it is on the outside. So the the metaphor that I prefer is one of blades. Um we have sharper and duller blades, and what I like about that is blades are subject to entropy, they get duller with use. And it kind of makes clear that like not all the blades have to be sharp. Um like if there's a blade which uh hasn't been used in months and you don't know when it's gonna be used again, uh it's it's it's not a very sound investment to spend time sharpening that blade in particular. But if you pick up a blade that you're about to cut with and you see that it's dull, or you try to cut with it, and you're like, crap, this is uh not going very quickly, then you have pretty clear justification for taking a minute and sharpening that blade before you cut with it. So I'm I've been on a campaign to try to get the world to switch from this technical debt metaphor to this blade metaphor.
SPEAKER_00Well, anything that we can do to get beyond debt, because I think it's the wrong one, and I think it's often weaponized against engineering teams. Well, they created so much debt, we shouldn't give them the time to go do that. The business needs to move on. Um, you know, changing the phraseology around that inside of a company is really, really important.
SPEAKER_01Yeah, that's a good point. And and it's maybe worth touching on that tension between the business concerns and the technical concerns, or at least the perceived well, there's there's the perceived um differences of interest and there's real tension. Like that's that's definitely true. Um and okay, one one one thing in that area that I really am bothered by is the idea of strategically taking on technical debt. Because there's never ever a plan to pay back that technical debt. It just uh uh it it it approximately never happens. You know, like 99% of the time, the payback doesn't happen. I don't think I've ever seen it in my life. I don't want to say it never happens, but I have never seen it. Um and and so to leave that that gap open where you can get a wedge in and say, let's strategically take on this debt, it really bugs me.
SPEAKER_00I think that's fair, and I think that's true. Um, I'd offer, and this is you know my you know CTO hat that I'm wearing right now. Um we might make a choice to move quickly in an area and not gold plate something because we're not really sure what part to gold plate. And so because of that, we will take on debt because, oh shoot, we need to re-engineer a portion because we didn't gold plate it here. And um, and that's an important thing for us to be able to do as a business so that we're not spending a lot of money, running out of money, running out of time, our competitors are eating on our lunch. Um, this, of course, can be weaponized, and the wrong leader can push an engineering team beyond beyond a point where uh the code is no longer maintainable or where um the debt's too high. Um, but in general, I think you know, uh a good to great leader, business leader or technical leader, will try to manage that uh on a relative basis to make sure that we're still um keeping up with the market um and not gonna close down because it's it's no no fun if we've gold played into this whole application and before we can launch it, the company goes under. And we have to layer everyone off, right? That's not great.
SPEAKER_01Right. Yeah, that raises an interesting question. Um okay, so I I think to tell me if I'm understanding part of your premise correctly. Um, in this premise, there's maybe a business direction to go to that is not a 100% sure bet. It's to some degree experimental. And so we need to make an investment in building out part of the software so that we can go in this direction, and then there may or may not be a payoff, or the payoff will be uh big or medium or or whatever. We don't know. And so we want to be prudent about the nature of the investment that we make in light of the uncertain nature of the payoff. Am I understanding that right? That's exactly right.
SPEAKER_00Yeah, and I think you know, if you think about it uh back into the building metaphor, uh, we start by building a one-story house, and then when the family grows, we try to build a second story. And sometimes that means you have to actually redo the foundation. Sometimes you have to put in steel beams to do a second story, and then a third story and a fourth story. But if we build the five-story house uh with you know a single person, that doesn't make a lot of sense. And so it's about trying to pick the right scale for the right time. And again, I think my point around framing is rather than think about that, you know, pouring a new foundation to support the second story as technical debt and think of it as a feature, you're gonna get the the organization to understand it better and say yes to it far more often, right? It's not the engineer's fault that we decided to pick uh to build a one-story uh house first. It was a business's choice. And so the business should also choose to build the foundations to allow us to lay that second floor.
SPEAKER_01Yeah, interesting. Yeah, these are complex, nuanced decisions, and it's almost too complex of a thing to talk about in an abstract way because a lot of these things have to be taken on a case-by-case basis. That's right. Um what I've seen a lot is this, and and this is something that I've been working on clarifying in my head more recently. Um it's almost as though uh every, let's say every week, um, you know, whatever you think the smallest uh unit of time is for a for a team, um you know, every week you get to do something. Um you're given this gift of time, and you can spend that uh uh gift of time in whatever way you choose. Um and so I think of that as like you're getting a coin every week. And you can spend that coin any way you choose, but you get the one coin, and if you don't spend the coin, it g goes uh falls into a crack and a drain, and you lose the coin forever. Um you you can't save up your coins. Um and what I unfortunately see so much over the last 20 years is people spending their coins as though they have a million of them. Um when all you get is one a week, and so you have to be really thoughtful about what you spend that coin on. Um and something of that I've been been trying to think of, you know, I I read that book, The Goal, recently, and so that got me thinking about value streams and all that stuff. And I'm trying to think of like how would I uh form a convincing argument around spending your coin on this versus that, and like a general strategy for choosing how to spend your budget. Um and here's my half-baked thought, and I'm curious your thought on this. Okay, um different investments have different costs and different expected payoffs. Um and I think there's no such thing as an absolute guaranteed bet, but there's sure bets and less sure bets. Um and and so you could have something where there's uh a 1% chance of getting a thousand X return on your investment or a 10% chance of doubling your investment, whatever. There's there's all these the significant thing there is uh what kind of payback do I expect and what's the wait what's the size of the payoff and what's the likelihood of the payoff? Those are the two things I think. Um and if you repeatedly make very sound bets, like there's a 99% chance I'll get a 2x return, that's a pretty good bet. Um and if I make that bet over and over, in fact, let me make it even simpler, there's a 100% chance uh I'll get a 2x return. And I make that bet over and over, that keeps compounding, and if you make like a graph of that, um that graph is gonna go up in in a really dramatic way. But then if you make um uh if if you make an investment every week where it's like there's a hundred percent chance of getting just a ten percent payback, then that line is gonna go up much more slowly. Um and so it really matters how you prioritize these things. And if you it sounds like I'm stating the obvious, which I guess I am, but this doesn't seem to be obvious to everybody. People prioritize risky bets over safe ones.
SPEAKER_00And I think businesses often have a risk thesis that is in the head of their CEO or their founder. That's not often communicated. And as organizations grow, you might take different bets with different teams. So I might say, hey, I've got a 100-person engineering team. With one of the teams, I'm gonna take a bet that that has, if it pays off, it's a 10x bet, but doesn't pay off that that effort is useless when the vast majority of my team are not taking as big a bet. They're taking that, you know, that single versus that home run. And I can diversify my bets across the team in a different way. But it also depends on that, you know, the risk profile of that CEO. You know, the CEOs that are very famous for doing wrong often had the wrong and misaligned risk profiles. But their investors were often saying, Yeah, yeah, yeah, double down. We're gonna go hard on this one. You know, yes, announce the product before we've even had a chance to get a proof of concept, you know? So it's really about that risk profile and how you want to tune it. It's the same with your own personal risks, your finances, etc.
SPEAKER_01I I agree with all that. And there's there's so many layers of of this. Um, I was actually, I didn't say it out loud, but in in my head, I was thinking of an even lower layer of like, let's refactor this piece of code or or something like that, where it's like um doing working in this area of the code. It's like, okay, maybe that would have some positive. Okay, here's a better concrete example. Like trying out some new design pattern you heard about last week. And let's take this code and redesign it to use this design pattern, instead of doing this other more boring thing where we have this area we know is a disaster, we know exactly what needs to happen to make it more clear, but instead of doing I I call it the difference between like working for a paycheck and playing the lottery. Um, like instead of digging a ditch to get a thousand bucks, people buy these five dollar scratch-offs. Sure. Uh, yeah, that's that's the behavior that I see a lot. And maybe my only purpose in saying this is just to gripe.
SPEAKER_00I mean, with engineering, I think you've got to take some of those bets. And you know, I'll go back to Google's 20% time when that was an actual thing. Um, that allowed the company to take very risky bets without affecting their core business. They just said, off the top, we're gonna take 20% of bench effort and we're gonna put it in this bucket, we're not gonna worry about it, right? But that allowed engineers to take tool bets, to take product bets, um, and not worry about the outcome. If it failed, okay, fine. It's 20%, no big deal. And so I do think as a advice to engineering leaders, try to think about the ways that you can allow your teams time to take those bets. Hey, let's try a new pattern, new framework. And if it fails, it's fine. No worries, move on to the next thing. But we've learned something through that, right? Like we've we've we've got the education of that. And so I always feel like the bets that fail, while there's a tendency as humans to try to bury them, hide them, not talk about them, just move forward. Didn't happen. We should actually be sharing those, what happened to the rest of the company. Hey, we learned something here. It cost us some money. We paid for an education. We learned that pattern is not a great fit for this thing, but maybe it's great for something else. And maybe there's lessons in that that we can all learn.
SPEAKER_01Yeah, that's a good point. The the distinction I would make there is the distinction between a conscious, uh risky bet and uh I don't know the other way I would put it. Making a risky bet when you think you're making a safe bet.
SPEAKER_00Yeah, with intent versus not intent.
SPEAKER_01Yeah, and and I still didn't say it quite the way I should. It's like I don't know, when you're just doing your like day-to-day work and trying to create value, I think that safe bets should be prioritized. At the same time, I totally agree that like there is value in making those risky bets, uh, as as long as you're not mixing up the risky bets with the safe bets.
SPEAKER_00Yeah. It's about understanding um the the downside if it fails. Right, it's like going to Vegas, right? I will, if I go to Vegas, which is very rarely don't like uh the cigarette smell so much, but if I ever gamble, I'll allocate a couple hundred bucks. And when I lose all that money, which it will always happen, I stop playing. Right? So I've hedged the risk, I've I've mitigated the risk, I've limited the damage of the risk. And so if you're able to be eyes wide open about the risks you're taking, know the downside, know the upside, and be willing to pull it when it's not working, then it's a good risk to take.
SPEAKER_01Yeah, I totally agree. Um, I wish we had a lot more time because there's so many things I want to dig deeper into with you. Um, but uh we're we're getting close to out of time. So um before we go, uh I want to spend at least two minutes, even though I wish we could spend two hours, um, on the business side of what you do. Um you work as a fractional CTO. Um, and the question that I ask everybody who does um service work, uh consulting, freelancing, that kind of thing, is how do you get your clients?
SPEAKER_00I'm very lucky to have done, you know, kind of engineering work for 20 plus years and built a great client network through just friends, relationships, colleagues. And so luckily it's all through word of mouth. Um I'm very unique to be in this position and very happy to be in this position. But um, yeah, I also do a bunch of writing on my Substack, uh WTF dot com. Uh, and I write a lot about AI strategy, how uh business leaders should think about AI. Uh we just did a survey of 15 companies from uh series A all the way to public uh and kind of come up with a bunch of lessons and learnings on how to implement AI correctly in your organization. And so a lot of my business comes through that work as well.
SPEAKER_01All right, and that's probably a good transition into my final question, which is where can people go to learn more about you and what you're up to?
SPEAKER_00Uh Oakenco.io or w tfdai.com.
SPEAKER_01All right. Well, Brad, thanks so much for coming on the show.
SPEAKER_00Thank you very much, Jason.