Code with Jason
Code with Jason
311 - Tyler Ewing, Creator of Ductwork
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode I talk with Tyler Ewing about Ductwork, his innovative workflow framework for Ruby and Rails. We explore its unique position between background job libraries like Sidekiq and heavier solutions like Temporal, and how Ductwork aims to streamline complex workflows with a focus on durability and usability.
Links:
A Snail Mail Newsletter For Developers
SPEAKER_02Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Email newsletters are a really nice way to keep on top of what's going on in the programming world. Except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more. That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right, I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like? You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming. The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary. I'm sure that's what you were going for, so just wanted to let you know that you succeeded. Looking forward to more. Drew Bragg from Philadelphia says Nonsense Monthly is the only newsletter I deliberately set aside time to read. I read a lot of great newsletters, but there's just something about receiving a piece of mail, physically opening it, and sitting down to read it on paper that is just so awesome. It feels like a lost luxury. Chris Sonnier from Dickinson, Texas, says just finished reading my first nonsense monthly snail mail newsletter and truly enjoyed it. Something about holding a physical piece of paper that just feels good. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsense monthly dot com. That's nonsensemonthly.com. I'll say it one more time. Nonsense Monthly dot com. And now without further ado, here is today's episode.
SPEAKER_03Hey, thank you for having me on. Excited to be here.
SPEAKER_02Yeah, um, so you created this thing called Ductwork. Tell us about that.
SPEAKER_03Yeah. Um Ductwork is something I've been working on for the last couple of months. Um it's basically just a pipeline or workflow framework. Um so you could think of like Sidekick or any of the other background job work, you know, background job libraries that exist for Ruby and Rails. Um, but it kind of takes things a step further, allows you to um like chain steps together, allows you to run things in uh concurrently, um, which we can get into like the actual concurrency model and how things run later, but um basically allows you to define just kind of like a workflow or pipeline using a declarative DSL. Um and then you just kind of trigger it and it runs. Um so yeah.
SPEAKER_02And what relationship, if any, or not relationship, but like analogy or similarity use case, whatever, uh, does this share with like things like GitLab pipelines, GitHub action, stuff like that?
Comparing To GitHub Actions And CI
Avoiding Heavy Workflow Engines
Real-World Use Cases For Workflows
SPEAKER_03Yeah, that's a good question. So you could set up um like your own CI with this if you really wanted to. Um those purpose-built, well, I I've never actually used GitHub or sorry, GitLab uh workflows before. Um, but I've used GitHub Actions quite a bit. GitHub Actions is pretty constrained in the sense that you have to like you know piece together um other libraries and uh what do they call them? I I guess just actions, right? Um and that's kind of like how you would run your CI or your pipeline or you know, build pipeline, whatever it is. Um this is I mean, this is you know, like any other framework, you just kind of like write your own code, you write your own steps, um, and then define your pipeline, you know, define those steps and how they run in what order, and again, concurrently, stuff like that. Um, so there's some similarities, but it wasn't exactly like the use case I was necessarily going for. Um, I was kind of targeting like a very generic use case because what really inspired this was I've used things like Apache Beam, um, Commandazeb, um and other just miscellaneous workflow engines. I've even played around with like Temporal I.O. Um, and they all just felt really heavyweight. Um so you can go kind of like the other direction and like build uh your pipeliner workflows with like side cheek batches or other um uh other things like now there's active job continuations, which is kind of kind of provides like step functionality. Um and I kind of wanted to land somewhere in the middle where it's like you could still build really powerful pipelines with like the more mature technologies, um, but it wasn't so clunky and so heavyweight that you needed to like stand up all of this extra infrastructure and or like learn BPMN and all of this other stuff. Uh BPMN uh for is just a modeling language. Um but yeah, so I kind of wanted to land in the middle, um, but still like you know, tackle real use cases like data pipelines or email campaigns, onboarding flows, order fulfillment, data migrations, things like that, stuff that like you run into on the day-to-day basis, um, that you'll probably reach for like a sidekick job for. I keep saying sidekick, but like any of the other job cues, um, but is just a little bit more complex than that. Um, but then again, not having to go to like the really big uh heavyweight enterprise, if you will, technologies.
Frameworks, Universality, And Rails Inspiration
Abstraction, Leverage, And Bad Platform Fit
SPEAKER_02Yeah. Um one reason this is interesting to me is like you said something like it's it's like a framework in the sense that you just write your own code. Um and this is something that really frustrates me about like um GitLab pipelines, GitHub Actions, any of these tools. And I know that it's like not exactly the same use case and stuff like that, but there's this general principle that that I like put my finger on at one point with these pipeline tools, um which is universality. So like if you're working with uh GitHub Actions, for example, there is your code, and your code is universal. You can do anything at all, like anything that code can do, you can write a program to do it. Um but then you have to like put it inside of this box, this GitHub Actions box, and then it's not really like completely yours anymore. Um you're like separated from it. Um and and you no longer have universality. There's there is universality, there's universality in the picture, but there's this membrane between you and the universality. Um whereas with, for example, Rails, it's the other way around. Um there's the framework, but then your custom code, your universal code, sits between you and the framework. The framework doesn't sit between you and your universal code. And that is a huge distinction, which has it makes all the difference in the world. And it it seems to me like you know, when I just looked at the snippets on the homepage for ductwork, um it gave me the impression that you you get it. Like you want to be able to have universal behavior and and not depend on the framework to like do the limited number of things that it could do.
Business Model And Open Core Strategy
SPEAKER_03Yeah, exactly. I didn't want to try to wrap existing functionality or anything like that. I won't yeah, like you said, it's just code, code can do anything. Um so yeah, very much like that. I would say it's kind of it's taken some inspiration from Rails in the sense that the framework doesn't want to get in the way. Or I I try to make the framework not get in the way. You're not, I mean, I know sometimes we can fight with the Rails framework quite a bit. Um, but at least with Rails, it always felt natural to me, like, oh, okay, cool. I just have a controller. Oh, I'll just make another model, and then like I just add my code to do the business logic, and everything just kind of works. Um and all of the I mean it's not exactly orchestration, but like the server and all of that sort of stuff is like taken care of for you, and you don't have to you don't have to really dig in there, like dig underneath the hood in the engine and um mess with that unless you really need to. Um so yeah, thinking about just like having the framework handle that orchestration for you, you don't have to do a lot of plumbing code. You you have to really just focus on the actual business use case or whatever your use case is and and uh implement that logic.
Early Customers And Feedback Loops
SPEAKER_02Yeah. Yeah, there's there's like two things that you want out of a framework, um, which are abstraction and leverage, and they're they're overlapping concepts, but like Rails abstracts away a lot of details. Um, like in one sense, like with scaffold generation, for example, um, it abstracts away work. Like you can just run a command and get all the files that you might have created by hand anyway. Um and you you still end up with the same stuff, like you're not spared any of those details, but you are spared the labor. Um so that's one way that you get leverage and abstraction. Um, and another way is like uh an action controller, for example, inherits from action controller base or whatever it is, um, and it abstracts away the the guts, uh, it abstracts away the the Rails code itself, and so you get that leverage. Uh you you don't have to think about those uh detailed internals, which you're probably never gonna have a reason to think about anyway. So you never have to look at that stuff, it only gives you the stuff that you're gonna want. And that makes a lot of sense. It's it's unlike uh I often pick on uh elastic beanstalk, which which gives you like the worst of both worlds. Um you're greatly limited, like you you're boxed in, you can only do what they allow you to do. Um and you don't really get much leverage either. And so it's just like a really unhelpful technology. Um unlike uh I don't know, uh unlike Rails, where for the reasons I just described, um, you do get that leverage.
SPEAKER_03Yeah, totally. Um yeah, I haven't used Elastic Bean stock in a while. It's fairly old, right?
SPEAKER_02Yeah.
SPEAKER_03Can I think it up the right thing?
SPEAKER_02Okay, cool. Um yeah, it's been a while. Yeah, it's old and it's time for it to die.
SPEAKER_03Uh classic AWS though, they never they'll support anything for a long time. So they probably have given them money.
Finding The Ideal Customer Profile
SPEAKER_02Yeah, there's probably enough people on it where they're uh you know committed to it, whether they like it or not. True. Yeah. Um, so I'm I'm curious. Yeah, go ahead.
SPEAKER_03I was just gonna say I like that uh leverage and abstraction uh for frameworks. Yeah, I uh that's a nice framing uh for thinking about it. Um yeah, sorry, that was just kind of a thought.
SPEAKER_02Yeah, um I'm curious about ductwork as a project and uh the business endeavor and stuff like that. I I did immediately notice that you have pricing and stuff, and when you said that you've been working on it for a matter of months rather than years or something like that, I was kind of surprised because immediately once I looked at your website, I'm like, oh, this is like something that clearly a lot of work has gone into. And so this is um probably something that's been around for a while and has a lot of customers and stuff like that, but then apparently it hasn't been around all that long. So I'm I'm curious about the the story of it as a as a business and the life cycle and stuff.
Early Go-To-Market Channels
Cutting Through Bad Business Advice
SPEAKER_03Yeah, yeah. Um maybe I should have been more vague and really uh take it till you make it sort of thing, like, yeah, it's been around for a long time. No, just kidding. Um, yeah, so I mean I've had the idea for a while. Um, again, just kind of like having worked at various companies and stuff, like I felt like I was building similar plumbing, um, similar job plumbing, workflow plumbing, um, or again, reaching for some of the more enterprise-like uh technologies. But I'd had the idea and kind of like toyed with it, you know, like any side project you build a little bit and then it sits on the shelf and then you pick it back up and and you try to like hunch it together again because it's a few months old and you try to update dependencies and all that stuff. So, but I really worked on it in earnest uh at the end of 25. But um I think it I think it feels like I I put a lot of thought into it. So maybe that's why it feels like it's like a more mature project, because like there's a lot of decisions I had kind of already made um going into it when I was building in earnest. Um but just just through you know the course of my career, it felt and it was like a really good opportunity to kind of take the jump and see if I could make some money off of this. Um and you know, like actually build it out proper, you know, provide real value for for companies doing similar things. Um so uh I keep I I know I've mentioned Sidekick quite a bit. I kind of was using their business model or their their being Mike, uh Mike's business model of like that open core. Um so you know, your your base tier or whatever you want to call it is just an open source project um that everything is kind of built around. And then, you know, like yeah, this pro uh the professional tier, the pro offering, um really just layers on extra functionality, um, changes, overrides certain things. Uh, for instance, they're like enhanced job durability, enhanced job or step durability, um, which I can go into a little more detail on. Um, but yeah, just like layers on extra stuff, more features, more uh more advanced things. Again, kind of modeling a lot of things after sidekick and like stuff that they've they've done and have found to be successful. Um so yeah, that's that's kind of where I was like standing up the business as a straight business model. You know, it's a per month or per year pricing. So um, you know, like that subscription model is there. But again, it's not just like you're receiving updates to the software. I'm also like willing to do a lot of support and design help and all that sort of thing. I I really want to be engaged, um, especially because the project is still pretty young.
SPEAKER_02I imagine that helps you just as much as it helps the person you're working with.
SPEAKER_03Yeah, yeah. As like it's building is by yourself is so interesting because like you're just kind of in your own echo chamber, and you're like, you know, that the whole time I was like, Am I even building the right thing? Like, is this what people want? You know, like if I'm trying to get quick customer feedback, but also like build out in a in a in a direction, right? Like you got to pick a direction and go and make some decisions. Um but um yeah, it was it's it's a good, it's it's it's fun, it's a lot of fun uh building in that in that way.
SPEAKER_02Do you have customers yet? Are you still working on getting your uh first paying customer? Where are you at in terms of that?
Psychology As The Backbone Of Marketing
SPEAKER_03Yeah, I got a couple customers, um, which is exciting. Uh a number of my people in my network uh are using the open source version just for like pet projects and side projects and things like that. Um I actually got some like good feedback from from from that, from them, um, which helped me kind of you know know what to build next, not just like here's some ideas I've had, I'm gonna like throw it in there. Um for context, my wife is a PM, so I'm very like very laser focused on like okay, get customer feedback, respond to them, what do they need, you know, and then iterate that way rather than just well, it's my project, I can add anything I want to it, you know?
How Buyers Really Choose CI Tools
SPEAKER_02Yeah. Um yeah. Yeah, it's um it's it's a tough thing. Um, so I don't know how much you might be aware, but I'm working on my own product slash business called Saturn CI. It's a CI platform, uh, continuous integration. Um, and I'm at the early stages uh from a business perspective. I have one paying customer right now, and there's another customer that I've been onboarding for like months. They're they're relatively bigger compared to the first customer. Um and so there's been a lot of like technical work that has needed to be done, and sometimes with sales, it just takes a lot of like time on the calendar to like uh send the emails, get the responses, have the calls, all that stuff. Um, and this is just one of those cases, which is fine. Um but one of the challenges I'm having, and I wonder if you're having a similar challenge, is to figure out who the product is even gonna resonate with. Cause like what you say is totally true. Like, I know what I would build into this product, and I think that I'm probably right, but I don't want to like bet the farm on that. And even if I know that I'm right, it seems premature to like put my focus on product development instead of sales and marketing. Cause like that's just uh that's just uh a wasteful use of resources. The longer you wait to do the sales and marketing, the longer it takes before you start getting the payback from that stuff. Anyway, um yeah, tell me about that. Do you have a firm idea yet of who exactly this is for? Are you still figuring that out?
Familiarity, Integration, And Risk In CI Choices
SPEAKER_03I'm still figuring that out. Um I have a friend who's he's kind of like, well, he's starting his own little like go to market shop type thing, uh, and like you know, builds marketing, tooling, and uh uh assets and stuff like that. So he I've been conversing with him quite a bit. And yeah, that's one, that's the one thing he keeps asking me. He's like, So, do you know who your ideal customer is yet? I'm like, not exactly. Um, and it's tough because like I've I purposely built the product to be to fit a lot of use cases. Um, like, yeah, you could build a CI pipeline with this if you really wanted to. You could also build a data enrichment pipeline. You could also build just like a really small pipeline that is two steps. You could build one that's a hundred steps, you know. So it like it's not geared towards any uh industry or anything like that. I mean, it's it's obviously geared towards Railshops, just because that's how it runs and integrates. Um But yeah, like I've not found, I'm not sure who my ideal customer is yet. I'm I'm thinking because the technology is still pretty young, uh, is the word I keep using. Um, I'm probably looking for younger businesses, like also business. As we were younger, looking to try out newer tech. I was just on Reddit, for example, talking with this one guy, and he was like, Well, he's like, I'm not actually who you're probably looking for. I'm more of a laggard. He's like, I like to use technology that's you know two years, uh two years no longer cool, sort of thing. Um, so he's like, I'm probably not who you're going for. I'm like, yeah, I guess I I guess I kind of like zeroes in a little bit more. Like I'm I'm looking for people who are more willing to jump on newer tech and try it out, you know, possibly spend a little bit more time up front to weed out some bugs, um uh to kind of have it pay off in the back end where I'm giving them like a lot of direct support and they're they're able to uh you know influence where the product goes and the features that might show up that I may build in the future and stuff. Um but yeah, it is it is hard. It's it's very like, you know, I don't I don't know where your career background is, but like being a product engineer, you know, is is the word I feel like I've I've heard the last few years. But it it's a really hard mode of thinking where you're like you want to build, you want to like to create and solve problems, but then you also have to like take a step back, think about the human and all of these like more squishy things that don't really fit into the the box that is programming.
SPEAKER_02Um it takes a lot of understanding of like psychology and economics and business and stuff like that. Um though like the the question of who the product is for in the early stages, because you know it's gonna be different at the early stages than three years from now, then ten years from now, and stuff like that. So the question isn't exactly who's this product for, that's too simple. It's who's this product for at this stage. Um and one dimension of it is like what kind of person or use case or whatever like is is this like you gave a couple examples. Can can you remind me like one possible use case?
SPEAKER_03Yeah, uh user data enrichment.
Emotion, Identity, And Technical Decisions
SPEAKER_02Okay, user data enrichment. Like uh that's that's one way that that that's one lens through which uh that could be looked at. But also um another way to look at it is like the people who are gonna use it early on are like people I know personally. Like that's a very different dimension. And I think that's the case with Saturn CI. Like both these two people who are who are onboarded and getting onboarded, like they're people I have known for years. And also every single one of the like, I don't know, five or ten people who have like at least started down that path of like kicking tires and and maybe being interested in being a customer, every single one of them was somebody I've known for years before now. And so like, and and their their motivations were all over the map. Like one guy was like, Yeah, I'm just doing this to support you. Like, I'm your friend, like I don't really give a shit about your product, but I'll I'll use this like to support you. Uh and that's great, and I think that can like get the flywheel going, you know. If if I just get a few people using it, I don't really care why they're using it. Um, and and the methods that I'm gonna use to get these early people are way different methods that I'm probably gonna be using six months, a year from now, that kind of thing.
Positioning Ductwork Between Categories
SPEAKER_03Yeah, yeah. So friends and family always come first, right? So um, yeah, I'm I'm kind of in a similar boat. Uh you know, the folks that I've had that are interested in it, they were like, yeah, you've either started something new or or whatnot, or like, oh, I I could see how this could fit in. So yeah, it's that's definitely where it starts. Getting getting beyond that though, like that that second layer of uh of connections, if you will, and then third and beyond, uh getting really getting the word out there is is tough.
SPEAKER_02Yeah, I found. Yeah, like you only have a finite number of five-year long friendships. Uh you you can't make a new one tomorrow because that that window closed five years ago. Um exactly.
SPEAKER_03Takes time.
SPEAKER_02Yeah, so once you go through all those people, you have to expand a bit. Um, what are you doing for for sales and marketing now? And what have you done and and what are you doing now?
Sidekiq Batches, Temporal, And Tradeoffs
SPEAKER_03Yeah. So I'm um I'm mostly just using, well, I guess tactically, I'm just using like I'm posting on Reddit. I I have a blog. I've only made like one or two posts on it so far. Um, but I'm posting things to Reddit. Um, I'm also in the Speed Shop Slack channel, the one by Nate Burkapec, posted in there. Um hosted uh let's see, where else? Um LinkedIn. LinkedIn is decent, but again, that's mostly like my connections. So like it's not the best place for it, but I know it would at least get some visibility there. Um I haven't done anything like paid advertising yet. I'll say yet, just in case I do go that route. Um but yeah, I'm I'm still trying to find what are good channels to use. Um I'm very new to this whole selling software thing. Like I've I've sh I've definitely like shadowed, you know, uh someone who's actually been in sales before, but like it's very different when you're like, oh, this is my product that I need to sell. Oh, yeah. And like there's no one else around me, you know. Like, so yeah, it it it's the whole landscape changes really quickly.
Durable Execution Explained
SPEAKER_02Yeah, and there's a lot of really bad advice out there. Um, I I kind of made a a rule some time ago, never listen to programmers about sales and marketing advice. Because it's all it's all just like regurgitated shit that, like, hey, here's uh the the conventional wisdom on what you should do in this area. I haven't done it. I just heard it from somebody else who also hasn't done it, who heard it from somebody else, and it's just this echo chamber of um uh bad advice that nobody is actually using or has experience with. So I ignore all that stuff. And these days I try to go more from first principles. Um, and over the years I've like tried to learn from uh sales and marketing people who aren't programmers. Um but like at some point I kind of like stopped listening to business podcasts or reading business books and and stuff like that. There's this whole fake economy around business advice. Um I think it's probably the case. This is just a guess because uh there's no way I could see the actual data or anything like that. But I would guess that the vast majority of people who consume business content never do any any actual business. Like they're they're just buying the stuff aspirationally. Um and most of the people creating this content are well aware of that, and so they're not targeting uh real, they're not targeting people who are actually doing it, they're targeting these uh, no offense, wantrepreneurs who who are just consuming it aspirationally, probably won't ever do anything. Uh, and and the real thing that's being sold is just like a feeling, um rather than actual like uh substantive advice.
SPEAKER_03Yeah. Yeah, it definitely feels almost like uh like like the self-help economy or something where it's you know, I'm the person is just there to sell their book and you know, to make you feel good, get you excited to buy it, rather than like, okay, what is it like what is your actual thesis? What do you what are you actually selling? Is it actual advice? Is it you know some real deep knowledge, some industry knowledge, or yeah, so uh I I tend to stay away from that. I I'm a newer father, so like I don't have a ton of time to consume a lot of of media like that, but um yeah, I I agree. There's a lot of it it definitely seems not like you can get one tidbit out of a whole book or something, and it's like that was that was the whole takeaway. That was the what the whole book was centered around.
Marshalling State And Reliability Guarantees
SPEAKER_02Like, okay, right, yeah, and that can be frustrating. I I will also say that um I did spend a lot of time consuming a lot of this material, and I don't regret it because like back when I didn't know anything, almost anything was better than nothing, you know? And I had to at least like learn the ABCs, and it didn't matter that much if 80% of what I was consuming was fluff or mistaken or or any of that stuff, just like getting some familiarity with all that stuff was was a bit helpful. Um, but honestly, like probably the most fruitful thing that I've I've studied is psychology because all this stuff is just psychology. How do people think? How do people make decisions? Why do people buy the things they buy and and stuff like that?
SPEAKER_03Yeah, that makes sense. Do you have any like specific books or podcasts or people to follow that around that specifically?
Why Workflows Beat Ad Hoc Job Chains
Saturn CI Examples And Uniform Strategy
SPEAKER_02Yeah, great question. Um yeah, so the something that came to mind earlier when I was just thinking about general business uh content was the I Love Marketing podcast. Um it's Joe Pile Joe Polish and this other guy, I forget his name. Um, but he has said, Joe Polish had said several times in that podcast that uh sales and marketing are applied psychology, which I think is is totally the case. Um of the psychology-related books that I've really found valuable are How to Win Friends and Influence People by Dale Carnegie. Um I went back and reread some of that recently. Um, you know, I first read it like 15 years ago, and then I've reread it several times, but not in recent years. And I realize now, you know, okay, so I was like pretty dumb when I was 25, 15 years ago when I when I read this. Um, but I realize now that I'm like older and smarter. Dale Carnegie's actually a really fucking smart guy. Like he has this like breezy writing style which kind of masks it, but now that I read it, it's now that I reread it, it's like wow, he actually like deeply understands psychology. This isn't just like self-help fluff, this is like real shit. Um, so that was one. Um, The Seven Habits of Highly Effective People is another one. Um, and then uh like evolutionary psychology. Um, everything in psychology can be tied back to evolution. Um, so How the Mind Works by Steven Pinker, The Blank Slate by Steven Pinker, um some of Richard Dawkins' stuff like The Selfish Gene, um uh what was the other one? Uh The Blind Watchmaker by by uh Richard Dawkins. That's a bit more adjacent. Um but all that stuff, and and just okay, so once you start to learn this stuff, you start to notice it everywhere and make observations, and then you can accelerate your learning just by observing in day-to-day life, and it starts to occur to you to ask questions that you never would have asked. Like, for example, um with CI products, why do people buy the CI products that they buy and how do they make those decisions? And maybe we we can even look at that for a second. Like, how do people decide what CI product to use?
SPEAKER_03Yeah, um, I guess from my perspective, like something I would be looking for would be one security. That that's that's I guess taking a step back, that's one thing I found building my own product. I was like, wow, I really care about security now. Like I cared about it before, obviously, but now I'm like, I really don't want certain things, you know, being out there. So security is one. Um performance, obviously, price, so kind of the the typical stuff, but especially with CI, extensibility is always interesting because I feel like it's always kind of been a little clunky. Um talking about 15 years ago, Circle CI, I feel like was huge. I don't know if it still is, I don't know if people are still reaching for it. Um but I remember like you could only do so much, kind of going back to that framework thing. Your leverage was really just wrapped functionality, and you could only like opt into certain things. If you wanted to like run your own code to you know to um extend the functionality, it was a lot harder to do, and there was like orbs and all of these weird things that um and and sorry if I may interrupt.
SPEAKER_02Um zooming back out a little bit. Um, you're obviously not starting with zero knowledge. Um you're not saying, okay, I need CI. What CI products exist in the world? Uh let's Google it and find out. Like, you're aware of some offerings. So like just to just to pinpoint those concretely, like, what are the CI products that that you're aware of?
Halts, Retries, And Pipeline Control
SPEAKER_03Yeah, um GitHub Actions. That's mostly what I use nowadays. Um Circle CI. Um I'm not as familiar because I just use GitHub Actions and kind of hold it quits. I haven't I'm not actually familiar with a lot of what's out there right now. Because there was a lot of con uh a lot of acquisitions and a lot of whatever uh purchasing of other companies. So I I don't know who all exists right now.
SPEAKER_02Um Yeah, and there's maybe other ones where like if somebody mentioned it, you'd be like, oh yeah, I I know about that. It's maybe not the first thing that comes to to your mind. Like GitLab pipelines, that's that's an option. Um uh there's like semaphore, which is maybe less well known, but uh, but that's another one people use. Um I'm curious, you know, you're using GitHub Actions now. How did that come to be? Was that like did you did you sit down and perform an evaluation and like create a spreadsheet of pros and cons and stuff like that? Or did you pick it out of more out of familiarity, or how did that decision go?
Who It’s For Now Versus Later
SPEAKER_03Um, at least for my own projects and products, uh it was more just familiarity and also just it was a nice experience having it with you know when I was storing my code with the repositories, um, and having it uh I guess integrated. You know, it's it's integrated, but it's like it's all one product, you know, part of GitHub. Um really that's that's about as much thought I gave it. I can you know configured a few projects, ran them, see how you know, see how they performed, and performed fine. So I was like, yeah, this is good enough. I'll roll with this for now. Um and then yeah, just kind of like as projects evolve, um, it wasn't too hard to maintain. Um again, kind of like framework not getting in the way. Not the you know, it was it was like it was good enough of a user experience that I was like, yeah, I'll continue to use this.
Next Steps And Where To Find Ductwork
SPEAKER_02Yeah, yeah. Let me um just think out loud about this. Um because I'm only traveling down this path for the for the first time. Um I'm I'm thinking about risk. Like when when you're gonna use a CI product or any product of any kind, um uh there there's like a risk reward trade-off. Um and you could take a risk and like say, you know, I'm familiar with GitHub actions. I know that it's like I don't know exactly how you feel about it, but let's say you feel like it's decent, you know, it doesn't make me jump up and down for joy, uh, but I don't think it's a complete piece of shit either. It's it's like decent. Um you could roll the dice and go like do some research and find out, hey, maybe there's something even better, but like would that be really be a rational thing to do to just be like there's this thing that I know is pretty good, but am I gonna like go out and do a bunch of bunch of research and set up something that might be better and like do all this work just on a maybe? Like, most people aren't gonna do that, and and rightly so, because like that could turn out to be a giant waste of time. Like, if I were you, I would probably make that same exact choice. Like, yeah, just GitHub Actions, why would I use something else? Does that seem like a fair characterization of how that decision went?
SPEAKER_03Yeah, for the most part. Um, I another kind of framing is like every every project needs CI for the most part, you know, there's an asterisk there, but it's more it was more like CI is almost like a necessity. So I'm gonna just pick something that I know doesn't get in the way and not yeah, like not think too too much about it. I'm not gonna treat it like a really important technical decision where I would make a pros and cons and you know uh trial them against other technologies and stuff like that. Yeah. It is just kind of something you're like, yeah, pick and move on, you know.
SPEAKER_02Yeah. And and by the way, um this is this is something you're you're surely already familiar with this principle, but it's one of the more useful things that I'd learned about psychology. Uh basically every decision and every behavior that that people make is emotional. Like we don't do anything for a rational reason. Um, like, I don't know. I I maybe this is a slightly bad example, but about five years ago I bought a pickup truck because I needed a truck. And I guess that's kind of let's set that part aside for a second, because you could call that a need. Because we have a farm, we need a truck. Um, but the specific truck I bought, like, I didn't need that. Like, I bought a brand new Toyota Tundra for like$52,000. T top trim level, like fanciest truck that I mean you can buy it like in you can buy a fucking$100,000 truck, but like a really nice truck that I did not need, absolutely didn't need it. I could have bought like some piece of shit a 20-year-old truck and it would have worked fine, you know? Yeah. Um, it was purely emotional. Um there's there's there's this phrase from Seth Godin that I read some years ago and it really stuck with me. Uh, people like us do things like this. Um I bought a Toyota and not a Ford or a Chevy because I'm not a Ford or Chevy kind of guy. Like I I sat in the Toyota and I'm like, I like this. The the interior matters a lot to me. Um, and I think the interior in Ford's and Chevy's, all American cars, unfortunately, is just fucking garbage. So I'm not gonna do that. Um, I forget where I was even going with this, but software decisions also, like they're very largely emotional.
SPEAKER_03Yeah. Um one kind of phrase I've heard that I like is vibe their data.
SPEAKER_02Vibe their data? What what does that mean? Um that oh wait, wait, vibes are data?
SPEAKER_03Yeah, vibes are data.
SPEAKER_02Okay, I thought you said vibe their data.
SPEAKER_03I'm like, Oh no, sorry. Vibes are data.
SPEAKER_02Vibes are data, okay.
SPEAKER_03Yeah. Vibes are just emotions, kind of feelings, uh, is a form of data. Um so yeah, that that totally resonates. Um where yeah, it's kind of you do end up making decisions, you know, based on emotion or something you like, or initial reactions end up being uh um a big player in in what you go with.
SPEAKER_02So yeah, and I want to say that that's that doesn't conflict with rationality. Like an emotional decision is not necessarily an irrational decision. Um, it's just not exactly the same thing. So I'm I'm curious about for ductwork. Like, how would somebody decide to use that? And is it like, are there other things that it's like apples to apples to? Like, are people deciding I'm gonna use this thing or I'm gonna use ductwork? Like, does it fit into a category?
SPEAKER_03Um, it kind of fits in in between categories, maybe. Um I might I might change that, but now that I think about it more. Um, yeah, I mean really it it kind of goes up against a lot of the big enterprise stuff. I mentioned temporal was one. Um somebody in my network asked about that. He's like, oh, so how does this compare with temporal? Um sorry, wasn't the original question? I I lost it. What was the question?
SPEAKER_02I think I was asking like if it's a if it fits into a product category.
SPEAKER_03Yeah, or yeah, like how would they pick this versus other you know, product Y. Um yeah, so I mean I I I'm trying to I guess like what I'm trying to do with Ductwork is I'm trying to lead with the use case. Where I'm like, you you've almost certainly run into this use case, here's how it fits. Which I know is kind of like just a simplistic uh viewpoint on, you know, like here's why you should use this thing. Um again, I'm still getting my sales and sales and uh marketing shops. So um yeah, I that's a good question. Um I'm like trying to like think and speak at the same time here.
SPEAKER_02Yeah. Um yeah, because okay, so some things are like all right, so so I'll give a brief bit of context. Um this is not my first attempt at building an online business. Um many years ago I did um software for hair salons. Um, and sometimes it was like a choice between a different product and my product, and and apples to apples choice. Sometimes it was a decision between like pen and paper and my product. Um and 37 Signals has made points about this. Like, sometimes you're the thing you're competing against is not other products, it's just a completely way of completely different way of doing things, no product at all. Um and I have really, I I feel like I've benefited from creating a product that neatly fits into a category. It's like, you know what Circle CI is, you know what GitHub Actions is. This is just another one of those. This is just like a better one of those. And so people are like, oh, okay, like I need CI, I'll I'll give your CI a product, I'll give your CI product a try because I know what that is. Um and it seems like I'm trying to figure out what's the case in your case. And and by the way, sorry, I just I want to be uh clear that I'm not trying to give advice uh because I hate when people do that. Um I'm just trying to like understand what the scenario is.
SPEAKER_03Yeah, yeah, yeah, totally. Um yeah, so I I guess I'd say it's kind of like in between uh in between other products because you want to go like if you think about like the most simplest use case, sidekick or any other background job service, great, yeah, you can use that and you could you could build ductwork ductwork like whatever uh using those technologies, but it's a lot of plumbing, it's a lot of extra maintenance, uh, you're gonna have to, you know, deal with bugs and um whatever. You have to build it yourself, you know, um like you do with libraries. If you go kind of the more the larger, more mature tech uh big enterprise technology route, temporal, Apache Beam, which is kind of similar-ish, um that's a lot of then you kind of get into like operational burden and maintenance burden there, because it's a lot of extra infrastructure. It's a lot, it's very complex. You likely have to train your team on the technology. Um and then not to mention expense thing, you know, things like Apache Beam, it's all open source, but you know, like how are you hosting it? Are you gonna use a cloud offering? Are you gonna host it yourself, etc.? So it kind of come back around. It fits in between those, and that's that's kind of what I wanted. I wanted to be able to build complex pipelines, complex workflows, but not have to not have a complex, uh, operationally complex system to do it. Um I guess kind of talking about competitors, the closest competitor would be sidekick batches. Um you can build pretty much any workflow you want with sidekick batches. Granted, I think it gets the ergonomics, I think, are very I don't want to say clunky, I'm not trying to like insult sidekick here, uh, because I think it's a great piece of software. Um and I think can work in tandem with with ductwork. But like it's it doesn't feel like it was purpose built for that.
SPEAKER_02Um what is sidekick batches?
SPEAKER_03Yeah, Psychick Batches is um allows you to. I don't want to do it justice.
SPEAKER_02Let me just look up the uh Yeah, my my guess is that it allows you to run jobs in batches, but I I don't really understand the exact use case.
SPEAKER_03Yeah, totally. Um yeah, like obviously you can build a bat and create a batch of jobs and then enqueue them. There's it it kind of goes beyond that where it's like you can monitor that group. So you get some control over that group of jobs. Um so for instance, you can you can create these workflows by uh it has a callback feature. So once all of these jobs complete, it'll fire off a callback. Um from so then from there, that callback, you're like, oh cool, I need to like go to the next step in my workflow. So then you can fire off another batch of jobs or a single job or multiple batches to get kind of like diverging branches of um of a workflow. Um so that was in service to say it has these features that allows you to build workflows, but like it's it wasn't purpose built for that. Um it's just like some of the features that are built into it you can you can do this with.
SPEAKER_02Um got it. And like Mike Parham probably doesn't have much incentive to go and like make sidekick batches as sophisticated as it can possibly be and and work for all your use cases. Like that's probably where it is now is probably about where it's gonna stay. So if you want something more sophisticated, that that's probably not the thing. Or or at least it's probably not the only thing.
SPEAKER_03Yeah, it's not the only thing. Um yeah, and like psychic batches is still great for you know batch uh having a huge batch of jobs or something, you know. So uh it definitely has its its niche. Um, but that's kind of where I was like, okay, yeah, I want I want that, but I want like better ergonomics and more durability over over top of that. Um and that's kind of where Duct Work came about. So that's kind of why I said it was like in between existing, you know, categories or whatever.
SPEAKER_02Um this seems like kind of an obstacle to me. Um, again, just thinking out loud. Like, I wonder how easy this product is for people to understand. For me personally, I I like half get it, but I'm like not sure. It it could be that my understanding is even totally wrong, and once I do get it, it turns out I didn't get it as much as I thought I did. Um but it sounds like there's also like existing you mentioned this Apache beam, beam?
SPEAKER_03Yeah, beam like a like a like an IBM, yeah.
SPEAKER_02Yeah, yeah, and that other tool I forget the name of. So there's like kind of analogous things existing.
SPEAKER_03Temporal. Yeah.
SPEAKER_02Yeah, what is temporal?
SPEAKER_03Yeah, temporal, so in the world of like workflow engines, the weird world of workflow engines, um there's like a subset of things called durable execution. Um so basically, I don't know if you've heard of like AWS step functions. Um okay, it's an it's analogous to that, uh where I have this piece of critical code. Say you're uh you're a financial, say you're a bank and you're um doing a transaction uh with like double entry accounting and stuff. You need to make sure that that transaction executes correctly. Um, you know, and if it if there is a failure of some sort, you either need to retry it or you need to fail loudly and like be alerted and like have manual intervention, et cetera. But I ideally like it just works, and like you have that guarantee that this is gonna, this transaction will execute correctly. Um so durable execution kind of gets at the base of that where it's like, okay, cool, like what are the problems that could go wrong while executing? Somebody pulls a plug, uh, there's a deploy, the process dies, a thread hangs, you know, all sorts of miscellaneous things where computers go wrong. Um and it tries to kind of mitigate those those issues. Um, so that's where temporal really dives deep into like durable execution and making sure like you you add this line of code, I will guarantee that this line of code runs correctly. Um, and then kind of builds that out with like, okay, now you have like these sets or checkpoints along the way where it's like, all right, I reached this, great. Now we can move on to the next one. Did everything execute correctly? Perfect. Keep you know, keep moving forward. Um so temporal dived really deep into that. Uh ducked work kind of it has it is durable in the sense that if you you know, if you're doing a deploy or the process dies, you don't lose any data. Um and that was kind of like a big, a big uh requirement that I had. Um sorry, I'm starting to meander a little bit here.
SPEAKER_02No, no, I'm I'm into this because like this actually overlaps with a lot of stuff that I have to think about with Saturn CI. Um because there's a lot of stuff where it like it has to happen, and if it doesn't happen, I need to like either either I personally need to know about it as the maintainer of the system, or the user needs to know about it because it just can't fail silently. Like if that's the worst scenario. Like if it works, obviously that's the best scenario. If it fails and you find out, then like okay, that's still a malfunction, but at least it's a malfunction that you know about. But just silent failure is is totally unacceptable.
SPEAKER_03Yeah, yeah, totally. Um yeah, and the way to like kind of mitigate that is just to like store a bunch of data, store it, store all the inputs, store all the outputs. Um, there's this really interesting article I just read by Julik. Julike? I I don't actually, I'm probably butchering the name. I apologize to the person. Um, but talked about durable execution and basically getting close to marshalling a call stack, which I thought was really interesting.
SPEAKER_02Um What does marshalling a call stack mean?
SPEAKER_03Basically, like um, well, I say basically, it's it's that it's kind of like a heady concept, in my opinion. Um taking like execution, stopping it kind of in its tracks, storing that somehow, process and then like you know, uh restarting a process, restarting a thread, restarting a machine, changing the machine, um, and pulling that context back up and then executing it again, or continuing that execution rather. Um, so if you were able to marshal uh just think of like Ruby's marshaling where it you know outputs objects to uh a data format, it serializes to a data format and deserializes. Um basically doing that, but for like computation, um and kind of like digging digging into the theoretical of that, um, and like how how would you do that sort of thing?
SPEAKER_02Interesting. Man, uh this is one reason I just like love having a podcast, is because I can have um people on the show with such diverse areas of knowledge, and I get introduced to things that I didn't even know that I didn't know. Like durable execution, like this is something I've been thinking about for like six months, a year. I didn't even know it had a name. Um so now I can be like, okay, durable execution. Um I I can Google that and like learn more about it. It's so great. Um, but back a little bit, um, we were talking about like uh getting people to understand what this product is and stuff like that. Um I feel like I personally am getting like closer and closer. Um how wrong would I be if I said like, oh, ductwork, it's all about durable execution. There's probably a lot more to the picture than that, right?
SPEAKER_03Yeah, a lot more to the picture. Um I guess the I guess the main point I'm trying to get across is like maybe, maybe with like I can illustrate it better with an example. It's like so many so often I would run into a thing where it's like, I have a I have a background job. It does a thing, but then like over time it gets more complex, you're calling out to more APIs, you're writing more data to the database, um, you're calculating more stuff, um, kind of going back to that data enrichment, like user data enrichment uh example. Um I want to fetch data from an API, enrich, you know, kind of collate that data with what I currently have, transform it probably somehow, um, and then store that data. Um like you could do that all in one background job, but then you start to lose item potency, you know, when you're like fetching from an API and prefixing data and there's all these side effects and whatnot. Um so it's like, how could you break that into a workflow where yeah, you can like go through each step that you need to do? Maybe some things could run concurrently. Um, say you're fetching data from multiple APIs, those could all run concurrently, and then you you know collate that data again. Um being able to just more uh declaratively express that and have it uh modified in a way that is readable, more maintainable, uh more durable. Um so each of each of those things would be like a step in ductwork.
SPEAKER_02Um Yeah, I'm starting to get it now. Like the the user, I didn't know what user enrichment, your user data enrichment. I'd never heard that term before now. Um but with with that example, like if you put all that work in one job, then you're like tightly coupling things that shouldn't be tightly coupled, uh, or or rather, there's like downsides to tightly coupling those things that you probably wouldn't want. And so if you decouple those and put them in separate, like I can imagine one part of that job failing, um, or one part of that job is like not perfectly reliable, and you'd want to put it in its own job that can be independently retried rather than having to like, I don't know what, retry the whole job and have like conditionals in there to be like, if this one failed, then do that one again, but don't do any of the other ones because we're only worried about this one on this run, like much simpler just to put that in its own job. So this kind of makes me understand like where the the question of like why would I need a workflow engine, like that that makes it a lot more clear for me.
SPEAKER_03Cool. Yeah, it it was very much yeah, like you just said, you end up either having like this huge job where yeah, you're doing conditionals and error handling yourself. Um, because yeah, it's like, well, if if this step, if this piece of code you know fails or doesn't return the value I'm expecting, now what do I do? Do I like fail the whole thing? Do I retry the whole job? Well, shoot, I already persisted some data uh further up, so like it's maybe it's not itempotent, so I can't run it again necessarily. Um, or I just want to run this single piece of code, and then you have like this job chain or something, you know, like one job spawns another job, spawns another job. Um, which in my opinion isn't a great pattern for reasons. Right. But yeah, like with a workflow, yeah, you can like um you can just have a whole pipeline halt. Like that's part of ductwork I kind of have. Like if a certain step fails n times, um, yeah, then the whole pipeline is halted and like nothing else moves forward, which I think you would you know kind of expect that. You wouldn't want to just like, oh, try to run the other steps and see what happens. Um so um but you know, pre-trial logic, durable execution, etc., whatever. Um but uh yeah, with a workflow, you have more control over the full. I loathe to word to I loathe to use the word workflow again, but that's really descriptive in what it is. Um a lot of the times, yeah, it isn't just a simple background job that you need to use, it's it's like this this whole group of steps that has some implicit relationships between each other um that you need to run. Um yeah, this is super interesting.
SPEAKER_02Um and I bet if you looked at Saturn CI and you could see all the like workflows involved in it, you would see so much stupid shit that I'm doing. Um because like some of this stuff is like, okay, when a when a test suite run runs, um, one of the first things that happens is there's uh a cache lookup that happens. It's like, is the Docker image for this application cached? Because if it is, let's not do a whole Docker build because that's expensive. Let's just um grab the the image from this other from the place where it's cached and just copy that file over because that's a lot cheaper. Um and there's a couple caching layers, and then once that happens, it it like runs the tests. And there's things that happen like a um like a um test count uh test count check. Like let's say my test suite has a thousand tests. Well, there are scenarios where something can go wrong, and it only it tries to execute just 996 tests. And that could run and pass. Like early on, I had situations where that would happen. It's like, hang on, the test suite passed, but it skipped four of them. Like, what the fuck? Like that's we can't consider that a success. Yeah. So now it does a check right at the beginning and it says, does the number of tests I'm trying to run match the number that I'm supposed to run? And if not, just halt and raise an exception because I don't know what's wrong, but something's wrong, and it's better to just blow up than to proceed. So a bunch of stuff like that. But they are managed in maybe kind of an ad hoc way on a case-by-case basis. I don't have like a super, I don't have like a common strategy for dealing with everything. And like, of course, there's retries and stuff like that, and these are like, you know, I try to keep it reasonably DRY and stuff like that, but there's repetition of these concepts, and and what I'm learning from this conversation is all that could probably be handled in a much more like uniform strategic way instead of all these one-offs.
SPEAKER_03Yeah. Um, I mean, obviously I don't I'm not looking at your code or anything, but um yeah. There's like yeah, there's like similar, I keep using the word plumbing. There's like similar plumbing that you have to do these, like do a retry or similar logic to when something fails, do this. Yeah. So um yeah, I'm just I'm like it is a half thought that I'm like saying out loud.
SPEAKER_02Yeah, this is really interesting. And actually at my day job, we have a bunch of these issues too. Um, because I work on the build and deploy team at uh an organization called Cisco Meraki. So we have a lot of really complex pipelines. Lines. Um yeah, interesting. Um and this is uh yeah, okay, so I'm I'm I think I get it now. I at least I get it way more than I got it before. And I'm I'm coming back to the same question of like who is this product gonna be for now at at this stage? Um yeah. Yeah.
SPEAKER_03Um yeah, it's it's always the question.
SPEAKER_02Yeah, yeah. I'm not expecting you to have an answer right now or anything like that, but I think you're gonna have a similar journey to me where it's like at least the approach I'm taking is I'm like lazy loading it. It's like first, first, who can I access? Like I just want to like try to grab people and be like, hey, you look at my thing. Do you like this? Um, and if so, great. And then once I once I get a handful of those people, it's like, okay, 10 of you are in now, you're using this thing, you haven't left. Like, why are you here? Why are you why are you into this? And then I can be like, oh, it's because blah, blah, blah. Now I can like put that on my marketing page and find more people like that.
SPEAKER_03Yeah, that's I'm definitely kind of hoping to add some of that to the site. Some some case studies, some, you know, uh user quotes, etc. Um and yeah, of the users that I currently have, which a few, um, really kind of like digging into them, like how are you using it? What do you like? What do you not like? It's that you know, it's the typical user research type stuff. Um so yeah.
SPEAKER_02Uh yeah. Um, well, we should probably start heading toward a conclusion here. Um, but if people want to go look at uh at duct ductwork and find out more about it, where can they go?
SPEAKER_03Yeah, so I have a website called get ductwork.io. Um that's just kind of the marketing site. Um it's under it's on GitHub, obviously. The project, the open source project is on GitHub, uh just ductwork ductwork, uh, ductwork organization, ductwork repo. Um, and then there's a doc site at docs.getductwork.io.
SPEAKER_02All right. Well, we'll put that stuff in the show notes. And Tyler, thanks so much for coming on the show.
SPEAKER_03Yeah, I appreciate it. Thank you.