Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Distributed Systems Engineering with Apache Kafka ft. Jason Gustafson
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Ever wonder what it’s like to be a distributed systems engineer at Confluent? Core Kafka Engineer Jason Gustafson dives into the challenges of working on distributed systems, particularly when it comes to a unique system like Apache Kafka®. He also discusses ways in which Confluent is working with the community to solve active problems and what it takes to be a distributed systems engineer.
As always, Confluent is looking for engineers who are interested in distributed systems, and you don’t have to have 10 years of experience to do it!
EPISODE LINKS
- KIP-392: Allow consumers to fetch from closest replica
- Kafka Improvement Proposals
- How to contribute
- How Confluent's Engineering Team is Building the Infrastructure for Real-Time Event Streaming
- Join us in Confluent Community Slack
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
There are a lot of specialties within the very broad vocation of software engineering, and all of them are hard to do. Distributed systems engineering is one corner of the discipline that poses a particular set of challenges. What's it like to build a distributed system? What special problems arise? How do you land a job doing it? And that's the conversation on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I'm your host, Tim Bergland, and I'm joined here today by my colleague Jason Gustafson. Jason, welcome to Streaming Audio.
SPEAKER_00Thanks for having me, Tim.
SPEAKER_01You bet. Jason, what do you do at Confluent or in open source? What's your work like?
SPEAKER_00Yeah, so at uh Confluent, I'm part of uh what we call our Kafka core engineering team. And uh our focus is really on um you know uh developing Apache Kafka. And um so the our focus specifically uh is on uh the broker side. So um the storage layer, the replication layer, like how how are we replicating the data between uh the different brokers? And we also focus on the um you know developing the uh standard clients that uh come with uh Apache Kafka. So most of our work um is is just within the Apache community.
SPEAKER_01Oh, nice. So you you work at Confluent, but you're primarily uh like a uh an open source Kafka developer. Yeah, that's right. Cool. Um what's uh what I really want to talk about today is uh uh what it's like to be a distributed systems engineer. So I don't so much want to deep dive on Kafka internals. Um I mean I know I I would I would enjoy that conversation a lot. I know it's a thing you can do, but I kind of want to just see what it's like uh to have the kind of job you have. There's a lot of different ways of being a developer, uh, and you're one of them. And I would just like to talk to you about that. So uh having said that, uh I I kind of would like to get an idea of like what's a thing that you've been working on in in Kafka recently. What's a what's a uh feature that you've been building?
SPEAKER_00Sure, yeah, I can give you some background on that. Um yeah, I and I I'll just I'll just mention mention one thing. Um so uh historically, uh Kafka, you know, we have a replication protocol. Um there you have a number of replicas. Among those replicas, you choose a leader, and uh all of your reads and writes go to that leader. So uh what we've been working on, you know, it's kind of been a proposal that's kind of been sitting around in the community for quite a while. Um, you know, people wonder why can't you have uh reading go to uh some of the other replicas? And uh the reason why that's interesting is because you know we have a lot of multidata center deployments of Kafka, and uh, you know, cross-DC traffic is expensive. And uh so if you can if you can read from uh one of the other replicas, you know, maybe one that's close, then um you can potentially save quite a lot of costs. So um, you know, obviously this is interesting to us in our in our cloud product, but it's a lot of interesting, you know, it's interesting to all our users as well who are trying to run in these kind of scenarios. So, you know, we we we put out a uh proposal in the community um which you know got adopted, and um, you know, we've been working on uh uh getting that code into place. So it involves changing um the way that the consumer behaves because it has to understand um you know how how how how close the replicas are relative to itself. Um and it has to change the you know the the broker uh semantics as well. You know, and there and you know we we could we could get pretty deep in this, I know you don't want to, but that you know there are a lot of uh subtle details to getting this all working, um such as you know, you're reading from a replica means that um you know you're not as you know you're not as far ahead in the replication. Um but how do we ensure that um you know we can we can still reason about the system? So that's been yeah.
SPEAKER_01Believe me, I do I do want to get into it. We we have other things to talk about, but uh so I I will regretfully leave that. Because I mean it sounds like there are interesting consistency problems that are gonna emerge there and uh the Yeah, that that's right. But the main thing is just about um saving costs. Um to w in in multi-TC deployments. And that affects uh obviously the broker, but it seems like a lot of that would be on the client side. So the Java client and my guess is lib rd Kafka uh both need to grow to make this happen.
SPEAKER_00Aaron Powell Yeah, as well as the other uh client clients that uh you know supported in the community. Um but yes, that's right. I think the way we did it actually is we we kind of pushed um a lot of the decision making to uh the broker side. So from the client's perspective, they can kind of identify themselves and where they are, but the broker is kind of the one that makes the decisions.
SPEAKER_01Oh, nice. Okay. Yeah. So and it it's I guess like a whole kind of the cluster topology model has has uh grown up a little bit here to make this work. Yeah, that's right. Suddenly, suddenly we have to know and care where things are. That's pretty cool. Do you also in your work uh you mentioned Confluent has a cloud service. Uh I hope that's not a surprise to anyone listening. But if it is, hey everybody, Confluent has a cloud service, Confluent.cloud. Check it out today. Um sorry, it's uh embedded commercial there. But yeah, we have a cloud service, and that cloud service uses uh, you know, is is hosted Kafka plus hosted Confluent platform things. Do you do anything uh outside of your core Apache Kafka broker work? Uh do any cloud problems ever come your way? And you have to think about those things.
SPEAKER_00Yeah, absolutely. Um so we we spend a lot of time um, you know, we're we're developing um, you know, in Apache Kafka and uh you know attempting to make it uh work better in such in situations such as what we have. Um, you know, and there are other people that are trying to run uh Kafka in a cloud service as well. So it makes sense for uh a cup Apache Kafka in general to be um you know more cloud native. And so I think there are a few different problems there related to multi-tenancy, making sure, you know, oftentimes uh usage of Kafka is multi-tenant, and we've got different or uh, you know, you may have different teams that are all trying to work on the same cluster, and you need to make sure that um they can all do so without stomping on each other's feet. So for example, this means um you know implementing uh quotas uh so that you can uh limit the utilization from any any particular user. Um and there are also um you know uh data balancing problems. Um so as you're as you're trying to run Kafka, you're trying to grow it, um, grow usage of it. And so that means that you kind of got to um move uh move the data around um when you need to grow. Um so yeah, we're we're focused on um you know addressing some of these problems as well.
SPEAKER_01Nice. Yeah, it it has occurred to me, uh, and I'm not uh, as you know, not a core Kafka engineer, but it has occurred to me that if you look at the time Kafka grew up when it was born, um, it's not like the cloud wasn't a thing, but the cloud wasn't a thing like it is now. I mean, you had S3, you had EC2. Um I'd have to you know look at a timeline, but there are probably uh you know small number of other Amazon services that were in existence and and people did that. You had Google App Engine back then, you know, people were using the cloud, but it was in what by today's standards looks like a very primitive way. And the the notion of a cloud native application, nobody said that. Uh it and in fact, it was this sort of painful time of taking all of our sort of classical on-prem architecture and deployment disciplines and saying, well, let's put that in EC2. And you know, it was it was it was difficult. And Kafka, there are some core design decisions inside Kafka that reflect that time period. And it sounds like what part of what you're doing is uh participating in the process of reconsidering those design decisions for the kinds of truly cloud native deployment architectures or you know, deployment situations we have today.
SPEAKER_00Yeah, that's exactly right. Yeah, I think um, you know, when when Kafka was developed at um LinkedIn initially, you've got you have a generally much more um uh you know cooperative environment. You you kind of know the teams that are using your system. And um so you know, multi-tenancy uh and this this kind of problems about like uh especially in a cloud service, you know, we're not at the forefront of uh a lot of the design decisions. Um so you know, now now you take a um a system which has has matured, at least from um, you know, like an operational or performance perspective, um, and trying to uh implement um you know some limitations in place for uh multi-tenant environments. Awesome.
SPEAKER_01What um what makes Kafka interesting to work on to you?
SPEAKER_00Yeah, Kafka has um, you know, from a distributed systems perspective, uh it has just about everything. Um so you know, at the heart of a lot of uh distributed systems is a uh you know uh log replication protocol. So for example, uh, you know, uh consensus is usually based on uh log replication. Um so Kafka, of course, that's like at the heart of Kafka is Kafka is one. That's right. Yeah, we log replication is is basically the really the core. And we you could you could say that's really like the focus of our team is really that log replication layer. And um, you know, building on top of that um are you know it it's already a uh quite a difficult model. You think a log, which is a simple data structure, but when you get down to it and you start um you know looking at uh the subtleties that come in when the fact that it's a distributed system and the different kind of semantics um that you're trying to achieve, exactly one semantics, for example, um then it you know it it's it's quite complex and uh quite interesting as well. Um turn turning a lot of this and you know, log compaction, for example, is another one, which um you know it's a lot of complexity and a lot of um you know interesting uh design trade-offs.
SPEAKER_01What let's take that. Let's take log compaction. Um what makes it hard? It when I so and I I ask, I pick it because when I think of log compaction, I think of that as um really a single broker proposition that it's just running in one place. And I know, you know, I I'm I'm aware of some of the distributed things that are going on in a cough cluster, of course, but I don't think of that as one of them. So what what makes it hard?
SPEAKER_00Yeah, I'll I I can give you um, you know, one one example, and this this is more about you know, so you really have to think about um, you know, when you're talking about a distributed system, the clients are part of your distributed system as well. Um and you need to be able to uh reason about what the clients are doing in addition to uh what the the servers are doing. So just to give you you one example of that, you know, the compaction process is basically you know going through the log and uh cleaning up um old entries for each given key. And um, you know, we we've got a notion of you know, you can write an old record, that means it's uh the the entries being uh deleted. So but one subtlety is um you know if you think about a consumer which is reading from the middle of the log, and meanwhile you've got the cleaner that's trying to clean it up. So you may have situations where the consumer sees a record, um, but then you know the cleaner kind of notices that it had been deleted, and even maybe will remove the tombstone for it. So in that case, if you're trying to follow along as a consumer, you may see um a record which had already been deleted, but you never see the deletion. So um you get into kind of an inconsistent state. So there are a lot of uh you know subtle details like that. And exactly when semantics adds like a you know a whole whole new set of wrinkles where you know that um you're you're trying to reason about um you've got data in the in the uh in the log which is either part of a committed transaction or an aborted transaction, and uh you're you're trying to uh you know clean it at the same time while reasoning about all of this. Um uh does that give you kind of a uh absolutely it does.
SPEAKER_01And and you know, you started it with uh clients are part of the distributed system, and that's a good reminder. And it's in all of those scenarios you were just walking through there, it seems like it's the state of the client that makes it hard. Uh otherwise.
SPEAKER_00Yeah, I think that's right. For compaction in particular, it's really that just the fact that you've got this uh consumer which is reading from really an arbitrary part of the log, um, but still trying to you know maintain the the semantics, the the log compaction uh semantics.
SPEAKER_01Yeah. Um and it's it's it's funny because when I asked that question, I was I absolutely had a completely broker-centric view of the price. You said log compaction. I'm thinking of, oh come on, a broker just does that and then replication happens like normal. How hard is that? But um yeah, actually, real hard because there's all this other state out in the world about the thing that you're changing, and and that's uh that's clients. So um excellent reminder, the clients are a part of the distributed system.
SPEAKER_00Yeah, it it it is it is fair that um you know from the perspective of the replication, anyway, there's um you know the the log compaction is is kind of handled at at you know individual broker layers. So most of the complexity is not not really uh you know any new any new wrinkles on the replication itself.
SPEAKER_01Right, right. Um how about you? How uh I want to ask you how you got into uh distributed systems engineering, and if maybe if you might take the long view of that question, like uh what was your educational preparation and you know what what was your career progression up to this point, if you don't, if you don't mind, uh talking a little bit about you, but how does one get into this sort of role? And your story is just your story, so if you're listening, this isn't the template for how to do it, but this is how one person did it.
SPEAKER_00Yeah, sure. Um yeah, sure. Excuse me. Um so I yeah, I didn't I wouldn't say, you know, when I was in in college that I I had a vision that I was one day going to be working on distributed systems. Um you know, uh I I I thought about it a bit more um, you know, when I was in uh graduate school, but still, you know, not not necessarily something that I I was setting out to do. Um you know, prior to uh Conflient, I was I was part of a uh a startup um called Gaikai. We were we were we were building game streaming. Um but we were we were dealing with a lot of uh you know, game streaming basically this is this is sort of like Netflix in the cloud for games. Uh right.
SPEAKER_01So the game runtime is executing in the cloud and streaming video to you and your streaming inputs back to it.
SPEAKER_00Yeah, that's exactly right. And I think um you know so the problems that we were dealing with, uh, you know, obviously latency is is always the big question for this. And uh so you kind of have to you're running a multi-data center uh environment in the first place, and you're you're you're trying to figure out how do you how do you get the user as close as possible to you know someplace where they can play and have have tolerable uh latency. So, you know, in in this process, we we were building this infrastructure to enable this, and uh, you know, we were also um quite heavy Kafka users. Uh and uh that's sort of how I became uh familiar with Kafka and uh familiar with uh the company. Um but you know we we we we were dealing with these distributed problems. I would say like most of these problems were um you know not not as stateful as as Kafka itself. So I wouldn't say like when I when I joined Confluent that I was already prepped and ready for all of the problems that we were going to face. Um it's more like you know, I had I had some background uh in uh you know understanding what the problem space was. Um but I had to you know learn quite a lot from um the other engineers who are here uh in order to um you know be able to contribute.
SPEAKER_01About about Kafka and internal.
SPEAKER_00About Kafka and about uh you know stateful distributed applications. I think they're you know, we we had done a uh a lot of a lot of distributed systems previously, um, but largely these these were um you know stateless. So in and you know, often in stateless systems, um, they're they're they're much simpler. Uh you know, oftentimes you can kind of restart them if anything goes wrong. Um but you know, stateful applications and persistent applications like like Kafka, they've got this whole other um you know problem space. You've you've got this on-disk state that um you know when you're starting up, you don't know what that is. And you so you you and and you're you're trying to uh you know reason about the system and at a at a you know much lower level, I guess. And um uh so you've got this persistent state that and you've got other rep uh you know um other nodes out there which also have some persistence state, and you're trying to get to some kind of um you know convergent position on what the current state of the system is. It's um you know, for for me anyway, it it took quite a lot of uh learning from um you know inside the Apache Kafka community and also from the other engineers here at Confluence. To to get to that point. Yeah, that's right.
SPEAKER_01And I I guess um I would add uh stateful systems that are dealing with mutable state. Um because one of the nice things that Kafka gives at the at its API level, you know, like what it exposes to a developer building business software on top of Kafka or something like that, um, it presents this abstraction of these logs of immutable events. Uh and so the user of Kafka gets to re reason about uh these immutable things, and that is a tremendous simplifying assumption. But of course, inside a broker uh ain't nothing immutable about a topic. You know, people are writing to them all the time. Uh and if they're if they're compacted, then they they change by themselves, even if you don't write to them, you know.
SPEAKER_00So uh Yeah, that's a good point. I think uh that's a you know, looking at it from the perspective of the user, you know, most mostly one of the advantages is that you do have this immutable log. Um, but you know, presenting that picture on the broker side uh has a whole host of challenges.
SPEAKER_01Exactly. You do that backbreaking work so Kafka users don't have to. Uh they should tweet you and say thank you. But uh seriously, that's the game streaming. Out of curiosity, how how long ago was that?
SPEAKER_00Yeah, uh that was um uh I want to say five years ago. You know, ultimately we were we were bought by uh Sony and that became the PlayStation Now service.
SPEAKER_01Oh, okay. That's fascinating. All right, and uh always want to say for posterity, we're recording this in the middle of 2019. So uh um I was gonna say game streaming five years ago is uh was super ahead of its time because all that stuff is you know from the the consuming public is kind of just coming online and and becoming a thing people do uh now. So super cool that you did that. Um so one one thing I um I uh uh a thing I I want to avoid is uh setting up some kind of hierarchy like distributed systems engineering is the cool kind and you know uh lowly front-end React work is the not cool kind of work. I I don't see the world that way. I think all kinds of software engineering is you know has the capability of pushing human beings to their intellectual limits and and personal limits, you know, it's it's hard work to do to translate uh uh ideas out in the world into things that machines can do. Um so having said that, what do you think makes distributed systems engineering different from other kinds of software engineering? Uh like I what when I worked as a developer, um we uh I didn't really say full stack engineer back then, but I mean I I was a full stack Java developer, web developer. What makes distributed systems engineering different from that, do you think?
SPEAKER_00Yeah, I think um you know, to full stack, obviously, like you need to be good at a lot of different areas. Um so distributed systems, you you need much more focused. Um so in distributed, stateful systems we were just talking about, um you you just you need to understand um you know the operating system at a very low level. How is how are the you know disks actually working and uh what guarantees are you getting from them? And then you know, distributed systems are are difficult in the sense that um you you know you're you're always kind of uh trying to adapt to a moving state. So the state of the system itself is evolving and you're kind of seeing like notifications mostly after something has happened. Um so you know, and if you if you're talking about like designing an application which is um you know on on a single a single machine, um, you know, or even to take it at an extreme, like a single thread, you you've got all the state in one place, and you don't need to have you know nothing else is modifying it. You take that to multi-threads, and uh you've got you know, you you you still have some operations that uh are some primitives that give you some re ways to reason about that, such as um, you know, like mutex or semaphores, stuff like this. Um now if you take that even further and you go to a distributed system, now you then your primitives, like you there's no there's no kind of like lock that you can use to make sure that you're synchronized. So mostly you're dealing with you know an asynchronous operations, and uh so the state of the system, the state of other nodes are are moving all over the place, and you're always trying to uh react to those. Um so it's it's like it's a very different mindset. And um you have to uh think about uh, you know, like the the low level, as I was saying, about the operating system, um, and this is like the stateful part of it, but then the distributed part of it is just you know, reasoning reasoning about these these uh you know. The moving state of the system and designing protocols which let you do so in a safely in a safe way. So I I think you know it's it I think it's mostly just a difference in in focus. I think, you know, for for example, uh a front-end engineer is is probably thinking a lot about um user experience and uh you know what what how how how should people interact with the system? You know, we're we're thinking about that at a different layer as well. So we're thinking about um the APIs that we're exposing to users and uh you know what semantics are actually achievable and what semantics are usable, uh and getting getting like you know, finding the nice nexus where you've got both usable and achievable is is always the uh desirable, you know, the the goal.
SPEAKER_01Yeah. You keep talking about state, and so would you summarize your view of distributed systems engineering as uh you know a discipline wherein you reason about state that is running on uns you know multiple unsynchronized processors?
SPEAKER_00Yeah, I think that's a fair way to look at it. I think you you know you you're the state is it's a distributed state, and and mostly um the the goal is to to reason about that state. And um, you know, given kind of the the hard thing is always that there are there's a propagation delays, and so something happens on like a broker crashes, it takes some some time for that uh that event to be processed on everyone that needs to know about it. And uh the you know the the really tricky thing is just the fact that by the time that you've processed it, you know, it may be that the state has actually evolved into something else. So the the broker may have been restarted. So uh your protocols need to kind of protect from all of these different edge cases where you you you think the state is one way, but it's actually some other way. Um and you you you your events can be uh received in the wrong order. Uh so you have to uh spend a lot more time with um you know validating the protocols to make sure that they're giving you the guarantees that you want and making sure that they're also simple enough to reason about. You know, I spend a lot of time um you know look looking at uh different different bugs that have popped up. And uh there they're just there's so many things in a distributed system that um can go wrong or happen in um unanticipated ways. So um, you know, we we think about often uh happy paths. Um but for a distributed system, you know, happy paths um you know it's not gonna get you very far. And you really need to be thinking about you know the corner cases. And even I would say you know you you reach the point pretty quick where just thinking about it is is not good enough. And uh so you you need more formal validations. Um and basically like you're you're trying to hit the system with as many possible states as possible, as close to the code as possible, um, and in order to you know verify that it actually works.
SPEAKER_01Yeah. Uh uh you mentioned that uh I forget exactly how you put it, but you know, things that things that might seem like they're not hard are hard in distributed systems. And uh I tell people this, I give a lot of talks at conferences and meetups and things like that. That's a big part of what I do. And I tell people and I play it as a joke, right? It's supposed to be a joke. And sometimes people laugh and sometimes they don't. Um, maybe it's not great material, but I always say, look, if you can possibly not build distributed systems, don't. And I'm talking to a group of application developers who are, say, building microservices on Kafka. Now, they're not building distributed systems infrastructure like you are, but they are, to your earlier point, you know, the clients count as distributed systems. Those are distributed systems developers. Uh they get to use infrastructure that hides a lot of ugly from them, but they're living the life. And I tell them, look, get a job where you can write a little bit of Python that runs on one core and owns the file system, and you'll go home at 3 p.m. and have plenty of time for hobbies and just lead a better life. Just go do that, you know, this kind of colic country sort of thing. Um, nobody ever takes me up on that. But and I I know I know you wouldn't. Um actually I'm gonna ask you why in a minute, but uh go on.
SPEAKER_00I was I was just gonna say, yeah, that there's just these the you know, these these trade-offs from you know have having monoliths versus having uh microservices, and that you know the the advantage of monoliths is that you know they're they're easier, um, you know, at le at least from some perspective. So from the perspective of uh you know testing them, they're often easier. Um, you know, you've got kind of in in some sense, you've got all the state in front of you. Um whereas in a distributed system, the state is kind of always behind you. You you're you're you're trying to think about the things that happened already. Um so yeah, it it's easier. Um you know, there then there are other concerns as well, like like you know, sometimes you create microservices just from scaling your application perspective, like just being able to scale it both organizationally and um scale it both uh like uh just in terms of uh performance and such.
SPEAKER_01Right. And I I I think managing complexity hopefully is the hopefully is the big driver there. But um a few things, and there's something that you said that I want to uh come back to if I can if I can uh bring it back to mind. It was about state. Yeah, so it's distributed systems engineering uh of the kind that you do is uh this process of managing state. And you kind of said that the state is all over the system. There, in other words, there are all of these opinions about what is true right now. And each one of the even the word now is a little bit fraught here, right? Like everybody has a clock and and some concept of what time it is, but um you don't really, you know, there is no sort of divine omniscient perspective that you're able to take. You just kind of have to fight it out among all of these imperfect views of what time it is and what's true.
SPEAKER_00Um Yeah, and often the goal is actually to introduce, you know, some some notion of a logical clock where you're you're you're reasoning about the system in a consistent way across the distributed systems, even given um you know disparities in times and such. Right.
SPEAKER_01And we a logical clock meaning a s a predictable sequence of events and not an account of quote unquote what time it is.
SPEAKER_00Yeah, more more like uh you know, like a Lamport clock, for example, where it's it's just like a sequential number which everyone agrees that you know one number happened before another number. Exactly. Exactly.
SPEAKER_01Uh yeah, the the question of the wall clock time is is you know maybe a thing that you can employ, but it's um it's a slippery concept in a system like Kafka. Because everybody has their own. Yeah, that's right. Um and I so like one one more final thought on that is that um really I guess what you're doing, if your view of distributed systems engineering is this uh coming to consensus on state, uh the the job of the infrastructure you deliver, that is, Apache Kafka, is to present a view to the client, to the user of the system, as if there is this one correct account of what is true. Uh there's like this froth in the middle uh in the system of everybody kind of arguing and and trying to make progress on consensus about what's true right now. But when you look in from the outside, it's just this nice, shiny kind of glass and stainless steel edifice that gives you an answer and it's a predictable answer, and you don't need to know anything about what's going on inside the building.
SPEAKER_00Yeah, that's exactly right. I think um, you know, and there there's kind of a difference between um, you know, some of the details which even even uh the clients are kind of hiding from the application. So um, you know, for for example, um in in we have a uh metadata API, and this is basically what the client uses to say, um, you know, like for a given topic partition, where's where's the current leader? Where should I go to get the data? Now you can get that view of the metadata from any broker out there, and the thing is that they're not always consistent. And you may fetch from one broker and see that the leader is, you know, uh node one, then you fetch from another broker, you see it's node two. So who's correct? Um, so we you know, we we have to kind of build um, you know, sometimes oftentimes in in cases like this, um you you know you're you're kind of exp abstract uh exposing our internal logical clock so that the client can reason about this. But from the perspective of an application which is uh using the client, of course, you're not aware that any of this is going on um behind the scenes. So really all you're seeing is um, you know, you subscribe to a topic, you fetch some data, um, and all the details about how it happens is is under the surface. Right.
SPEAKER_01As it should be. Uh a surface that you provide. So if somebody uh wanted to do this, like if they thought distributed systems infrastructure sounded attractive, um what advice would you give them? Like how what what's a good way to prepare? How do you get a job doing this kind of stuff? Um how do you get in?
SPEAKER_00Um yeah, it's a good question. Um I'm not sure I have a great answer for it. I think uh, you know, if you're interested in it, um, you know, the the definitely there's there's some um literature, the some background literature. I wouldn't say it's like required, but it's useful to have. Um, you know, the a lot of a lot of the the you know key results from distributed systems, they go way back. Um I mean like 30, 40 years back. Um that you don't have to know all of that. I think mainly um, you know, come in uh, you know, if for example, if if you want to work at at Confluent, um, you know, we come in and you just expect to learn. And uh, you know, there there's a lot of that I learned while I was um you know from from other people who had a lot of experience. Um and I think as long as you come in with an open mind, um you're ready and you understand that what you're getting yourself into, like that this this really is a hard problem. And um there there are often you know no no silver bullets, really it's about um you know finding the acceptable trade-offs. But you know, come in with with an open mind and uh willingness to learn um and you know prop possibly do some some um background uh research. Um and I and I think you you know you'd you'll do fine. Um yeah, I don't I don't think there's it it really needs to be much more than that. And that that's mostly what we look for in in a lot of um you know new people is just you know kind of a willingness to learn.
SPEAKER_01That's uh well said about trade-offs. Um as I say, in in any engineering discipline, there aren't any solutions, there are only trade-offs. Um and that's you know, you kind of have to have the maturity to to know uh when how to make them well.
SPEAKER_00Yeah, I think in distributed systems also sorry, sorry to interrupt you. I think um the the other thing is just you know there there are there are some problems which are actually you know provably impossible. So it's obviously useful to kind of have an understanding or at least an intuition about what those problems are so that you're not you know wasting a lot of time. Um so then that gets you kind of closer to what you know the trade-offs are that you can that are actually possible. So it you know, trade-offs which are impossible are not so useful to reason about.
SPEAKER_01Right, right. So you should know things. Uh if this is a discipline you want to get into, there are some of those famous results. Like, well, like you know, you said impossible. There's actually the the FLP impossibility result. Um that's a nice thing to know about. Like you should understand, particularly for a system like this, um you know, the atomic broadcast problem and what what is provably not doable. Um, and there's like that small, small list of the big results. Uh that that would be good background. But you wouldn't say that, for example, a PhD in uh distributed systems is not required to do this kind of work.
SPEAKER_00Yeah, that's absolutely right. Yeah. Um we we we we definitely don't expect that. We we just expect that people come in and um you know they're they're willing to kind of like it we you know with any Kafka is a complex system, uh like like any system like this. Um you know you have to start small and you you know when you're initially uh working on a a very small part of it, and over time you kind of uh grow the things that you understand that you think you understand well and that you can work on. And then be then you reach a point where you know you can actually um you know reason about the system, uh, you know, the full system and uh kind of the core improvements about it. Um but it takes time. It does, it does indeed.
SPEAKER_01And uh yeah, to be fair, there there isn't a small number of PhDs on the engineering team at Confluent, but no, that's not not a requirement. Um there's uh there's plenty of ways to be good at this other than that.
SPEAKER_00Yeah, and and even um, you know, we you know some some PhDs on our team, uh I wouldn't say their focus in uh was actually um distributed systems. So um, you know, probably what they've actually seen before coming to Confluent may be more like um you know, you know, maybe maybe some some work uh from more like master's equivalent type work in distributed systems. Um so most of them I think even you know they're often still having to learn quite a lot.
SPEAKER_01Cool. Uh PhD is in literary theory, not uh not computer science. No. All right. I'm gonna have several coworkers angry at me now. Uh everybody with PhD, I love you. And I know it's in computer science. Uh okay, so when you do interview, just ask more directly, I I assume you you get assigned to interview applicants. What kind of stuff do you look for?
SPEAKER_00Yeah, we we you know we we look for um you know I think the first thing is uh you know we understand uh people are coming in with uh different backgrounds. Um so if you're coming out of uh college, for example, we're we're we're just not expecting that um you you have a lot of system uh experience working in a system like Kafka. We're expecting really that um you know you're you're going to need to be able to uh grow on the job. Um whereas if you're coming in and you know you've got 10 years of experience and you've already been doing a lot of this stuff, or you know, obviously we expect you to know it. Um so uh it it it depends quite a lot on your background. I would say like um, you know, aside from that, you know, we're we're we're looking for um you know people that we want to work with um and you know people who we think we're gonna collaborate well with. Um so it's always good in an interview, you know, you you can establish a uh you know a good rapport fairly quickly. Um and in you know, in all all these systems, um, when you're doing designs, when you're doing uh code reviews, you know, you're you're working with with people and and you need to be able to uh work with people. And that's always like a kind of a give and take. So um, you know, some someone makes a suggestion and and you know, you you you you spend some time thinking about it and you may not agree with it, and you but you explain yourself um and sometimes you you know you you you agree with it and uh you change yourself. So I think we we we we kind of expect that people are you know in general good listeners and um and take feedback and uh you know taking feedback doesn't just mean that you you do what we say, it means more that you think about it, um and you you you come to your your your own conclusion. So we're looking for for that. I think uh, you know, and a lot of this stuff is is extremely difficult to try to evaluate in the context of an interview. Um, you know, stuff like uh you know, we expect people to come in and you know be be pretty proactive. Um, you know, t take some initiative and uh you know you don't understand the full system, so it takes a little bit of time. But um when you see a problem, you you know, you you try to fix it or um you try to come up with some way, uh, decide whether it's worth fixing. Um so you know, initiative and willingness to work with others. And if you've got all of those, um, you know, depending on depending on your level, then um, you know, we'd we'd really like to work with you.
SPEAKER_01That sounds great. There's a lot of uh a lot of non-technical uh properties you just described.
SPEAKER_00Yeah, it's a little bit harder to describe um you know the technical uh properties. Uh I think I think like um you know, for experienced people, yeah, obviously we expect you to be uh some pretty familiar with um you know running multi-threaded uh or or reasoning about multi-threaded systems. You should you know you know understand how to use a mutex, for example, and when you would you do it. And um if you're coming in with um you know less less experience, you know, we're we're we're you know trying to evaluate um you know how how well you think about problems and um you know we the kind of kind of traditional. I think our the the coding problems that we do, they're um you know, we're not trying to ask quick trick questions. Um we're we're mostly just it's a you know treat them as like a practical question. And uh how do you how do you get to some answer? And getting getting to some answer off often involves like, you know, uh asking some questions, uh playing with some examples, um if we see kind of all of this stuff, like even if you don't finally get to the optimal solution, um, you know, I actually we we probably still want to work with you.
SPEAKER_01Nice. That sounds good. What is um this this sounds like such an HR kind of interview question, and I I don't want it to be like that, but um I'm interested in a thing that has formed you that was like a a negative experience, something that well just say something that you screwed up at or that went wrong or that uh that that didn't go well in your capacity as a distributed distributed systems infrastructure engineer. Uh what's a technical failure that you have grown from? Jason says we do those.
SPEAKER_00Yeah, well I mean I've I've I've made plenty of this m mistakes. Uh um, you know, um when you write a lot of code, I think you kind of get used to it. Um daily process of beating. So yeah. Yeah, I I think um you know that there are there are some some cases where um you've got a um a problem which may maybe you're just initially a little naive about and um you you don't kind of understand um the the full implications of of changes. And you know, often they come back to come back to bite you. Um so and I I I don't know if I'm I'm you know racking my brain to think of something that's uh really specific. I I think this is more like a um you know something in in in Kafka. You know, we've we've had a lot of um you know misses or bugs in the replication protocol, some edge cases basically that you know we th we had thought really hard about it, we thought we had handled them. Um then you know we we we pushed out fixes, and then it's got a report, you know, sometime not too long after that saying it was you know there's still some problem. And then had to like really kind of go back and um you know uh find more formal ways to uh reason about the system to avoid just you know take like the human errors uh out of the picture as much as possible. Um I think that I'll also just you know you know, working in a community, um, so that this is this is not like actually something that technical, but just more like collaborative. Um you know, working in a community, like Apache community, um, you know, you you you need to really be receptive um to uh or or or listen well to uh what what people are trying to do and um you know and and how you communicate on on code reviews and all of this stuff is important and you know if if you're if you're not kind of careful with it, and you you can really um you know uh people will lose and lose interest kind of in what they're doing and just uh kind of go away. And you really don't want that. You you want to be um you know make make the community friendly and um provide feedback in in ways that are helpful. Um so yeah, I would say early on, uh you know, I I think I didn't do a great job of that in some cases. Uh I think I've gotten better. Um but uh you know that that's something that I I think had to learn kind of by by uh doing a lot of work in the in the community. Sure, learn learn by uh uh by trial.
SPEAKER_01And on that that last point, um, and even this would be a good thing to close with, um Kafka as a community, you know, this this is an Apache Software Foundation project that you spend a lot of your time working on. Um how uh you know what are the what are the wins there that you've seen uh in terms of contributions and discussion and just how does that how does that go? Uh you as an engineer, your full time is on this. What's it like to be a part of that community?
SPEAKER_00Yeah, um, so I think that you know the the um the contributions we get from the community are always great. So uh people are people are running the system. You know, oftentimes we get um JIRA's people who have spent you know, clearly a considerable amount of they ran into some problem and they spent a considerable amount of time uh trying to understand how it worked and what the problem was. And um, I mean that those those contributions have uh a huge amount of value. Um you know, oftentimes we we also kind of get the other side of it, which is you know, someone ran into uh a problem and just kind of threw it into a JIRA. Um but but you know I I would say we we do our best with those, you know, but uh the ones that are more helpful are the ones that were people who have done a lot of investigation. And we also get just um people who are you know kind of expanding Kafka to uh new use cases. And uh new use cases often mean new features and uh you know, finding a way. I think uh uh the Apache Kafka community has a nice process for uh you know implementing new features. Uh this uh we've got Kafka improvement proposals. Um so anyone can come up and write one of these. Um and you know, it may or may not get accepted. You know, you uh I think come if if you have that in mind, you have to come kind of with an open eyes and and that understand that there's there's a history of the project, and what you're asking may be something which has already been thought of ten times and been rejected ten times.
SPEAKER_01Um there are leaders in the project and you may not be they're yeah, that's right.
SPEAKER_00And and and just there's n the history about the you know, this decisions that were made, and often they were made you know just for technical reasons, and you know, you should spend a little bit of time uh like re-reviewing that and seeing, well, has have they thought about this before? And if they have, what conclusions did they reach? And maybe your conclusion, maybe there's something that we didn't think of or something, and and you have uh a different approach. Um but yeah, in in general, the unit you know um uh we we welcome people in the community and uh uh we welcome the contributions, and I think we get a lot of value out of it.
SPEAKER_01My guest today has been Jason Gustafson. Jason, thanks for being a part of Streaming Audio. Thanks, Tim. It was my pleasure. And there you have it. I hope that was helpful to you. If you've got questions, you can ask me at TL Burgland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time.
