Distributed Systems Engineering with Apache Kafka ft. Jun Rao

Jun Rao (Co-founder, Confluent) explains what relational databases and distributed databases are, how they work, and major differences between the two. He also delves into important lessons he’s learned along the way through the transition from the relational world to the distributed world. 

To be successful at a place like Confluent, he outlines three fundamental traits that a distributed systems engineer must possess, emphasizing the importance of curiosity and knowledge, care in code development, and being open-minded and collaborative. You may even find that sometimes, the people with the best answers to your problems aren't even at your company! 

Originally from China, Jun moved to the U.S. for his Ph.D. and eventually landed in IBM research labs. He worked there for over 10 years before moving to LinkedIn, where Apache Kafka® was initially being developed and implemented. 

EPISODE LINKS

• Get 30% off Kafka Summit London registration with the code KSL20Audio
• Join the Confluent Community Slack

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

There are a lot of specialties within the very broad vocation of software engineering, and all of them are hard to do. Distributed systems engineering is one corner of the discipline that poses a particular set of challenges. What's it like to build a distributed system? What special problems arise? How do you land a job doing it? That's the conversation on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome to another episode of Streaming Audio. I am your host, Tim Bergland, and I'm delighted to be joined in the virtual studio today by a man who is one of the co-creators of Apache Kafka and a co-founder of Confluent, Jun Rao. June, welcome to Streaming Audio. Thanks, team. Thanks. Now, a lot of people who follow Kafka know who you are. But I know even I don't know much of your story. I'd love to hear just how did you get uh how did your career get to be here? Uh what where did you get started and and what was your journey?

SPEAKER_01 1:09

Yeah, I can talk about it myself. Um I grew up in It's easy, right? Yes, it should be easy. Yeah, I grew up in China. I finished college in computer science there in in Beijing. Then I came to US uh uh for graduate school. So I spent about five years at Columbia University doing my PhD there. Um my PhD is in databases. So I liked the you know database part of that, you know, dealing with uh systems, um performance optimization, and this kind of stuff. And while while I was doing my PhD, I did a few internships. And then one of the internships I did uh was uh at IBM's Air Madden Research Lab down in San Jose. Um so that's actually a group that has a little bit of history, uh, in especially in the database world. So this is actually a place where the relational database technology was invented back in the 1970s. Um so worked with quite a few sort of database legends uh in that lab where I was doing internship. Uh I like that lab. So after I graduated in 2000, um I took a job there. So that's how I started. So I started, I spent about like 10 years in that uh IBM's research lab. Um initially I worked a lot on some of the DB2 stuff. This is actually IBM's relational database technology. Um but uh towards uh sort of the end uh of uh my IBM career, I think I started looking into some of the open source software stuff. That's when I got uh interacted with some of the newer things that's coming out around that time initially, some of the Hadoop stuff, then some of the key value store stuff like Cassandra, HBase, uh, and even some of the search index, uh uh like Lucene and other things. So what I found pretty interesting in that thing is I think the way you build systems is quite different from how DB2, you know, a conventional database build their systems. Um a lot of things we did uh in DB2, you know, is just trying to optimize very, very hard about the code pass, right, to squeeze in the last 5% performance. Um as you can imagine, you know, if it's a system that's 20 uh or 30 years old, it's kind of hard to get that thing in the complex system. Right. A lot of those a lot of those uh optimizations have already been made by the time you come around. And also the system is more complicated, right? I think when you optimize one component, you have to make sure it doesn't impact other things. Right. So but a lot of the sort of new other distributed open source software, they took uh just a slightly different uh perspective. I think the way they do that is not to like squeeze like the last bit of performance just on individual node performance, but try to build a more scaled-out um distributed architecture. So then now if you want to get some more performance, you know, it's pretty easy, right? You just get more hardware, which is getting cheaper and cheaper over time. So that changed a little bit, you know, how I think about things in software development. Then around 2010, that's when I thought, yeah, I think I I want to pursue some opportunities outside of IBM. So that's when I got the opportunity to work at uh LinkedIn. So at that point, um LinkedIn was just starting to build Kafka. So I was able I was fortunate to be able to essentially work on Kafka almost from the very beginning um uh and at LinkedIn. And then we were able to get it finished and then uh get it used at LinkedIn. And that turns out to be uh pretty successful for LinkedIn's usage, and then we open sourced that initially to just GitHub, but uh eventually to Apache Foundation. And uh that's when we saw just significant growth of adoption of Kafka, uh which eventually led to the starting of Confluent and uh the uh the stream event streaming platform we built uh off this platform.

SPEAKER_00 5:36

Which leads us rapidly up to the present day. That's right. Um so you said uh you said some good things in there. And and by the way, I didn't I didn't realize you were uh at IBM earlier in your career. These these are things that a good host, just research a good host would have done before recording. But I can learn directly from June just like you can in the listening audience. So that is that is super cool. Especially that lab. I mean, like you said, that's that's foundational work that that impacts the life of like every software developer happened there.

SPEAKER_01 6:07

Yeah, because that's uh yeah, that's the part I felt fortunate because you know that's that lab is not that big, is you know, about 20 to 30 people. But that's a lab with like a big uh sort of concentration of just uh the brain powers in a database area. You know, I got to be able to talk to um, you know, inventors of uh C core, early inventors of the like the recovery mechanism, uh algorithm, and um people who did the like initial version of uh parallel or distributed the version of the database. So I learned a lot from those people.

SPEAKER_00 6:46

Yeah, yeah, I imagine you did. Um you uh this isn't really a question, but you just you talked about the transition for you from uh relational database developer to distributed database developer and that uh you know, all that stuff that was happening around 2010 and the few years before in the open source world. And it was an interesting observation that you know, in the in the relational database world, which is fundamentally a single server game, right? You can do multi-core things, and multi-core was starting to become a thing around then. Um but it you know, the the technology fundamentally relies on at some point there's a lock on a data structure, and you can you can take a lock on that data structure, and therefore then you can do things with with certain guarantees. So that whole game, like you said, is all about optimizing code paths and how where can I get another five percent? And that sounds like a grind, you know, like I could see where you could maybe get tired of that. Um, and the approach we take in the distributed systems world is it's okay, it can be Java, and maybe you know, there could have been uh native languages that were faster, or we could not contend with garbage collection, or you know, whatever. A lot of that a lot of that stuff was Java, like like uh, or at least JVM, like Kafka. But the approach in the distributed things is just scale out. Like it's okay if the single node performance isn't optimal. Um anyway, I'm just kind of reflecting on that. And even in the the NoSQL space, that has played out. There's there's Cassandra and there's the uh native you know C version Scylla DB, which is like the the Cassandra with fast single node performance. So it's weird how that that debate still kind of plays out there. But anyway, uh taking a long time.

SPEAKER_01 8:36

Yeah, I I I should just clarify. I think um understanding of the performance, right, and then care about performance even on a single node is still important. Um I think you know, even if you build, even if you're building a distributed system, as long as it stores data, it retrieves data, is actually very useful to be understand, to be able to understand your code path, you know, what's your critical path, how does it use disk, uh, how does it store the data in an efficient way. Right. I mean so read the release notes if you don't if you don't believe it. I mean that's that stuff gets done. That's right. So I think I think what distributed system helped is because of now you have this uh scaled out capability, you don't have to over-optimize your system um just to get through what you want, but at a cost of complicating your design. Right.

SPEAKER_00 9:28

Right. There you go. That's uh I think a much more precise way to put it. Um and you also maybe replace the grind of relentless code path optimization in all cases for the grind of now you're a distributed systems developer, and so you've inherited a new set of problems, which is kind of what I want to talk about. Is um you know, you made that transition to being a distributed systems engineer, and your I mean, your presence on the podcast here today is a part of a series on distributed systems engineering. So I kind of want to talk to you about that. You know, you've made the jump, and now you interview and hire and lead uh and work with a number of distributed systems engineers. So what really is the difference? I mean, what what makes a distributed systems engineer that and not something else?

SPEAKER_01 10:18

I think first of all, I think they're just the good trade off, you know, just most of the engineers, right? I think a lot of engineers, I think, you know, you just you have to have clear logic, right? So you have to have you know good high-level logic uh when you do the design, and then when you do the implementation, then a lot of those is you know having good practice, having uh sound logic, and then be careful, you know, where uh with the code you are written. I think uh for the distributed system engineer, I think what's making that a little bit harder is a lot of the coordination across different servers. So there are different models there. You know, do you have like a centralized coordinator to coordinate everything, or do you do the peer-to-peer thing, each sort of have its own challenging. So that a lot of the challenges, you know, at the end against time, then you have to reason about which state you are in, right? How and how that is uh consistent or different from the state in other servers. So that's part of uh things to make that uh a little bit tricky because you know you not only you have to think about the the local thing that you are working on, but the impact that you would have on other related uh uh nodes in the service.

SPEAKER_00 11:45

The evolution of state and the reasoning about state is a topic that has come up in our previous discussions on the podcast about what it means to be a distributed systems engineer. So that state problem uh seems to be a rather central thing. Um what uh uh when you uh are hiring, uh when you're thinking about what makes a good distributed systems engineer, when you look at your own development uh in this discipline, what is what are the the properties, like what skills should I be good at?

SPEAKER_01 12:20

Yeah, I think uh of course, you know, there are lots of aspects, but personally, for me, uh I'm mostly looking uh at like uh three things. These are the things I think I particularly value, and uh I think I also see that in a lot of other sort of uh good distributed engineers. The first thing I would just say uh just uh uh the curiosity. You know, this is the curiosity of knowing the code and knowing it really, really well in detail, especially the core parts, right? Um so this uh can happen when you write a code, but can also happen when you read a lot of time, you know, existing codes, you know, through reviews and other things. Um if there are things that you don't understand, uh you try to uh ask around, especially experts in this area, right? Um what this logic means, does that match your expectation? And the same thing applies um just to the operational part of that. Um when you see an issue in production, uh are you curious in terms of understanding what's really causing it all the way to the bottom of it so that you can fully explain the situation. So that's I would say probably the first thing. The second thing what I value a lot in a lot of the distributed engineers is uh they care about uh not only the development of code, which of course is often a given, but also the operational part of the system. Um so a lot of the engineers think they would uh uh uh they would not be shying away from like encore. You know, they want to be part of the operational thing. And uh when there are issues, they try to figure out exactly uh what's causing the issue. And uh as part of this, I think it's just I think a lot of the skills they have to have, right? It's uh of course uh primarily they are developer, but to be good at operations, they also have to be a little bit broader just in terms of the whole system. So that includes things like understanding a little bit how the network layer works, right? Uh what does the TCP connect uh timeout means sometimes? Uh what does uh RAID means, you know, the difference between RAID 5 versus RAID 10 in terms of performance and uh storage need. And uh things like uh Java uh profiler, right? Garbage control, uh garbage control, and uh where is the hotspot through through profiling? And uh sometimes you know even lower layer, uh lower layer than uh the TCP, even at the network layer, you know, how do you uh figure out if uh if there's a potential issue due to um like uh uh uh like denial of service because you you you are being rate limited through some of the low-level tracing uh at a network layer. So that's sort of the second thing, which is you know a lot of the uh uh operational expacts uh of the distributed system is probably also hard, and it's but it's all equally important. So being able to um have the ability to reason about operational part of the system, that's also very important, I think, to me.

SPEAKER_00 15:58

And those those things you're describing there are all uh it occurs to me what they have in common is that they are lower levels of abstraction. There are APIs that wrap all those things that a normal developer normal, like distributed systems developers are as normal, right? You know what I mean. That's right. Uh a non, just an application developer, uh like the kind of code I I uh was writing when I was uh working as a developer. As an application developer, you have APIs for those. You're you know, you're using Spring and annotating a method uh so that REST stuff happens, and you you're aware that there's an HTTP uh API there and that there's TCP under the covers. And you know, you have to know some response codes when things go wrong, but you know, you're operating uh probably two layers of abstraction up from there. Likewise, the file system. You know, you've got uh some kind of file API, uh like in Java, you've got Java.io file or Java NIO or whatever, but you're all of those are higher levels of abstraction, which are necessary for people to be able to write applications. But what you're saying is in the in the distributed systems world, uh you can't. You have to know those things.

SPEAKER_01 17:08

That's right. Because I think, yeah, I think from an application developer's perspective, a lot of uh what do you need, right, uh provided is sort of given in the aspect, uh, the description provided by the API, right? So you sort of know what it's capable of doing and uh what it entails you to do. But I think a lot of the sort of system engineer or distributed system engineer challenges, you know, a lot of details are hidden inside those APIs, right? Because a lot of things can happen to serve a particular API request. And uh if it doesn't um work as expected, uh whether it's performance or other things, then you have to dig deeper into all those uh potentially related components.

unknown 17:55

Right.

SPEAKER_01 17:56

Uh somebody has to answer for that. Yeah, so the last part I would say, I think, in terms of trade, uh, you know, just uh being open-minded. So I particularly like that quote from Beer Joy, who is uh one of the co-founders of Intel. So he once had this quote saying that it doesn't matter um what problem you are trying to solve, sometimes, a lot of times, the best person um to solve a particular problem often works outside of your company. So, you know, you know, Kafka uh uh we started as an open source project and we continue to make that open source. I think one of the things I just learned through the interaction with the open source community is just that even though uh personally for me, I think I uh had a sort of a uh deeper background in Kafka. But when I try to do reviews or interacting with other people or trying to develop new features, I find myself I learned a lot just from all those people in the community. Um sometimes, you know, they even if they may not have been involved in Kafka um that long, but sometimes they they actually share pretty good perspective uh just based on the background they had. I think a lot of that, if you are just being a little bit open-minded, you end up with a better or simpler solution uh just by listening to others.

SPEAKER_00 19:22

Excellent advice. And uh good point about when you're in the environment of an open source project, uh you do have a community. I mean, that's not just people, well, it is people working for other companies, and maybe maybe those companies are paying them to work on the open source thing. I mean, that's that's uh fairly typical. But um the the the reason that network of people knows each other and interacts with each other all relates to their involvement with the project. And if you give them uh a place to talk, like a mailing list or Slack organization or something like that, um, then you know that community is no longer an abstract thing, but it's a very concrete thing and concrete place where you can go to to ask questions and get insights and be surprised at how many people there are smarter than you are and and that whole thing.

SPEAKER_01 20:14

Yeah, in the uh in the end, that uh makes the projects better longer term.

SPEAKER_00 20:18

It does. It does because you can't, you know, you're you're a company like Confluent. You're you're trying to employ lots of Kafka talent and lots of distributed systems engineering talent, because we're you know we're building a event streaming platform on top of Apache Kafka, and so naturally we want to be good at those things, and you know, you're good at those things, and that's that's kind of what this uh podcast is all about. Um but you know, you can only employ so many people, and there's this whole ecosystem of equally brilliant people out there, and so giving them a place to interact and support each other and get to know each other is just hugely valuable. And obviously that's a passion of mine. I mean, that's part of what I do here, is is uh developer relations and and uh community. So I'm I'm glad to see that be a thing. That's great. Yeah. Um so from if you remember uh the your transition from database developer in, I'll just say the most probably world-leading uh elite database development lab um in the known universe, uh going from there to uh becoming a distributed systems engineer working on on Kafka at LinkedIn. What are the things that you ran into as difficult and that that felt like lessons learned? What did you uh maybe you struggled with something? I mean, what what was that like?

SPEAKER_01 21:41

Yeah, I think uh wow, I think when I made uh my career change right from IBM to LinkedIn, uh it's actually a rather big change for me. Where first of all, I think I moved from a 100,000 person company to a 600-ish person company, um, just in terms of size. And And I also moved from research, which was the lab I was in, to to development. So those are actually big changes for me. But for me, I think at that point, what I think I valued, you know, is just uh the uh first of all is just uh the experience to be able to um see what you build get used uh as uh firsthand experience. Right. So I seldom get that opportunity at IBM because it's a big uh uh company. It's also um a company with a product that has like long release cycles, right? So you know you put out a release, you do you add a feature in a release that's only uh getting released maybe two or three years later. So um and then after that, because you are so decoupled from your end users, you seldom hear any feedback from them. So what I wanted you know, just to be able to be at a place that, okay, I can build some interesting stuff, but also see a bit how users are using that thing so they can have those more direct feedback. And I clearly get that uh at LinkedIn.

SPEAKER_00 23:11

Yeah, because a lot of that initial Kafka building was was really I mean you were building data infrastructure for use internally. So you're building it, somebody's using it, you talk to them, game changing.

SPEAKER_01 23:23

Yeah, so I think in terms of a hard problem, I think, yeah, there are there are problems that's hard sort of just on the on the technical side. I think a lot of those are sort of operational related issues. This happens probably in our early development and also happens when we load out some of the earlier features. So just like provide like maybe a more anecdotal thing. So that's probably 2012-ish. Um, so that's when we just released like a new version of Kafka that supports uh like replication. So then the issue we have, you know, just uh starting from some time, you know, every Wednesday, uh 10 a.m., what we'll see, you know, just we'll have to see like a uh a window, like five minutes, where all our clients who are talking to that shared Kafka clusters start hitting timeout. And then they will keep reconnecting, and then which will add more load to the server. Um it will be in this sort of a uh uh weird state for just about 10 uh five minutes. And then without us doing anything, um everything just uh uh switched back to normal um afterwards. And this happens every week, exactly at Wednesday at uh 10 a.m. Yeah, so this puzzled us quite a lot, I think. You know, it's uh just from looking at the volume of the data. It's not like at that time there's like a big increase in terms of volume or anything. Uh so it does took us quite a lot of time just to figure out what's causing this. Um so what we did uh to to investigate this issue, you know, just we look at all the like metrics we have, right, uh at Kafka, and uh around that time, and we do see that some of the latency metric, you know, like uh the time we spend serving like a produce request or maybe sometimes you know consumers, is longer. Um but uh we just don't know what's causing that to be long. And uh just from uh some of the um jVm metrics, and then maybe even just uh uh system metric like CPU, it doesn't seem there's like a big change in anything around that time. So uh so what we well what we uh discovered in the end is after some point, um what we just realized just from the fire system, it just we noticed that around that time there happens to be a lot of fire deletions around that window. That's one of the sort of uh interesting things that we noticed that. So then later on, I think we look at some of the configuration um in Kafka. We just realized, okay, at that point, Kafka has this uh property that controls how often we roll a log, you know, if the log hasn't been like uh actively uh written so that it will hit the size limit. So that default value is uh seven days. And we also have another configuration which is you know controlling the retention, you know, which is how long you keep those data in Kafka. And the default setting for a lot of topics is also seven days. So what ends up happening is um uh uh around that you know Wednesday at 10 a.m. thing, a lot of segments have been rolled, which will cause them to be deleted like seven days later at exactly the same time. So because there are so many, uh there are probably around thousands of those files like deleted around that window. So so that actually added a lot of metadata pressure in our underlying file system. So even though there's not a lot of additional new data just because of metadata pressure, that actually slowed down a lot of the IOs in our system, which essentially um put pressure back to our clients. And the reason why you know it lasted like for five minutes, you know, it's just I think we have this cluster which often gets like rolling restarted around the same time. So then that same effect will happen just gradually from one server to the other until the whole cluster is exercised. So that's actually one of the hard things for us to figure out and uh uh and discover. Um took us probably uh multiple weeks just to get uh DNA done because you know to to do some analysis, you have to wait another week for this to happen again. Um but once uh we identify the issue, the fix is actually pretty simple. So we just add a little bit of jittering effect when we do log rolling or doing the log deletes so that we can smooth out the metadata overhead over some period of time.

SPEAKER_00 28:33

Yes. So you you weren't in essence uh DOSing the metadata management of the file system. That's kind of what you're doing. Excellent. Go on. What's what's another uh these are these are great stories, another lesson.

SPEAKER_01 28:45

Yeah. Yeah, so that's that's sort of more like just uh like operational uh I think there are quite a few of those, I think, just because of you know they are distributed systems in nature and they are large scales, um often I think it uh it does take time to figure out those. I think another thing uh that's that's pretty hard, I think, um in the days of LinkedIn, I think, it's just I think the sort of uh scale of the data and the customers I think we have to manage. Once we deployed Kafka at LinkedIn, that became essentially the hub for pretty much all the microservices at LinkedIn. Around that time, there are like hundreds uh of those microservices. So essentially they are all our customers. So then we only have like a relatively small development plus like SRE team to manage all the coffee classes at LinkedIn. So the challenge, of course, in the early days is uh because Kafka is used pretty much in every application, and then often it's like the entry point, how they interact with the data. If there's any issue in the application, the first thing they would think of of blaming is Kafka. So you know, just that's a lot of the time we spend is just to answering those applications uh where they may have suspicion on there's a problem with Kafka. Often, you know, we just after some investigation, we just have to point out, okay, the issue is not in Kafka, it's somewhere else. Uh maybe sometimes it's you know the application problem, and uh the application owner has to fix it. So um it's just I think having to deal with all those usage at scale is pretty challenging. So what we uh helped a lot in the end is so we build this uh tool called Auditing Tool. Now I think at Confluence we are pure we are sort of generalizing that concept and build a similar tool in uh our product offerings called Control Center. So what this tool does essentially is just to add a little bit like instrumentation in all the publishers as well as in all the consumers as they are publishing and then reading the data. And then we collect all those uh statistics of those instrumentation and build a service off that. So with this service, then we can in real time check whether the how the how the data is flowing through all the publishers all the way to our consumers. We can tell you know what's the percentage of data coverage, whether there's any data delay or uh data missing, and then how late the consumer is with respect to to the publisher. And this is actually built as a dashboard, everybody in the company can check that. So with this, this actually helped a lot. So now if uh an application has an issue, right? Let's say they have an issue for a particular topic and they think it's a Kafka problem, often now answer you know, just go check that dashboard. Because that dashboard has that uh monitoring and for the instrument, instrumental data for all the topics at LinkedIn across potentially all the uh consumers. We can just say, okay, um, is your consumer up there, right? Uh is that delayed or not uh from a Kafka perspective? And if so, are you the only consumer being delayed? Because if you are the only one, chances are the delay is probably caused by your application, not the infrastructure at the Kafka. Uh but if everybody is delayed, then we can take a closer look because that may be something that's related to the server. I think that just having that dashboard, having that tooling around the system and make that globally accessible helped reduce a lot of the load we have early on.

SPEAKER_00 32:57

Awesome. Tell me about the role of simplicity in the life of a distributed systems engineer.

SPEAKER_01 33:04

Yeah, I think that's actually uh also interesting. That's sort of one of the things uh I also learned uh along the way. Um I think um a lot of what I I learned from the literature and also learned from other uh engineers, you know, you just you want to, of course, um uh strive for simplicity, right? Um the simpler you have, the easier it is to build, uh, and then the easier to uh to debug, probably, and then probably the easier it is uh to monitor other things. So a lot of the design and the implementation, I think, in Kafka, I think, especially in the early days, what we strive for is try to make it as simple as possible. But I think over time, what I learned is I think simpli the sort of ultimate simplicity is not necessarily the ultimate goal uh or the best thing to have. What you want is essentially you want to have simplicity, have simple stuff, but not at the uh at a cost of having uh having you sacrificing in some other aspects. So I'll just uh provide like maybe a concrete example to illustrate on that. I think there are quite a uh a few examples we have learned in this space. Um one of the things I think uh uh if you're familiar with Kafka, um you will know in Kafka Consumer, we have this model called the group consumption, which allows multiple instances of consumer to jointly subscribe to a topic or multiple topics as a group. So then we have this distributed framework to divide up the data among those uh uh subscription instances uh evenly. And uh we also monitor the liveness of those instances. If any of them goes down or some new ones joined, we'll trigger this rebalance protocol to make sure the data is uh balanced again with the new seven members. So when we try to design that, uh we know we have to have some protocol between the client and server to coordinate this behavior, right? Because in the end, they have to be jointly agree on the final assignment uh for the data of those topic partitions uh in the same group. So initially, what we try to do is try to try to use just one round of remote request protocols between the client and server. Because that's simple, right? Uh that's a simpler thing uh that we can do. So we try to squeeze as much as we can so that we can make all the decisions just in one round of this RPC between the client and server. And it's we in the end, we sort of quite we sort of figured out a way of doing that, but it does have some implications. So one of the implications is uh this actually makes it uh harder to debug. For example, because we only have like limited rounds of communication, the the server side actually doesn't know the final assignment of those uh uh topic partitions among those instances. All the clients know it. But since we don't have another round of communication, the server actually doesn't know that. So this is actually one of the impacts, and then there are a few others. So uh we just think try to thought about it in a harder and hand, but you know, we can't get over of this limitation. And in the end, what we settled on upon is uh it's not the simplest form. We settled on a two-round of RPC between the client and server to complete the rebalance in this uh group consumption model. So what this gave us is a lot more flexibility. First of all, this actually allowed the debugging and monitoring a lot easier because now the server does have the information about all the assignments, which is actually really helpful when you want to figure out uh some of the performance issues or potential bugs. The second thing is it actually allowed us to plug in different policies of how you want to do the assignment. So it's not just limited to a subset of the assignment policy that uh you can rely upon uh those distributed consensus. So it actually can be applicable to an arbitrary way of assigning the system. So this actually made the group protocol for those consumption uh much more general and much more powerful. So just looking back, I think to me, just um in the end, I think that model, by the way, I think at the at this uh at this moment, you know, after two to three years of working on that, I think worked pretty well. Um so looking back, I think what I think I uh learned from this is just sometimes um the ultimate simplicity is not always necessarily the best. I think you have to look at it within the context. I think you want to uh get to certain level of simplicity, but without having your hands tied up, uh too tied up for a lot of other things. This can be expressiveness or uh ease of use or ease of monitoring of debugging.

SPEAKER_00 38:37

Very well put. Um there's uh I I'm kind of of two minds on this. I mean, having read code that other people have written, and even the code that I've written, um I think our collective drive for simplicity maybe is not as great as we would like to think it is. Uh, I mean, a lot of people make some really baroque things that are just needlessly complex. But we definitely say, like our stated belief as software developers is that simplicity is a is a priority. Whether we live that out or not uh is another question. But I think your you know, this sort of mature approach you're advocating here reminds me of um Occam's razor. You know, the the way Occam's razor, that principle is normally stated, um, refers to a uh medieval monk named William of Occam, who well, we'll get to what he said. What most people say when they're talking about Occam's razor is well, the simplest explanation is the one that's true, or something like that. Um, which is not what Occam said. What Occam said is that one ought not to multiply entities needlessly, uh which is a very different statement, right? Needlessly is the key point there. You you shouldn't make something more complex. And he had a very particular philosophical context in mind uh when he said that, but you shouldn't make things more complex than you need to. But like don't don't don't lose it here. We don't want to break things or or um unnecessarily limit the the future evolvability or current behavior of the system in a drive for simplicity.

SPEAKER_01 40:13

Yeah. So I have another example on that just in the Kafka land. I think that's also related to the consumer side. So uh so one of the things I think uh we're trying to do in the Kafka consumer is we try to reduce the number of background threads we have in the consumer. So we thought, okay, it won't be easier if the user threads who is consuming the events, that's the only thread who is driving everything. So this will be the threat who will be using, who will be responsible for processing all the data once you get those events. But that's also the threat who will be driving the communication between the client and the and the brokers, potential multiple of them, and then multiplex mounds. So that's what we we had initially, um, which uh um definitely has the merit that you know, I think the fewer threads you have to manage in the application, probably the better. But what we realized um later on is because this single-threaded model, it actually complicates a little bit the how uh how the application interacts with the server. Because one of the uh main issues we realize is even though Kafka is designed for like real time, some sometimes when you apply that in application, sometimes applications have some downstream dependencies. And uh in the case when there's like high volume of data, it actually may take some time for them to finish processing all those events once they get them. And that time can vary. And it's it's it's kind of hard to have to set up like a static session timeout to accommodate for all those use cases. So once the first version of this uh sort of single-thread consumer is out, I think the common thing we have seen from a lot of users is uh uh they see a lot of the timeout of their consumer session. I think when we looked into that, a lot of the cases is not you know, Kafka is itself, the library or the library is doing some actual work. It's really the application is doing something that sometimes is not fully in their control. So we iterate on that like a few times. In the end, we just realized, okay, so maybe we just should uh just uh uh uh give up on that uh single-threaded model. Because that actually uh even though it adds some additional threat, but that will make the system much more easier and flexible to build. If you had allowed multiple threads in the consumer, okay, which would be a that'd be a huge change. That's right. Well, what we did is uh you know we just add one more thread, which is essentially a background thread that allows the consumer library to do things like heartbeating to the server. So this is actually separated from the thread who is doing all the processing of those events. So this way, I think we sort of decoupled the processing of those events from the liveness detection of the client. So they are sort of uh independent. So even though we added a little bit more complex in terms of like managing those different threads, but architecture wise, this actually made it uh a lot easier to build applications using this uh consumer new consumer library because uh it's more stable. Um, it's easier to to understand whether the issue is in the application versus in the library itself.

SPEAKER_00 43:58

That makes a whole lot of sense. That's definitely a trade-off of complexity for just kind of sanity elsewhere, because you know you have another thread, but now a lot of things work better. I mean that that having those having heartbeat and consumption in the the same thread led to some well-known problems. Um another question. Tell me about usability. Uh some people say user-facing design, but what where do you think usability enters into being a distributed systems engineer? And I ask that because it's not something that people normally think of. But do you think there's a relationship?

SPEAKER_01 44:33

Yeah. Yeah, that's also a good question. I think that's sort of another sort of lesson I think I learned uh those years, you know, we're working on Kafka and some other distributed system, which is as a as a developer, right? If you are a developer of a distributed system, I think you have a not of a lot of knowledge of all the components of the system, uh, how they work internally, um, and uh what's the meaning and then what's the trade-off of those uh uh different components. But I think sometimes you know when you uh when you build a system for the user to use, right, you have to define some user-facing uh properties so that the user can understand. So often I think as an engineer, as a distributed system engineer or developer, you tend to sort of uh design those user interfaces based on the knowledge you have as you are and as you understand uh how the internal system works. But um sometimes I think that may not be the best thing for the user, um, because it just adds probably too much complexity or giving too much information to the user that they don't necessarily need to know. So I'll just give like a more specific example of that. So one of the things in the in Kafka when we uh this also uh uh a few years back, when we try to add a new property in the producer. Because one of the things uh the producer does, right, if you know how the producer works in Kafka, it's actually fairly involved and has multiple components. So if a user try to send the event through the producer API, it actually has to go through multiple phases. So first thing it has to do is it has to be added into this uh in-memory buffer pool the producer manages if it has space. If it doesn't have space, it will potentially block that for some configured amount of time. And once you you are in this buffer, we typically wait a little bit so that you can hopefully accumulate enough bytes so that they can group together to be sent to the server. So this is actually to improve the performance. So once you have enough uh bytes uh to form a group, we take it out and then we send it to the server. Now, on the server side, depending on your acknowledgement mode, maybe you know you have to wait until the data is fully replicated to multiple brokers, or maybe you know just a single broker. So uh and uh if it's uh waiting for multiple brokers, then you are also sensitive to the delay in potentially the communication among those different brokers. Then when this comes back, the producer will get an acknowledgement. Um so as you can see, this whole uh latency uh of publishing an event is tied to the availability of the memory, the amount of batching you wanted to do, your chronological mode, uh, and the interaction, uh, the communication between the client and server and then potentially among servers themselves. So it's pretty involved. So uh what we want is uh is to add like a timeout uh in the producer such that um if things are not going well, I think uh the client can get uh notified earlier, so that instead of like having to wait forever. Then the question is you know, is what timeout we should be giving back to the client. In some of the earlier discussions, I think a lot of the uh discussion is okay, earlier we had uh request timeout already on the producer side, which essentially controls, you know, once you send a request from the client to the server, how long you wait before you actually get an acknowledgement uh from the server. But then that's not enough because earlier I mentioned the events could also be sitting in the producer side buffer, waiting for more data to come in to be batched, right? So some of the delay could be because of that. So our initial thinking was okay, maybe we should just add another timeout, which is you know how long this event is like waiting in the buffer, right? Um and then uh and then uh this of course has impact with other things uh with retries. So as we are uh discussing this feature, so what we realize in the end is just from the end user, if you are building like a new application and you want to publish some events to Kafka, right? From that user's perspective, they don't have all the knowledge of the underlying implementation details in Kafka. You know, what are all the stages this event has to go through, and uh what are some of the parameters that's involved with that. All they care about is just from the time I send the event to the time I get an acknowledgement. How long I'm comfortable of waiting for that. Now, how that time is distributed uh or segmented into those components is really up to uh the library or the or the system to deal with. But uh, which can have you know multiple separate uh uh parameters. But just from the user perspective, all they care about is really just one number, uh, which is the from the time when the event is generated to the time and acknowledgement is received. So uh in the end, I think after some discussion, we just realized just from a user perspective, right, that's actually the user-facing configuration we should be exposing instead of just adding more and more detailed uh parameters for each of the segments um that we have in our implementation. And in the end, I think that uh uh that feature seems to be uh much more well received uh than we would have uh gone like otherwise. Because without this feature, earlier what we have you know is every time we add a new separate timeout, a few months later, some other people will be asking for another timeout because there's another component of the latency that's not accommodated by some of the existing ones. But with this, I think um it's much easier for the user to use because um they just care about from their perspective how long they are waiting to uh to wait. Right.

SPEAKER_00 51:26

And I would add to this, okay, so these are these are product decisions that you're making. And they're one way of describing them is they're highly technical things, but they're fundamentally product surface area usability decisions, which normally for an application developer, when there are usability decisions that you made, there's usually a designer or a product manager or there's somebody in the loop who focuses on that and lets the developer focus on implementation and you know their own craft. Uh, and someone else kind of has the calling of making sure this thing is is pleasant to use and understandable. So the user you're talking about is another developer. That's the thing. When you're a distributed systems engineer, you're in some sense, you're making pipes, you're making plumbing. Now the pipe analogy applies particularly well when it comes to Kafka, but you know, you're you're always making infrastructure that some other developer uses. And so, you know, on the one hand, that's nice because you're also a developer, and so you can think clearly about what constitutes good usability for other developers, but um, this is, I think, a key thing about being a distributed systems engineer that you have to be thinking about usability. You have to, because this is these are these are not simple tools generally. Distributed systems are hard and you want to make them comprehensible for other developers. And the buck stops with you. There's no PM or designer gonna help you. Uh, you're the one who knows.

SPEAKER_01 52:53

That's right. Yeah, because I think, yeah, I think as distribute engineers, right, I think you are pretty good, I think, a lot of the internal details. But I think when you design at least APIs or interfaces or user-facing configurations, you sort of have to put the application developer's mind in place to see what uh they would think from their perspective when interacting with your system.

SPEAKER_00 53:18

My guest today has been June Rao. June, thanks for being a part of Streaming Audio.

SPEAKER_01 53:22

Thanks a lot. Uh, thanks for uh for having me here.

SPEAKER_00 53:26

And there you have it. Hey, it's Kafka Summit time again, and you get another discount code for listening all the way to the end. Kafka Summit London is coming up on April 27th and 28th of 2020, and you can get 30% off your registration if you go to Kafka-summit.org and use the discount code KSL20Audio during checkout. Just enter KSL20 Audio while registering at Kafka-Summit.org, and that 30% off is all yours. I would love to see you there. And anyway, I hope this podcast was helpful to you. If you want to discuss the podcast or ask a question, you can reach out to me at TL Burgland on Twitter. That's at T L B E R G L U N D, or you can leave a comment on a YouTube video or reach out to us in Community Slack. There's a Slack sign-up link in the show notes if you want to join that group. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. If you subscribe through iTunes, be sure to leave us a review there that helps other people discover the podcast, which we think is a good thing. Thanks for your support, and we'll see you next time.