Apache Kafka 2.3 | What's New in This Release + Updates and KIPs

Tim Berglund (Senior Director of Developer Experience, Confluent) explains what’s new in Apache Kafka® 2.3 and highlights some of the most important Kafka Improvement Proposals (KIPs).

EPISODE LINKS

• Read the blog
• Watch the video version of this podcast

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

Copy Transcript Copied

SPEAKER_00 0:00

Apache Kafka 2.3 is here. Honestly, it takes a lot of work to keep up with what's in each release and figure out what matters most. So we rounded up a list of the tips that we thought were most important and recorded a short explanation of each. Check it out on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hi, I'm Tim Berglund with Confluent. I'd like to tell you a little bit about what's going on in Apache Kafka 2.3. Uh I'm going to do that by just walking you through some of the kips that are included in this release, and I'm going to break them into three categories. There's the core Kafka kips, there's Kafka Connect kips, and there's a bunch of cool stuff that happened in Kafka streams. I happened to be in Tel Aviv this week, and we found a suitable hotel patio from which to tell you these things. Let's get started with KIPS 351 and 427. These have to do with under-replicated partitions. Historically, you've got the under-min ISR metrics group and the under-replicated metrics groups, and those are telling you which partitions have not enough follower partitions that are replicating. That is, they're broken in some sense. You're not going to be able to do a produce with axe equals all to one of those partitions. So something is wrong. But what about when we're on the threshold of something going wrong? It's not broken yet, but it's kind of getting there. Kip427 adds a group called atmin isr. So if you're atmin ISR, you can still produce with axe equals all, but if you lose another one, if another one falls behind in replication, then you won't be able to anymore. Kip351 adds a switch called undermin ISR to the described topics command. And that lets you see which topic partitions are below the min in sync replicas and need addressing. So both of those helping you with under-replicated or almost under-replicated partitions. Maximum log compaction time. Now you've always been able to set a minimum log compaction time. So that's an amount of time that a log segment is guaranteed to stick around before it's compacted. So that's like minimum amount of time that a key will survive compaction. Max log compaction time lets you say the maximum amount of time that you want a log segment to hang around. So this is good for GDPR compliance, right? If there's a certain amount of time by which you need a key to be deleted when you've tried to delete it, this makes sure that it gets compacted within that period of time. Kip 402 makes the socket server a little more fair. So historically, socket server prioritizes new connections over existing connections. And if something has gone wrong in the network, in the cluster, or with a client where there's now a connection storm going on to a broker, it prioritizes all those new connections and will let existing connections fall behind. So basically, you can very quickly DOS an individual broker that way. So with this improvement, KIP402, this balances the priority between new connections and existing connections so that that's a little bit harder to make happen. KIP 461 improves the replica fetcher. Now, in a broker, any given broker is the leader for some number of partitions and the follower for some number of partitions as well. So for those followers, the replica fetcher is actually actively fetching stuff from whoever the leader is. There's a thread pool of a configurable size based on number of cores and however you like to size thread pools that tells you how many threads will be running replica fetchers. Historically, before this kip, if one of those replica fetchers fails for some reason, all of the replicas that it was responsible for goes away. I mean, you might only have five threads in the pool, and maybe you've got a thousand uh partitions that you're follower for. So if one of those fails, you could lose replication on 200 of those. That's a super bad deal. So this improves replica fetcher such that when one partition fails, only that partition fails, and all of the other replicas that it's fetching go on the way they normally should. There are some other kips. Let me walk you through them real quick. KIP339 makes it easier to change configs on a broker incrementally. So if you just want to change a few configuration values that you have to change right away, you don't want to bounce the whole broker, this makes that easier to do. And importantly, the Alter Configs API has been marked for deprecation as of this KIP. KIP341 updates the sticky assigners user data protocol. So this makes consumer partition assignments a little more stable. It makes sure that we don't assign the same partition to multiple consumers in the same group. KIP361 makes it a little harder to enable autotopic creation. See, it used to be just a broker configuration that you used to switch that on and off. Now there's also a consumer configuration that has to be agreeing with the broker in order to enable auto-topic creation. Kip417 allows you to use SSL for JMX tool to connect to a secure RMI port. That's a good thing. KIP421 adds externalized secret resolution to the abstract config base class. So that base class gets used by a number of things like streams applications, consumers, producers. So if you've got external config variables, which were enabled in KIP297, now all of those things get to play in 297's playground. They can more easily use externalized configuration variables. There's kip 425 that added some log4j appender properties to producing to secure brokers. KIP430 changes the admin client now to allow users to determine which operations they're authorized to perform on topics. And kip436 adds a new metric called start time in MS, which is the Unix time at which this process started. So that's your core Kafka configs. Now let's take a look at Kafka Connect. The first Connect KIP is 451, incremental cooperative rebalancing. So historically, if you change the configuration of a connector, that causes a rebalance across the cluster, and that rebalance stops all of the other running tasks. So now when Connect has to do a rebalance, it does it incrementally. It doesn't stop the world on all the other connectors. It only stops activity that it absolutely has to to service that part of the rebalance. It makes configuration changes a much more pleasant thing. KIP 449 adds connector contexts to the log. Now there's a few different threads and thread pools that are involved in Kafka Connect activity, and they're all emitting logging data asynchronously. So to untangle that can be a little bit of a pain since they're not necessarily in the order that you want them to be in. Well, Kip 449 adds some context, as you can see here on the slide, tells you about the task and the connector, sort of what's going on, and that lets you string those log messages together in a reasonable way. A few other tips for Connect. There's 454, that expands the Connect cluster state interface to help make the configuration change API a little bit more uniform and includes additional items like configuration connectors, configuration of tasks, ID of the Kafka Connect cluster in that interface. This cooperates with KIP285, which enhanced Connect REST extension plugins. So this is a continuation of improvements to the way we configure things in Connect. KIP 458 is called Connector Client Override Policy. What it really does is lets you set a different principle by which each connector will authenticate itself. And so that lets you set their authorization policies in a more fine-grained way. Normally you get a list of the connectors, and then for each one, you query each one. So you have to do this n plus one rest interaction, which is unpleasant. Now you can add this expand query parameter, and that will return you an array of those things so you get them all in one shebang. So that's Connect. Next, let's take a look at what's new in streams. The first streams kit is 258, and this adds the timestamp, the event timestamp, to the stuff that's stored in RocksDB or to the Kafka Streams state store. And really, this is laying the groundwork for a bunch of new future features. There are some new things we're going to be able to do with K-Tables. So by itself, it's kind of like this infrastructure change under the covers. But you'll see a lot of future kips refer back to this one. So super important. If you care about streams, if you watch the internals of streams, keep an eye on this kip. Things are going to happen as a result of this. Kip 428 adds an in-memory built-in Windows Store. So one of the cool things about State Store and Streams is that it's pluggable. And some of you have plugged in your own implementations either just to play around or because you've had to do something that you didn't like what the default RocksDB implementation did. So that's all totally pluggable. Now you have this default in-memory thing. This is great if, say, you're just playing around locally, you don't want the overhead of RocksDB running, or you have some other reason that you want things to be in memory, you've got that implementation already there for you. KIP313 was halfway implemented in Kafka Streams 2.2, and now the other half lands in 2.3. The first half was a method called Flat Transform. Now, FlatTransform is a lot like FlatMap, except it kind of brings processor API functionality into what we call the Streams DSL. Now you used to be able to break out into the processor API, have access to the state store, do all that processor API stuff, and you could do flat map-like functionality in there, but not in a type safe way like FlatMap gives you. FlatTransform is a method that gets you access to the state store, gets you access to the processor context, lets you do all those processor API type things, but it's type safe just like flat map is. So you have a defined transition from this key, this value, to that key and that value in flat transform. So that was again in 2.2, that's already there. In 2.3, this work has continued with flat transform values. So flat transform values is to flat transform as flat map values is to flat map. In other words, it doesn't touch the key. So this is a way of transforming only what's in the values in this type-safe way that kind of bridges you to the processor API world and is not going to touch the key. So the key remains the same, there's no repartitioning, you're just transforming the values. So kind of a cool thing and interesting to look at the bridging of the processor API world and the DSL world. Kip 328 lets you suppress updates for K-tables. So when a K table is changing and there are new events coming in and they're being processed, they're being put in the table, by default, for every event that comes into that K table, an event is emitted, a message is emitted for that K table. This lets you suppress that until the end of the window. So if you just don't want the underlying topic to be touched, you don't want that flurry of messages going out to it to be compacted later, you can suppress that until the end. KIP 331 adds default implementations to close and configure to the serializer, deserializer, and SARD interfaces, leverages the new Java default interface stuff to implement close on all those by default. Kip 442 makes threading a little bit more stable by returning to the default max poll interval in streams. Kip 443 cleans up the way streams repartition topics are created. Remember, sometimes streams will have to repartition something, and if it's doing that, it's creating topics under the covers, and it the default segment milliseconds and segment index bytes are returned to the defaults. So the way log segments are managed in those topics will be the broker default and not anything special. KIP453 gives us a close method on the RocksDB config setter that might help clean up a few memory leaks. It's nice to have that close method hanging around if there are resources you have to dispose of. And finally, Kip 462 switches us to a thread local ID. This is totally an internal change to the threading model, but it's gonna make it easier to support static membership in a future release. So this is another one of those kips that's like this little infrastructure change under the covers, but future kips are gonna refer back to it. There's things we're gonna be able to do in the future that will draw on that. So, anyway, that is 2.3 in a nutshell. That's the stuff happening in core Kafka. Connect Kafka streams. I hope that gives you an idea how to think about this release, how to focus on what's important, and helps you make cooler things with Kafka. And there you have it. I hope that was helpful to you. If you've got questions, you can ask me at TLBurgland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time.