Confluent Platform 5.3 | What's New in This Release

A quick summary of the most important features in Confluent Platform 5.3. We discuss improved Kubernetes and Ansible support, improvements to Confluent Control Center that give you better insight into the data in your cluster, and an important new set of security features—Role-Based Access Control—aimed at making complex deployments more secure.

EPISODE LINKS

• Read the docs
• Read the blog
• Watch the video version of this podcast (featuring an actual stream)
• Download Confluent Platform 5.3
• Join us in Confluent Community Slack

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

Copy Transcript Copied

SPEAKER_00 0:00

Confluent Platform 5.3 is here. As I said before, it takes a lot to keep up with new releases like this, so we rounded up all the features in the release we thought you should think about first and recorded a short explanation of all of them. And we did it from an appropriate location. Check it out on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the Cloud. Hi, I'm Tim Berglund with Confluent. I'm here in front of this stream to tell you all about what's new with Confluent Platform 5.3. The new stuff in 5.3 breaks down into three categories. Number one, we've added some features to help your deployments be both more agile, that's small A agile, like moving around fast, not like stand-ups and post-it notes. They want to be more agile and also more cloud native. Second, we're giving you more tools to understand and manage your event streams, to really get some insight about the data that's inside your cluster. Third, we've got some exciting new security features that help you manage deployments that involve a large number of people with complex permissions. Let me break these things down one at a time. To remind you, Confluent Platform is the event streaming platform built by the original creators of Apache Kafka. It's built on that solid foundation of Kafka as that core event streaming platform. It's got development and stream processing tools layered on top of that, and of course, a lot of good things to help you deploy and manage and secure Confluent Platform at scale. All of this can run on machines in your data center or machines in the public cloud, and of course, you've always got options in the managed service Confluent Cloud. Confluent Platform is always built with the latest release of Apache Kafka. In that case, this is Kafka 2.3. I'm not going to tell you too much about those features in particular because we already have another video that breaks those down pretty well. There's also a great blog post by Colin McCabe that describes those things in print if that's what you prefer. Now let's talk more about these agile cloud native deployments. 33% of Confluent customers plan soon to be running Confluent Platform on Kubernetes. There's no denying that Kubernetes is a very important trend, and I think for good reasons. Kubernetes really was originally created to help manage workloads like lots of stateless microservices. We have all these containers, we need some way to orchestrate them. But once people started doing that, they found that their stateful workloads, their databases, and their event streaming platforms like Confluent Platform made a lot of sense to run on Kubernetes also. There's kind of an economy of scale in getting all of your work into a Kubernetes cluster once you dip your toe into that water a little bit. But to make a stateful workload like Confluent Platform work well on Kubernetes, you need a thing called an operator. Now an operator is a Kubernetes component that helps take the configuration that you give it and turn that into real life inside the cluster. And to do that with something stateful like Kafka and the other components of Confluent Platform takes a little bit of work. You need a custom operator to do that. That's what the Confluent Operator is all about. This lets you deploy Confluent Platform in a cloud native way to Kubernetes. And that's any of the components of the platform. Brokers, your Zookeeper cluster, Kafka Connect, KSQL, Schema Registry, Control Center, Replicator, all of that stuff. And it doesn't matter where you're deploying it. There are a lot of places you can run Kubernetes. There's Pivotal Container Service, there's Google Kubernetes Engine, there's your own Kubernetes cluster that you might manage on hardware in your own data center. It doesn't matter. Any of those is a place where Confluent Operator can help you. The key lifecycle operations like scaling things, you know, we've always said we have elastic scalability in Kafka and Connect and KSQL and all these things. Well, now that's a configuration change in Kubernetes and Confluent Operator makes those happen. Also, rolling upgrades. You don't want to go back to the bad old days of, hey, it works, we never have to version this again. With cloud native deployments, we expect things to be kept up to the current version. And with Confluent Operator, you get support for rolling upgrades. Now, if you have engineering resources and you know Kubernetes well, this is a thing you can build yourself. But what we've done is to take our growing body of experience operating Confluent Platform in the cloud through Confluent Cloud, our own hosted service, and we've literally encoded that in the code of Confluent Operator. So this is a way of drawing on the most extensive body of Confluent Platform experience in the world running in Kubernetes and making that work for you. Now what if you don't run Kubernetes and you're deploying the components of Confluent Platform on servers again, either on-prem or hosted servers in the cloud, but you're managing each one of those yourself. Well, for some time we've had on a trial basis Ansible playbooks available. Those use Ansible to deploy your configurations in an automated way into those servers that you run. In Confluent Platform 5.3, these are now production ready and fully supported. And we didn't just flip a switch saying, okay, they're ready, you can use them, we'll support them. There's actually some new security features. As you see here, we have proper certificate authority-based TLS and Kerberos support baked in. If that's a cake you've ever tried to bake yourself, you know it's nice to have another chef helping you there. And again, these cover all the components of the platform: Zookeeper, Kafka itself, Connect, KSQL, Schema Registry, Control Center, whatever it is, they're supported by the new production-ready Ansible playbooks. Now let's talk about features that help you understand and manage your event streams. Since about Confluent Platform 5.0, you may have noticed that Control Center has been growing some new features that are helping it dip its toe into the waters of being a development tool. Now, Control Center was conceived as an ops tool to help you manage and monitor your Confluent Platform cluster. But increasingly it's getting some dev support also. This is a great thing. It's grown features like topic inspector, schema registry support, the KSQL UI, and so forth. So continuing this trend, we're giving you an extensively redesigned Control Center interface. All kinds of new things to help you visualize the data in your cluster better. It's also normalized with the interface that we see in Confluent Cloud. So whether you're managing your own deployment of Confluent Platform or you're using Confluent Cloud, those tools are going to look more and more the same, which is a great thing for switching back and forth between those two platforms. This is a part of our plan of giving you freedom of choice to be able to run Confluent Platform wherever you want. Fully managed in Confluent Cloud, completely on-prem, in actual metal servers that you rack yourself, or in managed servers in the cloud. Wherever you do that, we want it to look the same and feel the same so all those skills transfer across your teams. And here's that new control center UI. The homepage now gives you this card view of available clusters. This lets you manage multiple confluent platform clusters from one instance of control center. Right here we see just one cluster. If you have a large number of clusters, you can search by name or hide clusters that are healthy and might not need your attention right now. When I click on our one cluster, we get the new overview screen. We've reworked this to get you a quick view of all Kafka metrics, the total number of brokers, cluster controllers, and whether Zookeeper is healthy. We also have an overview of the number of partitions and how many are under-replicated or out of sync. Next, let's look at the new view of topics. The topic inspector gives us a summary of the data in the topic. There's a table view and what we call the card view of the messages, where each message is displayed as a discrete card. You can navigate through the topic by scrolling to see the older messages and newer messages. We can also inspect the schema of this topic. This is retrieved from the Confluent schema registry. And we have improved case equal integration. Here we're registering a topic as a case equal stream, so we'll be able to perform queries against it. We select what we want the key for this stream to be, what field contains the event time, and we indicate that we want the stream to be encoded as Avro data. The rest we leave with the default. With that stream created, we can run a continuous query on it. That's what this select is, to view the data in that stream as it arrives. You can see we're getting new messages, and those messages are available in a tabular view and card view, just like in the topic inspector. Finally, here's the new view of consumer groups. We can drill down into one group and see information about the consumer lag for that group. The most recently committed offset is clearly indicated, and the display gives you a sense of where the offset is relative to the newest message in the topic. For about the past two years before this release, Confluent has had a CLI, the Confluent CLI. This has been a development tool. It's been super handy for me. It's a nice little way to spin up a local environment. If I don't want to use Docker at the moment, I can just have this local environment, do things with it, bring it up, bring it down, blow away all the data, bring up a new one. Great for development and demos and all that kind of thing. Confluent CLI, though, in 5.3 has just gotten a major upgrade. It's now a production-ready tool. And all the things that you might have gotten used to in the previous CLI, those things are still supported in the new one, plus new features, particularly role-based access control, which I'll talk about more in just a minute. The Confluent CLI is your gateway to that functionality. You should keep an eye on this. It's likely to grow a lot more features in the near future. Now finally, let's talk about granular secure access to your Confluent platform cluster. Imagine you have some number of people, developers, roles, you know, principals, something like that, accessing your Confluent platform cluster. Each of them is going to have different levels of access. Like some developers might need to be able to read and write these topics. Some of them might be able to deploy new connectors to the development connect cluster. Then you've got your administrators, and they're the only ones able to deploy, say, new connectors to the Connect cluster or new queries to the KSQL cluster in production, or something like that, right? It's clear that you're going to have these different roles and responsibilities across the group. And maybe with a small number of principles, you can manage that on your own. Alice is an administrator, she can touch the production stuff, Bob is a developer, he can touch the development stuff. Now imagine there are 250 Alice's and 250 Bobs. Suddenly your life is a lot different. Role-based access control is a way of managing granular access to all cluster resources with a large group of people. It allows you to assign those people to roles and then permissions to those roles. So Alice is an Alice, Alice is an administrator, and as an administrator, she gets to access the production cluster. Bob isn't just Bob, he's really a developer, and as a developer, he gets to access the development resources. It's that kind of thing. So this is a resource that makes that kind of management much, much easier. This is configured with that new Confluence CLI that I told you about just a minute ago. And I need to stress, this is in preview as of 5.3. It's a huge new feature, touches literally every component of the platform. We would love for you to try it out and let us know what you think. If you're already a customer, of course, you know you've got those channels to be able to talk to us. If you're not and you've just downloaded this and you've played around with it and you want to talk to us, always, always join us in Community Slack. You can talk there, you can comment on this video, and you can reach out to us on Twitter at Confluent Inc. Any of those channels would be great and we would love to hear more. So that is 5.3 in a nutshell. You've got more agile, more cloud native deployments, you've got a better ability to understand and manage your event streams, and you've got a preview of role-based access control. Download it now, check it out, and I hope to hear from you soon. Hey, you know what you get for listening to the end? A Kafka Summit discount code. Kafka Summit is coming up on September 30th and October 1st in downtown San Francisco, and you can get 30% off if you go to Kafka-summit.org and use the discount code AUDIO19 during checkout. Just enter Audio19 while registering at Kafka-Summit.org, and that 30% off is all yours. I'd love to see you there. But hey, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TLberglund on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks for your support, and we'll see you next time.