Understand What’s Flying Above You with Kafka Streams ft. Neil Buesing

Neil Buesing (Director of Real-Time Data, Object Partners) discusses what a day in his life looks like and how Kafka Streams helps analyze flight data.

EPISODE LINKS

• Using Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL and KSQL
• Kafka: The Definitive Guide by Neha Narkhede, Gwen Shapira & Todd Palino
• Read the Confluent blog
• Join the Confluent Community Slack

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

My colleague Victor Gamoff recently caught up with Neil Busing to talk about how he uses Kafka streams in his day-to-day work. Neil is a fantastic member of the Kafka community whose work you really should follow. Check it out on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.

SPEAKER_02 0:24

Hello and welcome to a streaming audio episode from Kafka Summit London edition. And uh today I have as my guest, the person that I will be interviewing, one of the speakers of Kafka Summit, Neil Busing. Hi Neil. Glad to meet you. First of all, thank you so much for agreeing to do in this uh the interview. And it's great that you're also doing uh Kafka Summit talk. But before we go into this one, uh I just want you to introduce yourself to our audience and talk about what you do, why you're here, uh, you know, obviously, and how Kafka affected your life.

SPEAKER_01 0:57

Well, I'm Neil Buzing from Object Partners, a company based out of Minneapolis, Minnesota, in the United States. We do software development primarily around all things Java, JVM. I'm the director of real-time data, so which basically means in the last year, all things Kafka, because that seems to be where everybody's interested in knowing more about. Today I get to speak about one of my passions. I used um Kafka Streams to do geospatial data comparison of aircrafts, either with airports or among each other, trying to show how I can use streams in a different way than I've seen used before.

SPEAKER_02 1:34

It's amazing. It's one of the one of the things that I as a developer advocate uh start appreciating is uh travel and learning about like aircraft and airports. So I'm actually looking forward to your talk. One of the uh one of the things that uh get me into some of the Kafka streams as well is the blog post from uh a couple of years ago. Uh Bill Bijak wrote the blog post about like flight delays predictions using Kafka streams. Uh so um and it was like very interesting to see how this uh like a small library can be used to do uh this kind of uh wasn't thing. So um how you see Kafka is affecting your customers since you're director of real-time data, you're definitely the person who can bring some light and talk about how the things are changing, use cases are changing for for your customers.

SPEAKER_01 2:24

Well, I mean, uh there's two things to being a consultant. One is the technology you're interested in because you like to be a consultant so you get to work on things you're passionate about. At least you hopefully.

SPEAKER_02 2:35

Once you're doing things that you like, you never have to work anymore.

SPEAKER_01 2:39

Exactly. Well, yeah. The other aspect is what do the companies want? And if you can blend those two together, you have a great career. So this past year, I've been for a little over a year now, the director of real-time data at Object Partners, and it exposed me a lot to companies' needs around doing things at scale. We have quite a few companies of scale in in Minnesota, and we have others that we work with elsewhere, but the the main thing they come to object partners is we need people to help us with Kafka. So I spend most of my days working on Kafka, and it's a technology that I've grown to love. And everyone's use cases seem to have a very similar trait to them in regards to I need to enrich data in real time. And that is why I've tended to specialize in Kafka streams as a developer, because it's been the natural tool set to use that and seems to fit nicely into what our customers want.

SPEAKER_02 3:39

So why you Kafka streams in this case? Like why you couldn't use any like a clustering technologies that's already kind of sort of doing stream processing. Why you said, like you said, it's a natural fit. Like why is that?

SPEAKER_01 3:52

It kind of fits into the wheelhouse of object partners where we're a Java development shop. We know Java very well. And being able to provide a tool set that is built on the JVM is an easy transition not only to our developers, but also our customers that have already built in a pretty robust infrastructure of microservices using the various technologies they already have at their company.

SPEAKER_02 4:16

Yeah.

SPEAKER_01 4:16

So it's it's just a natural progression for things.

SPEAKER_02 4:19

Nice. Yeah, okay, so it makes a lot of sense. So since it's a Java library and uh JavaScript will be natural. So what's your opinion about other languages? It's like you send it Java, but is it like you're limited to Java or like a JVM thing or um the company's focus has been a full development stack experience.

SPEAKER_01 4:39

So we do mobile, we do iOS, we do JavaScript front end, we do various back-end technologies. We're only a company of 100 to 120 people, so you tend to want to have some focus to what you do. And the JVM technology stack has been that. We we do do other languages. There's people that do Go, Golang, for example, a lot of JavaScript, as I mentioned. Um, one thing to point out, Object Partners is a partner with Confluent, and as part of that partnership, I've been able to lead engagements. And when I leave the Midwest, I see a lot more use of Go, Golang. And what's nice about Kafka is it works with all those languages. And what I love about KSQL, it allows people using those languages to get the rich streams aspect without having to go down the GIS. Now, there one of the things I'll go into my talk when I talk about KSQL, there are certain things you cannot do. And I'm showcasing what I can do with streams because that's where my passion is, that's where I do my development. But most of what I'd achieved in the algorithms that I'm showing, I was able to do them in both Streams and KSQL, which is really starting to showcase the growth and maturity that KSQL is providing. And I'm excited to see where the Apache community and the confluent open source communities take that to the next level.

SPEAKER_02 6:07

So have you uh have you looked into these like uh ways how you can extend KSQL? Like you can do like UDFs uh and the kind of I've seen some people doing even some of the machine learning functions uh using uh uh KSQL UDFs.

SPEAKER_01 6:22

Um what's was your experience with this one since since you Well part of my talk when I do what I call the nearest airport lookup, I use a UDF to show how I can make a RESTful call directly out of KSQL. Interesting, yeah. And I also look at the existing uh UDFA function around collect set, which is available to me, which then made my coding even easier. So, and then at Kafka Summit, New York, Tim Berglund and I had a chat, and I was talking about my talk, and he actually gave me some ideas of it's not part of the public API, but looking at how to potentially get at the stateful sets within KSQL. So once I'm done with this talk, that's what I'm exploring. Because if I can achieve and find a way to get at that stateful stats within KSQL, it's gonna be easier for me to advocate for it. Yeah. And I would want to push to making that part of a standard operation available.

SPEAKER_02 7:16

You're talking about the things that we know as uh um interactive queries in the Kafka streams. Yes. That that is not part of Key SQL right now, uh, but uh it is a potentially interesting uh beat that allows to get access to internal state of the application so you can query application, not your database. So obviously we know that uh the pipelines are awesome, Kafka Connect is awesome. Sometimes you just need to get this data out.

SPEAKER_01 7:42

Right, because one of the things that I wanted to show is I built in an app without any other data store other than Kafka Streams. So when I visually display aircrafts on the screen using D3 map projections, I query and pull the data from Kafka Streams. The trick is there may be many streams applications I have to pull from. So I have to discover them all, query them all, and return the data. I probably would never go to a pro a customer mine and recommend doing it because it's just not reliable yet. If the consumers are rebalancing, I'm not gonna be able to access my state stores and stuff like that. But the growth there is is fun. And and fortunately for me, I picked a talk that was mine that I could just play with and have fun with.

SPEAKER_02 8:26

Right. Yeah. So uh the the things what the Neil mentioned here is that um when the rebalance is happening, state store is not available because of the way how the Kafka streams consistency model works. It also relies on uh the Kafka uh metadata through rebalance, and it actually piggybacks the metadata about the the state stores and keys that stored in particular keys state stores on this um uh on this rebalance, uh rebalance data. So uh now you start hearing some some music, meaning that the Kafka summit is uh about to start. So um, Neil, let's talk a little bit about not technology aspect of your life. What you do um in general? So what's what what's what's uh what gets you excited? Like you do any like sports, maybe?

SPEAKER_01 9:12

Well, my passion around sports is running. Um I run, my family may say I run too much, I say not enough. I have run three marathons and I need to get back to that. So I hope to someday come back here and run one. At least that would be uh a passion of mine to try to achieve, not get there. The other passion is my family, my wife and and kids. They mean the world to me. They supported of me coming flying overseas by myself to a summit, jealous of that fact, and that yet they were very happy for me and that. So it's really my family and and running tend to take up the rest of my my day. The balance some days, especially preparing for this, may not be always the correct balance, but but it's worked out pretty well.

SPEAKER_02 9:56

So um one of the things that I like to ask people when I I talk, so one of the things that I started doing a lot last year is is reading. And every time when I uh talk to some interesting person, I would like to get some advice for any book. Like you can say like any technical book or non-technical book, the book that makes impact on you, or you would recommend to people to read, or you read something very you know, yes, or something on the plane, or we were reading right now. I don't know.

SPEAKER_01 10:28

Um that's a good question. My my wife's the passionate reader in the family, she reads a lot. I read a lot of technical stuff. If I have free time, I'll read the blog. I'll read articles. Constantine's learned blog? Very much so. Um there was an excellent blog and there was excellent blogs there just the other day. As far as reading, I have some running books that I I read occasionally, but I'm not passionate about reading about running. I'd rather go out and run. I don't feel like I'm the expert to say here's the here's the book to go read. Um I'd probably go plug something like the definitive guide or or of Kafka or something like that.

SPEAKER_02 11:04

Yeah. So Neil is talking about the definitive guide of uh for Kafka, obviously, by uh Gwen, Neha, and Todd Polino. All right, so uh Neil, thank you so much for finding time. I hope you will have a great talk and uh good luck to you and thank you again to being part of the Kafka Summit London. Thank you very much. Thank you. And uh as always, you can find uh all these episodes in iTunes. Uh, don't forget to subscribe and support us there. And as always, have a nice day.

SPEAKER_00 11:31

Hey, you know what you get for listening to the end? A Kafka Summit discount code. Kafka Summit is coming up on September 30th and October 1st in downtown San Francisco, and you can get 30% off if you go to Kafka-summit.org and use the discount code AUDIONINTE during checkout. Just enter AUDIO 19 while registering at Kafka-summit.org, and that 30% off is all yours. I'd love to see you there. But hey, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at at TLBurglund on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks for your support, and we'll see you next time.