Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Ask Confluent #4: The GitHub Edition
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Want to see a feature implemented in KSQL or other Kafka-related project? Gwen answers your questions from YouTube and walks through how to use GitHub issues to request features. This is the episode #4 of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube and elsewhere.
EPISODE LINKS
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
Do you have a feature in KSQL or some other Kafka-related project you'd like to see implemented? We'll walk you through some GitHub issues and more on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Welcome back everyone. I'm Tim Berglund. Today my co-host, Gwen Shapira, answers your questions from YouTube and walks you through how to use GitHub issues to request features. Let's get to it.
SPEAKER_01Welcome back to UsConfluent. I'm your host, Gwen Shapira, and I'm all on my own today answering a few questions from the community. And I'll show you around the Confluent GitHub, so to teach you some things about interacting with the community via GitHub. So let's get started. First question from Twitter. A few days ago I got asked, Gwen, in your paper recommendations for deploying Apache Kafka on Kubernetes, there is a mention of Helm charts named CP Kafka and CP Zookeeper. Are those publicly available? So thanks Javier for the question. And when you posted the question, they were actually not available, but they became available magically two days after you posted the question. So if you go to our blog, you can see our announcement of the Helm charts, some tips on how to use them, links to the Helm chart source code over on GitHub, and then also links to two excellent papers on how to use Kafka on Kubernetes. One is the paper you already read by me, and the other is the Kafka on Kubernetes reference architecture by my colleague Victor. So read the papers, play with the Helm charts, send us feedback. I'm very glad you're interested. Okay, next question from YouTube. Ayman Patel asked, and this is a response to the YouTube I published, I think maybe almost half a year, maybe more back, about how to use Confluent Control Center to monitor Kafka. And Ayman Patel asked, Can I use monitoring as a freelance developer? As a freelance developer, you can definitely monitor Kafka, and that actually makes a lot of sense. You can use Confluent Control Center, but Confluent Control Center is proprietary. So you get a certain day trial license that you can use. But after that, since you're a freelance developer, you kind of have to convince your customer to spring up for control center, or he will be unable to continue monitoring Kafka. If that doesn't work for you, there are open source monitoring solutions out there. So you can kind of decide what works for you. Next up, question for our previous episode in AsConfluent3. Vitali Zborvo Zborovsky, I hope I got it right, said great stuff. I would appreciate it if you commented on issue. And here it is, a link to our Confluent Kafka Connect JDBC connector. And he wanted me to look at an issue and comment. So let me switch over to the GitHub issue and take a look and see what we think. Okay. So here we are: the Kafka Connect JDBC. Kafka Connect JDBC is an open source connector. It plugs into the Apache Kafka Connect framework. And what it does is gets data from a JDBC database, pushes it to Kafka, or it gets data from Kafka and pushes it into a database using the JDBC protocol. And 21 days ago, someone called Mismatch, which I'm kind of guessing maybe our friend Vitali, asked, basically said, I have a table which purposes to receive data and stores them to one of partition tables via insert trigger. And then he describes how he's using the insert trigger to create a partition table. And says this causes the JDBC SYN connector to fail due to a check in flash method of buffered record class. It would be good to add configuration option to skip this check. Or is there another way to use the connector and circumvent this scenario? So I don't know another way to circumvent this scenario. And if you already found the exact line that is failing, you can definitely add a configuration and add this check. One of the things that I noticed is that this pattern is kind of specific to Postgres. 90% of the time we kind of often recommend for people not to have triggers in their databases. So I would say that because it's kind of a specific scenario and we don't really want to clutter up JDBC with special options for every single database in the world, it may make sense to fork the JDBC connector, create a specific Postgres JDBC connector, and you can add this option, and maybe other people can add data types that are specific to JDBC and other specific things for this specific database. And sometimes when I recommend to people to fork and create something specific for their database or use case, they go like, oh, wouldn't Confluent be upset if I create a derivative? But actually, no, we created JDBC as something generic. We expected people to take that and create things more specific for their databases and their use cases. And if you create a connector and you're happy and you have users and you want to grow your own user community, we have the Confluent Hub where you can post your connector and be part of the entire connector developer community out there. So I hope you're going to go that route and create your very own Postgres connector. Next question from I think it's Kartikeyan. Cartikeyan. Or join one stream with two tables. That's an excellent question. And actually the answer that right now you cannot. You'll have to do it in two steps. Join a stream with one table and then take the result and join it with another table. But I want to show you how you can express your desire to get this feature. So let's move back again to GitHub. KSQL is an open source project, so we have KSQL GitHub. And as you can see here, we have a lot of KSQL GitHub issues. So we currently have 319 open issues on the KSQL GitHub. You get to it by just going to confluentinc slash KSQL on GitHub. And you can see we have lots and lots of issues with a lot of tags and labels. Now you basically asked about multi-way join, so let's go and search for multi-way join. And you can see that we already have a request, KSQL should support multi-way joins. Since the request already exists, I'm not going to ask you to open a new request, which one would if the request did not exist. But what you could do is say, hey, I've seen this request and I also really want this feature. And one way to do it is by voting for a request. How do you do it? You click on the smiley face up there and say plus one. And here we now have an extra vote asking for this feature. If there's more votes, the feature is actually more likely to get added. So I talked yesterday to approver. You can see Approva right here is one of our chief KSQL developers. We have a few of them. You'll see them as you interact with the community. And Approver said that the way that he prioritizes which of those issues to work on is by selecting looking how many people voted for an issue, but also how many use cases exist for every feature. So if you have a story, why do you need this feature? I want to do multi-way join because I have a star schema, and in order to do enrichment, I need to take this click stream and enrich it with five different K-tables. Fantastic, that's a story, and you add a story. And if you have large number of stories and large number of votes, more likely that the feature will get added. So go ahead, comment on the issue with your stories, vote on issues that you think are important, and maybe even if you find an easy issues that you want to work on, you can actually do it yourself. So one way to do it, if you go to issues, you can see that we have a label good first issue. Click on that and you can see all the issues that we think are useful for beginners. You can pick one and you too can become a KSQL contributor. Next question is also a KSQL question on the level up your KSQL by Confluent. I think Tim will be thrilled to know that so many people are interested in leveling up their KSQL and are really engaging with the whole KSQL and improving their skills process. And this is a name I'm not going to try pronouncing, I'm sorry, but this dude said that's awesome, but I'm still stuck with KSQL because it is hard to integrate with Ranger and my system. Now we don't know a lot about Ranger, it's not a confluent product, but we do want to help you. So again, another way to do it would be to go back to the issues list. Let's see if anyone already asked about Ranger. So let's search. And you can see that nobody asked about Ranger yet. So you can be the first to open an issue about the lack of Ranger support. Now, because we don't have tons of experience with Ranger, we need you to be very detailed about what would you need, what would make you happy. So link to the Ranger project and say, for me, good Ranger integration would be if I can control who runs KSQL queries using Ranger. Or if Ranger would show me logs of everyone who is running KSQL queries. I don't really know because again I'm not a Ranger expert, but I'm sure you know. So let us know what you need in as much detail as possible and we'll try to make you happy. Okay, another question, this time not from the KSQL video. This is from Monitoring Kafka using Confluent Control Center video. The demo in the video used the Confluent CLI. So Sagar Bandai asked, can we install Confluent CLI on a Windows machine? And I asked Constantin, who is our lead CLI developer, and he said that you can try. In order to try, you're going to need to run Bash on Windows, which is available. The GNU project has GNU Bash for Windows. But we've never tested it. So, like anything that is not very tested, there is some possibilities that you will run into issues. If you do, let us know, but I don't know that we have Windows internally to reproduce them. So we may need to have some discussion about how it works on Windows. But definitely give it a try. Who knows? Maybe it will just all work straight out of the box. Wishing you good luck. And last, again, from the level up your KSQL video. Oh my god, those videos are probably so popular. Alephiao said the videos are very easy to understand, really good and useful set of tutorials for such great projects. Thanks. Oh my god, I'm Tickle Pink. We you think that KSQL is a great project, which is fantastic because we worked kind of hard on that, and Tim will be thrilled to hear that you love his videos. So I'll make sure to let him know. Thank you so much. For us, this kind of good feedback totally makes our day. We work kind of hard on those videos, and we're glad to hear that they're useful. We work hard on KSQL and we're glad to hear that this is useful as well. So thank you. And that would be it for today. That was kind of fun. We had lots of really good questions from the community. Uh keep up uh with the all the questions. I'm looking forward to see more issues and more votes and more user stories on the KSQL GitHub. And see you all next time.
SPEAKER_00And there you have it. I hope that was helpful to you. If you've got questions, you can ask me at T L Burgland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can ask Gwen at GwenShap, that's G W E N S H A P. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there.
