Ask Confluent #13: Machine Learning with Kai Waehner

Gwen and Kai chat about machine learning architectures, and whether software engineers and data scientists can learn to get along.

EPISODE LINKS

• Blogs on deploying machine learning workloads: 
  • Machine Learning with Python, Jupyter, KSQL and TensorFlow
  • How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
  • Using Apache Kafka to Drive Cutting-Edge Machine Learning
• KIP-392: Allow consumers to fetch from closest replica 
• Watch the video version of this podcast

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

When people think first of Apache Kafka, they don't usually think second of machine learning. Gwen and her guest might convince you otherwise on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Welcome back, everyone. Today my co-host Gwen Shapira interviews Kai Vahner, who was in town from his home in Germany, to talk about event streaming and machine learning. Now, bear in mind, Gwen and Kai do some whiteboarding in this episode, so if you're listening to this in audio only, you might also want to check out the video version of Ask Confluent at youtube.com slash Confluent. Now, let's get to it.

SPEAKER_01 0:44

Welcome back to Ask Confluent, where we answer questions from the internet. I'm your host, Gwen Shapira. With me today I have a special guest, Kai from Germany. Kai is an SE here at Confluent, and he's kind of known inside Confluent and I think also wide outside Confluent as kind of a machine learning expert. Like the first time after we hired him, a bunch of people told me, Oh my god, you're working with Kai! Yeah, that kind of thing. So clearly I'm super glad to have you here on the show and to answer questions about AI.

SPEAKER_03 1:22

Okay, great to be here, Gwen. It's really amazing and it's a hot topic. We see that many customers which also want to combine Kafka with machine learning because it makes a lot of sense for many use cases. And I think the questions we have today from our audience is exactly discussing these things, what's the challenges, the trade-offs, and what you can benefit from this combination.

SPEAKER_01 1:39

Yeah, so you're actually like first time that happens to me on the show, you actually asked people, hey, do you have any questions? And you got a bunch of responses. So we have lots of questions, let's get to it. So the first set of questions is from Quora. And at first when you pinned them over to me, I was like, oh my god, there are so many different topics. And then on the second look, uh it was like, oh, they were actually kind of around the same set of problems. So the first question was, how do you take a machine learning model to production? And someone else asked, is Java a good choice for building machine learning applications? I think Java is a good choice for everything, but I've been writing Java for the last uh 15 years, so maybe it's not that fair. And then the last one was what is the best approach for building a machine learning ETL pipeline for training? Or should I use Kafka for this? And I'm like, why can't we have both? So why are all those questions in your mind per basically the same question?

SPEAKER_03 2:46

Yeah, because all of these companies and projects have the same challenges. Only one side they have the data scientists which build analytic models. And typically data scientists use Python. Some of them use R, but Python mostly. And with that, you build algorithms to train models on big data sets. That's the one part. On the other side, um, you don't want to just train models, you want to deploy them to production, to leverage them, to add business value, to reduce risk or cost or something like that. And that's typically not done with Python because Python doesn't scale to millions of messages to be highly reliable and available.

SPEAKER_01 3:22

Python totally scales to hundreds, thousands of messages.

SPEAKER_03 3:26

Yeah, but I mean it's a it's a different kind of how you deploy it, and and many customers simply have their reproduction environments in Java, right? That's the point. And therefore, they think about how can we combine this? The data scientists build models, they train with Python technology, but then you want to deploy it somewhere else in a Kafka, in a Java environment. And that's in the end um where this comes together, so we see this impetus mismatch, like it's with a database, right? Or object-relational mapping. And the same is here for this model building and training and deployment.

SPEAKER_01 3:53

And the mismatch goes really deeper beyond just the language, right? Like data scientists are good at training models, they don't necessarily know what has to be done in order to write production code.

SPEAKER_03 4:04

Exactly. I mean, um it sounds funny for us as engineers because we wrote a lot of Java code in the last years, but most data scientists don't even know what a version control system like GitHub is. They don't use continuous integration pipelines like Jenkins, and so it's a completely different lifecycle how we do production deployments for 24-7 compared to building these scripts in the end which train models on your local laptop or even on with a big data uh cluster under their hood.

SPEAKER_01 4:27

Yeah, that's a kind of it's totally different things. What about the reverse? Do you ever have software engineers who want to train the models and they're wondering how do I do it in Java?

SPEAKER_03 4:37

Um yeah, I mean we see this trend also, and also there is more and more technology. So on the one side, you can, as data scientists, use your Python frameworks, but there is also frameworks emerging like Deep Learning4j, which is an open source framework, which is really built for software engineers to build and process and train models. So with them, it's much easier for software engineers because they can keep it in their technology. And they can use Java or the Java platform for all of that, for training, for pre-processing, and for deployment. And therefore, we have these different frameworks and options for solving the different kinds of use cases.

SPEAKER_01 5:07

That's fantastic. So you can actually have because machine learning has different parts to it, different stages. There's the ingest, there is the training, there's the deployment. You can actually mix and match different languages and really different people with different skills. You can build a comprehensive team.

SPEAKER_03 5:26

And one last thing which is important here, there's a lot of frameworks like H2O or others, um, where you have both interfaces. So the data scientists can use Python to train a model, but the binary, the model in the end, is then also Java code, which you can deploy in your Java application, and that makes it much easier. And that's also how we think about um then bringing that to production at scale for something like in combination with Kafka.

SPEAKER_01 5:45

That's uh quite reassuring, like to know that if you pick a good framework, you can actually out of the box enjoy both paradigms.

SPEAKER_03 5:52

Yeah, exactly.

SPEAKER_01 5:53

Okay, so next question is from LinkedIn, which I think the first time we're featuring LinkedIn questions here at AsConfluent. And Michail something like this. Something like this. Um who is apparently a machine learning and data science at Google Cloud, so he actually works for Google Cloud. That's really cool. Google Cloud is absolutely the coolest. We had good announcements with Google this week. It's been a very Googly week for us here at Confluent. So Michael from Google asks model training and model deployment each have their own complexities and considerations. Yeah, no doubt, they're each and entire world in their own. Can you discuss in detail how Kafka and other tools fit within each of the phases of machine learning project? So maybe I think we should start by actually discussing the two different phases because I think so many people, even like presenters at big conferences, totally get it wrong and mix everything up into like a big mishmash together.

SPEAKER_03 6:56

Yes, absolutely. And that's very important to really separate the model training from the model deployment to the predictions. It's two completely different things. And that's a good thing because you can separate them easily. And so we will use the whiteboard here to explain this in more detail. And let's about uh think about a use case where we have some kind of producer which produces some data. This can be, for example, sensor data from connected cars or mobile app for any other use case, whatever data you generate. Um, this data has to be processed. So, what we see here a lot here is of course that you use um Kafka as an gestion um pipeline because with this you have all the adventures of Kafka, which you of the audience of this um session here should know, like high scalability, high throughput, and so on.

SPEAKER_01 7:38

And also decoupling the producers from the consumers.

SPEAKER_03 7:41

Exactly, that's another point because typically you have not just one producer but many of them, and and you can have back pressure and all these things. And the main point, however, is then um that you want to ingest all the data into a big data store to train the models. Um on the way there, typically, as data scientists, you have to do a lot of pre-processing.

SPEAKER_01 7:58

So basically, I'm sending the data over here, and it will land on something like let's say S3. Exactly. So have Google's here. Where do Google? BigQuery?

SPEAKER_03 8:06

Do we put it in? No, that's in the end that Google Cloud Storage. Google Cloud Storage.

SPEAKER_01 8:09

Let's do maybe Google Cloud Storage.

SPEAKER_03 8:11

Exactly. So that's where you want to store it, um, where you can train your models. Before you do that, typically, a big part of the data scientist is to do pre-processing of data at scale, because you have to do it for the millions of data sets. And therefore, that's again good for Kafka because you here you can use things like Kafka streams or for example KSQL to do that. Here you can filter or transform or something like this, and then send it to another Kafka topic. And from here, one of the consumers can be your big data storage. In parallel to that, of course, any other consumers can process it in real time.

SPEAKER_01 8:42

But then when you do the training, you do it on this, right? So I have some kind of a process. Exactly.

SPEAKER_03 8:48

So the point is now here. Um when you think about the machine learning part, which is the fancy frameworks like TensorFlow, for example, they really are used not for the ingestion pipeline. Um that's really why you use it here on the cold data, let's say. So here you have something like TensorFlow, which is used to analyze the data. Um it takes some time, sometimes hours or even days, and then you get a model out of that. So the result of analyzing this historical data, that's in the end what is a model. And a model that's very important to understand, that's in the end some kind of binary, or depending on the technology, an XML file or Java code. And that's the result of the training part. And that now can be used somewhere to deploy to do predictions on top of that.

SPEAKER_01 9:28

So basically, when the when we have a model and maybe the model has been tested and validated a bit, the data scientists will say, Hey, thank you, I'm done, right? You got my results, I'm now going to work on a different project.

SPEAKER_03 9:41

Exactly. I mean, first of all, typically not the m first model is the best. So and that's another great use case for Kafka because um you can consume the data from the Kafka log several times to train different models, sometimes even with different technologies, not just TensorFlow, but maybe H2O or some other ones.

SPEAKER_01 9:55

Um everyone tells me that Keras is like the coolest new thing. Yeah, but that's the end of wrapper on TensorFlow. Oh, Keras is just a wrapper for that?

SPEAKER_03 10:03

Yeah, it makes it much, much easier, yes. It's embedded into the TensorFlow.

SPEAKER_01 10:06

What about PyTorch?

SPEAKER_03 10:07

Is that also that's a different framework, yeah. But you could, for example, train models with TensorFlow and with PyTorch and compare them. And depending on which model is best, you use that as your final model, and then the data scientist is done for the moment and says, this is our model which we want to deploy to production to reduce risk, to make more revenue, or whatever the use case is.

SPEAKER_01 10:25

Okay. And then the data engineers take over from here.

SPEAKER_03 10:28

Or the software engineers take over from the It's always um depends on how you define these terms. So sometimes this is also the data engineer which does the pre-processing. So it's not so much about which role is it, but the different kind of process.

SPEAKER_01 10:41

So it's very important that the production process starts here.

SPEAKER_03 10:44

Exactly. And that's the other point. So that's really important. It's really separated from the training part. And so here now we have a model and now we want to deploy that to production. So that's a different part of that. So think about that. We want to produce a new app, and in this app, we want to use this model which we trained here. So this is an application, right?

SPEAKER_01 11:03

And this is that would be something like maybe people you may know, or uh car batteries you may want to buy, or yeah, in the in the end, so here now we have again a producer.

SPEAKER_03 11:15

So the funny thing is the pipeline in the end is very similar to the pipeline for training the model. Because let's say we have sensor data for um cars. This was used for training a model, and now we use the same pipeline. And sometimes we can reuse the ingestion pipeline like Kafka Connect, like pre-processing with KSQL, the same way here, because here is input. We need the same input to process it.

SPEAKER_01 11:36

So if it's for example, something about the battery behavior in the car, here you have battery behaviors, this exact same data goes here, but this time instead of training, we basically check does your battery need replacement.

SPEAKER_03 11:48

Exactly. And here in either it can be in this case directly deployed here. So here really we have our TensorFlow model, our binary. We deploy this, for example, into a KSQL application which does continuous processing of the sensor data. And then after that, on the one side it can send information back, like a prediction, hey, please stop your car, it engine will break or something like this, with predictive maintenance use case, right? And we actually have a few customers which are doing this already. And that's also where you see for these kind of use cases, you need at scale 24-7 availability, and that's perfect. And on the other side, however, it's completely separated from the training model. So we see more and more customers which do the big data training in Google Cloud, for example. But therefore the application itself, it runs at the edge. It could be in the car or in a gateway and in the specific region.

SPEAKER_01 12:34

For example, the notification can show up in your phone, it can show up on some in an engineer's desk saying, hey, you know, the new batteries don't perform as well as you expect them to. It should have lasted a year, but we are seeing those issues after two months.

SPEAKER_03 12:48

Absolutely. I mean that's that's in the end back then.

SPEAKER_01 12:50

And you can have a lot of different consumers to the resort. Absolutely.

SPEAKER_03 12:53

And I mean this could also go back here to another Kafka topic, right? And from here it goes to the mobile app or maybe to your Elasticsearch installation or whatever. And it's always b-directional. You can analyze data here and then also send it back. Depending, sometimes it has to be real, real time, then you deploy an application close to the edge or maybe even in the cloud. So that depends on your use case.

SPEAKER_01 13:11

That's uh I really like this as an architecture. Like it because it because it's so flexible. By separating this into two different stages, you can kind of really fit the best technologies on each side, which is really kinda kind of amazing.

SPEAKER_03 13:25

Absolutely.

SPEAKER_01 13:26

So that's something that people already do in production.

SPEAKER_03 13:29

Yes, they do. And uh the key point really is the first challenge we discussed about um how to combine um the data scientist with Python zone with the production environment and how to build this pipeline at scale and all these things. And we have several customers which do that already in production. Um with Kafka, it's not that hard because for ML, I mean you combine it with technologies like TensorFlow, but the same the idea is the same like for any other business use case which you build around Kafka in Confluence.

SPEAKER_01 13:52

So since people do it in production, obviously it's not all roses. What's the hardest part? What's the thing that everyone always trips on when they try to do this kind of thing?

SPEAKER_03 14:00

Um when they try to do this, of course, it's it's simply really hard to understand at high scale how to run this highly available and with good performance, with zero data loss. But that's really the common challenges for machine learning from a production perspective. It's the same like for any other Kafka application. It's a Java environment. Um so the challenge is really to understand Kafka well, to scale that in a production way. And then if you have that running, um, you can do that the same way for ML. So what I always talk about in use cases and in presentations at conferences is that it's not just about the fancy things like you think about self-driving cars and about image recognition. That's cool new deep learning stuff. But I would say 80% of the use cases today is more about improving traditional business processes, like fraud detection, like cross-selling.

SPEAKER_01 14:44

So you can start with something very simple, get the architecture right, and then start iterating on fancy modules.

SPEAKER_03 14:50

And improve it more and more, and then um, this is what we discussed training and then deployment. And of course, this is then a continuous process because um this has to improve. You ingest new data, you train a new model, you improve it, and redeploy that.

SPEAKER_01 15:01

And here's basically we can close off the process like that.

SPEAKER_03 15:04

Exactly. So it's uh and in best case, and we have some customers, they automate this in a continuous integration. And some customers even um share the new model updates via a Kafka topic again. So they send it to uh to a Kafka topic here, and from here the consumer, like the application, consumes the new model and deploys it internally.

SPEAKER_01 15:21

Something like that.

SPEAKER_03 15:23

Exactly. And in this way, you have a continuous way to improve your model. That again depends on the use case. Some models are updated once a month, but others uh once an hour or so.

SPEAKER_01 15:31

This also deals with what I see as the biggest challenge. Most people on the first iteration, the data that they're getting in is crap. Like it's absolutely crap. Not just because it's dirty that you can deal with, but just uh, you know, those cars that have the sensors are quite new on the road. So in the beginning you just don't have that much data. It's just from the top end of the population, maybe just from a few cities. It takes a long time to actually collect enough data to make really good predictions. So you really need to get this cycle uh going. So as new data comes in, your predictions will continuously improve rather than continuously become less and less relevant.

SPEAKER_02 16:08

Yes, exactly.

SPEAKER_01 16:09

Cool. That's that's amazing. I really like that. And then last question on that topic. Um Mitch Seymour from MailChimp says I saw your Kafka streams and KSQL examples, and I think you had a bunch of examples on GitHub, right? So we'll post them on the channel here. And you deploy H2O and TensorFlow models. Can you please discuss the trade-offs of embedding an analytical model directly in the streaming app versus using an RPC to a model server? So over here we kind of bundled everything in the app.

SPEAKER_03 16:45

Yes.

SPEAKER_01 16:45

But there is another option.

SPEAKER_03 16:47

There's another option. And actually, this is the option which we often see from a Kafka perspective, which makes a lot of sense because we train our model somewhere else, it's just a binary, and then we simply embed it into our application, like Kafka Streams, that's just Java code. You can embed, in best case, a Java class like in H2O, or in TensorFlow if it's a protobuf or whatever technology you use, you can just deploy it here. And the huge advantage of that is that's really here in the application. That could be at the edge, so you don't have to communicate to another kind of server to get the prediction because it's directly in the app. So it's um decoupled from other systems, and it also processes things in real time because you don't have to do remote calls for that. So that's the huge advantage of this kind of deployment, and therefore you also can scale it up easily.

SPEAKER_01 17:30

So just so I'll get it. The model is basically going to be codified in a jar. Is that like a jar?

SPEAKER_03 17:37

Or it depends on what the technology is. Sometimes it's a zip file, sometimes it's a Java class. Um that depends what it is.

SPEAKER_01 17:42

Okay, so yeah, I'm just trying to figure out like how hard it is. Like I'm an engineer, obviously I don't I've never even seen a model. How hard would it be for me as a Java engineer to call the model? Because if it's like a jar, I can you know dynamically load it. It's an API, I know how to do those. If it's a zip, like what do we need to do?

SPEAKER_03 18:02

That again depends on the framework you use, but there's always also an API to use it here. So for example, for TensorFlow, you have protobuf models, which are binary, which you embed here, but then you also have the Java API. So from your Kafka streams application, it's just one more line of code. Um, load this model and then apply to do a prediction on your incoming data into the case.

SPEAKER_01 18:19

So I need to use like the H2O or TensorFlow library in my app basically. Exactly. Okay.

SPEAKER_03 18:25

There's to be more clear, here's also standards like um ONNX or something like this, where you can standardize it so that you don't rely on a specific framework. That's another option there.

SPEAKER_01 18:33

Yeah, that sounds kind of nice because those frameworks seem to kind of get grow in popularity and then wan in popularity quite rapidly.

SPEAKER_03 18:41

There's always a trade-off between standards and using the direct API, right, with all its pros and cons, and depending on the use case you have to evaluate there's ONX, PMML, and so on.

SPEAKER_01 18:49

Yeah, that like basically using the an API can have a better performance because it's actually someone who knows what they're doing implemented it.

SPEAKER_03 18:57

Yeah. Yeah. So this is the one option. Deploy the binary model in your application and use it from here. The other option, however, is, and that's what most data scientists uh know from the beginning, is that you also on this side you have a model server. So here you have your model server. And that's um something in case of TensorFlow, for example, you have something like TensorFlow serving, which is the model server specifically for the TensorFlow framework. That's in the end something which you host somewhere, either close to the edge or in the cloud, where it's hosted by Google, for example. Um and then you deploy your TensorFlow model here, your binary.

SPEAKER_01 19:31

Oh, I like that. So if it's on the edge, like my self-driving car can actually communicate directly with the serving model and not have to go all the way to my data center.

SPEAKER_03 19:42

However, the main difference between a model server is that here you always have to do an RPC call. Um in TensorFlow they use GRPC, um and some other technologies use HTTP, which is even worse. But the main architectural difference is that here, as you see, you always have to do a call and a request response. Um that makes it easier for some things because some people simply know how to do a request response call. Right? So that's the easy part of that, so it's good. And also another advantage is that it the model server has specific ML features, like you can easily do versioning and A B testing. If you want to do these things here, you have to implement them in your own template or something.

SPEAKER_01 20:15

Which we most of us really already have a way to do, like we in a confluent cloud we do a lot of feature flags. Yeah. So we already know how to do feature flags.

SPEAKER_03 20:25

Sure, it's just you have to think about that that you have to do that. And the other big point here is that um it's not just about um the technology like HTTP that you have to do more communication, but the big problem is what do you do, for example, in case of failure? If this communication does not work, right, how do you handle that here? Um and so there is always the trade-offs, and also we will share the GitHub links, I've discussed it in much more detail, um, where you can take a look at that. So it has all its pros and cons. Um it's not that one solution is always the best.

SPEAKER_01 20:51

I love the idea of using model server for things that are on the edge, where my clients, for example, the mobile app can just talk to you directly, and I don't have to do that much extra work. And there's no real RPC calls between different data centers and that kind of thing, versus using embedded if it's an app running on my data center if I don't have an edge. Which a lot of use cases really don't have an edge to them. Yeah, absolutely.

SPEAKER_03 21:15

And so I mean that's also the main strategy of Google, right? I mean they um either host this model server for you in your Google ML engine in the cloud, or they provide it to you, for example, a stocker container, like in my GitHub example, where you can run it anywhere. I do it on my laptop.

SPEAKER_01 21:27

So I can actually have it as a sidecar. So I can write my app and take the model server, run it as a sidecar on the same Kubernetes pod, and get the benefits of all the AP API calls are basically like it's request response, but it's on the local machine, so I get all the that's amazing. That's amazing. That's great.

unknown 21:46

Cool.

SPEAKER_01 21:48

Okay, so the next question is actually not about machine learning at all. Um Aditya Varun Chadha had a question for a Apache Kafka folks, which we both are. Sure. We have been doing Apache Kafka inc for a while now. How long have you been at Confluent?

SPEAKER_03 22:07

Uh over two years already. So I was one of the first hundred or something like this.

SPEAKER_01 22:11

Yeah, defin definitely enough time to be an Apache Kafka folks.

SPEAKER_03 22:14

I think so.

SPEAKER_01 22:16

Can consumers be assigned to read from replicas? Meaning followers, because consumers always read from the leaders, that's what they do. If not by design, as in if we there is a good reason we prevent it, are there any documentation or links explaining why not? And then it kind of explains many replicated systems use replica for read scaling in addition to fault tolerance when eventual consistency is enough. And then I've been CC'd, and that's how I find the question and got it over here. So that I think a good question because a lot of people actually wonder why does Kafka always insist on reading from the leader while m things like Cassandra, which you worked with before, will allow you to read from just about anywhere.

SPEAKER_03 23:06

Yeah. Sure, and that's a pretty good question. Um the bad news is it's not possible today to read from the followers, right? The good news is there is a Kafka improvement proposal, shortly KIP 392. So take a look at that, we will also provide the link. And so actually, um we are working on something like this today already, and we want to contribute that back to Apache Kafka. So in the future, you will be able to consume and fetch data from the followers. And with that, it's exactly the benefits which you already described in the tweet. And actually, it's also still related to machine learning again, because there if you build high-scale architectures over different kinds of regions, then there it's very important, right? Even if you do a prediction somewhere and um send it back to a topic, everybody else can consume it much easier because you don't have to consume it just from the follower.

SPEAKER_01 23:49

I mean, yeah, so you can consume the model at the edge, as we discussed earlier. But and you can also, I just thinking like in general, you usually train models, especially like in large companies, over very large clusters. And you if you want to do it in something like Kafka streams, for example, which I don't know if I would, but suppose that we want to make Kafka streams more useful for that, you can start by scaling it, basically scaling it out and doing kind of more local processing on the machines that actually has the data. So it's kind of a cross between more Cassandra HGFS model and Kafka.

SPEAKER_03 24:26

So it's a huge feature, and hopefully it's coming very soon in one of the next few releases in Apache Kafka.

SPEAKER_01 24:31

Yeah, but I think it's worthwhile talking about why it took us so long and also why is the feature actually complicated to add. Because like I think the key here, and if you read a kip, you'll see discussed in ridiculous level of details because that's how kips work. Uh but the idea is that even though we allow you to read from replica, Kafka will remain a consistent system. That's incredibly important. Like the whole reason that right now we don't allow you to read from anyone except the leader, and it's not only that, the real leader will not let you read events that were written to the leader but were not replicated to all the in-sync replicas. That's the big meaning of in-sync replica. That until your message is on all of those, you cannot even read it. And the reason is that we don't want you to read a message, and then the leader will go away, election will happen, something will happen, and next time you connect, the message is no longer there. That breaks our consistency model, that breaks our delivery guarantees. Not gonna happen. And so this is true today, and we do it by just forcing you to read from the leader, and the leader knows how many replicas got it, and uh we guarantee safety that way. But the entire keep is how we take this idea of high watermark, those messages that are safe to read, and make it distributed so all the replicas will know about the high watermark, so they can also serve data safely. So we are doing it by not with not breaking the consistent guarantees. So it's not going to be like another Cassandra.

SPEAKER_03 25:57

Yeah, that's good, that's important, yeah.

SPEAKER_01 25:59

Okay, great. So totally different model and great, great question, Aditya. And don't forget to check out the keep because it's awesome. Like you learn a lot about Kafka by reading those keeps because of the level of detail of the discussion. So we are done with questions. Now it's time for the traditional patting ourselves on the back. So Mohammed Humein commented on our previous, previous video, Ask Confluent 11, and said, Gwen, you rock. Thank you, Mohammed. It made my day. I like knowing that I rock. Next time I may bring a guitar and see how much that rocks. Yeah, thank you. Next comment is from Prasad Penumarty. He commented on Niha's keynote on Kafka Summit 2017, which was Go Against the Flow, Database and Stream Processing. Have you been in Kafka Summit 2017?

SPEAKER_02 26:53

Yeah, actually.

SPEAKER_01 26:53

You remember that was this talk was when we announced KSQL.

SPEAKER_02 26:57

That was a great announcement at that time.

SPEAKER_01 26:59

Yes, and you remember when we did the way we demoed it was actually with those cars? I was just saying how we came full circle because we created kind of a cool demo with race cars, and you get metrics of how they do laps around a racetrack. And next thing you know, we're collecting data from real cars and cleaning it up, training models, surfing models.

SPEAKER_02 27:19

Yeah, that's cool.

SPEAKER_01 27:20

It's so cool. Yes, so Persad said uh it's rocking Chris presentation, loved it. And obviously, it's Kafka Summit, it's Nana Kitty, it's KSQL, and it's cars. So what's not to like? Obviously, it's perfect.

SPEAKER_03 27:36

What do you want more?

SPEAKER_01 27:37

Yeah, exactly, right? So, yeah, it's really good video. I actually went back to watch it after I saw this comment.

SPEAKER_03 27:43

And the great thing is that many of these um videos of Kafka Summit are even interesting after two years or so, right? And you can watch all of them free on our website, on the Kafka Summit website.

SPEAKER_01 27:51

So you know how they said the future is here but not evenly distributed? A lot of times you go through talks when someone said something two years ago, and you're like, oh, now I get why he gave very important tips that solves my problem right now. So a lot of those are really evergreen. On the other hand, a lot of them really aged because the technology keeps moving and a lot of the tips are no longer all that relevant. Um in New York uh last week, Jason did a talk basically revisiting exactly once two years later, which was always kind of interesting. Like there is the feature you imagine when you just release it, and then there is all the lessons learned after two years, what worked, what didn't work, recommendation, etc. So that was pretty cool. Everything is obviously available online Kafka Summit website. So fresh comment. Jake Reps last week gave a talk at Kafka Summit New York City. He gave the keynote, which was titled Events Everywhere, which by the way, I love the title, it's just so inspiring and it's very, very true. Everywhere and also all the time, everywhere and continuously. And Duff Duby commented, unfortunately, not on Jay and not on the content of the keynote, but he said, Tim has such a trailer guy voice that the first time I watched this video, I thought it was surely dubbed over by a professional voice actor. So yeah, it's hilarious. Apparently, it's not just that he has a professional voice, we've been told that he sounds just like Jeff Goldproof. I don't know if you agree or not, but possibly. But I was thinking, like, we're a startup, we save money by hiring amazing directors of community who also double as voice actors.

SPEAKER_02 29:34

Absolutely.

SPEAKER_01 29:36

But yeah, it's also worthwhile watching in order to get the whole new event-driven paradigm and how Kafka really enables new and really cool use cases, so not just for team's voice, which is a small bonus. And uh Nyanar Kiddy gave uh the keynote at Spring One Platform 2018, which I missed, but I saw all over Twitter people were just all over the top, excited about how amazing this uh keynote was and how it was the best keynote ever. So I kind of feel bad for missing out. But we have the video up on uh YouTube. And Tanmei Nike commented, can we have Kafka Summit in India? Because clearly there is big demand for seeing this kind of content and opportunity to interact with Kafka experts anywhere in the world. So I asked our Kafka Summit team. I bet maybe you want to know is it going to be Kafka Summit Germany anytime soon?

SPEAKER_02 30:31

That would be awesome.

SPEAKER_01 30:32

Yes. So the answer is that right now we don't have any plans expanding into India, but we are growing Kafka Summit a lot, and we are always looking for new places and new markets to get into, and new locations, new people, new communities. So basically, watch the space, follow hashtag Kafka Summit on Twitter, follow Confluent on Twitter, uh, follow the kafkasummit.org website. And when we decide on new locations, this year we had three locations: New York, London, and San Francisco. So maybe next year we'll have four, and who knows, maybe India, maybe Germany.

SPEAKER_03 31:14

We'll definitely grow.

SPEAKER_01 31:15

Yes. Yeah, it's and especially like demand in Asia is growing like crazy. So I can totally see somewhere in the Apex region a Kafka summit growing. That was not a promise, by the way. That was me giving crazy predictions. Okay, that's it for today. Yeah. That was a lot of fun. Thank you so much for joining us and telling us all about how to do architectures for machine learning. Yeah, that was great.

SPEAKER_03 31:41

And um, I really want to ask the audience also if you have any other feedback or questions about machine learning, Kafka, just let us know and we want to discuss with you. Comments. Because it's absolutely a hot topic everywhere.

SPEAKER_01 31:51

And we are super excited about just talking more about things that are kind of relevant and interesting to our users. Uh just remember learning is not just for machines, you have to learn too. And subscribing to our channel is a good way to stay on top of the latest topics and learn more about Kafka and related ecosystems architectures and topics. So, join us next time. Okay, see ya. Goodbye.

SPEAKER_00 32:18

And there you have it. I hope that was helpful to you. If you've got questions, you can ask me at TLberglund on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time.