Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
MySQL, Cassandra, BigQuery, and Streaming Analytics with Joy Gao
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Joy Gao chats with Tim Berglund about all things related to streaming ETL—how it works, its benefits, and the implementation and operational challenges involved. She describes the streaming ETL architecture at WePay from MySQL/Cassandra to BigQuery using Apache Kafka®, Kafka Connect, and Debezium.
EPISODE LINKS
- Cassandra Source Connector Documentation
- Streaming Databases in Real Time with MySQL, Debezium, and Kafka
- Streaming Cassandra at WePay
- Change Data Capture with Debezium ft. Gunnar Morling
- Join the Confluent Community Slack
- Fully managed Apache Kafka as a service! Try free.
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
Kafka is a key part of modern streaming analytics pipelines. What's one of those, you ask? Well, rather than extracting data at night or during off-peak hours, whatever those even are, streaming analytics systems process events as they occur and make whatever analytical computations the business needs in real time, right when the event happens. Joy Gao is in charge of just such a system at WePay, a payment service provider designed for web-based platforms. She tells us all about that pipeline and in particular how it integrates with the organization's relational databases and the quite interesting problems that arise from those integrations. On today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.
SPEAKER_01Thank you for having me here.
SPEAKER_00Joy, you work at WiiPay as a senior software engineer. Of course, that could mean a lot of things. What do you do there?
SPEAKER_02Yeah, so I work on the data infrastructure team here at WePay. Specifically for data infrastructure, I kind of uh break down what our team does into two areas. Well, uh we essentially move the data from production data houses into our analytical data warehouses, uh, sorry, production databases into our analytical data warehouse. And uh it's broken down into a traditional kind of ETL batch-oriented approach as well as a real-time streaming approach. So uh most of my work from the day-to-day has been kind of um maintaining this these data pipelines from various sources and then getting the data into various destinations.
SPEAKER_00Gotcha. So for listeners who might not be familiar with ETL, and it's it's possible to be uh a productive application developer and just never touch ETL stuff at all. Uh describe a traditional ETL process and then a streaming ETL process and just kind of tell us tell us how streaming ETL is different.
SPEAKER_02Yeah, totally. So ETL stands for extract, transform, load. Uh the extract part essentially referring to uh extracting the data from some kind of a source database. The transformation is when the data needs to be transformed into a format that the downstream um sync can understand, uh or the downstream database can understand, and then uh the load part is once the transformation has been completed, we can load the data into the data warehouse or some kind of downstream sync. So specifically, the typical approach for ETL is that let's say uh at least in the WePay scenario, the way we traditionally have handled ETL, is in our MySQL databases, we uh uh ask our developers to specify uh create timestamp and a modified timestamp for every single piece of data. And we have these jobs that would periodically um query the database for change, and they would specifically look at the modified timestamp. And if in some past time frame, uh some interval of time, the data has been changed, this will be picked up, and this information will essentially be what's being loaded into the downstream data warehouse. And we run this job on a regular interval to do these kind of incremental loads of data from our uh production databases into our data warehouse.
SPEAKER_00And you've got kind of an organization-wide commitment to a particular schema there, which is created at and updated at. And everybody who has a table, every application that has a table and does things to a table uh is committed to have those two fields. Is that right?
SPEAKER_02Uh yeah, so specifically for us, most of our database traditionally have always been MySQL. So um we use Airflow to do the ETL load. And uh Airflow essentially will take the MySQL data um using some kind of database client, and then it will transform the data into a schema that our BigQuery, our data warehouse understands, and then uh load the data into BigQuery.
SPEAKER_00Gotcha. Um Airflow, and so terrible question on my part. I feel like I should know this, but I know it's an Apache project. Yes. And I know it's something vaguely ETL related, but um, so it's funny. Every once in a while, you know, I'll be asking questions because I'll think, well, a listener might want to know that. And then other questions were there questions because I don't know. So tell me what Airflow does.
SPEAKER_02Yeah, so uh you're completely right. Airflow is an open source uh project, uh, it's an Apache project that was originally created by Airbnb. And the goal of Airflow was essentially to manage complex workflows and data pipelines. So it's designed uh for ETL process, but over time, um as Airflow evolved, it's being used kind of like a Chrome, but on steroids, because it's got all kinds of features that are uh being added to it, so that it's essentially a job scheduler with a uh a lot of um DAG dependencies and uh allowing uh users to orchestrate uh jobs with very complex logic.
SPEAKER_00Gotcha. So you could, for example, set up a graph of this extraction needs to happen, then this extraction, and you know, then this transformation can be done, and you know, then this other load. So the the graph of stuff like that.
SPEAKER_02Exactly. So in our case, for our ETL pipeline, we have one task that will convert the data from MySQL uh into uh Google Cloud Storage, which is just a cloud storage place where we store the data that we're about to upload into BigQuery. And then we would have a second task that depends on the success of the initial task, uh, which then it will take that data and then load it in batch into BigQuery.
SPEAKER_00Gotcha, gotcha. And your destination, uh, you know, where you're loading all of this data in the ETL process is BigQuery. BigQuery is where your analytics take place.
SPEAKER_02Exactly, yeah. There are other, yeah, there are other um other applications or other databases where the data could also go, but in which case, traditionally, we had to build a separate data pipeline for every single destination, which has been eliminated by introducing Kafka.
SPEAKER_00Ah, okay, which I guess is what we're talking about next. We that's that's kind of the batch ETL process that you described. And even what you've got is frankly a pretty modern batch ETL process. If you look at classical um turn-of-the-century ETL or you know, what what was state of the art at the turn of the century, um, the data darn sure didn't go into BigQuery.
SPEAKER_01That's very true.
SPEAKER_00And uh you it sounds like you have the ability to do pretty small batches, like you could run that hourly, the system you described, and it it shouldn't get too upset, right?
SPEAKER_02Yeah, we are able to run that actually as frequent as every 15 minutes for some of the tables, uh especially some of the larger tables. So um yeah, it's been it's been uh efficient, but not optimal.
SPEAKER_00Not optimal, right? Yeah, so compared to uh 10 years ago, ETL, you know, that's that's still pretty close to space age. That's that's a good system. But uh it's not streaming ETL and it's not literally real time, which um sounds like what the streaming ETL is all about. So talk about your streaming pipeline from a high level.
SPEAKER_02For sure. So the way our streaming data pipeline works at WiiPay is it's built on top of this, uh on top of Kafka. So our source is MySQL, our sync is BitQuery as before. Um, and then the way we feed the data from MySQL into Kafka is through an open source project called Debezium, which is built on top of the Kafka Connect framework. And the way we get data from Kafka into BigQuery is via Kafka, is via KCBQ, which stands for Kafka Connect BigQuery. This is um a sync connector that we have created or built at WiiPay um in order to stream data from Kafka into BigQuery.
SPEAKER_00Gotcha, gotcha. Um, you know, I forgot to ask you uh for the benefit of the listeners who might not know, what does WePay do?
SPEAKER_02Yeah, so we are uh in the fintech space and we provide payment APIs for platform and businesses to make payment for their businesses, essentially.
SPEAKER_00Gotcha. Gotcha. So uh I I think we'll get to I don't know, I don't know how much you'll be able to talk about, but there's probably a number of real-time analytics where the difference between 15 minutes ago and you know uh a second and a half ago is a huge difference.
SPEAKER_01Yes.
SPEAKER_00Yeah. Uh that's usually the case. Usually payment payment people are keenly interested in right now.
SPEAKER_02Yes. And I think with data in general, the closer the data is to now, the more valuable it is. So as the data age over time, it actually loses some of its value. So by allowing a streaming data pipeline, it allows us to extract the maximum amount of value from the data that we have.
SPEAKER_00Yeah, yeah. Um uh that makes a lot of sense. So you've got uh Kafka Connect and uh Debezium specifically, the Debezium connectors taking data from uh MySQL and loading it into BigQuery. That's the essence of the streaming data pipeline. Uh can you say what kinds of data you do that with? Is that like everything? Or is that um and by the way, stop me if I ask you a thing that is secret sauce and you can't answer because I'm sure, again, being a fintech company, there's any number of things I know you can't talk about.
SPEAKER_02Sure. Yeah. So for us right now, we stream data from all of our microservices. So WiiPay was historically built with a monolithic database, and over time we have refactored that into various microservices. And the data in our microservices are mostly using MySQL as their database. So we've been able to kind of build this data pipeline for every every microservices that is being created or that is existing.
SPEAKER_00Okay, so these microservices are all using their own MySQL databases basically?
SPEAKER_02Exactly.
SPEAKER_00Okay. And you stream all of their data uh you extract, stream through Kafka and load into BigQuery. Um is there any, do you at any point use that extract process to get events from one microservice to another? Is there like actual application integration happening here?
SPEAKER_02Uh yes, we do. So we have a separate set of Kafka topics, which we call the queuing topics, which allows us to do cross-microservice communications. And we follow a similar um approach that extract uh using using KCBQ that extract the data from Kafka into BigQuery as well.
SPEAKER_00Gotcha. So that's uh you one microservice is producing to a Kafka topic and another service is directly consuming that topic, but there's also the other stream data.
SPEAKER_02Exactly. If there are any kind of analytical needs for that information to be exposed into BigQuery, we will then have the capability of exposing that uh by um adding this into our connector.
SPEAKER_00Gotcha, gotcha. And uh let us agree that in most cases uh there is an analytical interest in that data for almost no matter what that data is.
SPEAKER_01It happens pretty frequently for sure.
SPEAKER_00Uh-huh. So you think, no, no, this is just between services, nobody cares. And then an analyst says, give me that, and uh and you come to the rescue. Okay, cool. I actually so I I knew um prior to this conversation, I knew about your streaming data pipeline stuff. I mean, you've talked about that at Kafka Summit, and you know, we've talked about it. Um, but I didn't know that you guys were doing event-driven microservices also. That is super cool. Um, that is uh, in my view, like the right way to build software, the the best, the best way of building systems of this kind of medium level of complexity that most of us are working on. And yeah, it's always nice to hear somebody doing it.
SPEAKER_02Yeah, there's an interesting um kind of component of that I want to bring up, which is I totally agree that event sourcing or uh approach for microservices is works really well. Um, except for one scenario that happens frequently in our in the payment industry that makes event sourcing not the most ideal approach. Um and that is the um idea of read-your write consistency. So read your write consistency means when you write a piece of data to some database and then you follow up with a read immediately to extract that information, um you're expected to read what you just wrote into the database. However, with the approach of event sourcing, which is the idea that you write your data into Kafka first, and then you have some database that's consuming the data from the Kafka log, in which case there is a delay from the data of the data to arrive into the database. So um if we do a read immediately after that write, it is potentially possible to get stale data. So this would be the one scenario where event sourcing doesn't quite cover this case.
SPEAKER_00Got it, got it. That makes sense. You'd have to do different things like um well, the reads, the reads would have to be happening against the service that took the rights and uh right, and in which case for Kafka, it's not that's not an API that is easily exposed. Right, right. Um yeah, okay.
SPEAKER_02Yeah, and it's very common in the payment space because we're talking about uh financial transactions where say we don't want user to extract money once it has reached zero, because then we'd be losing money. So um these would be the scenario we have to watch out for. But other than that, event sourcing uh has been really great.
SPEAKER_00Excellent, excellent. I'm just kind of conventional application stuff. It seems like the right way to the right way to do things. Um so okay, you've mentioned Debesium. Um we have recently and uh recently recorded, as of the time you and I are talking, uh recently recorded uh an episode about Debezium, and that will probably launch before this one does. So, listeners, if it hasn't, um, you know, there's a little bit of asynchrony here in the process. So look to the near future to that episode on Debezium, but it has probably happened in the last couple or two or three weeks, uh, which is good news. And given that, you've got that hopefully that uh good deep dive into Debezium. But uh Joy, tell us for maybe anybody who didn't hear that or somebody who just needs a refresher, uh, what's the very quick intro to Connect and Debezium?
SPEAKER_02So uh Debezium essentially uh is built on top of Kafka Connect. It's specifically it's the source connector. And what Debezium does is that it uh it's uh it essentially leverages the concept of changing to the capture, um, which it extracts the changes of events from the source database and then sending these events into Kafka. Um and it leverages the source connector as a way to to essentially implement this mechanism.
SPEAKER_00Cool. And one of the big things is like there's a there's a I always want to call it more standard, but it's it's it's not more standard, it's just another uh maybe simpler connector called the JDBC connector, where uh you know you you can tell it, hey, go run this query on this table every uh second, and if you see records with a changed timestamp, a new timestamp, um, produce them into this topic, right? Uh the difference with Debezium is it is usually reaching in some sense directly into the database, like reading uh you're using MySQL, so the Debezium connector. I and I actually I forget how that works. It's either like reading the bin log directly or there's some API that MySQL exposes, but it it has direct access to the actual mutation events that the database processes, right?
SPEAKER_02Yeah. So what it uh specifically for the case of MySQL, what Debesium does is that it actually pretends to be a MySQL replica. That's right. So that's right. So yeah, it's essentially streaming the MySQL bin log as if it's a replica, but instead of storing that data into itself, its database, as a database, it will actually send the data to Kafka. So it's a pass-through. Yeah.
SPEAKER_00Producing it to a Kafka topic. So it is kind of replicating. It's you know, from one change log to another slightly more durable change log, uh, or in some cases, uh, much, much, much more durable change log, depending on what your retention policy is. Uh so yeah, you use that uh Debesium connector, you get stuff into Kafka. And then what else goes on? I mean, I think everybody can basically imagine uh here's the MySQL connector, it's acting like a replica, uh, it's producing uh messages into a topic, and then there's some consumer on that topic that writes them into big data, and that's either a Kafka connect, uh, sorry, BigQuery, uh big data, that's a great product name, writes them into BigQuery, and that's either another connector or your own consumer uh interface or whatever, but that the extract and the load, I think, makes sense. In the streaming pipeline, do you end up doing a lot of transforming or does that just end in, you know, does that come out in the wash of the big table queries? How does all that work?
SPEAKER_02So, yeah, uh so the Debezium connector specifically will read the data. Um, and because it's uh written in Java, so it will uh basically receive this uh object as some kind of Java object. Um, and then it will convert this Java object in in our case into Avro. And then this Avro information uh is being propagated into uh being sent into Kafka. And on the Kafka Connect BigQuery or KCPQ side, it's taking the data or it's consuming the data from Kafka, and then it's going to do another transformation uh that converts it into a schema that BigQuery understands. So I I can say there are two transformations that are happening through Avik.
SPEAKER_00Gotcha. One is one is you know uh API object that the MySQL API gives you to Avro, and then the other is Avro to um to whatever Big Table is happy with. Yeah. And you did say, I think you said this before, the the Kafka to Big Table end of things is the connector. It's the um Kafka Connect Big Table connector.
SPEAKER_02Uh yeah, there's it is also using a Kafka sync connector.
SPEAKER_00Cool. Okay. So or uh yeah, sync connector. So you're you're uh you're big connect users. Um is there so in your particular case you're you've got a lot of uh tables, you know, basically that you're uh doing change data capture on and you're streaming the data into BigQuery. Is there any domain-specific transformation? So just to frame that, it used to be the case in the good old days of star schema relational data warehouses, um, that you know, you would pre-compute things and you would cleanse data and you'd you know, maybe uh rationalize customer records and and do some like lightweight version of master data management or something, where there's actually a lot of computation and uh you know, computation of the usually of the pain in the butt variety in the T, in the transform. It's not just, oh, you're in this one serialization format, I want you to be in this other serialization format, but stuff to do where computers get warm. And in your pipelines, do you do that or do you or you do you have the privilege of just kind of moving things through?
SPEAKER_02That's a really good question, actually. So what we do at WiiPay is probably slightly different or unconventional comparing to most of the ETL or most of streaming ETL pipelines. So our pipeline, the streaming pipeline itself, actually doesn't do a whole lot. It takes the data. Um, specifically for the B Z. It has a couple sections. It will take the data from MySQL and it will generate a before, an after, and some metadata about this particular data. And then on the KCPQ side, it actually doesn't do any kind of transformation at all. It will take the event from Kafka as is and send it to BigQuery. And because BigQuery handle nested data, KCPQ will store any sort of nested information as is. What we do that's uh that actually do more of the domain-specific transformation happens after the data have arrived in BigQuery. So we have these raw BigQuery tables with data that are not particularly useful. They're duplicated, they have all these before, after, and metadata fields. And then we would lay these views on top of the BigQuery raw table. So specifically, we have a two, we have two sets of views. We have a full view, which handles the actual deduplication of data and compaction of data. And then we have a clean view, which will do transformation of data, including masking of fields, or uh deleting certain uh or removing certain columns or converting an actual value into a Boolean field and things like that. So transformation that are more done on a business logic level. Um but in terms of the pipeline itself, we our goal is to keep it as generic as possible so that it can be applied easily as new uh data arriving to the system.
SPEAKER_00Got it. Got it. That makes sense. Um is MySQL the only database in your life there? Is everything living there?
SPEAKER_01So uh traditionally, MySQL has been, I would probably say, the only database. Um this was when I first joined Wii Pay back in 2016.
SPEAKER_02Uh we've definitely expanded our suite of databases. Uh some of them are in the cloud uh uh that are provided by uh Google Cloud. Um and then the one that we do have in-house, uh which still runs on Google Cloud, but um is not a Google Cloud product, is uh Cassandra.
SPEAKER_00Okay, Cassandra. And I uh I definitely have some history with Cassandra myself. So uh is that also can can you describe what Cassandra is doing and like what services it's powering?
SPEAKER_02Yeah, so uh Cassandra um for uh when we first introduced Cassandra, the main purpose of it is to uh handle um data that simply cannot fit into a single MySQL uh instance. At the time we had the option of either sharding the MySQL database or we can choose another database that essentially is a NoSQL database that will be able to scale horizontally.
SPEAKER_00Yeah, that'll help you do that sharding basically.
SPEAKER_02Exactly. So after some evaluation, it becomes the most optimal option for us. So we went down that path. And the kind of data that tend to be on the higher um volume are things like fraud signals, where for every single API that is being called, we want to generate signals that can be used for us to do fraud detection. Um another would be something like notification, which are um events that are being sent again for every single API call. So any of these kind of generic um high volume um events are the things that we would like to use in OSQL database.
SPEAKER_00Gotcha, gotcha. Um so high volume, low latency, still transactional. You know, this is not an analytic thing, but this is these are events happening in the business and you needing to do things right now. All sounds like pretty classical Cassandra sort of use case. And of course, um your Debezium MySQL uh connector pretending to be a replica, subscribing to uh replication events, all that goodness is not going to happen. And if I recall correctly, uh Debezium right now is very good at MySQL and Postgres, and there's SQL Server support, and I know whispers of other databases being added, but I don't think Cassandra is in there. So how are you doing your Cassandra change data capture?
SPEAKER_02Yeah, so there's a reason that Cassandra hasn't been in there yet. Um, and that reason is all of the database that are currently being supported by DZBD by Debesium tend to be more of a master slave architecture where there is one node that is the source of truths that's replicating to all of the other nodes. In Cassandra's case, um, it's a peer-to-peer database, which means that all of the nodes in Cassandra are equal, which also means that if I try to read from a single node of a Cassandra cluster, I'm not going to be able to extract all of the information. So instead, you have to read all of the data from all the nodes in order to actually get a complete picture. So that's one of the challenges of introducing Cassandra to Debezium because it doesn't quite follow the same conventional philosophy of the Debezium pipeline.
SPEAKER_00Yeah, yeah. Okay, can I just riff on that for a second? Just some Cassandra uh background for anybody who doesn't know. So it's a distributed database. So you can just imagine there's you know five, eight, ten, twenty nodes, whatever, however big your database is. And it's like like Joy said, all the nodes are peers, and clients don't need to be picky about where they connect. They can they can theoretically a client can connect to any node and issue a write. Um, and that write gets you know gets forwarded to the right node. But the you know, the equivalent of the bin log, there there still is one of those. I mean, every every database starts with a write-ahead log at its at its heart. Um but there's one of those on every node, uh which is what you just said, which kind of stinks all of a sudden. So yeah.
SPEAKER_02Yes.
SPEAKER_00All right, so what do you what do you do?
SPEAKER_02So yeah, our our approach um to handle this right now. So initially we built this Java JVM process that we are planning to um kind of drop one of them into every single one of the Cassandra node. And we we call this a CDC agent. So what this agent does is that it maintains the local uh lifecycle of the commit log as it as it processes each file. So it will detect when a new file has arrived or has been flushed to disk. It will process that commit log and converts them into individual events, similar to what Debesium does. And then once it's done, it's going to delete the log. And this is happening on every single node in the Cassandra cluster.
SPEAKER_00Got it. So you install this agent on every node. And does Cassandra know it's there? Like is there an API that you can get those commit log changes, or are you watching the directory where the Cassandra.yaml says commit logs restored?
SPEAKER_02Yeah, so you'd have to essentially watch the directory yourself. Okay. What they what Cassandra does provide is a commit log reader and a read handler. So it will do the deserialization of the binary data for you. But how you extract that information is up to you.
SPEAKER_00Nice. Okay, so that's a a dependency, a Cassandra library that the agent depends on to do the deserialization. But you're you're chilling there, looking at a directory, looking at files, uh reading them in when they happen, and doing your thing. Okay. Uh that's ingenious. I mean, that that seems like that cooperates quite well with the way Cassandra logs changes and on paper. On paper. Okay, so you're you're building this, I take it.
SPEAKER_02Yeah. Uh we're this is a currently working process, but the reason I said on paper is because um there are so many other nuance about the Cassandra uh way of uh Cassandra bin uh Cassandra commit log that we have to work around that was kind of all given to you in the MySQL world. So some of the examples are um in the MySQL world when there is a schema change, that schema change will also be recorded as a row into the MySQL bin log. So then the Debezium is able to update the schema for that particular record. In the Cassandra land, the data that is um being written into the Cassandra commit log doesn't include any kind of schema information. So the way to actually know when a schema change happens is you have to um pull the database.
SPEAKER_00You have to query a table, right?
SPEAKER_02You have to query the essentially you have to query the table for change. And this makes this is going to result in some level of latency from the point when the schema change has happened to the point where um the actual schema change happened has been detected.
SPEAKER_00Right. Do you also have some built-in latency uh with the commit log? Like commit log segments don't get flushed all at once, they get appended to, so you're getting flushes very frequently, right?
SPEAKER_02Yeah, so the I believe the default size of a commit log is 32 megabytes. So in in Cassandra's case, it's we have to essentially wait for the commit log to fill up, which then gets dropped into the CDC directory. And that's the point at which we are able to process that log. That said, uh in Cassandra 4.0 and above, there has been a change that's uh being introduced where they provide an API that allows you to actually pull the commit log before it's being dropped into the CDC table. So the data can actually be pulled in real time and the offset can be recorded as you pull the data. Nice. Um but this is not currently possible in this 3.x land, which is the Cassandra uh that we're working with.
SPEAKER_00Cassandra that you're running and may be running for the foreseeable future, but there's there's hope of a lower latency connector in 4.x.
SPEAKER_02Yeah. There's a workaround that I believe some companies do in order to make sure that the data is getting flush all the time, because the concern is that the the commit log sits there and no data is being written into any of the tables for some time. So some of the old changes might never be reflected. And what they would actually do is uh populating fake data to some arbitrary table just as a way of getting this data flushed on a predictable interval. Which is not ideal, but it's not ideal, that's a hack.
SPEAKER_00Uh yes. The assumption is under conditions of load, uh, you know, that's that's uh a big giant database with a bunch of writers and everything evening out and and um you know the central limit theorem doing what it does, and you know, you're getting this nice clockwork flushing of logs, it's great, but if things slow down, it just is not great. So that makes sense. So it sounds like you've learned a lot about uh a certain part of the Cassandra write path.
SPEAKER_01It's been really fun.
SPEAKER_00Yeah, no, this is fun. That's fun code digging digging into things. Um is this a thing? Um, I mean, your big Debezium users, is this a thing that will become a Debezium connector over time?
SPEAKER_02Yes, so actually, I'm actually currently working the process of porting our CDC agent uh and contributing contributing it back into the Debezium um ecosystem. Specifically, the goal is to first introduce it into um the DBZM incubator, which is actually a separate repository that some of the newer connectors are being introduced. And once they've been stabilized, they will then graduate from the incubator and be moved into the actual um Debezium repository in the long run.
SPEAKER_00Awesome. Awesome. Do you have any sense of uh what the timeline is on that?
SPEAKER_02Uh not a very exact timeline, but it will be sometime um, I would say within the next couple of weeks to months.
SPEAKER_00Oh wow, okay. So potentially soon. Uh you should look on the the Debesium site for that, right?
SPEAKER_01Yeah.
SPEAKER_00Yeah. That's exciting. That's totally cool.
SPEAKER_01Yeah, for sure.
SPEAKER_00So um imagine a newcomer to Kafka. What's the the one thing you want them to know based on your experience in the last few years?
SPEAKER_02I think what I would um advise for someone new coming to Kafka, I feel like there's so many uh documentation out there for Kafka that majority of the ecosystem has been pretty mature. And um there's basically it's it's very easy to configure um just to kind of start off your your basic kind of Kafka pipeline or something.
SPEAKER_00You can find stuff to get you to Hello World.
SPEAKER_02Yes. The area is as Kafka scale, especially on an enterprise level, I think the area that I find to be um pretty interesting is regarding to uh Kafka topic naming convention. It's kind of an area that I feel like is typically um overlooked or ignored because it's a semantic, right? You're just saying um this is the topic that the Kafka is uh Kafka data is going into. But from our personal experience is that we like to have the Kafka topic into kind of some uh form into a uh predefined format so that searching the data uh and extracting the data from Kafka becomes easier as the amount of data that's being added. And also on top of that is just the semantics of data. That's that's where the Kafka topics will help to be able to understand when you have some pre-selected uh format for your Kafka topic. And the area on top of this is schema evolution. Sometimes when you make certain um data changes, you might have to actually modify your, you might have to create a completely new Kafka topic. And then this might require versioning of your Kafka topic on top of just like your on top of just renaming and starting from scratch. So um this I think these are just the areas that I feel as in working in an enterprise um more so company where as the data scale, we have to be just more aware of in general and mindful about it.
SPEAKER_00That makes a lot of sense. If you could sum it up in one word, you'd say schema. Uh topic naming, topic naming is like the step before schema. Uh and you know, you're saying name them in such a way that people can understand what's in them.
SPEAKER_01Exactly.
SPEAKER_00That's fundamentally a schema question. So uh I think that's great advice. Because once when your pipeline is small, the the first version of it you're building is always from one thing to one thing. And maybe that one thing is really four tables or whatever, but it's it's one chunk of the domain. And it just goes off into someplace where you can do the analytics on it. Um, it's basically point-to-point. But if it's successful four years down the road, it's not that. It's a lot more complex. And so just trying to be able to keep track of oh, it's like labeling your pipes. You know, if you're in a a parking garage or something, you know, you'll see the pipe that says drain water and the pipe that says fresh water and all that stuff. You know, you want the pipes to have labels on them.
SPEAKER_02Exactly.
SPEAKER_00Yeah.
SPEAKER_02It's information essentially organizing your information in a way that is easily extractable.
SPEAKER_00Right. And hey, you know what? There's a thing uh that when I was introducing you, uh, there's this other amazing qualification you have that I completely failed to mention at the beginning. And so those of you who have have stayed with us till now, this is your reward. Uh, you're also a program committee member for the Kafka Summit.
SPEAKER_01Yes.
SPEAKER_00Uh and let me just take this opportunity publicly to say thank you for doing that. You're amazing, and uh I really appreciate it.
SPEAKER_01Oh, thank you. It's been a lot of fun as well, uh, to be able to partake in this process.
SPEAKER_00Yeah, yeah, and especially with three summits a year, right? That's uh yeah, that's that's a little bit more program committeeing than you bargained for, but uh uh that's how it goes. So uh and the next one, uh so just so everybody knows, uh I'm the program committee chair and and Joy's member of the program committee, and all the voting is done and has been done for a couple of months. So we're we're you know on Kafka Summit vacation for the year, or at least the rest of the program committee as I still I still have uh uh trials to endure there. But Kafka Summit San Francisco is coming up on I think September 30th and October 1st. This yeah, this September 2019. So um you should be there. Uh Joy helped pick uh the talks that made it in, so chances are they're gonna be really good. Um and um you'll you'll hear more on this program about how to register for Kafka Summit. Don't worry, commercials will come up uh for that. You have nothing to worry about. Um anyway, Joy, any other thoughts about streaming pipelines, WiiPay, Kafka, Kafka Summit?
SPEAKER_02Yeah, I think just in general, like streaming data is just an area that a lot of people I think are somewhat intimidated by, right? They feel that uh they're comfortable to do things in an offline manner, but when it comes to using uh real-time data as your source of truth, people get just a little bit nervous about. Or when it comes to uh streaming data in real time, people just there's some aspect of this real time. It's kind of like writing an email is a little bit easier than having a conversation face to face sometimes. Um, it's in a similar manner, I would kind of think of streaming is more difficult because everything's in real time and it becomes kind of unpredictable, um, and maybe uh just kind of scary. So I guess the final thought that I've had around this is just it doesn't have to be scary. Um, what we've done at WePay specifically is that we started off doing streaming data pipelines for uh our microservices one at a time. We've gained these operational knowledge and kind of insights over time, figuring out what's not like what are the things that we need to watch out for when we work with streaming data. And in the case we make a mistake, we can always go back to say, uh let's start this over and let's do a rebootstrap and try this again. And then over time, as we get more comfortable with streaming data, we feel that there are basically less operational like fires that we need to fight. And in general, it's just adds way more value to our organization on the whole. So I would strongly encourage people to not be scared of it.
SPEAKER_00If it seems scary, it's actually not as bad as you think. I think that sounds like your message.
SPEAKER_01Yes.
SPEAKER_00My guest today has been Joy Gao. Joy, thanks for being a part of Streaming Audio.
SPEAKER_01Thank you for having me again. It's been fun.
SPEAKER_00And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay for anything. If you're a new user and you go through the regular signup process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first three months after you sign up. So that's $50 per month of serverless Kafka for three months at no cost to you. So go to the sign-up link in the show notes, I don't want to read you the URL, and sign up now. I think the only thing I could really do more is write your code for you. And I think we can both agree that's too much to ask. So check it out, and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can reach out to us on Twitter at Confluent Inc. or reach out to me at TLbergland. That's T L B E R G L U N D. Or you can hit us up in Community Slack. There's a sign-up link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.
