Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Change Data Capture with Debezium ft. Gunnar Morling
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Friends don’t let friends do dual writes! Gunnar Morling (Software Engineer, Red Hat) joins us on the podcast to share a little bit about what Debezium is, how it works, and which databases it supports.
In addition to covering the various use cases and benefits from change data capture (CDC) in the context of microservices—touching on the outbox pattern in particular, Gunnar walks us through the advantages of log-based CDC as implemented through Debezium over polling-based approaches, why you’d want to avoid dual writes to multiple resources, and engaging with members from the community to work collaboratively on Debezium.
EPISODE LINKS
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
One of the distinguishing features of the world is that there's a lot of data that isn't in Kafka yet. If that data is in a relational database, there are some special challenges associated with getting it from there into Kafka. We're going to talk about how an open source project called Debezium solves those problems. On today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the Cloud. Hello and welcome back to another episode of Streaming Audio. I'm your host, Tim Berglund, and it's my pleasure to have on the show today uh Gunnar Morling. Gunnar, hello.
SPEAKER_00Hi, Tim. Thank you so much for me for having me.
SPEAKER_02Oh, you bet. You bet. It's great to finally have you on the show. You are, correct me if I'm wrong, a software engineer at Red Hat. Is that right?
SPEAKER_00Exactly right. That's what I am.
SPEAKER_02Yes. And of course you're known, I say of course, maybe everybody doesn't know, but I know you for a particular project, and that is Debezium. Right. So I want to ask you a few things, actually a lot of things about Debezium. Because you're you're the guy to ask. But um first, what is it? And then what is your involvement with it?
SPEAKER_00Right. So Debezium really is a platform for change data capturing. So essentially what it means is it taps into your database, into the um change log on to the transaction log of a database and gets all the changes out of it. So whenever something happens there, a new record gets created or something gets updated, something gets deleted, Debesium will know about it and then it will get a change and it will produce a change event which describes this and um send it to Apache Kafka, right? And then you can connect all sorts of consumers to those topics and um yeah, react to those change bits. But really, Debesium is enabling change data capture for a wide variety of databases.
SPEAKER_02Got it. So if I uh assuming that we we all know what Kafka Connect is, you know, the data integration framework for Apache Kafka, is it is it precise to say Debesium is a connector?
SPEAKER_00Right. So it is a essentially it's a set or a family of Kafka connect um connectors. Um and that's what people or that's the way most of the times it is used. Um but you also even could use it as a library just within your application. So let's say you're after some sorts of reactive streaming use case and you would like to react to those data changes right within your Java application. That's also something you could do.
SPEAKER_02Ah, okay, okay. Um so and what's your relationship to the project? How do you uh how do you come to be involved here?
SPEAKER_00All right, yeah, so I'm the lead of the project, um, but really I don't try to not put too much emphasis to this. So in my opinion, it's like a I mean, definitely it's a community and team effort. And I mean there's multiple people um involved in this, and I just you know try to steer, and maybe if something needs to be decided, I would do it. But yeah, that that's my role.
SPEAKER_02Good deal. Yeah, I appreciate your humility on that point. It is, although, let me let me not let you have all that humility because product project leadership um is such a valuable thing for for open source things. It's you know, and and I want to talk later, I kind of want to pick your brain later about community and what you've done to uh you know make it a positive place to contribute and get and help people, you know, think it's a good idea to get involved and all that. But um and and it it relies, you know, those contributions are the lifeblood of an open source project, but somebody needs to be driving and and right uh just having the vision for where the thing should go and calling people to follow them. And so that's that's important work, and I'm glad that you do it.
SPEAKER_00Right, yeah, thank you.
SPEAKER_02Let's um so I mean I I kind of want to talk about how you use it, what you what you do with it, but um, I'm super curious to dig into how it works. Right. And there are uh I I say this often on the show. Uh sometimes I'm asking questions because I want to get you to say something to help everybody in the audience understand uh a thing. Sometimes I ask questions because I don't know, and I'm curious. Right. And you know, nobody really needs to know the difference, right? But every once in a while I'll say, I'm asking, dang it, because I don't know and I want to know. But yeah, talk to me about how it works. Like let's just kind of dig into it. I uh if we could just again make the assumption that basically everybody knows what a connector is, maybe we can talk about that in a little bit of detail, but but walk us through Debesium.
SPEAKER_00Right. So um yeah, the the the uh let me say change to the capture by itself, it's not a particular new idea, right? So this concept has been around for some while, and databases um offered, or some databases offer APIs for for getting changes out of them. But really, that's that's already the problem there. I said APIs, uh deploy, right? So there's no one single way or one canonical format which you would just use, and then you would have CTC for all databases which are out there, right? Instead, it's a different and specific effort to make this happen for each particular database. So let's talk about uh MySQL, for instance. That's one of the databases we support, right? So there uh the log, it's called the bin log, and this is structured in a specific way, and there's the client library which essentially exposes the bin log to a remote client. So that's the API informat we need to deal with in case of MySeqr. For Postgres, we have the write hair log. So that's um, I mean it's this it's the same thing in terms of concept, right? But it's just different in terms of how you access it. So instead, for instance, for Postgres, you actually need to have one small component, which is called a logical decoding plugin, which needs to run within the database itself. And then this logical decoding plugin will stream the changes to our connector, then which runs in Kafka Connect. So it's a little bit different, and those APIs and formats are different. Um, but then this is actually the good part or the good thing, which is Debisium is doing. So we are doing all those explorations, right? So we do this heavy lifting, we understand or we try to find out how to get how to get changes out of MySQL, out of Postgres, out of MongoDB, and so on. And then the events which we produce and which we send to Volskafka, they are as much as possible in a generic and more abstract structure, right? So for a consumer then of those events, um, they don't have to care too much. Does this particular event come from MySQL or Postgres or any other database? So that's that's the basic idea.
SPEAKER_02You just end up with Avro in the topic and and kind of go from there.
SPEAKER_00Oh, right. Yeah, I mean so exactly. So that's again, that's um like an offagonal concern, right? What's the serialization format? So I would say most people use Avro, but uh I know some people from the Debiesium community, they are very uh strong believers in the Google protocol buffers. So they've implemented their own serializers for protobuf, and that's what they are using. So you have some flexibility there.
SPEAKER_02That's that's strange. That that topic has never come up of protobuf versus Avro. You're the first person ever to mention that.
SPEAKER_00Um Okay, I see. I mean I I saw it coming up a few times, not too often. Maybe maybe maybe once or twice.
SPEAKER_02Yeah, yeah. Yeah. Um, that's a that's a common uh you know schema registry question, is uh is protobuf support, and that's one of those debated things that is likely to happen someday, but but who knows. Right. Okay, so let me let me um back up a little bit. The uh it's interesting that uh the two you you mentioned two databases, Mongo, uh Postgres, and MySQL, but MySQL and Postgres already expose their commit logs or or write-ahead logs in entirely different ways. Uh MySQL actually offers uh a client API that you can basically subscribe to remote updates and it it it will handle doing that over the network and and you just can't connect to the yeah. So that's that's cool. Postgres, that agent that you install, I forget what you called it, that that lives in the code.
SPEAKER_00Logical decoding plugin, yeah.
SPEAKER_02Logical decoding plugin. Is that a part of Debesium or is that like a standard thing that you just need to set up?
SPEAKER_00Oh yeah, so there it's getting complicated. So the thing is um there's not one standard logical decoding plugin, let's say. So and so there's multiple ones. And we have one um which we maintain ourselves, which is um called decoder buffs, and the name gives it away. So this actually uses protobuf um as the transport format between this plugin and our connector. Um, and then there's uh another one which we support, which is called well to JSON. So this is um based on JSON. And the reason that we have support for multiple ones is that um, well, in order to install those plugins in the database, um obviously you need to have some kind of elevated permissions, right? And this means typically you won't be able to do this in, for instance, in a cloud database, right? So if you are on Amazon RDS, let's say you just cannot install your custom plugins, so we need to support what's given there. And while to JSON happens to be available there, so that's um what we support. So we have those two currently, and we are working on another one which is called PG Output. And this one actually does exist in every um Postgres database since version 10, I think. Um so because it's used for replication internal purposes. So that's something which we explore right now, and um one member from the team is working on this, and if we had this as another option, this would be pretty cool in terms of portability because as I said, it's it's just available everywhere.
SPEAKER_02Right. Excellent. Okay, that makes a lot of sense. Yeah. Um another thing. Oh, go ahead.
SPEAKER_00No, no, no, that's that's fine.
SPEAKER_02Okay, uh another thing I wanted to back up on. I I feel like uh just in case you know you're you're new to the podcast, we've done whole episodes on Kafka Connect before. Uh I strongly recommend anybody who's new and doesn't know what we're talking about, uh, go back through the archives and and find an episode on Connect. We've we've gone through it in some detail. But briefly, uh a connector, we've talked about a connector, that's uh think of that as a jar file. It's a it's a little component that implements this thing called the Kafka Connect API and plugs into this runtime framework that then you know runs the connector, which makes the connection to this external database, uh does whatever transformation needs to be done on serialization formats, and then produces stuff into a Kafka topic. Um and Debesium is fundamentally a source connector, right? It's it's for getting data in.
SPEAKER_00Right. So it's yeah, it's a set, so it's different source connectors for different databases, and then we have a couple of those SMT single message transformations. Um and yeah, we try to add up some more things which you know would add some more value, which go a little bit beyond the level of the basic connectors, let's say.
SPEAKER_02Gotcha. So um, next question. You used the phrase uh change data capture, and in the event that there's somebody who doesn't know what that means, let me uh ask the question. I want to ask it this way, sort of uh by way of being devil's advocate. There's a standard Kafka connect connector, well, standard in terms of the connectors that Confluent makes. I guess standard standard from my perspective, but there's this Confluent connector called the JDBC connector.
SPEAKER_00Right.
SPEAKER_02And using that, I can connect to any database, and if I you know if I want to see what changes in the table, um the the JDBC connector will do that same thing and produce those records into a topic. Um so what is uh what how does how does Debezium differ from that? And in the process, if I could just ask you to define change data capture.
SPEAKER_00Right. So um yeah, that's that's a great question which comes up um um yeah um often definitely. Um so if you have the chance to add a picture to the podcast somehow, I don't know. So Robin, your colleague Robin, he just did a very nice um um meme on this. Um you you know you this this picture of the couple where this guy is with the new um girlfriend and the other one he looks to you know that one attractive boyfriend meme. Yeah, yeah, exactly. That's the one, right? And and Robin did this, and so like the the new um person he looks to, that's that's log-based change data capture. And then the the not so nice um attractive um girlfriend is interesting, that's um query-based um um change data capture. So that's that's um something which pretty much describes this. So there's a couple of advantages to the log-based approach, which we do. And the first of one, so if you are doing the polling stuff, which the um JBC source is doing, right? Um essentially you have a conflict of interest there. So you would like to run this query as often as you can, right? Because you don't want to miss any updates which happen between two polling attempts. Um but now if you do this uh too often, maybe at some point your DBA comes and says, please don't run this query every 10 seconds or whatever you are doing, because it just creates too much load on a database. So there's this conflict between creating load, which this query will do, and having up-to-date data. And then the problem is no matter how often you run this query, you're just not sure that you might not miss any intermediary updates, right? So if there are two updates which um refer or which apply to the same record, which happened in the closed time, you just might miss the first two out of those three. So that's the that's the first thing. Then um you cannot capture deletes with a lock, uh with a polling-based approach. I mean, if you just poll and you run this query, if something has been deleted um since you asked the last time, well, you just won't see it. So you won't be able to produce a delete event.
SPEAKER_02Then you will delete deletes simply will not happen by polling. Unless you keep a separate copy of everything in the database, right? Then you can reason about delete. But yes, you probably don't want to do that.
SPEAKER_00Exactly. So if you have some sort of soft delete concept, right, then you then it kind of would work, but if it gets physically deleted, well then it then it's just just gone, right? The the next thing is you need to have some marker columns in your model which allow you to identify those records which um you haven't extracted yet, right? So there must be like an update timestamp column, something. So there's an impact on your actual model. Whereas with the log-based approach, this is just not the case. So we can essentially capture any table, and you're not uh required to add those kind of columns to your tables. Um yeah, so there's that, and there's a couple of other reasons, but really that that pretty much sums it up.
SPEAKER_02Cool. No, that's uh that's outstanding. Yeah, so the reasons are uh deletes is uh deletes is always the biggie that comes up, right? Right. Um because you you can get inserts and updates to happen um as long as your model supports it. Like you need an ID column, you need a internet column. So you can get there if you have some control over the model, but even so, as you say, um a number of of updates, any updates that happen between polling periods, you'll miss. And so some people, if they only need inserts and updates and they don't care about that kind of thing, right? Polling can work. Right, it's it's always a second class approach.
SPEAKER_00Exactly. And that that's that's where we come back to the distracted boyfriend meme, right? So I really I really like this idea.
SPEAKER_02It's I mean, that's the thing about that meme is that it we we are all uh that guy, uh no matter whether you're a man or not. That there's always you know you have this good thing and but you're not really satisfied with it, and you're looking to some better thing uh that that you know you don't quite so it's it's everything's so universal, the appeal of that thing.
SPEAKER_00Um so by the way, there's one more thing which we can do with the log-based approaches, and this is like extracting some metadata, like um the transaction ID, for instance, and and and maybe some position in the log file, and this also could be or or for instance the query which um uh was causing a specific change, we might be able to get this in some cases. So this all this kind of metadata usually you won't be able to get with a query-based approach. And depending on use case, this might be something for interesting topics.
SPEAKER_02And some of that metadata can be available in the in the transaction log?
SPEAKER_00Exactly right. So this depends uh on the database. Does it have this particular you know capability? And then also often it's a matter of the configuration. So for instance, in MySQL, you need to enable um the fact that queries are um, you know, the causing queries are present there. And if you have done this, then we could just emit this. Um I think we use a um uh yeah, just like it's part of the message uh metadata where where you have the query, and this might be interesting.
SPEAKER_02Right. No, absolutely might be interesting. So cool. Um okay, so that's that's the the basics of Debesium. Everybody, you should think of it as a Kafka connect connector that knows how to talk to the database commit log, whatever that whatever that database calls its own commit log, it's able to extract those mutations directly from the log and produce them as messages uh to Kafka topics. So it is the no nonsense um you know best way to get changes to a table into your database. Now, you mentioned in terms of the databases supported. Um I mean I I knew off the top of my head that MySQL, Postgres, and Mongo were supported. Is there is there anything else?
SPEAKER_00Right. So there is um a connector for SQL server um which we have. Excellent. And then, right, so this this is uh was added not too long ago, but we see some uptick there, so people are trying it out and and you know reporting feature requests and so on. So this is gaining traction. And now, of course, the elephant in the room is Oracle, right? So everybody would like to have some. Well, I mean, there's there's already a solution there, so I don't know why you'd even well um yeah, so there's some point um to be made about licensing and cost for this, right?
SPEAKER_02So potentially and that's that's Oracle Golden Gate. It's it's it's totally okay to name the product. I mean, it's not like we can't uh it's not it's not Voldenworth, we can we can say its name. Um but that's Golden Gate is Oracle's CDC solution that integrates with Kafka, right?
SPEAKER_00Yes, exactly. Exactly. So and actually, so we have a connector which is work in progress, and then you can use an API which um is called Xtreme, and which as far as I understand also is used by Golden Gate in some way. So you can use this, and our current connector is um using this. The caveat is you still need to have a Golden Gate license in order to do this.
SPEAKER_02To use that API.
SPEAKER_00Right, exactly. It's still interesting for some people, and I I know some people are are using this. Um, but we are actively exploring some um truly free and open alternatives. There's actually uh, and this comes a bit uh back to the community. There is people from the community exploring this uh or who are working on this, and yeah, I'm hopefully at some point we might have something which would not have this licensing um requirement. So that's uh that's um those are the five we have. So MySQL, Postgres, Mongo, Secret Server, they are stable, they are there, Oracle is in the workings, and then um there's another one um which is happening right now, and actually I think I'm not sure whether I can tell it or not, but one of your next guests, she's working on this, and she might share some news about this coming up very soon.
SPEAKER_02I like it. Let's just leave it at that. That's tantal. Exactly.
SPEAKER_00So uh right. I wanted to do some teasing around it.
SPEAKER_02There you go. Good, good, good. No, that's that's exciting. Um yeah, so pretty pretty broad support. I mean, that's most databases that most of us are using for the kinds of systems that would be interesting to do this sort of integration. Uh you're pretty much covered there. So that's uh Right.
SPEAKER_00So yeah, DB2 comes up um uh every now and then. Um so now there's this um yeah, uh potential acquisition happening, right? So um we will see how this goes. Um that's definitely also something which is asked for.
SPEAKER_02Subject to regulatory approval. Uh yeah, I don't think anybody's really thinking of it as potential. So uh and it just uh for posterity's sake, this is being recorded in the middle of June 2019. So uh that acquisition was announced months ago. Has not closed yet. But exactly.
SPEAKER_00That's that's why I'm saying it like that. It's not fun.
SPEAKER_02That's just uh that's Gunnar's uh hashtag safe harbor there.
SPEAKER_00Exactly.
SPEAKER_02Um and what are those those? Oh uh uh Golden Gate. I just wanted to say X Stream, that's the name of the API, right? That's the most amazing API name I've heard in a while, right? Because it's like Xtreme, yeah. Xtream and it's all edgy and everything, which is exactly you know consistent with the branding we normally think of when it comes to Oracle. Uh yeah. Anyway, great name there. That's fantastic. I just love it.
SPEAKER_00Yeah. Let's say the name of the API is great, and uh let's leave it at that.
SPEAKER_02There you go. There you go. Strictly positive here. Uh I appreciate that. Um, but this it's uh uh just to kind of reframe um Debesium, really, now that we know what it is and what it integrates with and a little bit about how it works, if you're new to it, uh if you're listening to this and it's a brand new thing to you, uh it re its function in you know the Kafka community is really as this beloved database integration layer. Right. I mentioned that there is a JDBC connector, and that thing gets used. It's not like that's um you know this this thing lying in the gutter over there. It totally gets used only only for very simple kinds of integration. For uh, you know, when when you've got a table where rights are landed. In an interesting way, and you need them to be in Kafka. Debezium is a thing people think about. And I just have never heard anybody say a bad thing about it. It's a thing with a sterling reputation.
SPEAKER_00Oh wow. Yeah, cool. Thank you. That's really cool. Yeah, I mean, so it's a small team working on this, right? And I I work oh, I have been working on it for for about pretty much exactly two years right now. And since then, I think it yeah um gained quite a lot um in terms of adoption and and traction. And I'm really excited whenever somebody blogs about it or um you know they are sharing at it at the conference that they are using it. So that's always really cool to see if people speak up or write about their adoption of it. Exactly.
SPEAKER_02Yeah, and that's I would expect I would expect you would get those kind of spontaneous conversations. Oh hey, Gunnar, just wanted to thank you because this thing's great. You know, it's it's that sort of thing.
SPEAKER_01Oh, yeah, yeah.
SPEAKER_02Now that I've said that, of course, uh, once this episode ships, I'm gonna get all the hate tweets for people who used it and had a bad experience. That's okay. You can tweet me, don't tweet Gunnar, you can tweet at me and vent and you'll feel better and we can talk it out.
SPEAKER_00So yeah, and I mean definitely, so we have we um are always very eager to have also this kind of feedback. I should I should say that. So if some somebody is trying it and it's not working, I definitely are very curious. Um, you know, why why didn't it work for them? What was the problem? Is there something which we which we can do? So also this kind of feedback this is um sometimes even more valuable, actually.
SPEAKER_02Yeah, no, I agree very much. Uh and uh to be clear, this is open source. I mean it's uh it's kind of sponsored by Red Hat. So GP is it GPL?
SPEAKER_00No, it's uh it's Apache Wii Two.
SPEAKER_02So it's uh sort of compatible with the Kafka universe. The um compatible with the Apache Kafka cinematic universe. Um cool. So let's uh let's talk about some what we like to call use cases. Um I mean obviously what you do with it is you take a table and you get it into uh a topic, but a little less mechanically than that. What are the kinds of problems that one solves?
SPEAKER_00Right, so exactly. I mean you don't want to just have you have the data sit in a topic, right? And then there it is, but you would like to do something with the with the change events. Um and so the way I see it on a very general sense is it's really like liberation for your data. So you have this data sitting there in your database, and it really would like to get out and you know be there for your disposal and change data capture is doing this. So, what are people using it for? So many people are using it just for replication, so across um different databases um and also across database or vendor boundaries, right? So let's say you're using some database in production and maybe it's like commercially licensed and expensive, and you would like to have uh just the same data in a free uh open source database uh somewhere else for you know analysis purposes and so on. So this is something you could use this for. I know people use it for to replicate data from their operational database into a data warehouse. That's definitely a big use case. And this also um um actually comes back a bit to this question about this polling base. So, for instance, one of the um users I know, they have their data in MySQL and they stream the change events to uh Google BigQuery as a data um warehouse, right, in the Google Cloud. And they have like a sub two-second end-to-end delay. So this means with uh with in less than two seconds, some when something has happened in a database, they already see the data in their data warehouse. So this is really cool, I would say. Um so this all is replication, right? And then um also a bit related to this is like full-text search. So very often people would like to use something like Elasticsearch or Solar to enable full text search because they are maybe not um happy with the search capabilities which are provided by their database. So they would like to have to change the data in Elasticsearch, and of course you somehow need to keep Elasticsearch or your search index in sync with the primary database, right? So that's something people use it for. Um you could use it for cache updates or cache invalidation. So maybe you have some, you know, some use case in your application which requires very short um latency and you cannot afford to go to the database. So you have some cache in there. And now, of course, if something changes in the database, either the data and the cache must be updated um accordingly or at least it needs to be invalidated, right? So that whenever the next access for this item comes, you would actually go to the database. So that's cache updates, cache invalidation. Um you can do it for exchanging data between microservices, so that's definitely something we could um you know dive into in in more depth. So um, I mean in microservices you would not share a database, right? Each of the services they should have their own database. Um, but still um very often those services need to have data from other services, right? And change data capture would allow you to propagate data changes amongst microservices.
SPEAKER_02Um go ahead. But I want to I actually want to dig into that one a little bit more. Absolutely, yeah. We can do that. Let's back up first, because everything you had said up till microservices, yeah, there was a pattern to all those things. All right. Um and I could again, uh playing devil's advocate, which I I actually don't really enjoy doing, but uh I'll do it just for for purposes of discussing architecture here. I could say, well, okay, there's an application that writes to the database, and you want to write to uh the the data store underlying BigQuery, yeah, uh just do that, right? Just just you're uh you're a programmer, do both rights. Or you know, you want to update your search index right to the database and write to the search index. You know, if I suppose I haven't spun up Connect yet. I don't have a Kinect cluster. Yeah, maybe I've got a Kafka cluster lying around, but I feel like ah, it's this extra infrastructure. Yeah, I don't want to learn it. Why not just do that in the application?
SPEAKER_00Right. So that's a that's a great question. And there's a problem there. And I mean, this might work, or you might perceive it works as long as um everything is nice and shiny, right? So you do those two rights and actually it it seems to work. But then um you need to think about failure scenarios, right? So what happens if you cannot, for some reason, you cannot access this elastic search cluster? Maybe you have like a network split or it's it's just not um available. Um so what do you do in this kind of situation? So you would either have to buffer this request so you can send it later on, or you would um somehow have to, you know, have some sort of batch which in the night comes and re-indexes all your data, um, or maybe you just reject this primary request you got, which let you change your data in the first place. So there's um you know all those complexities. And um, this is what I call would call um you have this tie-in in terms of availability. So as soon as one of those resources isn't available, um, then you have a problem, right? So that's something you need to somehow solve. And then there's another problem which is even more subtle, and this is consistency. So this um so let's say you receive two requests to your application. Maybe it's about purchase orders, right? And they those two requests they would like to update the same purchase order. And they apply to the same order. And so let's say we name those requests A and B. And now in the database you apply A before B. But now you also send those two corresponding requests to Elasticsearch. And now, due to the way this all works, um there's HTTP involved, and those requests might be routed differently. It might happen you apply B before A in the search index. And now this puts you in a very bad position because your data in the search index doesn't reflect the data in the database, right? And this is something you need first to be aware of, and then you somehow again need to resolve it. So this uh already is a is a big problem. And that's why uh I always try to make the point and say friends don't let friends do dual rights. So just don't do those dual rights, which means don't try to update resources which are not part of one shared transactional context. And now change data capture um allows you to nicely address this problem because if you cannot update multiple resources in one go, well, you just update one resource, and this would be your database, the database of your application. And then you use CTC to stream the changes from this database to Elasticsearch, to your cache, to the um data warehouse. And now this all will be consistent, but it will actually be eventually consistent, but you will not um have this situation where you um you know you're you're not tied to the availability of Elasticsearch because whenever it will be there again, um this pipeline will just catch up and you will end up in a consistent state.
SPEAKER_02Is there yet a t-shirt um with the Debezium logo that says friends don't let friends do the right?
SPEAKER_00That's actually a great idea. Um I I will see an order one uh right after we finish this uh recording.
SPEAKER_02At any point I can uh get access to such a t-shirt. I will wear it. I will tweet uh selfies with it.
SPEAKER_00That's great. That's great.
SPEAKER_02That is uh yeah, that's a that's a great uh line. And I felt uh, you know, I said I was playing devil's advocate. I said I don't like playing devil's advocate, and I felt dirty asking the question, but you answered it very well. That's uh that really is the deal. Because when I'm you when I'm sort of at a blank whiteboard and talking about architectures with people, um I usually take a fairly Kafka-centric approach. So I'm saying, hey, look, uh, you know, produce something to a topic and then get it where it needs to go. Well, right, you know, that's great, but there's all kinds of situations where you just can't do that, where you know there is an existing application, there's an existing database, or for some other reason, the you know, prudent engineering judgment is that this data goes to a database first.
SPEAKER_00Right.
SPEAKER_02And that has to be the one right.
SPEAKER_00Exactly.
SPEAKER_02And that is whatever gets to do too.
SPEAKER_00Right. And I just to give one reason why you might want to do this is um you m very often you would like to have what's called like um instant read your own write semantics, right? So if you have Kafka, or if you read something, it's by definition asynchronous, right? So what it means is you do this update to the to the database. Maybe you place your new purchase order, and then the next second you go to this dialogue of this webshop where you'll see all your all your purchase orders, and of course you expect that this order which you just placed that shows up there, right? And um, this can be really tricky to do if there's asynchronicity involved, and you don't have this if you just write to the database because once the transaction has been committed, if you select for the data at the very next second, you will be guaranteed to see it.
SPEAKER_02You do have read-your-own rights consistency, exactly. So when you need that property and the scale of your system allows you to keep everything in a single relational database, then boom, you know, you're you're there, and that's that's a good place for that data to land. And you use Debesium and Kafka to get it to other places. Now the system is still eventually consistent, like search is, but uh there comes a point where you don't have that choice to make.
SPEAKER_00Right. And I also think it's fine, right? Um something like um downstream consumers, separate systems which just would like to have the data for you know replication purposes or whatever. Usually there it's just fine if you have those um eventually consistency, um, eventually consistent semantics there, whereas it might not be so desirable if you just look at the application itself.
SPEAKER_02Right, right. Or like you've got you've got maybe uh streaming analytics, you real-time analytics you want to do in case equal on this data, then it gets into a topic and you it's there, it's in Kafka. You know, the whole platform gets to operate on it and and it's amazing. Um, but if the one right is to a database, the one right has to be to that database, and you don't have the option of or let me let me put it this way. I mean, your your slogan is is pithier and it was it, it's the one that goes in the t-shirt. Um, but the way I, you know, my devil's advocate architecture could have been solved with two-phase commit, right? That's a that's a thing that we could have used distributed transactions to solve. Anytime somebody puts an architecture suggestion on a whiteboard uh that has a problem that could be solved with distributed transactions, that we could consider that a smell that it's probably the wrong architecture, right?
SPEAKER_00So yeah, so the reason why I'm not excited about it is that it again brings it back to this availability concern. So as soon as one of the resources you would like to access in this distributed transaction isn't there, well then you just cannot do this uh transaction, right? And this is uh obviously a problem in terms of your own availability.
SPEAKER_02Yeah, yeah, yeah. No, it's a it's a complete non-starter as a solution. I'm I my point is the architect the the devil's advocate architecture I I put up there is a candidate for two-phase commit or for distributed transactions, which means it's probably the bad the wrong idea, right? Just just rule it out and uh and get back to the friends don't let friends do dual rights. Um and microservices, yeah. I I uh I stopped you there. You you can't I guess you summarized it, and what you were saying was, of course, we shouldn't have microservices share a database for all kinds of reasons that are that are pretty well known at this point. Um but if you have each service well, and let me just I actually frame it this way it's fairly easy um when we're you know you're you're decomposing a monolith into microservices, it's fairly easy to find the functional uh uh boundaries inside that monolith, right? Like, well, here's order validation and here's uh shipment and here's the the uh customer loyalty program uh analysis service. And you know, that that's all obvious, but um data doesn't play that game. Uh the functional decomposition is simple, but those pieces want to share each other's data.
SPEAKER_00Exactly.
SPEAKER_02And once we once we impose the discipline on ourselves of not sharing a single database, which is a discipline we have to impose on ourselves, we have to figure out how to get the data back and forth.
SPEAKER_00Right. And I mean, yeah, so change capture is something you can use there, right? So coming back to this example, the shipment service, it somehow will need to have the information about the ordered items, right? So they can create the shipments. So you somehow need to get this data there. So there uh then there's different ways you could play this. So there's the obvious, or let's say one solution, and you would just let's say capture changes to this purchase order table, right, in the order systems database. You could do that, and I I know people are are doing that, but then there's um some um yeah issue you need to be aware of, and this it kind of exposes the internal structure of this database, right? So let's say in the the order team they are working on this on this order application, and now they would like to somehow restructure how they perceive a purchase order should look like, and they they alter the structure of this table, and now by just capturing changes to this table, this also gets apparent to downstream consumers, right? And they need to be adjusted and so on. So there might be a concern that you would not like to do this. And actually, there's a very interesting pattern um which I recommend in this case to uh prevent this um kind of issue, and this is called the outbox pattern. And so um have you heard about it or are you could I not by name, no. Alright. So actually, also again, it's not a new thing, but I think uh the name is pretty pretty catchy and it uh it gains some traction there. And the idea really is that of an outbox. So within your application, um you update your actual business tables. So you update this um purchase order table, you insert a new record there, or you update a record there. And now at the same time, you also produce an insert into a outbox table, which exists right in um the same database. And this outbox table, it really just is uh essentially a container for events. So the application which writes there, it produces already an event, it could be like a JSON event or it could be even an ever or whatever, which describes this change it would like to export. And now the CDC pipeline and Debesium, it would just capture changes to this Outbox table. And then people would subscribe to the topic which contains the events from there. And the idea there is that this event, which has been written into this outbox table, that did that this is a very consciously designed and well-defined public contract of this uh order application in this case, right? So now when they changed something in their internal model and their internal purchase order table, um they could do that. And then at the same time, they might alter this contract of the events which they send through the outbox table, but they would do it in a backwards compatible way, and they would be very careful, you know, about not breaking any consumers of this event. And the beauty of this model is that again, this all happens within a single transaction, right? So the outbox table and the actual business tables, they exist in the same database, so again, you don't have this issue with with dual rights.
SPEAKER_02Uh you uh I it it it sounded like you had an issue with dual rights. So walk me through why you don't, because I have to I have to create you know, I I insert or update a purchase order. That's my right. Right.
SPEAKER_01Yeah.
SPEAKER_02And uh then I have to create I have to make sure this outbox message gets created. But that I do through I don't do that just naively through Debesium because then I'm exposing my schema changes directly.
SPEAKER_00Right. So you this would be a little bit of logic which uh exists in your application. So this is um something where your application needs you know to cooperate. And um the key is that the application is doing this insert into the outbox table in the very same transaction where it updates the business tables. And now this means if something goes wrong, the transaction rolls back. Well, then there also would be no event in the outbox table because the transaction has been uh you know uh undone.
SPEAKER_02So it's it's uh it's two uh rights in the sense of two inserts, but it's an atomic transaction, so we're fine.
SPEAKER_00Exactly. So we came across this pattern in the last year, and I really felt okay, this is really uh you know something which we should expand and we should have support there. So we built a demo um which showed how you could do this um with a little bit of glue code. For instance, one of the issues we have to solve is um by default, Debezium will create one topic per table it captures, right? Or it will send the events from one table to one topic in Kafka. And now if you have this Outbox approach, there you might have events of different kinds, of different types, right? You might have something like an order event or a customer event or whatever. And very often you would like to have those events in different topics because different consumers are interested in those events, right? So what we did there is we built an SMT, a single message transformation, which routes the events from this outbox table to different topics based on some configurable column within this outbox table. So the source application, it can say, okay, this is a customer event or this is an order event, and then this SMT will make sure that those events go into the right topic. And this actually is something which someone from the community built. So we just did this demo, then we blogged about it, and people were really interested because they figured, okay, this solves the problem which we have. So maybe we are doing this tool right and we we shouldn't do it. And so someone from the community stepped up and said, Okay, I would like to, you know, make the concepts from this blog post and from this demo into an ready-to-use SMT, which now is part of Tibesium.
SPEAKER_02That makes a lot of sense. That sounds like a nice middle ground between um, you know, on the one hand, you've got conventional microservices that are all using their own little isolated Postgres and talking to each other over GRPC or something like that, you know, which is which has uh which which can be made to work, but it has its issues. On the other extreme, you've got you know the the Kafka only microservices that are using Kafka streams and materializing K-tables, and maybe there's a database, but probably not. Um, and that works well, but is uh uh is very different, right? That's great for maybe a greenfield thing or an aggressive refactoring. Yeah in the middle, you've got this hey, look, I'm a microservices developer, I'm using a database, I'm like a normal person, but I still want the advantages of integrating my services through messaging. Exactly. Debesium is that is that kind of middle ground in that argument.
SPEAKER_00Exactly. Nice to have that. Yeah, definitely. And I'm really excited about it. So since we have support for this, people come and ask just right now. Someone is working on you know some expansion to this feature. So this is always the best which can happen. So you add something like a new feature, and you are interested. Okay, will people adopt it? Is it useful? And then actually people come and you know improve it. Um, so that's that's really always great to see.
SPEAKER_02Yeah. And that, by the way, is a great segue to the other thing I wanted to talk to you about, which was community in general. I mean, you're you're running a I would say a fairly high-profile, well-adopted open source project. Thank you. What has that been like? And and how do you maybe just kind of give us your philosophy of community in the sense of developer community? Uh yeah.
SPEAKER_00Yeah, um, so I mean, yeah, there's many things to be said about so the reason the thing I see it, yeah, really trying to be what we try is we try to be a community and you know have an equal playing field. Someone comes up with an idea, um, they are very much encouraged to you know to explore this or to to work on this. So for instance, um someone comes with a bug report, the very first thing well, we will try to acknowledge whether the bug exists or not, right? But then if you think, okay, this is a bug, the first thing we will usually ask is, hey, wouldn't you be interested in fixing this yourself and send a pull request to resolve this problem for you and for everybody else? And this really works well. So I don't know, maybe in like in a third of the cases or so, people will actually follow up and send a pull request. And maybe it's not perfect. Yet and you need to you know improve it and give some guidance to to polish it, but this definitely helps a lot. Um, and then the other thing really is we try to encourage people to you know just essentially own um ideas or own topics. So, for instance, with the outbox stuff, um so Renato that's that's the um uh person from the community, so he was interested, and so we said, Hey, wouldn't you be you know up to exploring this and take the lead on on see how we could have some sort of um outbox support in the museum out of the box? And this really I think makes people, you know, they are motivated and they would like to do this actually if they kind of own um something and then it's like their you know their baby which they created. So I think that's definitely something which is important.
SPEAKER_02Right. How have you um have you done anything to intentionally solicit external contributions? I mean, you're talking about people who come to you and how you treat them, but how how do you reach out or do you reach out?
SPEAKER_00Oh yeah. Right. So yeah, we do a couple of things. Um so we um we always try to have um issues which are tagged with a certain label, which I uh is like, I don't know, suitable for starters or something like that. Because every now every now and then it happens, someone comes to the chat or to the mailing list and they say, I'm interested in Demuseum. I don't have any bug I have to report right now, but I just like would like to help out. So it helps if you have you know this the these issues which are like a good starter because they are all you may maybe pre-analyzed and there's some guidance in the tickets, so it really is just a matter of doing it essentially. So we have this. Um we definitely um tweet about stuff on Twitter. I think this also actually is how the Outbox um stuff uh came to happen. So we were sharing this and we're asking people um if some is somebody interested, and yeah, then people will come. We try um so yeah, we were thinking about doing a Google Summer of Code project this year. Um for some uh different reasons. We didn't do it in the end, but this again would you know be one vehicle to get new young people students into the project, and um you know they will um make a difference for the project, but also for them, it's very nice. Um they will learn and they will get some experience, so this is like a two-way street, right? So this is always this important. You just you must not just try and get stuff out of people or so, but um you really they should have something from this as well, right? So it's a learning experience and so on.
SPEAKER_02That's fantastic. Um what do you in terms of your community goals, like uh where are you hoping to go next? Uh what's what's the next big thing that you'd like to see happen community-wise in Debesium?
SPEAKER_00Yeah, so about this um connector, one of your next guests is talking uh going to talk about. So they're real, the idea is that they will that this will be like a community-led connector, right? So um it will be under the be under the Debezium umbrella, but um they will, you know, they will do it and and decide how how it's done. So that's definitely, I think, a major step in terms of community because so far we just didn't have this. All the connectors um were worked on by the core team, but this doesn't really scale um, you know, uh so well. At some point, you just cannot, or you just must not spread out yourself um too thin. So that's definitely something also for the Oracle connector. This is very much worked on by the community, and I'm really excited um about to see people from the community coming and taking ownership of these things.
SPEAKER_02My guest today has been Gunnar Morling. Gunnar, thanks for being a part of Streaming Audio.
SPEAKER_00All right, Tim, it was a pleasure. Thank you so much for having me.
SPEAKER_02Hey, you know what you get for listening to the end? A Kafka Summit discount code. Kafka Summit is coming up on September 30th and October 1st in downtown San Francisco, and you can get 30% off if you go to Kafka-summit.org and use the discount code AUDIONTINE during checkout. Just enter Audio 19 while registering at Kafka-summit.org, and that 30% off is all yours. I'd love to see you there. But hey, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TLbergland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks for your support, and we'll see you next time.
