Streaming Audio: A Confluent podcast about Apache Kafka®

Diving into Exactly Once Semantics with Guozhang Wang

April 22, 2019 Confluent, original creators of Apache Kafka® Season 1 Episode 29
Streaming Audio: A Confluent podcast about Apache Kafka®
Diving into Exactly Once Semantics with Guozhang Wang
Chapters
Streaming Audio: A Confluent podcast about Apache Kafka®
Diving into Exactly Once Semantics with Guozhang Wang
Apr 22, 2019 Season 1 Episode 29
Confluent, original creators of Apache Kafka®

It has been said that in distributed messaging, there are two hard problems: 2) exactly once delivery, 1) guaranteed order of messages and 2) exactly once delivery. Apache Kafka® has offered exactly once processing since version 0.11, which allows properly configured producers and consumers to make the guarantee that each message will be processed exactly one time. 

In this episode, Kafka Streams engineer Guozhang Wang walks through the implementation of transactional messaging in Kafka in some detail, including the idempotent producer API, the transaction coordinator responsible for managing the transaction log and consumer configurations. It’s a complex topic, but he takes us through it carefully and completely.

EPISODE LINKS




Show Notes

It has been said that in distributed messaging, there are two hard problems: 2) exactly once delivery, 1) guaranteed order of messages and 2) exactly once delivery. Apache Kafka® has offered exactly once processing since version 0.11, which allows properly configured producers and consumers to make the guarantee that each message will be processed exactly one time. 

In this episode, Kafka Streams engineer Guozhang Wang walks through the implementation of transactional messaging in Kafka in some detail, including the idempotent producer API, the transaction coordinator responsible for managing the transaction log and consumer configurations. It’s a complex topic, but he takes us through it carefully and completely.

EPISODE LINKS