Kafka transactions

A Kafka transaction makes records written to several partitions, and the consumed offsets, visible together or not at all.

The diagram as text

1. Begin on the transactional producer → Produce records to any partitions
2. Produce records to any partitions → Send the consumed offsets with the group's metadata
3. Send the consumed offsets with the group's metadata → Commit the transaction
4. Commit the transaction → A read_committed reader sees the records, and the offsets count as committed: markers written
5. Send the consumed offsets with the group's metadata → Abort: records discarded, offsets unmoved, the input is read again: on error
6. One transactional id belongs to one live producer: initializing it again fences the earlier one

A transactional producer is named by its transactional.id. It runs one transaction at a time. Records produced inside it land in the log at once, and the commit writes markers that make them visible.

A reader at the read_committed isolation level, librdkafka's default, sees a transaction's records only after the commit and never after an abort. A reader at read_uncommitted sees them as they are written.

The consumed offsets can join the transaction. Then the input positions move exactly when the output becomes visible, which is the consume-transform-produce shape of KIP-447. An abort leaves the offsets where they were, so the input is read again into a new transaction.

A transaction open longer than transaction.timeout.ms (one minute) is aborted by the coordinator. A producer started with an id already in use fences the earlier one: that producer's next call fails.

The defaults and the fencing rule are librdkafka's (transactional producer, configuration).