Skip to main content

Lifecycle

KafkaBroker::new records configuration, connect probes the cluster, startup opens a consumer per subscription, and shutdown flushes.

The broker's three states as types, and what each transition does on KafkaBrokerOn KafkaKafkaBroker::new(servers)ConnectedKafkaBrokerClosedKafkaBrokerunflushed_records()Configuration only,no I/OMetadata probewithin connect_timeoutEach subscription:a consumer, joined andassigned its partitionsThe first publishopens the producerFlush in-flightrecords withinflush_timeoutA publisher that outlives the connectionanswers KafkaError::Closed, never a silent success
The diagram as text
  1. Broker: KafkaBroker::new (servers); ConnectedKafkaBroker; ClosedKafkaBroker unflushed_records()
  2. On Kafka: Configuration only, no I/O; Metadata probe within connect_timeout; Each subscription: a consumer, joined and assigned its partitions; The first publish opens the producer; Flush in-flight records within flush_timeout
  3. KafkaBroker::new (servers) → ConnectedKafkaBroker: connect(self)
  4. ConnectedKafkaBroker → ClosedKafkaBroker unflushed_records(): shutdown(self)
  5. Configuration only, no I/O → Metadata probe within connect_timeout
  6. Metadata probe within connect_timeout → Each subscription: a consumer, joined and assigned its partitions
  7. Each subscription: a consumer, joined and assigned its partitions → The first publish opens the producer
  8. The first publish opens the producer → Flush in-flight records within flush_timeout
  9. A publisher that outlives the connection answers KafkaError::Closed, never a silent success

new does no I/O, so the service's builder stays synchronous. The runtime connects the broker when the app starts. Connect fetches the cluster's metadata through a short-lived client built from the producer settings, within connect_timeout (30 seconds). An unreachable cluster or a producer property librdkafka refuses fails the start.

Each subscription's descriptor resolves once, against the connected broker. A group subscription opens its consumer and waits until the group has assigned it partitions and the start of each is known, up to assignment_timeout (one minute). Until the runtime starts reading, the consumer keeps answering the group, so a rebalance does not wait for it. A subscription that names no group, under a broker with no default_group, fails the start.

A service that only consumes runs no producer. The first publish opens the shared one.

Shutdown marks the connection closed, then flushes what the producer still holds within flush_timeout (30 seconds). ClosedKafkaBroker reports the records the flush left behind. A consumer closes when its subscription is dropped: it leaves the group, and unless its subscription is transactional it commits its stored position on the way.