Skip to main content

Consumer

The librdkafka consumer fetches ahead on threads of its own, hands records out one poll at a time, and commits a stored offset on a timer.

The librdkafka consumer: fetch threads, the local queue, the application's poll, the offset store and the commit timerlibrdkafka threadsApplicationKafka clusterpartition leaders, group coordinatorFetchassigned partitionsLocal queuefetched aheadAuto-commiton a timer,every 5 sPollthe next recordProcessthe recordOffset storein memory,a positionper partition
The diagram as text
  1. librdkafka threads: Fetch assigned partitions; Local queue fetched ahead; Auto-commit on a timer, every 5 s
  2. Application: Poll the next record; Process the record; Offset store in memory, a position per partition
  3. Kafka cluster partition leaders, group coordinator → Fetch assigned partitions: records
  4. Fetch assigned partitions → Local queue fetched ahead
  5. Local queue fetched ahead → Poll the next record
  6. Poll the next record → Process the record
  7. Poll the next record → Offset store in memory, a position per partition: on hand-over
  8. Process the record → Offset store in memory, a position per partition: or later
  9. Offset store in memory, a position per partition → Auto-commit on a timer, every 5 s: read
  10. Auto-commit on a timer, every 5 s → Kafka cluster partition leaders, group coordinator: commit

librdkafka fetches the records of the assigned partitions into a local queue before the application asks for them. queued.max.messages.kbytes (64 MiB) bounds that queue. When it is full, fetching for a partition pauses for fetch.queue.backoff.ms (one second).

A poll takes the next record off the queue. rust-rdkafka's StreamConsumer does that through recv, an async call that is safe to cancel.

The offset store is an in-memory position per partition. By default (enable.auto.offset.store) a poll stores the offset of the record it hands out, before the application has processed it. The dashed arrow is the other way: the application turns the automatic store off and stores an offset once the record is processed.

A timer commits the stored positions every auto.commit.interval.ms (five seconds) while enable.auto.commit is on, and once more when the consumer closes. A crash loses what was processed since the last commit, and it is read again. Under the default store, a record stored on hand-over and not yet processed is skipped after a crash.

The behaviour and the defaults are librdkafka's (offset management, configuration).