Skip to main content

Lanes

One subscription reads all its partitions through one consumer; workers(n, by_key) spreads the records over lanes keyed by partition.

One subscription's dispatch: three partitions, one consumer, one stream and two lanes keyed by partitionPartition 0Partition 1Partition 2The subscription's consumerevery partition the group assigned itKafkaSubscriber streamone record at a timeworkers(2, by_key)lane = hash of the partitionLane 1partitions 0 and 2,each in orderLane 2partition 1, in orderUnder LaneKey::RecordKey a lane is keyed by the record key:one partition spreads over lanes, each key in order
The diagram as text
  1. Partition 0 → The subscription's consumer every partition the group assigned it
  2. Partition 1 → The subscription's consumer every partition the group assigned it
  3. Partition 2 → The subscription's consumer every partition the group assigned it
  4. The subscription's consumer every partition the group assigned it → KafkaSubscriber stream one record at a time: fetched records
  5. KafkaSubscriber stream one record at a time → workers(2, by_key) lane = hash of the partition: one at a time
  6. workers(2, by_key) lane = hash of the partition → Lane 1 partitions 0 and 2, each in order
  7. workers(2, by_key) lane = hash of the partition → Lane 2 partition 1, in order
  8. Under LaneKey::RecordKey a lane is keyed by the record key: one partition spreads over lanes, each key in order

A subscription's stream yields one record at a time, whichever partition it came from. Without workers, one loop runs the handler for each record in turn, so every partition keeps its order and nothing runs side by side.

workers(n, by_key) runs n lanes. A record goes to the lane its key hashes to, and a lane runs its records in order. The key is the record's partition by default (LaneKey::Partition), so a partition never spreads over two lanes and Kafka's order is kept. Lanes fewer than partitions serve several partitions each.

LaneKey::RecordKey keys a lane by the record key instead. Records of one key keep their order, and one partition's keys run side by side. A record without a key has no lane key, and such records rotate over the lanes.

Acks from parallel lanes arrive out of order. Under Commit::Tracked the committed offset still never passes a record that has not settled, as Settlement draws.

A batch handler takes one record and whatever librdkafka has already fetched, up to the size the mount site names, so a batch is often smaller than that size.