Skip to main content

Lanes

One subscription reads all its topics and partitions through one consumer; workers(n, by_key) spreads the messages over lanes keyed by their partition key.

One subscription's dispatch: three partitions, one consumer, one stream and two lanes keyed by the partition key-partition-0-partition-1-partition-2The subscription's consumerevery topic and partition it readsPulsarSubscriber streamone message at a timeworkers(2, by_key)lane = hash of the partition-key headerLane 1keys A and C,each in orderLane 2key B, in orderA message without a key has no lane key:such messages rotate over the lanesAcross processes, a KeyShared subscriptionkeeps each key on one consumer
The diagram as text
  1. -partition-0 → The subscription's consumer every topic and partition it reads
  2. -partition-1 → The subscription's consumer every topic and partition it reads
  3. -partition-2 → The subscription's consumer every topic and partition it reads
  4. The subscription's consumer every topic and partition it reads → PulsarSubscriber stream one message at a time: pushed messages
  5. PulsarSubscriber stream one message at a time → workers(2, by_key) lane = hash of the partition-key header: one at a time
  6. workers(2, by_key) lane = hash of the partition-key header → Lane 1 keys A and C, each in order
  7. workers(2, by_key) lane = hash of the partition-key header → Lane 2 key B, in order
  8. A message without a key has no lane key: such messages rotate over the lanes
  9. Across processes, a KeyShared subscription keeps each key on one consumer

A subscription's stream yields one message at a time, whichever topic or partition it came from. Without workers, one loop runs the handler for each message in turn, and nothing runs side by side.

workers(n, by_key) runs n lanes. A message goes to the lane its key hashes to, and a lane runs its messages in order. The key is the message's partition key, which a delivery carries in the partition-key header, so the messages of one key keep their order. A message without a key has no lane key, and such messages rotate over the lanes.

Lanes order the work inside one process. Across processes, a KeyShared subscription keeps each key on one consumer, so several instances of a service share a topic and still keep each key's order.

A seek through a PulsarSeeker moves the whole subscription: every topic and every partition its consumer reads.

A batch handler gets batches the runtime assembles in the service, because the client hands over one message at a time. A batch holds at most the size the mount site names. It closes when it is full, or 10 milliseconds after its first message, the descriptor's batch_wait.