Skip to main content

Path

The way of a Kinesis record through a RustStream service: the read, the delivery, the handler, the reply and the checkpoint, one at a time or in batches.

The path of a record, one at a time with its reply on the left and in batches on the right, from the read to the checkpointOne at a timeBatchesGetRecordsup to 1000 recordsChannel64 deliveries, every shardKinesisMessageheaders, a placeHandlerdecodes and runsReply: PutRecordthe resolved keyackcheckpoint if contiguousGetRecordslimit n, up to 10,000Channel64 deliveriesA batchn records, or 50 msBatch handlerone call per batchEach elementsettles on its own
The diagram as text
  1. One at a time: GetRecords up to 1000 records; Channel 64 deliveries, every shard; KinesisMessage headers, a place; Handler decodes and runs; Reply: PutRecord the resolved key; ack checkpoint if contiguous
  2. Batches: GetRecords limit n, up to 10,000; Channel 64 deliveries; A batch n records, or 50 ms; Batch handler one call per batch; Each element settles on its own
  3. GetRecords up to 1000 records → Channel 64 deliveries, every shard: records
  4. Channel 64 deliveries, every shard → KinesisMessage headers, a place
  5. KinesisMessage headers, a place → Handler decodes and runs
  6. Handler decodes and runs → Reply: PutRecord the resolved key
  7. Reply: PutRecord the resolved key → ack checkpoint if contiguous
  8. GetRecords limit n, up to 10,000 → Channel 64 deliveries
  9. Channel 64 deliveries → A batch n records, or 50 ms
  10. A batch n records, or 50 ms → Batch handler one call per batch
  11. Batch handler one call per batch → Each element settles on its own

A shard's reader asks for up to 1,000 records per GetRecords. Each record becomes a KinesisMessage: the payload is copied out of the SDK's answer and the envelope is read off it when there is one. The partition key, the sequence number and the shard id become headers. The delivery also takes its place in the shard's watermark, which decides its checkpoint. These are a few allocations per record, and Benchmarks publishes what a record costs.

Every reader of the subscription sends its deliveries into one channel of 64. A reader waits while the channel is full, so the handlers set the pace of the reads. A read is stamped before it is sent: if the subscription repositions while the read is in flight, its records are dropped on their way out of the channel and are never handed to a handler.

A batch subscription makes its batch size the limit of every read, capped at the 10,000 a read can return. The batch closes when it holds that many records, or 50 milliseconds after its first record, so a batch is often smaller than the size. A batch may hold records of several shards. Every element settles on its own, with its own checkpoint.

A reply is encoded into a buffer the dispatch loop reuses. The partition key comes from the publish call, then from a partition-key header, then from the policy, and a record nobody keyed gets a key that spreads records over the shards. User headers go into the envelope, and a reply without them is the encoded buffer as it stands. The KinesisPublisher writes it with one PutRecord, through the SDK client of the runtime that publishes.

The delivery settles after the reply. An ack writes a checkpoint only when it moves the shard's watermark, which Settlement draws.