Skip to main content

Streams

A Kinesis stream is a set of shards, each an ordered log of records, and every consumer reads every shard for itself until retention ends.

Producers put records into the three shards of one stream, and two consumers each read every shardStream ordersProducersthe partition key picks the shardShard 012345Shard 1123Shard 21234Consumer billingan iterator per shard,a checkpoint per shardConsumer auditan iterator per shard,a checkpoint per shardA record stays for every reader until retention ends:24 hours by default, up to 365 days
The diagram as text
  1. Stream orders: Shard 0; 1; 2; 3; 4; 5; Shard 1; 1; 2; 3; Shard 2; 1; 2; 3; 4
  2. Producers the partition key picks the shard → Stream orders: PutRecord
  3. Stream orders → Consumer billing an iterator per shard, a checkpoint per shard: reads every shard
  4. Stream orders → Consumer audit an iterator per shard, a checkpoint per shard: reads every shard
  5. A record stays for every reader until retention ends: 24 hours by default, up to 365 days

A producer writes a record with PutRecord: the stream's name, a partition key and a data blob. Kinesis hashes the key with MD5 into a 128-bit number. Each shard owns a range of those numbers, so the key picks the shard, and the records of one key stay in order. Order holds inside a shard and nowhere else.

A record holds a data blob and a partition key, and nothing else: there are no headers. The key is 1 to 256 Unicode characters, and the blob with its key is at most 10 MiB. A stream is named by 1 to 128 characters of letters, digits, _, . and -. This crate names a stream by its name.

Kinesis numbers each record with a sequence number when the record arrives. Within a shard the numbers grow over time. The crate keeps track of the order in which a shard delivered its records and never compares the numbers themselves.

A record stays in the stream until its retention ends, whether anyone read it or not: 24 hours by default, up to 365 days. Every consumer reads every shard on an iterator of its own, so two consumers each see every record. What a consumer has read is one position per shard, its checkpoint. Kinesis stores no checkpoints and no acknowledgements: the consumer keeps them, in a lease table.

User headers have no place in a record, so the crate writes them into a small envelope in front of the payload, and only when there are any. A record published without headers is the plain payload, which any consumer reads.

The model is Amazon Kinesis Data Streams' own (key concepts, PutRecord, retention). How shards split and merge is on Shards and resharding, and how a consumer reads one on Reading.