Skip to main content

Log

A Kafka topic is a set of partitions, each an ordered log of records numbered by offset, and every consumer group reads all of it.

A producer appends to the partitions of one topic, and two consumer groups each read the whole topicTopic ordersGroup billingGroup auditProducerthe key picks the partitionP0012345P10123P201234Member 1: P0, P1Member 2: P2Member 1:P0, P1, P2Each group commits its own offset per partition:the record it resumes from
The diagram as text
  1. Topic orders: P0; 0; 1; 2; 3; 4; 5; P1; 0; 1; 2; 3; P2; 0; 1; 2; 3; 4
  2. Group billing: Member 1: P0, P1; Member 2: P2
  3. Group audit: Member 1: P0, P1, P2
  4. Producer the key picks the partition → Topic orders: appends
  5. Topic orders → Group billing: reads every record
  6. Topic orders → Group audit: reads every record
  7. Each group commits its own offset per partition: the record it resumes from

A producer appends a record to one partition of the topic. A record with a key goes to the partition its key hashes to, so the records of one key keep their order. A record without a key goes wherever the producer's partitioner puts it.

Each partition numbers its records from zero, and its offsets grow apart from the other partitions'. Order holds inside a partition and nowhere else. A record stays in the log until retention deletes it, whether anyone read it or not.

A consumer group reads every record of the topic once. The group hands each partition to one of its members, so the members share the work. A second group reads its own copy of every record, and the two never affect each other.

What a group has read is one number per partition: its committed offset, the record it resumes from. Kafka stores no per-record acknowledgement.

The model is Apache Kafka's own (concepts and terms). How a group shares its partitions is on Groups, and how the client reads and commits on Consumer.