Groups

A consumer group hands each partition to one member, and a member joining or leaving moves partitions in a rebalance.

The diagram as text

1. Member 1: Reads P0, P1, P2; P2 is revoked; Reads P0, P1
2. Coordinator: Rebalance: a new assignment; A member that stops heartbeating for session.timeout.ms leaves the group, and its partitions move the same way
3. Member 2: Joins the group; Reads P2 from the committed offset
4. Joins the group → Rebalance: a new assignment: join
5. Rebalance: a new assignment → P2 is revoked: revoke
6. P2 is revoked → Reads P0, P1
7. Rebalance: a new assignment → Reads P2 from the committed offset: assign

The group coordinator is a broker of the cluster. When a member joins or leaves, it starts a rebalance and the group agrees on a new assignment. A partition taken from one member is revoked there and assigned to another, which resumes from the group's committed offset. Work the first member did after its last commit is read again by the second. A partition the group has never committed starts where auto.offset.reset says: the end of the log by default.

A member that stops sending heartbeats for session.timeout.ms (45 seconds in librdkafka) is removed from the group. A member that does not take a new record for max.poll.interval.ms (5 minutes) leaves the group too.

partition.assignment.strategy decides how a rebalance moves partitions. The eager strategies, range and roundrobin (librdkafka's default is both, range first), revoke every partition and assign them anew. cooperative-sticky moves only the partitions that change owner, and the rest keep flowing during the rebalance (KIP-429). The two families cannot mix inside one group.

A consumer can also take partitions by name instead of joining the group protocol. It then never rebalances, and nothing moves its partitions.

The defaults are those of librdkafka (configuration, consumer groups).