Skip to main content

Topics

A Pulsar topic lives in a namespace of a tenant. A persistent topic keeps its messages in BookKeeper, and a partitioned topic is one topic per partition.

A tenant holds a namespace, the namespace holds a partitioned, a persistent and a non-persistent topic, and a producer publishes to the partitioned oneTenant acmeNamespace acme/orderspersistent://acme/orders/paymentspersistent://acme/orders/creatednon-persistent://acme/orders/ticksProducera key picks the partition-partition-00123-partition-101-partition-20120123kept inBookKeeper ledgersnothing kept: only consumers connected nowA bare name is a persistent topic of the default namespace:orders means persistent://public/default/orders
The diagram as text
  1. Tenant acme: -partition-0; 0; 1; 2; 3; -partition-1; 0; 1; -partition-2; 0; 1; 2; 0; 1; 2; 3; kept in BookKeeper ledgers; nothing kept: only consumers connected now
  2. Namespace acme/orders: -partition-0; 0; 1; 2; 3; -partition-1; 0; 1; -partition-2; 0; 1; 2; 0; 1; 2; 3; kept in BookKeeper ledgers; nothing kept: only consumers connected now
  3. persistent://acme/orders/payments: -partition-0; 0; 1; 2; 3; -partition-1; 0; 1; -partition-2; 0; 1; 2
  4. persistent://acme/orders/created: 0; 1; 2; 3; kept in BookKeeper ledgers
  5. non-persistent://acme/orders/ticks: nothing kept: only consumers connected now
  6. Producer a key picks the partition → persistent://acme/orders/payments: publishes
  7. A bare name is a persistent topic of the default namespace: orders means persistent://public/default/orders

A topic's full name is {persistent|non-persistent}://tenant/namespace/topic. Tenants and namespaces are the units Pulsar administers: permissions, retention and other policies are set on them. PulsarTopic parses a name before any I/O, and a bare name such as orders becomes persistent://public/default/orders.

A persistent topic writes every message to BookKeeper ledgers. A message stays there until every subscription has acknowledged it, and then for as long as the namespace's retention policy says. A non-persistent topic stores nothing: a message goes to the consumers connected at that moment, and a consumer that connects later never sees it.

A partitioned topic is a set of internal topics named <topic>-partition-<n>, each a log of its own, each owned by one broker of the cluster. The producer picks the partition for every message. This crate's producers hash a message's key to its partition, so the messages of one key keep their order, and rotate over the partitions for a message without a key. A delivery from a partitioned topic reports the partition in its topic name.

The server creates a persistent topic the first time a producer or a consumer names it, unless the cluster turned automatic creation off. A partitioned topic is created by an administrator, with its number of partitions.

The model is Apache Pulsar's own (topics, multi-tenancy, partitioned topics, non-persistent topics). How consumers read a topic is on Subscriptions.