Design

Where ruststream-pulsar sits: between the RustStream runtime and the pulsar client, one consumer and one driver task per subscription, one producer per topic.

The diagram as text

1. Your service: #[subscriber] handlers; RustStream::new(..) .with_broker(..)
2. RustStream core: Runtime: dispatch, lanes, settlement, batches
3. ruststream-pulsar: PulsarSubscriber and its driver task; PulsarPublisher a producer per topic; PulsarMessage; PulsarSeeker; PulsarBroker configuration; PulsarSubscription, PulsarTopic
4. The pulsar client: Consumer one per subscription; Producer one per topic
5. #[subscriber] handlers → Runtime: dispatch, lanes, settlement, batches: mounted
6. Runtime: dispatch, lanes, settlement, batches → PulsarSubscriber and its driver task: polls the stream
7. PulsarSubscriber and its driver task → Consumer one per subscription
8. PulsarPublisher a producer per topic → Producer one per topic
9. Consumer one per subscription → Pulsar cluster
10. Producer one per topic → Pulsar cluster
11. Under the testing feature, an in-process transport can take the client's place

The crate implements the core's broker traits over the pulsar client. The runtime above it is the same for every broker. What is particular to Pulsar is in this crate: the descriptors, what each outcome sends, the dead-letter policy and the positions a seek moves to.

A connected broker holds one client. Every publisher of the connection shares that client and its producers, one per topic, opened by the first publish to the topic.

Each subscription opens one client consumer, and a task of its own drives it. The client acknowledges through the consumer, which only one owner can use at a time, while a delivery has to settle from wherever the handler runs. So the driver task owns the consumer. It passes each delivery on through a channel of 16, and takes back the acknowledgements and the seeks the deliveries send it. A PulsarMessage carries the way back to its driver, and a PulsarSeeker uses the same way.

The client hands over one message at a time. A batch handler gets batches the runtime assembles in the service, as Lanes draws.

A retry is the broker's: the delivery is negatively acknowledged and the broker delivers it again. Nothing is published from the service to bring it back, so a registration on this broker has no retry publisher to declare, as Settlement draws.

How the pieces start and stop is on Lifecycle, and the way of one message on Path.

The in-process transport

With the testing feature, TestApp::start connects the same broker to an in-process transport instead of a server. The descriptors, the policies and the deliveries are the production ones, and so are their settings. The transport models full topic names; subscriptions over one topic, a list or a pattern; the four subscription types; negative acknowledgement; the client's dead-letter policy; delayed retries on the runtime's clock; and the retained log a seek moves over. It refuses what a server refuses: a payload over max_message_size, a second consumer of an Exclusive subscription, a handle used after shutdown.

What is the server's own is left to a live run: KeyShared hash ranges, which consumer of a Shared subscription the server picks, the redelivery after the acknowledgement timeout, partitioned topics, and messages published while a subscription has no consumer.

Without the feature the transport is not compiled. In a production build the broker's transport is exactly the size of the client, and a subscription's stream and a delivery's settlement are exactly the size of the channel they wrap. The compiler checks those sizes.