Skip to main content

Design

Where ruststream-rdkafka sits: between the RustStream runtime and the rust-rdkafka client, one consumer per subscription, one shared producer.

The service, the RustStream runtime, this crate, the rust-rdkafka client and the Kafka cluster, top to bottomYour serviceRustStream coreruststream-rdkafkarust-rdkafka and librdkafka#[subscriber]handlersRustStream::new(..).with_broker(..)Runtime: dispatch, lanes, settlementKafkaSubscriberyields KafkaMessageKafkaPublisherand transactionalpublishersCommit trackerEosPipelineKafkaBrokerconfigurationKafkaTopic,KafkaTopics,KafkaPartitionsStreamConsumerone per subscriptionFutureProducerone per connectionKafka cluster
The diagram as text
  1. Your service: #[subscriber] handlers; RustStream::new(..) .with_broker(..)
  2. RustStream core: Runtime: dispatch, lanes, settlement
  3. ruststream-rdkafka: KafkaSubscriber yields KafkaMessage; KafkaPublisher and transactional publishers; Commit tracker; EosPipeline; KafkaBroker configuration; KafkaTopic, KafkaTopics, KafkaPartitions
  4. rust-rdkafka and librdkafka: StreamConsumer one per subscription; FutureProducer one per connection
  5. #[subscriber] handlers → Runtime: dispatch, lanes, settlement: mounted
  6. Runtime: dispatch, lanes, settlement → KafkaSubscriber yields KafkaMessage: polls the stream
  7. KafkaSubscriber yields KafkaMessage → StreamConsumer one per subscription
  8. KafkaPublisher and transactional publishers → FutureProducer one per connection
  9. StreamConsumer one per subscription → Kafka cluster
  10. FutureProducer one per connection → Kafka cluster

The crate implements the core's broker traits over rust-rdkafka, which wraps librdkafka. The runtime above it is the same for every broker. Everything Kafka-specific is in this crate: the descriptors, what an ack means, where a retry copy goes and how a transaction commits.

Each subscription opens a librdkafka consumer of its own, so a subscription is one member of its consumer group. The publishers of a connection share one producer, which the first publish opens. A transactional publisher opens a producer of its own, under its transactional.id.

A setting this crate does not type is a librdkafka property. The config passthroughs reach every one of them, and a property left unset keeps the librdkafka default.

The commit tracker turns acknowledgements into stored offsets. The EosPipeline commits consumed offsets inside a producer transaction. How the pieces start and stop is on Lifecycle, and the way of one record on Path.