Skip to main content

Pipe

The stdio client reads standard input on one thread and writes standard output on another, one line per message, for the whole process.

How the stdio client carries lines in from the stage before and out to the stage afterThe stage beforestdin threadreads and parses a lineDispatch by stream key:a line with no keygoes to every consumerStdioConsumerone per subscriptionStdioPublisherenvelope if neededstdout threadnumbers, stamps[time | key | seq]payloadThe stage afterOne stdin and one stdout thread per process:a shutdown ends every stdio stream in it
The diagram as text
  1. The stage before → stdin thread reads and parses a line: lines
  2. stdin thread reads and parses a line → Dispatch by stream key: a line with no key goes to every consumer
  3. Dispatch by stream key: a line with no key goes to every consumer → StdioConsumer one per subscription
  4. StdioPublisher envelope if needed → stdout thread numbers, stamps
  5. stdout thread numbers, stamps → [time | key | seq] payload
  6. [time | key | seq] payload → The stage after
  7. stdout thread numbers, stamps → Dispatch by stream key: a line with no key goes to every consumer
  8. One stdin and one stdout thread per process: a shutdown ends every stdio stream in it

A line is [timestamp | stream_key | sequence | shard_id] payload, and every field in the brackets is optional. The timestamp reads YYYY-MM-DDTHH:MM:SS, with fractions of a second allowed. A line that starts with [WORD] is read as the stream key WORD.

The first subscription starts the stdin thread, which reads a line, parses its fields and hands it to the consumers of its key. A line with no key goes to every consumer, so a stray line on standard output of the stage before reaches every subscription of this one. When the stage starts, the crate holds standard input until its first consumer exists, so a line already waiting is not lost.

The first publish starts the stdout thread. It numbers each key from zero, stamps the time of writing to the second, and prints the line. It skips an empty payload, so the publisher refuses an empty message instead of reporting a line that never left. A payload a line would change goes inside the envelope.

The dashed arrow is loopback: the stdout thread also hands each line to this process's own consumers. It runs a stdio service in one process, as a test does.

Both threads belong to the process, not to a broker. A shutdown ends every stdio consumer and producer the process holds, after each subscription has delivered the lines it already read. When standard input ends, the stdin thread stops and the subscriptions stay open. A line that is not UTF-8, or one whose brackets do not parse, aborts the process.

The model is sea-streamer-stdio's own (the client). How a subscription reads its key is on Lines.