Skip to main content

Backpressure

A subscription reads its socket at most read_ahead messages ahead of the handler; past that it stops reading, and the wait reaches back to the peer's send.

A subscription's read-ahead: the message goes down the left column, and a full channel's wait climbs back up the right one to the peerPeerPUSH or PUB sendSocket buffersof the kernelSocket taskreads one messageChannelread_ahead deepHandleror workers(n) lanesThe handler is behind:the channel is fullThe task waitsand stops readingThe buffers fill, andthe TCP window closesThe peer's sendwaits for roomOn PUB/SUB the PUB waits for its slowest matchingsubscriber; a message no filter matches is dropped
The diagram as text
  1. Peer PUSH or PUB send → Socket buffers of the kernel
  2. Socket buffers of the kernel → Socket task reads one message
  3. Socket task reads one message → Channel read_ahead deep
  4. Channel read_ahead deep → Handler or workers(n) lanes
  5. Channel read_ahead deep → The handler is behind: the channel is full
  6. The handler is behind: the channel is full → The task waits and stops reading
  7. The task waits and stops reading → The buffers fill, and the TCP window closes
  8. The buffers fill, and the TCP window closes → The peer's send waits for room
  9. On PUB/SUB the PUB waits for its slowest matching subscriber; a message no filter matches is dropped

A subscription's task reads one message at a time off its socket and puts it into a channel. The channel holds read_ahead messages, 1000 by default, the receive high-water mark libzmq gives a socket. The broker carries the setting, so every subscription of that broker reads that far ahead.

When the handler falls behind, the channel fills and the task waits for room. A task that waits reads nothing, so the frames stay in the kernel's buffers until they fill. Then TCP stops the sender, and the peer's send waits until the handler catches up. A slow handler slows the sender down, and this process's memory stays bounded.

On PUB/SUB the waiting sender is the PUB socket. It writes to every matching subscriber in turn, so one slow subscriber holds back the messages for all of them. A message no subscriber's filter matches is still dropped, as Queues explains.

The handler takes one message at a time unless the mount site asks for more. workers(n) runs n handlers side by side, and a free worker takes the next message from the channel. ZeroMQ messages carry no key, so workers(n, by_key) has nothing to hash, and the messages rotate over the lanes. A batch closes at the size the mount site names, or 20 milliseconds after its first message.

A publish waits the same way when the peer reads slowly. A queue publish can be cancelled while it waits, and the next publish completes its send. A fan-out publish, a reply and a request cannot, so give them up through a timeout.