Skip to main content

Settlement: retry, retry_after, dead letters

charge answers each refusal of the payment provider with the outcome it needs: a retry now, a retry later, a drop, or the dead-letter topic.

Every delivery ends in one outcome. ack finishes it, drop discards it, and retry asks the broker to deliver it again. retry_after delivers it again after a delay, without holding up the deliveries behind it. A retry that never succeeds would come back forever, so the mount site caps the attempts. Past the cap, the delivery goes to a dead-letter topic, where an operator or another service can look at it.

The subscriber​

The payment provider refuses a charge in three ways, and each needs a different outcome. Busy means its rate limit is used up for now: an immediate retry would be refused again, so retry_after waits a second. Unavailable means it did not answer: retry tries again at once. Declined means the card does not cover the amount, and another try gets the same answer, so the payment is dropped. The provider's client comes from the app's state, which chapter 7 opens.

payments.rs
/// How long a busy provider asks its callers to wait.
pub(crate) const BACKOFF: Duration = Duration::from_secs(1);

/// Charges an order through the payment provider.
#[subscriber("payments.requested", reply)]
async fn charge(
    request: &PaymentRequest,
    State(gateway): State<Arc<Gateway>>,
) -> Result<Paid, HandlerOutcome> {
    match gateway.charge(request).await {
        Ok(()) => Ok(Paid {
            order: request.order,
            customer: request.customer.clone(),
            lines: request.lines.clone(),
        }),
        // Past the rate limit: a redelivery right now would be refused again.
        Err(Refusal::Busy) => Err(HandlerOutcome::retry_after(BACKOFF)),
        // No answer: try again at once, up to the attempt cap.
        Err(Refusal::Unavailable) => Err(HandlerOutcome::retry()),
        // A refused card stays refused.
        Err(Refusal::Declined) => {
            tracing::warn!(order = request.order, "card declined, payment dropped");
            Err(HandlerOutcome::drop())
        }
    }
}

The cap and the dead-letter topic​

max_attempts counts every delivery of one message, the first included. The fifth delivery is not retried: dead_letter publishes it to payments.dead, payload and headers as they arrived. The router decodes the requests as CBOR and encodes the reply the same way.

payments.rs
/// The payments side's subscriber: five deliveries at most, then the dead-letter topic.
pub(crate) fn routes() -> impl RouterDef<MemoryBroker, Shop> {
    Router::new()
        .with_codec(INTERNAL)
        .include(charge)
        .max_attempts(nonzero!(5u32))
        .dead_letter("payments.dead")
        .out_reply(Publish)
        .codec(INTERNAL)
        .build()
}

The in-memory broker redelivers by itself, after the delay retry_after asks for. A broker that cannot delay a delivery gets a copy published back to the subscription; where a retry copy goes covers it. Chapter 8 tests all three paths, the delay included, without waiting for it.