Skip to main content

Retries and dead-letter queues

This guide retries an SQS message now or later, caps its deliveries and lets SQS move what keeps failing to a dead-letter queue.

SQS does both jobs itself. A retry is the same message made visible again, with its receive count intact. The cap and the dead-letter queue are the queue's redrive policy, and the mount site declares it.

The example is an invoices service that charges invoices to customers.

Retry now and retry later​

src/main.rs
/// The largest invoice charged without a credit check.
const CREDIT_LIMIT_CENTS: u64 = 100_000;

/// How long an invoice waits for the credit check before it comes back.
const CREDIT_CHECK: Duration = Duration::from_secs(30);

#[derive(Debug, PartialEq, Serialize, Deserialize, Outgoing)]
#[outgoing(name = "invoices")]
struct Invoice {
    id: u64,
    customer: String,
    amount_cents: u64,
}

#[subscriber(SqsQueue::new("invoices").create_if_missing())]
async fn charge(invoice: &Invoice) -> HandlerOutcome {
    if invoice.customer.is_empty() {
        // Another try gets the same answer.
        tracing::warn!(invoice.id, "invoice without a customer dropped");
        return HandlerOutcome::drop();
    }
    if invoice.amount_cents > CREDIT_LIMIT_CENTS {
        tracing::info!(invoice.id, "invoice waits for the credit check");
        return HandlerOutcome::retry_after(CREDIT_CHECK);
    }
    tracing::info!(invoice.id, invoice.amount_cents, "invoice charged");
    HandlerOutcome::ack()
}

An invoice over the credit limit waits for a credit check. retry_after sets the message's visibility timeout to 30 seconds, and SQS hands it out again when they end. SQS counts whole seconds, so a fraction of a second rounds up. SQS keeps a message hidden for at most twelve hours from its receive, so a delay ends within them. A longer delay is shortened to end with them, and the crate logs a warning that names the queue. The message stays on its queue the whole time, so a deferred retry survives a restart of the service.

retry sets the visibility to zero, and the message is handed out again at once. An invoice without a customer gets the same answer on every try, so charge drops it.

The cap and the dead-letter queue​

src/main.rs
#[ruststream::app]
fn app() -> impl App {
    RustStream::new(AppInfo::new("invoices", "0.1.0")).with_broker(
        SqsBroker::new()
            .endpoint("http://localhost:4566")
            .region("us-east-1")
            .test_credentials(),
        |b| {
            // Four deliveries of one invoice, the first included; then SQS moves it.
            b.include(charge)
                .max_attempts(nonzero!(4u32))
                .dead_letter("invoices-dead");
            b.include(review);
        },
    )
}

max_attempts and dead_letter together are the queue's redrive policy. The cap becomes its maxReceiveCount, and the dead-letter queue becomes its target. The subscription writes the policy onto the queue when it opens. From then on SQS counts the receives and moves a message that ran out of them; the service publishes nothing. Here an invoice is delivered four times, and on the fifth receive SQS moves it to invoices-dead.

The two steps go together. A cap without a dead-letter queue, or a dead-letter queue without a cap, stops the service at startup with SqsError::IncompleteRedrive, which names the missing half. SQS takes a maxReceiveCount of at most 1000.

The dead-letter queue must exist when the subscription opens, unless a descriptor of the app creates it with create_if_missing. It must be of the same kind as its queue: a FIFO queue takes a FIFO dead-letter queue. Writing the policy needs sqs:SetQueueAttributes on the queue and sqs:GetQueueAttributes on the dead-letter queue.

drop deletes a message. On a capped queue, a drop of the delivery that used up the last receive returns the message instead, so SQS moves it to the dead-letter queue rather than losing it.

Reading the dead-letter queue​

src/main.rs
/// Reads the invoices SQS moved away after their last delivery.
#[subscriber(SqsQueue::new("invoices-dead").create_if_missing())]
async fn review(invoice: &Invoice) -> HandlerOutcome {
    tracing::error!(invoice.id, invoice.customer, "invoice needs a review");
    HandlerOutcome::ack()
}

A dead-letter queue is an ordinary queue, and review reads it like any other. The message arrives as it was sent, with its attributes. A queue with no reader keeps its messages until its retention period ends, so a person can also look at them in the console.

Testing the delay​

The tests in the example file run the app in process with a paused clock, so tb.advance(CREDIT_CHECK) lets the 30 seconds pass at once. One test runs an invoice through its four deliveries into invoices-dead, another checks that a cap without a dead-letter queue does not start. Testing an SQS service covers the harness.