Getting started

This guide installs ruststream-kinesis, starts a local Kinesis and runs a first service against it.

Installation

Cargo.toml

[dependencies]
ruststream = { version = "0.7", features = ["macros", "json", "logging"] }
ruststream-kinesis = "0.7"
serde = { version = "1", features = ["derive"] }
tracing = "0.1"

[dev-dependencies]
ruststream-kinesis = { version = "0.7", features = ["testing"] }
tokio = { version = "1", features = ["macros", "rt"] }

ruststream is the framework: macros gives the attributes, json the default codec, and logging prints the service's log to the terminal. ruststream-kinesis is the Kinesis broker, built on the official aws-sdk-kinesis client. Its testing feature adds the in-process Kinesis that tests run on.

The broker crate has these features, all off by default:

Feature	What it gives
dynamodb-lease	DynamoLeaseStore, so several instances of a service share the shards (Several instances)
asyncapi	the Kinesis details of each channel in the generated AsyncAPI document
testing	the in-process Kinesis behind TestApp

A local Kinesis

LocalStack runs Kinesis in one container:

compose.yaml

services:
  localstack:
    image: localstack/localstack:4.14.0
    ports:
      - "4566:4566"
    environment:
      SERVICES: kinesis
      KINESIS_LATENCY: 0

docker compose up -d

The stack answers on http://localhost:4566 and starts with no streams. KINESIS_LATENCY: 0 turns off the delay LocalStack adds to every Kinesis call by default.

The first service

Macros

src/main.rs

#[derive(Debug, PartialEq, Deserialize)]
struct Order {
    id: u64,
    amount_cents: u64,
}

#[subscriber(KinesisStream::new("orders").create_if_missing(1))]
async fn take_order(order: &Order) -> HandlerOutcome {
    if order.amount_cents == 0 {
        tracing::warn!(order.id, "order without an amount dropped");
        return HandlerOutcome::drop();
    }
    tracing::info!(order.id, order.amount_cents, "order taken");
    HandlerOutcome::ack()
}

#[ruststream::app]
fn app() -> impl App {
    RustStream::new(AppInfo::new("orders", "0.1.0")).with_broker(
        KinesisBroker::new()
            .endpoint("http://localhost:4566")
            .region("us-east-1")
            .test_credentials(),
        |b| {
            b.include(take_order);
        },
    )
}

Manual

src/main.rs

#[derive(Debug, PartialEq, Deserialize, JsonSchema)]
struct Order {
    id: u64,
    amount_cents: u64,
}

struct TakeOrder;

impl Handle<Order> for TakeOrder {
    fn handle(
        &self,
        order: &Order,
        _outs: &(),
        _ctx: &mut Context<'_>,
    ) -> impl Future<Output = Result<(), HandlerOutcome>> {
        if order.amount_cents == 0 {
            tracing::warn!(order.id, "order without an amount dropped");
            return ready(Err(HandlerOutcome::drop()));
        }
        tracing::info!(order.id, order.amount_cents, "order taken");
        ready(Ok(()))
    }
}

fn app() -> impl App {
    RustStream::new(AppInfo::new("orders", "0.1.0")).with_broker(
        KinesisBroker::new()
            .endpoint("http://localhost:4566")
            .region("us-east-1")
            .test_credentials(),
        |b| {
            b.include(
                subscriber(KinesisStream::new("orders").create_if_missing(1), TakeOrder).build(),
            );
        },
    )
}

fn main() -> ExitCode {
    run_main(app)
}

take_order reads JSON orders from the orders stream. It acknowledges an order with an amount and drops one without. KinesisStream names the stream, and create_if_missing(1) creates it with one shard when the subscription opens. A production service leaves that step out and takes its streams from its infrastructure. KinesisBroker::new only records the configuration. endpoint, region and test_credentials point it at the local stack. The SDK client is built when the service starts. #[ruststream::app] turns app into main; in the manual form, run_main does the same.

Run the service:

cargo run -- run

The subscription opens at the tip of the stream, so it reads what is published after the service started. Publish an order from another terminal with the AWS CLI:

export AWS_ENDPOINT_URL=http://localhost:4566 AWS_REGION=us-east-1 \
  AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test
aws kinesis put-record --stream-name orders --partition-key customer-7 \
  --cli-binary-format raw-in-base64-out --data '{"id":1,"amount_cents":1250}'

The service logs order taken with the order's id. Ctrl+C stops it.

The first test

The example file also holds two tests. Each starts the same app with TestApp::start, publishes an order and checks how take_order settled it. KinesisBroker connects to an in-process Kinesis there, so the tests need no stack:

cargo test

Testing a Kinesis service walks through these tests and what the in-process Kinesis models.

Coming from aws-sdk-kinesis

The Rust SDK reads a stream one shard and one call at a time, and a service on it writes the consumer around those calls. Here is where each part goes in RustStream:

aws-sdk-kinesis	RustStream
aws_config::load_from_env and Client::new	KinesisBroker::new, or from_config with an SdkConfig of your own (Running in production)
ListShards and a GetShardIterator per shard	the stream name in KinesisStream::new; the subscription finds the shards and follows splits and merges
a GetRecords loop per shard	a #[subscriber] handler (Consuming a stream)
decoding record.data()	the handler's parameter type, decoded by the codec
a checkpoint written after the work	the handler's outcome (Checkpoints and delivery guarantees)
a lease table so that several instances share the shards	DynamoLeaseStore (Several instances)
GetShardIterator with TRIM_HORIZON or AT_TIMESTAMP	start_at at the mount site, or the seek handle in the handler (Replay and seeking)
PutRecord with a partition key	a reply or an Out parameter, sent through KinesisPublish (Shards, partition keys and ordering)