Rust in production: the compiler is not your architect

5 min read

Rust in production is sold as a safety net. The borrow checker, the type system, the "if it compiles, it ships" pitch. Then you ship to a real backend handling GOV.UK sign-in flows or a Monzo-style payments stream, and the compiler is suddenly the least interesting thing in your stack. The borrow checker never told you that your retry storm would melt Postgres. It never warned you about the unbounded channel, the silent integer truncation in a JSON deserialiser, or the 6GB RSS on a service that should sit at 80MB. The compiler is a syntax-level safety tool. Production is a systems problem. This is what actually bites teams who learn that lesson the hard way.

The compiler checks your code, not your system

Move semantics, lifetimes, exhaustive matches. Lovely. None of them fire when your Axum handler spawns a tokio task that holds an unbounded mpsc::channel and a cloned Arc<Database> for the lifetime of the process. I have seen a small Rust service at a London fintech push a queue to 4 million pending messages because the producer was faster than the consumer and nobody added backpressure. The compiler was perfectly happy. The on-call rota was not.

Memory safety inside the process is solved. Memory pressure, file descriptor exhaustion, FD_SET limits on Linux (default 1024 per process unless you raise ulimit -n and bump net.core.somaxconn), TCP TIME_WAIT storms under a load test from the Gatwick edge, those are not borrow-checked. Treat the compiler as a hygiene layer, not an SLA. Production hardening still belongs to you.

Async Rust is where the real footguns live

Tokio makes a async fn feel free. It is not. Every .await is a yield point where your task can be parked indefinitely, where a cancelled request can drop a guard mid-flight, and where a panic inside a spawned task can silently take down the worker if you used tokio::spawn without a JoinHandle or a supervisor. NHS Digital's published incident retrospectives repeatedly call out the same shape: a handler that looked correct in isolation, ran fine in staging at 50 RPS, and then wedged at 3,000 RPS on a Tuesday morning.

Three patterns I now treat as mandatory on any new Rust service:

• Bounded everything. mpsc::channel(cap) with a real cap. tokio::sync::Semaphore around downstream calls. bb8 or deadpool pools with hard limits, not "max 1024, lol".
• Explicit cancellation handling. Use tokio_util::sync::CancellationToken on every long-running task. Treat JoinError as a first-class error, not a log line.
• Timeouts at the edge, not the core. Wrap outbound HTTP with reqwest and tokio::time::timeout. Wrap DB calls with sqlx query timeouts. A 30-second default is not a default, it is a latent outage.

A tiny but honest handler

This is the kind of Axum handler that compiles cleanly and still misbehaves in production. Note the version with backpressure and a deadline:

use axum::{extract::State, http::StatusCode, response::IntoResponse, Json};
use std::time::Duration;
use tokio::time::timeout;
use tokio_util::sync::CancellationToken;

#[derive(Clone)]
struct AppState {
    db: sqlx::PgPool,
    shutdown: CancellationToken,
}

// BAD: unbounded, no timeout, no cancellation awareness.
async fn fetch_user_bad(State(st): State<AppState>, id: i64) -> impl IntoResponse {
    let row = sqlx::query!("SELECT email FROM users WHERE id = $1", id)
        .fetch_one(&st.db)
        .await
        .unwrap();
    Json(row.email).into_response()
}

// GOOD: bounded, time-bounded, cancellable.
async fn fetch_user(State(st): State<AppState>, id: i64) -> impl IntoResponse {
    if st.shutdown.is_cancelled() {
        return (StatusCode::SERVICE_UNAVAILABLE, "shutting down").into_response();
    }
    match timeout(Duration::from_millis(250), sqlx::query!(
        "SELECT email FROM users WHERE id = $1", id
    ).fetch_one(&st.db)).await {
        Ok(Ok(row)) => Json(row.email).into_response(),
        Ok(Err(e))  => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
        Err(_)       => (StatusCode::GATEWAY_TIMEOUT, "db timeout").into_response(),
    }
}

Observability is not optional just because you use Rust

The "Rust is fast so we need less monitoring" take is a myth peddled by people who have never paged at 03:00. A Rust service can be CPU-idle and still returning 500s because a shared DashMap has hot-pinned mutex contention, or because the connection pool is exhausted, or because a feature flag silently flipped a code path to a panic-prone branch. The BBC's iPlayer team has talked publicly about instrumenting Rust services with the same rigour as any other backend: OpenTelemetry traces, Prometheus exemplars, structured logs with tracing spans propagated into the HTTP layer.

Concrete things that pay for themselves inside a week:

• Per-route histograms, not just request counts. axum's into_make_service plus tower_http gives you http_requests_total and http_request_duration_seconds with a label for status class. Alert on p99, not averages.
• Tokio runtime metrics. tokio::runtime::Handle::metrics() exposes live num_alive_tasks, worker_busy_duration_total, and remote_schedule_count. Export them. When alive_tasks climbs while busy_duration stays flat, you have a parked-task pile-up, almost always backpressure-shaped.
• Allocations on the hot path. jemalloc via the tikv-jemallocator crate, plus metrics-allocator, gives you allocated_bytes_total. A flat line is a healthy line. A sawtooth that climbs every restart is a leak, and the borrow checker will not notice it.

The ecosystem is younger than your incident playbook

This is the bit UK engineering leads underplay in pitch decks. Go has a decade of operational lore: pprof tooling, well-trodden context cancellation patterns, SRE books. Rust is catching up fast, but the production playbook is still being written. sqlx is excellent but its migration story is rougher than flyway. tonic for gRPC is solid, but load shedding and circuit breaking are still on you, usually via failsafe or a hand-rolled tower layer. rustls has had real CVEs in the last 24 months and you must keep up with point releases, the same way HMRC's payment platforms track OpenSSL advisories as a daily ritual.

What this means practically: budget the same on-call and reliability engineering effort for a Rust service as you would for a JVM service. Do not assume "memory safe" means "operationally safe". Write runbooks. Drill failure modes. Load test against the real Postgres, not a stub. If you cannot articulate what your service does when the DB is at 100% connection saturation, the compiler has not helped you and neither has Rust.

Build with the compiler, run with discipline

Pick Rust because it gives you predictable performance and removes whole classes of CVE. Do not pick it because someone promised it would let you skip the boring parts of backend engineering. The compiler is your floor, not your ceiling. Backpressure, cancellation, timeouts, observability, dependency hygiene, and incident response are still your job. Treat them as such, and Rust in production is genuinely excellent. Skip them, and you have just written a C++ service with extra steps and a smugger team.

FAQ

Is Rust actually safer than Go for backend services?

At the language level, yes: Rust prevents data races and use-after-free at compile time, which Go does not. At the operational level, both languages have similar failure modes: leaks, pool exhaustion, retry storms. Safety is not a reason to skip observability.

What is the most common Rust production incident you see?

Backpressure failures in async pipelines. Unbounded channels, missing timeouts on downstream calls, and tasks piling up on a saturated Postgres pool. The compiler will not catch any of these.

Which crates should every Rust backend standardise on?

For HTTP: axum with tower and tower-http. For async: tokio plus tokio-util for CancellationToken. For DB: sqlx. For resilience: failsafe for circuit breakers. For telemetry: tracing, tracing-subscriber, and opentelemetry. For memory: tikv-jemallocator with metrics-allocator.