Supervisors

A supervisor is a long-lived Tokio task that the framework starts at boot and restarts automatically when it exits. Supervisors are for "always-on" work: background heartbeats, metrics collectors, connection warmers, periodic sweepers, or any async loop that should never stop running. They are distinct from queue workers, which consume discrete Job items from a queue. A supervisor has no job queue - it owns its own loop and decides when to sleep, wait, or act.

The SupervisorRegistry starts every registered supervisor as a detached Tokio task, watches each task's JoinHandle, and restarts it according to its RestartPolicy when it exits - whether by returning Err, returning Ok, or panicking. Restarts are separated by an exponential backoff that starts at 100ms and caps at 60 seconds, so a crashing supervisor does not spin-loop and flood logs.

Quick Start

Define a supervisor, register it via inventory::submit!, and call SupervisorRegistry::start_all() at bootstrap.

src/supervisors/heartbeat.rs:

use async_trait::async_trait;
use std::time::Duration;
use suprnova::supervisor::{RestartPolicy, Supervisor};
use suprnova::{FrameworkError, SupervisorEntry};
use tokio_util::sync::CancellationToken;

pub struct LogHeartbeat;

#[async_trait]
impl Supervisor for LogHeartbeat {
    fn name(&self) -> &'static str { "heartbeat" }

    async fn run(&self, cancel: CancellationToken) -> Result<(), FrameworkError> {
        loop {
            tokio::select! {
                _ = cancel.cancelled() => return Ok(()),
                _ = tokio::time::sleep(Duration::from_secs(60)) => {
                    tracing::info!("supervisor heartbeat tick");
                }
            }
        }
    }

    fn restart_policy(&self) -> RestartPolicy { RestartPolicy::Always }
}

// Use the re-exported `suprnova::inventory` so a scaffolded app doesn't need
// to add `inventory` as a direct dependency.
suprnova::inventory::submit!(SupervisorEntry {
    factory: || Box::new(LogHeartbeat),
});

src/bootstrap.rs:

use suprnova::supervisor::SupervisorRegistry;

pub async fn register() {
    SupervisorRegistry::start_all().await;
}

That is the full setup. The LogHeartbeat supervisor starts at boot, logs every 60 seconds, and - because RestartPolicy::Always restarts on both Ok and Err exits - is restarted immediately if the loop ever exits for any reason.

Restart Policies

Each supervisor declares its RestartPolicy via the trait method. The default is OnError.

fn restart_policy(&self) -> RestartPolicy { RestartPolicy::OnError }   // default
fn restart_policy(&self) -> RestartPolicy { RestartPolicy::Always }    // daemon loop
fn restart_policy(&self) -> RestartPolicy { RestartPolicy::Never }     // one-shot

When to pick Always vs OnError. An infinite loop supervisor (loop { ... }) should use Always - if the loop ever returns Ok(()), something unexpected happened and a restart is the correct response. A supervisor that does finite work and returns Ok on success (e.g., refreshing a cache once) should use OnError so that a clean finish does not trigger a restart.

Never for one-shot work. Prefer queue workers or scheduled tasks for work that runs on a schedule. Use RestartPolicy::Never when the supervisor pattern is convenient for something that must run once at startup and never again.

Panic Handling

Panics inside run() are caught by the registry and treated as errors - a panicking supervisor is restarted with backoff rather than crashing the process. The registry monitors each supervisor's JoinHandle and detects panics via the standard Tokio join mechanism.

From the restart-policy perspective, a panic is always treated as an Err exit regardless of the policy:

• OnError - restarts after a panic (panic counts as error).
• Always - restarts after a panic (same as any other exit).
• Never - does not restart after a panic (same as any other exit).

The panic is logged at error! level with the supervisor name before the restart backoff begins.

Backoff

When a supervisor exits and its policy says to restart, the registry waits before spawning the replacement:

The backoff resets after a healthy run. The delay doubles on each consecutive restart up to the 60 s cap, but a run that stays up at least 60 s (the cap duration) is treated as healthy: the next restart drops back to the 100 ms floor instead of inheriting backoff that climbed during an earlier burst of failures. So a daemon that ran cleanly for hours and then blips restarts promptly, not after a 60 s wait it accumulated long ago.

The reset is liveness-based, and deliberately conservative: only a run that outlives the maximum possible backoff counts as healthy. A run that exits before that threshold carries the current backoff forward, so a genuinely flapping supervisor - one whose runs never reach the threshold - still ramps all the way to the 60 s cap and stays there. The reset never masks a supervisor that is crash-looping.

The 60-second cap prevents a permanently-broken supervisor from sleeping indefinitely or hammering external dependencies on every retry. Combine with error!-level logging to alert when a supervisor enters the high-backoff band.

Graceful Shutdown

Supervisors receive a CancellationToken as a parameter to run(). The framework cancels this token on Ctrl-C / SIGTERM as part of Server::run's shutdown sequence. Supervisors that want to flush state, finish in-flight work, or otherwise exit cleanly should tokio::select! on cancel.cancelled():

async fn run(&self, cancel: CancellationToken) -> Result<(), FrameworkError> {
    loop {
        tokio::select! {
            _ = cancel.cancelled() => return Ok(()),
            _ = tokio::time::sleep(Duration::from_secs(60)) => {
                tracing::info!("supervisor heartbeat tick");
            }
        }
    }
}

The framework drains the supervisor JoinSet with a 5-second grace window after cancellation. Supervisors that do not honor the token within that window get aborted via JoinSet::abort_all. The drain runs after the WebSocket handler drain (so WS connections clean up first) and before telemetry buffers flush.

Supervisors that ignore the token entirely will run until the 5-second window expires and then be forcibly aborted. If your supervisor holds resources that need flushing (open file handles, in-flight HTTP requests, partially-written records), always select on cancel.cancelled() and clean up before returning.

Embedders and integration tests

Server::run calls SupervisorRegistry::shutdown(...) for you. Code that calls SupervisorRegistry::start_all() outside of Server::run (embedders driving the framework from a custom binary, or integration tests that spin up supervisors directly) must also call SupervisorRegistry::shutdown(timeout) at teardown, or supervisor tasks will leak past the lifetime of the test:

use std::time::Duration;
use suprnova::SupervisorRegistry;

// Test setup
SupervisorRegistry::start_all().await;

// ... exercise the supervisor ...

// Test teardown - cancels the shared token, drains the JoinSet up
// to `timeout`, then `abort_all` for stragglers.
SupervisorRegistry::shutdown(Duration::from_secs(1)).await;

shutdown is a no-op if start_all was never called, so it is safe to call from teardown unconditionally.

Observability

Every error-path restart emits an error!-level log entry with structured fields:

• supervisor - from Supervisor::name().
• error - the error message from run()'s Err return value, or "panic: <payload>" for a caught panic, or "join error: <detail>" for an unusual join failure.
• backoff_ms - the backoff delay in milliseconds before the next spawn.

Panics are reported through the same error log - there is no separate "panicked" message:

ERROR suprnova::supervisor: supervisor errored; restarting after backoff supervisor=heartbeat error=connection refused backoff_ms=400
ERROR suprnova::supervisor: supervisor errored; restarting after backoff supervisor=heartbeat error="panic: \"deliberate test panic\"" backoff_ms=800

RestartPolicy::Always returning Ok(()) emits a warn! (not error!) with the same supervisor / backoff_ms fields and the message "supervisor returned Ok under Always policy; restarting" - useful for spotting daemon loops that exited cleanly when they shouldn't have.

Supervisors do not get an automatic tracing span around run() - the registry spans the lifecycle (start, restart) but not the interior of the task. Emit your own info_span! or instrument your loop body if you want span context on work done inside the supervisor:

async fn run(&self, cancel: CancellationToken) -> Result<(), FrameworkError> {
    loop {
        tokio::select! {
            _ = cancel.cancelled() => return Ok(()),
            _ = async {
                let span = tracing::info_span!("heartbeat.tick");
                let _guard = span.enter();
                do_work().await.ok();
                tokio::time::sleep(Duration::from_secs(60)).await;
            } => {}
        }
    }
}

Why Suprnova diverges

Laravel has no direct equivalent. PHP's request-per-process model makes always-on in-process daemons impossible - long-lived work has to live outside the request lifecycle, typically as a supervisord-managed worker process consuming a queue or a cron-scheduled command. Laravel's queue worker (php artisan queue:work) is the closest analogue, but it is still a one-shot CLI process that an external supervisor restarts.

Suprnova runs on Tokio inside a single long-lived process. Always-on background tasks fit naturally as supervised Tokio tasks alongside the HTTP server - no extra process boundary, no external supervisor, no separate IPC channel for state. The Supervisor trait is the in-process equivalent of supervisord, scoped to the framework's own task tree, with the same restart-on-exit + backoff guarantees.

Queue workers (which Laravel has) still ship - see Queues - for discrete-job work. Supervisors cover the "always tick" case that Laravel pushes out of the framework boundary entirely.

Out of v1 Scope

The following items are intentionally deferred:

• Supervisor trees (parent/child). There is no hierarchy - all supervisors are peers under the single SupervisorRegistry. Structured supervision (where one supervisor owns and restarts child supervisors) is orchestrator territory.

• Resource limits (cgroup, memory, CPU). Apply resource constraints through systemd unit files (MemoryMax=, CPUQuota=) or Kubernetes resource requests/limits at the pod level. The framework does not impose process-internal resource limits on individual supervisor tasks.

• Multi-machine supervision. Supervisors run within a single process on a single machine. Distributing supervision decisions across machines is orchestrator territory (Kubernetes, Nomad, systemd on multiple hosts).

Reference

The four primary types - Supervisor, RestartPolicy, SupervisorEntry, SupervisorRegistry - are re-exported at the crate root (suprnova::Supervisor, etc.) in addition to the longer suprnova::supervisor::* path. The two free accessors stay under suprnova::supervisor::*.

Next

• Queues - supervisor-vs-queue-worker decision and the discrete-job alternative
• Scheduling - for periodic work that doesn't need a long-lived loop
• Workflows - for stateful, long-running work that needs durable resume
• Broadcasting - uses the same shutdown sequence (drain ordering)
• Request Lifecycle - where Server::run and the shutdown drain fit in