Supervisor

Overview

The Supervisor behaviour is the fault-tolerance mechanism that ensures the Prismatic Platform's 404+ agents and services keep running even when individual components fail. Supervisors monitor child processes and automatically restart them according to configurable strategies when crashes occur, implementing the BEAM VM's "let it crash" philosophy where processes are designed to fail fast and recover through supervision rather than defensive error handling.

The Prismatic Platform's supervision tree is hierarchical -- top-level supervisors manage application-level supervisors, which in turn manage domain-specific supervisors, which finally supervise individual GenServer processes. This tree structure isolates failures: a crash in one agent does not propagate to unrelated agents, and the supervision strategy determines whether siblings are restarted as well. The platform's PrismaticSupervisor application provides dependency-aware startup ordering and domain-based supervision grouping, ensuring that storage adapters start before the applications that depend on them.

DynamicSupervisor extends this pattern for the platform's agent system, where agents are started and stopped dynamically based on operational needs. Each agent runs under its own supervision with automatic restart on failure, and the PrismaticAgents.AgentSupervisor tracks all active agents through an ETS-backed registry. This architecture means the platform can add, remove, and recover agents at runtime without affecting the rest of the system.

Key Features

• Restart Strategies: :one_for_one (restart only the failed child), :one_for_all (restart all children when one fails), :rest_for_one (restart the failed child and those started after it)
• Max Restarts: Configurable restart intensity (max_restarts within max_seconds) to prevent infinite restart loops that would mask persistent failures
• Child Specs: Declarative child process configuration with start functions, shutdown timeouts, restart policies (:permanent, :temporary, :transient), and type declarations
• DynamicSupervisor: On-demand child process management for runtime-spawned agents and workers that cannot be known at compile time
• Shutdown Ordering: Graceful shutdown in reverse start order, ensuring dependent processes stop before their dependencies
• Automatic Recovery: Self-healing without manual intervention -- the platform recovers from transient failures (network timeouts, database disconnects) automatically
• PartitionSupervisor: Distributes children across multiple partitions to reduce contention in high-throughput scenarios
• Telemetry Events: Supervisor restart events emit telemetry for monitoring and alerting on process instability

Platform Integration

Supervision trees provide automatic fault recovery across the platform. The agent supervisor manages the entire agent lifecycle through a combination of static and dynamic supervision.

defmodule PrismaticAgents.Supervisor do
  use Supervisor

  def start_link(opts) do
    Supervisor.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @impl true
  def init(_opts) do
    children = [
      {Registry, keys: :unique, name: PrismaticAgents.Registry},
      {DynamicSupervisor, name: PrismaticAgents.AgentSupervisor,
                          strategy: :one_for_one},
      PrismaticAgents.Coordinator,
      PrismaticAgents.HealthMonitor
    ]

    Supervisor.init(children, strategy: :one_for_one, max_restarts: 10, max_seconds: 60)
  end
end

Individual agents are started dynamically and supervised independently. If an agent crashes, only that agent restarts -- the coordinator and other agents continue unaffected:

defmodule PrismaticAgents.AgentManager do
  @doc "Start a new agent under the dynamic supervisor"
  def start_agent(agent_spec) do
    child_spec = {PrismaticAgents.Agent, agent_spec}

    case DynamicSupervisor.start_child(PrismaticAgents.AgentSupervisor, child_spec) do
      {:ok, pid} ->
        Registry.register(PrismaticAgents.Registry, agent_spec.id, pid)
        {:ok, pid}

      {:error, {:already_started, pid}} ->
        {:ok, pid}

      {:error, reason} ->
        {:error, reason}
    end
  end

  @doc "Gracefully stop an agent"
  def stop_agent(agent_id) do
    case Registry.lookup(PrismaticAgents.Registry, agent_id) do
      [{pid, _}] -> DynamicSupervisor.terminate_child(PrismaticAgents.AgentSupervisor, pid)
      [] -> {:error, :not_found}
    end
  end

  @doc "List all running agents with their PIDs and metadata"
  def list_agents do
    DynamicSupervisor.which_children(PrismaticAgents.AgentSupervisor)
    |> Enum.map(fn {_, pid, _, _} -> {pid, GenServer.call(pid, :get_state)} end)
  end
end

The PrismaticSupervisor application adds dependency resolution on top of standard supervision, ensuring correct startup ordering across the umbrella:

defmodule PrismaticSupervisor.DependencyResolver do
  @moduledoc "Resolves startup dependencies between umbrella applications"

  @doc "Compute a valid startup order respecting all dependencies"
  def resolve_start_order(apps) do
    graph = build_dependency_graph(apps)
    case topological_sort(graph) do
      {:ok, ordered} -> {:ok, ordered}
      {:error, :cycle} -> {:error, :circular_dependency}
    end
  end

  @doc "Build a directed graph of application dependencies"
  defp build_dependency_graph(apps) do
    Enum.reduce(apps, %{}, fn app, graph ->
      deps = Application.spec(app, :applications) || []
      Map.put(graph, app, deps)
    end)
  end
end

Architecture

The platform's supervision tree follows a hierarchical structure that maps to domain boundaries, with each layer providing isolation and recovery at its scope.

The restart flow for a failed agent:

Agent crash -> DynamicSupervisor detects exit -> Restart agent process
                                                      |
                                              Agent.init/1 called
                                                      |
                                              State rebuilt from ETS/DB
                                                      |
                                              Agent operational again
                                              (typically < 100ms)

Performance Characteristics

Supervision adds negligible overhead to the platform while providing automatic fault recovery.

Configuration

# Static supervision tree configuration
Supervisor.init(children,
  strategy: :one_for_one,   # Only restart the failed child
  max_restarts: 3,           # Max 3 restarts...
  max_seconds: 5             # ...within 5 seconds before supervisor crashes up
)

# DynamicSupervisor for runtime-managed processes
DynamicSupervisor.init(
  strategy: :one_for_one,
  max_restarts: 100,         # Higher tolerance for dynamic children
  max_seconds: 60,
  max_children: 500          # Limit total dynamic children
)

# Child spec with explicit configuration
%{
  id: MyWorker,
  start: {MyWorker, :start_link, [args]},
  restart: :permanent,       # Always restart (:permanent, :temporary, :transient)
  shutdown: 5_000,           # Milliseconds to wait for graceful shutdown
  type: :worker              # :worker or :supervisor
}

Best Practices

• Design the supervision tree before writing code -- the tree structure determines failure isolation boundaries and must be planned deliberately
• Use :one_for_one by default -- most children are independent; :one_for_all should only be used when children share critical state that becomes inconsistent on partial restart
• Set appropriate max_restarts -- too low and transient failures crash the supervisor; too high and persistent failures cause restart storms that mask bugs
• Use DynamicSupervisor for runtime-spawned processes -- static Supervisor is for processes known at compile time; DynamicSupervisor handles processes created at runtime
• Never link processes across supervision boundaries -- a linked process crash propagates outside the supervisor's control, defeating isolation
• Register supervised processes -- use Registry or named processes so other parts of the system can find and communicate with supervised children
• Use :transient restart for Task-like processes -- processes that should only restart on abnormal exit (not when they complete successfully)
• Monitor supervisor restarts with telemetry -- frequent restarts indicate an underlying issue that supervision is masking rather than resolving

Comparison with Alternatives

OTP Supervisors provide process-level fault isolation within a single BEAM node, complementing container-level orchestration tools like Kubernetes. The Prismatic Platform uses both: OTP Supervisors for intra-node process management and Docker/Fly.io for inter-node deployment orchestration.

The platform's PartitionSupervisor usage in high-throughput paths, such as the telemetry event processing pipeline, distributes work across multiple supervisor partitions to eliminate single-process bottlenecks while maintaining the fault isolation guarantees of standard supervision.

Related Technologies

• GenServer - The primary supervised process implementation for stateful workers
• Erlang/OTP - The supervision framework and OTP design principles that underpin the pattern
• BEAM VM - Process lifecycle management and the "let it crash" philosophy
• ETS - Registry storage for tracking supervised processes by name

Related Apps

• All 90 Prismatic Platform applications use Supervisor trees
• prismatic_web - Web application supervision with endpoint and PubSub children
• prismatic_agents - DynamicSupervisor-managed agent lifecycle for 404+ agents

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator