Telemetry & Metrics

Overview

Observability in the Prismatic Platform is built on Erlang's :telemetry library -- a lightweight, dynamic dispatching library for metrics and instrumentation. Unlike traditional logging-based observability that relies on parsing unstructured text, telemetry provides structured, typed events that are emitted at specific points in the code and consumed by pluggable handlers. This architecture decouples instrumentation (what to measure) from reporting (where to send measurements), enabling the platform to simultaneously feed Prometheus for metrics, OpenTelemetry for distributed traces, and Phoenix LiveView dashboards for real-time visualization -- all from the same instrumentation points.

The Prismatic Platform emits telemetry events from every significant operation across its 90 umbrella applications: agent executions, storage queries, EASM discovery scans, security rating calculations, supervision tree restarts, and event store operations. This comprehensive instrumentation enables the platform to maintain its quality gates with evidence-based performance budgets rather than guesswork.

Why Telemetry Over Traditional Logging

The Logging Problem

Traditional observability relies on Logger.info/1 calls scattered throughout the codebase, producing unstructured text that must be parsed (often with fragile regular expressions) to extract metrics. This approach has fundamental limitations:

1. Performance overhead: String formatting and I/O happen on the hot path, regardless of whether anyone is consuming the logs.
2. Tight coupling: Changing the logging format breaks downstream parsers. Adding new fields requires coordinated changes across producers and consumers.
3. No aggregation semantics: Logs are individual records. Computing aggregates (p99 latency, error rates, throughput) requires external systems that parse and window the log stream.
4. Binary choice: Either you log everything (high overhead, noisy) or log selectively (miss important signals).

The Telemetry Solution

Telemetry inverts the model. The application code emits lightweight events (a list of atoms for the event name, a map of numeric measurements, and a map of metadata). If no handler is attached, the emission cost is effectively zero -- just a pattern match on an empty handler list. If handlers are attached, they receive structured data that can be directly aggregated without parsing.

Telemetry Architecture

The platform's telemetry system operates in three layers: emission (instrumentation points in application code), routing (the :telemetry dispatch mechanism), and consumption (handlers that transform events into metrics, traces, and alerts).

flowchart TD
    Layer1["Layer 1 — Emission<br/><i>Application Code</i><br/>:telemetry.execute/3 or :telemetry.span/3"]
    Layer2["Layer 2 — Routing<br/><i>:telemetry library</i><br/>Pattern-matched dispatch to all attached handlers"]
    Metrics["Metrics Handler"]
    Traces["Traces Handler"]
    Alerts["Alerts Handler"]
    Dash["Dashboard Handler"]
    Audit["Audit Log Handler"]
    Prom["Prometheus"]
    OTLP["OTLP"]
    PD["PagerDuty"]
    LV["LiveView"]
    ES["Event Store"]

    Layer1 --> Layer2
    Layer2 --> Metrics
    Layer2 --> Traces
    Layer2 --> Alerts
    Layer2 --> Dash
    Layer2 --> Audit
    Metrics --> Prom
    Traces --> OTLP
    Alerts --> PD
    Dash --> LV
    Audit --> ES

    classDef emit fill:#1e293b,stroke:#3b82f6,color:#fff
    classDef route fill:#1e293b,stroke:#a855f7,color:#fff
    classDef handler fill:#0f172a,stroke:#10b981,color:#fff
    classDef sink fill:#0f172a,stroke:#475569,color:#e2e8f0
    class Layer1 emit
    class Layer2 route
    class Metrics,Traces,Alerts,Dash,Audit handler
    class Prom,OTLP,PD,LV,ES sink

Event Naming Convention

The Prismatic Platform follows a hierarchical naming convention for telemetry events that mirrors the umbrella application structure:

[:prismatic, <app_domain>, <operation>, <lifecycle_stage>]

Where:

• app_domain identifies the umbrella application (:perimeter, :agents, :storage)
• operation identifies the specific operation (:discovery, :scan, :query)
• lifecycle_stage is one of :start, :stop, :exception

This convention enables both fine-grained subscriptions (e.g., only perimeter discovery events) and broad subscriptions (e.g., all :stop events across all domains for latency monitoring).

Instrumentation Patterns

The Span Pattern

The most common instrumentation pattern wraps an operation in :telemetry.span/3, which automatically emits :start, :stop, and :exception events with duration measurements.

defmodule PrismaticPerimeter.Discovery.Engine do
  @moduledoc "Asset discovery engine with comprehensive telemetry instrumentation."

  @spec discover(String.t()) :: {:ok, Surface.t()} | {:error, term()}
  def discover(domain) do
    :telemetry.span(
      [:prismatic, :perimeter, :discovery],
      %{domain: domain},
      fn ->
        case do_discover(domain) do
          {:ok, surface} = result ->
            measurements = %{
              assets_found: length(surface.assets),
              domains_resolved: surface.domains_resolved,
              certificates_found: surface.certificates_found
            }
            {result, measurements}

          {:error, _reason} = error ->
            {error, %{assets_found: 0, error: true}}
        end
      end
    )
  end
end

This single instrumentation point automatically produces three events:

Manual Emission for Custom Events

For events that do not fit the start/stop lifecycle, the platform uses direct :telemetry.execute/3:

defmodule PrismaticPerimeter.Rating.Calculator do
  @spec calculate_rating(String.t()) :: {:ok, Rating.t()} | {:error, term()}
  def calculate_rating(domain) do
    start_time = System.monotonic_time()

    with {:ok, factors} <- collect_rating_factors(domain),
         {:ok, score} <- compute_score(factors),
         grade <- score_to_grade(score) do

      rating = %Rating{domain: domain, grade: grade, score: score, factors: factors}

      :telemetry.execute(
        [:prismatic, :perimeter, :rating, :calculated],
        %{
          duration: System.monotonic_time() - start_time,
          score: score,
          factor_count: length(factors)
        },
        %{
          domain: domain,
          grade: grade,
          previous_grade: get_previous_grade(domain)
        }
      )

      {:ok, rating}
    end
  end
end

Supervision Tree Instrumentation

The supervision trees emit telemetry events for every lifecycle action, enabling monitoring of restart rates and process health.

defmodule PrismaticTelemetry.SupervisorHandler do
  @moduledoc """
  Handles supervisor telemetry events and converts them into
  platform-standard metrics for monitoring restart rates and
  detecting supervisor instability.
  """

  @spec handle_event([atom()], map(), map(), term()) :: :ok
  def handle_event(
    [:supervisor, :restart_child, :stop],
    %{duration: duration},
    %{id: child_id, supervisor: sup_name, reason: reason},
    _config
  ) do
    :telemetry.execute(
      [:prismatic, :supervision, :restart],
      %{duration: duration, count: 1},
      %{
        supervisor: sup_name,
        child: child_id,
        reason: categorize_reason(reason)
      }
    )
  end

  defp categorize_reason(:normal), do: :expected
  defp categorize_reason(:shutdown), do: :expected
  defp categorize_reason({:shutdown, _}), do: :expected
  defp categorize_reason(_), do: :unexpected
end

Metrics Pipeline

Prometheus Reporter Configuration

The platform defines its metrics using the Telemetry.Metrics DSL, which maps telemetry events to Prometheus metric types.

defmodule PrismaticTelemetry.Metrics do
  @moduledoc """
  Defines all platform metrics as Telemetry.Metrics specifications.
  These are consumed by TelemetryMetricsPrometheus for exposition.
  """

  import Telemetry.Metrics

  @spec metrics() :: [Telemetry.Metrics.t()]
  def metrics do
    [
      # ---- Perimeter Metrics ----
      counter("prismatic.perimeter.discovery.count",
        tags: [:domain],
        description: "Total number of discovery operations"
      ),
      distribution("prismatic.perimeter.discovery.duration",
        tags: [:domain],
        unit: {:native, :millisecond},
        reporter_options: [buckets: [10, 50, 100, 250, 500, 1000, 5000]],
        description: "Discovery operation duration in milliseconds"
      ),
      sum("prismatic.perimeter.discovery.assets_found",
        tags: [:domain],
        description: "Total assets discovered"
      ),

      # ---- Agent Metrics ----
      counter("prismatic.agents.execution.count",
        tags: [:agent_type],
        description: "Total agent executions"
      ),
      distribution("prismatic.agents.execution.duration",
        tags: [:agent_type],
        unit: {:native, :millisecond},
        reporter_options: [buckets: [1, 5, 10, 50, 100, 500, 1000]],
        description: "Agent execution duration"
      ),
      last_value("prismatic.agents.active.count",
        description: "Currently active agents"
      ),

      # ---- Storage Metrics ----
      distribution("prismatic.storage.query.duration",
        tags: [:adapter, :operation],
        unit: {:native, :millisecond},
        reporter_options: [buckets: [1, 5, 10, 25, 50, 100]],
        description: "Storage query duration"
      ),
      counter("prismatic.storage.query.count",
        tags: [:adapter, :operation],
        description: "Total storage queries"
      ),
      last_value("prismatic.storage.pool.size",
        tags: [:adapter],
        description: "Connection pool size"
      ),
      last_value("prismatic.storage.pool.available",
        tags: [:adapter],
        description: "Available pool connections"
      ),

      # ---- Supervision Metrics ----
      counter("prismatic.supervision.restart.count",
        tags: [:supervisor, :reason],
        description: "Supervisor child restart count"
      ),
      distribution("prismatic.supervision.restart.duration",
        tags: [:supervisor],
        unit: {:native, :microsecond},
        description: "Restart duration"
      ),

      # ---- Event Store Metrics ----
      counter("prismatic.events.appended.count",
        tags: [:stream_type, :event_type],
        description: "Events appended to the store"
      ),
      distribution("prismatic.events.append.duration",
        unit: {:native, :millisecond},
        description: "Event append latency"
      ),

      # ---- VM Metrics ----
      last_value("vm.memory.total", unit: :byte),
      last_value("vm.memory.processes", unit: :byte),
      last_value("vm.memory.ets", unit: :byte),
      last_value("vm.total_run_queue_lengths.total"),
      last_value("vm.system_counts.process_count")
    ]
  end
end

Application Startup

The telemetry system is initialized early in the application supervision tree to ensure all events are captured from the first moment of operation.

defmodule PrismaticTelemetry.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Prometheus exporter (must start before any event emission)
      {TelemetryMetricsPrometheus,
       metrics: PrismaticTelemetry.Metrics.metrics(),
       name: :prismatic_metrics,
       port: 9568},

      # VM metrics poller (emits vm.* events every 5 seconds)
      {:telemetry_poller,
       measurements: [
         {PrismaticTelemetry.VMPoller, :dispatch, []},
         {PrismaticTelemetry.PoolPoller, :dispatch, []}
       ],
       period: :timer.seconds(5),
       name: :prismatic_poller},

      # Custom event handlers
      {PrismaticTelemetry.AlertHandler, []}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: PrismaticTelemetry.Supervisor)
  end
end

Complete Telemetry Event Catalog

Perimeter Events

Agent Events

Storage Events

Event Store Events

Distributed Tracing with OpenTelemetry

While :telemetry provides metrics, OpenTelemetry provides distributed tracing -- the ability to follow a single request as it traverses multiple processes and services. The Prismatic Platform integrates both, using :telemetry events as span boundaries within OpenTelemetry traces.

Trace Context Architecture

defmodule PrismaticTelemetry.Tracing do
  @moduledoc """
  OpenTelemetry integration for distributed tracing across
  umbrella applications. Trace context is propagated through
  process dictionary and explicit context passing.
  """
  require OpenTelemetry.Tracer, as: Tracer

  @spec trace_discovery(String.t()) :: {:ok, Surface.t()} | {:error, term()}
  def trace_discovery(domain) do
    Tracer.with_span "prismatic.perimeter.discovery",
      attributes: %{"perimeter.domain" => domain} do

      # Phase 1: DNS enumeration
      assets = Tracer.with_span "prismatic.perimeter.dns_enum" do
        PrismaticPerimeter.Discovery.DNS.enumerate(domain)
      end

      Tracer.set_attribute("perimeter.assets_count", length(assets))

      # Phase 2: Certificate transparency
      certs = Tracer.with_span "prismatic.perimeter.cert_transparency" do
        PrismaticPerimeter.Discovery.CertTransparency.search(domain)
      end

      # Phase 3: Parallel asset scanning
      Tracer.with_span "prismatic.perimeter.scan_all",
        attributes: %{"perimeter.scan_count" => length(assets)} do

        Task.Supervisor.async_stream_nolink(
          PrismaticAgents.TaskSupervisor,
          assets,
          fn asset ->
            # Propagate trace context into spawned tasks
            ctx = OpenTelemetry.Ctx.get_current()
            Tracer.with_span "prismatic.perimeter.scan_asset",
              attributes: %{"asset.id" => asset.id} do
              PrismaticPerimeter.Scanner.scan(asset)
            end
          end,
          max_concurrency: 20,
          timeout: 30_000
        )
        |> Enum.to_list()
      end
    end
  end
end

Cross-Process Context Propagation

When work is distributed across processes (as with agent executions), trace context must be explicitly propagated:

defmodule PrismaticAgents.TracedExecution do
  @moduledoc """
  Wraps agent execution with OpenTelemetry trace context propagation.
  Ensures that agent work appears as child spans within the initiating trace.
  """
  require OpenTelemetry.Tracer, as: Tracer

  @spec execute_with_context(Agent.t(), map()) :: {:ok, term()} | {:error, term()}
  def execute_with_context(agent, context) do
    # Attach the parent trace context in the new process
    otel_ctx = Map.get(context, :otel_ctx, OpenTelemetry.Ctx.new())
    prev_ctx = OpenTelemetry.Ctx.attach(otel_ctx)

    try do
      Tracer.with_span "prismatic.agent.execute",
        attributes: %{
          "agent.id" => agent.id,
          "agent.type" => to_string(agent.type),
          "agent.tier" => to_string(agent.tier)
        } do
        result = agent.module.execute(agent.config)

        Tracer.set_attribute("agent.result",
          case result do
            {:ok, _} -> "success"
            {:error, _} -> "failure"
          end
        )

        result
      end
    after
      OpenTelemetry.Ctx.detach(prev_ctx)
    end
  end
end

Real-Time Dashboard Integration

The telemetry system feeds directly into Phoenix LiveView dashboards, providing real-time visualization without polling external metric stores.

defmodule PrismaticWeb.MetricsLive do
  @moduledoc """
  Real-time metrics dashboard using LiveView.
  Subscribes to telemetry events through PubSub bridge
  and updates the UI in real-time.
  """
  use PrismaticWeb, :live_view

  @refresh_interval_ms 5_000

  @impl true
  def mount(_params, _session, socket) do
    if connected?(socket) do
      # Subscribe to metric summary updates
      Phoenix.PubSub.subscribe(PrismaticPubSub, "metrics:summary")
      Process.send_after(self(), :compute_kpis, @refresh_interval_ms)
    end

    {:ok, assign(socket,
      kpis: compute_kpis(),
      discovery_latency_p99: 0,
      agent_success_rate: 100.0,
      active_agents: 0,
      events_per_second: 0
    )}
  end

  @impl true
  def handle_info(:compute_kpis, socket) do
    Process.send_after(self(), :compute_kpis, @refresh_interval_ms)
    kpis = compute_kpis()

    {:noreply, assign(socket,
      kpis: kpis,
      discovery_latency_p99: kpis.discovery_p99_ms,
      agent_success_rate: kpis.agent_success_rate,
      active_agents: kpis.active_agents,
      events_per_second: kpis.events_per_second
    )}
  end

  defp compute_kpis do
    %{
      discovery_p99_ms: PrismaticTelemetry.Aggregator.percentile(:discovery_duration, 99),
      agent_success_rate: PrismaticTelemetry.Aggregator.rate(:agent_success, :agent_total),
      active_agents: PrismaticTelemetry.Aggregator.gauge(:active_agents),
      events_per_second: PrismaticTelemetry.Aggregator.throughput(:events_appended, :second),
      storage_pool_usage: PrismaticTelemetry.Aggregator.gauge(:pool_checked_out),
      supervisor_restart_rate: PrismaticTelemetry.Aggregator.rate(:restarts, :minute)
    }
  end
end

Alerting Pipeline

The telemetry system drives the platform's alerting through dedicated handlers that evaluate conditions and dispatch notifications.

defmodule PrismaticTelemetry.AlertHandler do
  @moduledoc """
  Evaluates telemetry events against alert rules and dispatches
  notifications for threshold violations.
  """
  use GenServer

  @alert_rules [
    %{
      event: [:prismatic, :perimeter, :discovery, :stop],
      condition: fn measurements, _meta ->
        measurements.duration > System.convert_time_unit(10, :second, :native)
      end,
      severity: :warning,
      message: "Discovery took over 10 seconds"
    },
    %{
      event: [:prismatic, :agents, :execution, :exception],
      condition: fn _measurements, meta -> meta.kind == :error end,
      severity: :critical,
      message: "Agent execution failed with error"
    },
    %{
      event: [:prismatic, :supervision, :restart],
      condition: fn _measurements, meta -> meta.reason == :unexpected end,
      severity: :warning,
      message: "Unexpected supervisor restart detected"
    }
  ]

  def init(_) do
    Enum.each(@alert_rules, fn rule ->
      :telemetry.attach(
        "alert-#{:erlang.phash2(rule)}",
        rule.event,
        &__MODULE__.evaluate/4,
        rule
      )
    end)

    {:ok, %{}}
  end

  def evaluate(event_name, measurements, metadata, rule) do
    if rule.condition.(measurements, metadata) do
      PrismaticNotifications.dispatch(%{
        severity: rule.severity,
        event: event_name,
        message: rule.message,
        measurements: measurements,
        metadata: metadata,
        timestamp: DateTime.utc_now()
      })
    end
  end
end

Performance Characteristics

The telemetry system is designed for negligible overhead. Measurements from the Prismatic Platform confirm sub-microsecond emission costs.

The key design constraint is that telemetry handlers must not perform blocking I/O. Handlers that need to send data to external systems (Prometheus, Jaeger, PagerDuty) do so through dedicated processes with buffering and backpressure, ensuring that a slow external system never impacts application latency.

Comparison with Alternative Observability Approaches

The Prismatic Platform uses the :telemetry + OpenTelemetry combination, getting the lowest possible emission cost for metrics through :telemetry while gaining distributed tracing capabilities through OpenTelemetry span integration.

Summary

The telemetry and metrics architecture of the Prismatic Platform provides comprehensive observability with sub-microsecond emission overhead. By building on Erlang's :telemetry library for structured event dispatch, Prometheus for metric aggregation and alerting, OpenTelemetry for distributed tracing, and Phoenix LiveView for real-time dashboards, the platform achieves visibility into every operation without sacrificing performance. The hierarchical event naming convention, aligned with the umbrella application structure, enables both surgical investigation of specific subsystems and platform-wide health monitoring. Combined with the supervision tree instrumentation and event store metrics, the telemetry system ensures that every aspect of the platform's behavior is measurable, alertable, and debuggable in production.

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator