Overview

Performance tracking is the discipline of systematically measuring, recording, analyzing, and acting on performance data generated by software systems in development, staging, and production environments. It encompasses everything from low-level function execution timing to high-level user experience metrics, providing the quantitative foundation for reliability engineering, capacity planning, and service-level objective (SLO) compliance.

In the Prismatic Platform, performance tracking is not optional -- it is a P0 absolute requirement. Every page must load under 250ms, every server-side render must complete under 100ms, and every LiveView event handler must respond within 50ms. These are not aspirational targets; they are enforced through quality gates, pre-commit hooks, and CI/CD pipeline checks that block deployment of any code violating these thresholds.

This entry covers the theory, methodologies, tools, and implementation patterns that make rigorous performance tracking possible in Elixir/Phoenix applications and beyond.

Definition and Importance

Performance tracking is the continuous process of collecting, aggregating, and analyzing quantitative measurements of system behavior -- latency, throughput, error rates, resource utilization -- to ensure that applications meet defined service-level objectives and to detect degradation before it impacts users.

The importance of performance tracking in production systems cannot be overstated:

User experience: Research consistently shows that every 100ms of additional latency reduces conversion rates by 1-7%. Users abandon pages that take more than 3 seconds to load.
Revenue impact: Amazon famously calculated that every 100ms of latency cost them 1% in sales. Google found that an extra 500ms in search results reduced traffic by 20%.
Reliability: Performance degradation is often the first symptom of deeper issues -- memory leaks, database contention, network saturation, or cascading failures.
Capacity planning: Historical performance data enables accurate forecasting of infrastructure needs, preventing both over-provisioning (wasted cost) and under-provisioning (outages).
SLO compliance: Modern platform engineering requires quantifiable service-level objectives backed by real measurement data, not assumptions.

Historical Context

Performance tracking has evolved dramatically from its origins in mainframe-era system monitoring.

1960s-1970s -- System Monitors: Early mainframe operating systems provided rudimentary CPU and memory utilization metrics. Operators manually inspected printouts and console displays.

1980s -- Network Monitoring: SNMP (Simple Network Management Protocol) enabled automated polling of network device metrics. Tools like MRTG visualized bandwidth utilization over time.

1990s -- Application Performance Management: The rise of web applications drove demand for application-level monitoring. Companies like Wily Technology (later acquired by CA) pioneered agent-based APM that instrumented Java bytecode to track transaction response times.

2000s -- Distributed Tracing Origins: Google's Dapper paper (2010) and its open-source descendants (Zipkin, Jaeger) introduced distributed tracing -- the ability to follow a request across multiple services and correlate latency contributions.

2010s -- Observability Movement: The term "observability" (borrowed from control theory) gained traction, expanding beyond metrics to encompass logs, traces, and events as complementary signals. The RED and USE methods provided structured frameworks for what to measure.

2020s -- OpenTelemetry and Convergence: OpenTelemetry emerged as the vendor-neutral standard for instrumentation, unifying metrics, traces, and logs under a single API. Elixir's :telemetry library brought the same principles to the BEAM ecosystem with minimal overhead.

Core Concepts: Key Metrics

Latency Percentiles

Latency -- the time between a request being sent and a response being received -- is the most user-visible performance metric. Raw averages are misleading because they hide tail latency. Percentile measurements provide a complete picture:

P50 (median): Half of requests are faster than this value. Represents the "typical" user experience.
P95: 95% of requests are faster. Captures the experience of users on slower connections or hitting more complex code paths.
P99: 99% of requests are faster. Reveals tail latency caused by garbage collection pauses, cold caches, or database lock contention.
P99.9: Critical for high-traffic systems where even 0.1% of slow requests affects thousands of users.

The Prismatic Platform enforces specific latency budgets:

Component	P95 Limit	Enforcement
Total page load	250ms	Merge blocked
Server-side render	100ms	Merge blocked
LiveView mount	150ms	Merge blocked
LiveView handle_event	50ms	Merge blocked
Health check	10ms	Merge blocked

Throughput

Throughput measures the rate at which a system processes requests, typically expressed as requests per second (RPS) or transactions per second (TPS). Throughput and latency are related but distinct: a system can have low latency but also low throughput (processing few requests, each quickly) or high throughput with high latency (processing many requests, each slowly due to queuing).

Error Rates

The percentage of requests that result in errors (HTTP 5xx, exceptions, timeouts). Error rates should be tracked both globally and per-endpoint. A spike in error rate often precedes a latency spike as retry logic kicks in.

Saturation

How "full" a resource is -- CPU utilization, memory consumption, disk I/O queue depth, connection pool usage, message queue backlog. Saturation metrics are leading indicators: they predict future problems before they manifest as latency or errors.

RED and USE Methodologies

RED Method (Request-Scoped)

Developed by Tom Wilkie at Grafana Labs, RED focuses on request-driven services:

Rate: Requests per second being handled
Errors: Number of failed requests per second
Duration: Distribution of request latency (histograms, not averages)

RED is ideal for microservices and web applications where the unit of work is a discrete request. In Phoenix applications, RED metrics map naturally to endpoint-level telemetry events.

USE Method (Resource-Scoped)

Developed by Brendan Gregg, USE focuses on infrastructure resources:

Utilization: Percentage of time the resource is busy (or percentage of capacity used)
Saturation: Degree to which the resource has extra work it cannot service (queue length)
Errors: Count of error events for the resource

USE is applied to every physical and logical resource: CPUs, memory, disks, network interfaces, BEAM schedulers, ETS tables, and database connection pools. USE and RED are complementary -- apply RED to services and USE to the resources those services consume.

Technical Deep Dive: Telemetry in Elixir/Phoenix

The Elixir ecosystem provides first-class performance tracking through the :telemetry library -- a lightweight, dynamic dispatching library for metrics and instrumentation built into the BEAM ecosystem.

Architecture

The :telemetry library follows a publish-subscribe pattern:

Instrumentation points emit events with measurements and metadata using :telemetry.execute/3
Handlers attach to event names and process measurements (aggregate, forward to backends, trigger alerts)
Reporters send aggregated metrics to external systems (Prometheus, StatsD, DataDog)

Phoenix, Ecto, and other libraries ship with built-in telemetry events, providing comprehensive observability out of the box.

Core Events in Phoenix

Phoenix emits telemetry events at every critical point in the request lifecycle:

[:phoenix, :endpoint, :start] -- HTTP request received
[:phoenix, :endpoint, :stop] -- HTTP response sent (includes duration)
[:phoenix, :router_dispatch, :start] -- Route matched
[:phoenix, :live_view, :mount, :start | :stop] -- LiveView mount lifecycle
[:phoenix, :live_view, :handle_event, :start | :stop] -- LiveView event handling
[:phoenix, :channel_joined] -- WebSocket channel joined

Ecto Database Events

[:prismatic, :repo, :query] -- Database query executed (includes query time, decode time, queue time)

These events provide the raw data needed to enforce the Prismatic Platform's performance standards.

Prismatic Platform Implementation

Performance Policy Enforcement

The Prismatic Platform enforces performance standards through multiple layers:

# Performance check mix task (simplified)
defmodule Mix.Tasks.Performance.Check do
  @moduledoc "Validates page load performance against P0 standards."
  use Mix.Task

  @page_load_limit_ms 250
  @server_render_limit_ms 100
  @live_view_mount_limit_ms 150
  @handle_event_limit_ms 50

  @impl Mix.Task
  def run(_args) do
    results = run_benchmarks()

    violations =
      Enum.filter(results, fn result ->
        result.p95_ms > limit_for(result.category)
      end)

    case violations do
      [] ->
        Mix.shell().info("All performance checks passed.")

      violations ->
        for v <- violations do
          Mix.shell().error(
            "VIOLATION: #{v.name} P95=#{v.p95_ms}ms exceeds #{limit_for(v.category)}ms limit"
          )
        end

        Mix.raise("Performance check failed. #{length(violations)} violation(s) detected.")
    end
  end

  defp limit_for(:page_load), do: @page_load_limit_ms
  defp limit_for(:server_render), do: @server_render_limit_ms
  defp limit_for(:live_view_mount), do: @live_view_mount_limit_ms
  defp limit_for(:handle_event), do: @handle_event_limit_ms
end

Telemetry Metrics Configuration

defmodule PrismaticWeb.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def start_link(arg) do
    Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
  end

  @impl Supervisor
  def init(_arg) do
    children = [
      {:telemetry_poller, measurements: periodic_measurements(), period: 10_000},
      {TelemetryMetricsPrometheus, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

  defp metrics do
    [
      # Phoenix endpoint latency distribution
      distribution("phoenix.endpoint.stop.duration",
        unit: {:native, :millisecond},
        reporter_options: [buckets: [10, 25, 50, 100, 150, 250, 500, 1000]],
        tags: [:method, :route]
      ),

      # LiveView mount latency
      distribution("phoenix.live_view.mount.stop.duration",
        unit: {:native, :millisecond},
        tags: [:view]
      ),

      # LiveView event handling latency
      distribution("phoenix.live_view.handle_event.stop.duration",
        unit: {:native, :millisecond},
        tags: [:view, :event]
      ),

      # Database query latency
      distribution("prismatic.repo.query.total_time",
        unit: {:native, :millisecond},
        tags: [:source]
      ),

      # VM metrics
      last_value("vm.memory.total", unit: :byte),
      last_value("vm.total_run_queue_lengths.total"),
      last_value("vm.total_run_queue_lengths.cpu"),
      last_value("vm.total_run_queue_lengths.io"),
      summary("vm.system_counts.process_count")
    ]
  end

  defp periodic_measurements do
    [
      {PrismaticWeb.Telemetry, :dispatch_ets_stats, []},
      {PrismaticWeb.Telemetry, :dispatch_pool_stats, []}
    ]
  end

  def dispatch_ets_stats do
    for table <- :ets.all(), info = :ets.info(table), is_list(info) do
      :telemetry.execute(
        [:prismatic, :ets, :table],
        %{size: info[:size], memory: info[:memory]},
        %{table: info[:name]}
      )
    end
  end

  def dispatch_pool_stats do
    case DBConnection.get_connection_metrics(Prismatic.Repo) do
      {:ok, metrics} ->
        :telemetry.execute(
          [:prismatic, :db_pool],
          %{
            idle: metrics.idle,
            busy: metrics.busy,
            checkout_queue_length: metrics.checkout_queue_length
          },
          %{}
        )

      _ ->
        :ok
    end
  end
end

Custom Telemetry Event Emission

defmodule PrismaticWeb.PerformanceInstrumentation do
  @moduledoc """
  Custom performance instrumentation for critical platform operations.
  Emits telemetry events with measurements and metadata for tracking
  against P0 performance standards.
  """

  def measure_operation(event_name, metadata, fun) when is_function(fun, 0) do
    start_time = System.monotonic_time()

    try do
      result = fun.()

      :telemetry.execute(
        [:prismatic, :performance | event_name],
        %{duration: System.monotonic_time() - start_time},
        Map.put(metadata, :status, :ok)
      )

      {:ok, result}
    rescue
      error ->
        :telemetry.execute(
          [:prismatic, :performance | event_name],
          %{duration: System.monotonic_time() - start_time},
          Map.merge(metadata, %{status: :error, error: Exception.message(error)})
        )

        {:error, error}
    end
  end
end

Benchmarking with Benchee

# Benchmarking critical render paths with Benchee
Benchee.run(
  %{
    "perimeter_dashboard" => fn ->
      conn = Phoenix.ConnTest.build_conn()
      {:ok, _view, html} = live(conn, ~p"/perimeter")
      html
    end,
    "asset_list_render" => fn ->
      conn = Phoenix.ConnTest.build_conn()
      {:ok, _view, html} = live(conn, ~p"/perimeter/assets")
      html
    end,
    "compliance_view" => fn ->
      conn = Phoenix.ConnTest.build_conn()
      {:ok, _view, html} = live(conn, ~p"/perimeter/compliance")
      html
    end
  },
  time: 10,
  memory_time: 2,
  reduction_time: 2,
  formatters: [
    Benchee.Formatters.Console,
    {Benchee.Formatters.HTML, file: "bench/output/performance.html"}
  ]
)

APM Tools and Integration

OpenTelemetry

OpenTelemetry is the vendor-neutral observability framework that the industry has converged on. The Elixir ecosystem provides full OpenTelemetry support:

opentelemetry_api -- API for creating spans and recording metrics
opentelemetry_exporter -- OTLP exporter for sending data to collectors
opentelemetry_phoenix -- Automatic instrumentation for Phoenix endpoints
opentelemetry_ecto -- Automatic instrumentation for Ecto queries
opentelemetry_liveview -- LiveView mount and event instrumentation

Commercial APM

Tool	Strengths	Integration
DataDog	Full-stack observability, APM + infrastructure + logs	`datadog-agent` + StatsD telemetry reporter
New Relic	Transaction tracing, error analytics, browser monitoring	`new_relic_agent` hex package
Grafana Cloud	Open-source stack (Prometheus + Loki + Tempo), cost-effective	Prometheus telemetry reporter
Honeycomb	High-cardinality event analysis, BubbleUp feature	OpenTelemetry OTLP exporter

Choosing an Approach

For the Prismatic Platform, the preferred stack is :telemetry (instrumentation) combined with Prometheus (metrics storage) and Grafana (visualization). This provides full control over data retention, avoids vendor lock-in, and integrates natively with the Elixir ecosystem.

Performance Budgets and Regression Detection

A performance budget is a quantitative limit on a performance metric that must not be exceeded. Budgets transform performance from a subjective concern into an objective, enforceable constraint.

Budget Categories

Time budgets: Maximum allowed latency for specific operations (e.g., 250ms page load)
Size budgets: Maximum payload sizes for JavaScript bundles, CSS, images, and API responses
Resource budgets: Maximum CPU time, memory allocation, or database query count per request
Count budgets: Maximum number of HTTP requests, database queries, or external API calls per page load

Regression Detection

Performance regressions are detected through:

Baseline establishment: Benchee runs capture baseline metrics for critical paths
Threshold comparison: New measurements are compared against baselines with defined tolerance (typically 10-15%)
Statistical significance: Random variation is distinguished from genuine regression using statistical tests
Automated alerting: Violations trigger notifications through monitoring pipelines

The Prismatic Platform's pre-commit hooks (Phase 8) and CI/CD gates enforce performance budgets automatically. Code that introduces a measurable regression is blocked from merging.

Real User Monitoring vs Synthetic Monitoring

Real User Monitoring (RUM)

RUM collects performance data from actual user sessions in production:

Advantages: Reflects real-world conditions (diverse devices, networks, geographic locations), captures long-tail performance issues, provides user-centric metrics (LCP, FID, CLS)
Disadvantages: Requires production traffic, privacy considerations (data collection consent), noisy data requiring statistical aggregation
Implementation: Browser Performance API, Navigation Timing API, or commercial RUM tools (DataDog RUM, Sentry Performance)

Synthetic Monitoring

Synthetic monitoring executes predefined transactions from controlled environments at regular intervals:

Advantages: Consistent baselines, early detection (works before production traffic), controlled conditions, no user privacy concerns
Disadvantages: Does not capture real-world variability, limited to predefined scenarios, may miss issues only triggered by specific user behaviors
Implementation: Lighthouse CI, WebPageTest, Playwright-based synthetic tests, or custom Benchee suites

Combined Strategy

Production systems should use both approaches. Synthetic monitoring catches regressions in CI/CD before deployment. RUM validates that real users experience acceptable performance after deployment. The Prismatic Platform uses Benchee-based synthetic tests in CI and :telemetry-based production monitoring for comprehensive coverage.