Definition

Performance Testing is a systematic engineering discipline within the Prismatic Platform that quantifies application behavior under controlled conditions to validate that latency, throughput, memory consumption, and resource utilization meet defined thresholds. Unlike functional testing, which verifies that software produces correct outputs, performance testing verifies that correct outputs are produced within acceptable time and resource constraints. In the Prismatic Platform, performance testing is not an optional late-stage activity -- it is an integral part of the quality gate pipeline, enforced through automated benchmarks, telemetry-driven assertions, and hard page-load performance limits that block merges when violated.

The platform enforces a P0-absolute performance standard: all pages must load under 250ms total, server-side rendering must complete under 100ms, LiveView mount must complete under 150ms, LiveView event handlers must complete under 50ms, and health checks must respond under 10ms. These are not aspirational targets but blocking merge criteria enforced by mix performance.check and CI pipeline gates. Violations at the 250-500ms level block merges; violations above 500ms trigger immediate rollback.

Interactive Performance Analytics

Quality DNA Evolution & Performance Metrics

Quality DNA Evolution

Performance Thresholds

Platform Fitness Score Progression (Generation 19)

89ms

Avg Page Load

Threshold: 250ms

67ms

Server Render

Threshold: 100ms

134ms

LiveView Mount

Threshold: 150ms

0.9995

Fitness Score

Apex Achievement

Historical Context

Performance testing as an engineering discipline evolved alongside the growth of web applications in the early 2000s. Tools like Apache JMeter (1998), Gatling (2012), and k6 (2017) established the practice of load testing as a pre-deployment activity. However, these tools treat performance testing as a separate phase -- something done before release rather than integrated into the daily development workflow.

The Prismatic Platform's approach to performance testing reflects a generational shift in methodology. Rather than testing performance as a late-stage activity, performance validation is integrated at every stage: micro-benchmarks during development, regression checks during pre-commit, load tests during CI, and continuous monitoring in production. This multi-layer approach was formalized during Generation 10 of the platform's evolution, when the AutoEvolve system began including performance fitness as a dimension of its evolutionary scoring function.

The BEAM virtual machine introduces unique performance testing considerations that do not apply to thread-pooled runtimes. BEAM's preemptive scheduler ensures that no single process can monopolize a scheduler thread for more than a fixed number of reductions (approximately 4000 function calls). This means that latency distribution in BEAM applications tends to have much tighter tail latencies than equivalent applications on the JVM or V8, because garbage collection pauses are per-process (microseconds) rather than global (milliseconds). Performance testing for OTP applications must account for these characteristics, particularly the relationship between process count, scheduler count, and reduction budgets.

Overview

Performance testing in the Prismatic Platform spans four distinct methodologies, each targeting different failure modes:

Benchmark Testing measures the execution time of individual functions and modules in isolation. The platform uses Benchee for micro-benchmarks, providing statistical analysis of execution times including mean, median, standard deviation, and percentile distributions. Benchmarks are run against baseline measurements to detect performance regressions before they reach production.

Load Testing evaluates system behavior under expected concurrent usage patterns. For the Prismatic Platform's LiveView-heavy architecture, this means simulating hundreds of concurrent WebSocket connections, each generating events that trigger server-side re-renders. Load testing validates that the BEAM VM scheduler distribution and process architecture handle concurrent workloads within latency budgets.

Stress Testing pushes the system beyond expected load to identify breaking points and degradation patterns. This reveals how the OTP supervision tree responds to resource exhaustion, whether circuit breakers activate at appropriate thresholds, and how gracefully the system degrades under extreme conditions. The platform's fault tolerance design must ensure that stress conditions cause graceful degradation rather than cascading failures.

Endurance Testing runs the system under sustained load for extended periods to detect memory leaks, ETS table growth, process accumulation, and other time-dependent degradation patterns. This is particularly important for long-running GenServer processes and ETS-backed caches that may accumulate state over time.

Technical Details

Benchee-Based Micro-Benchmarking

The platform uses Benchee as its standard benchmarking library, with conventions for structuring performance tests alongside functional tests:

defmodule Prismatic.Performance.BenchmarkRunner do
  @moduledoc """
  Executes Benchee benchmarks with standardized configuration
  and regression detection against stored baselines.
  """

  @type benchmark_config :: %{
    warmup: pos_integer(),
    time: pos_integer(),
    memory_time: pos_integer(),
    reduction_time: pos_integer(),
    parallel: pos_integer()
  }

  @default_config %{
    warmup: 2,
    time: 5,
    memory_time: 2,
    reduction_time: 2,
    parallel: 1
  }

  @spec run(String.t(), map(), keyword()) :: :ok
  def run(name, scenarios, opts \\ []) do
    config = Map.merge(@default_config, Map.new(opts))

    Benchee.run(
      scenarios,
      warmup: config.warmup,
      time: config.time,
      memory_time: config.memory_time,
      reduction_time: config.reduction_time,
      parallel: config.parallel,
      formatters: [
        {Benchee.Formatters.Console, extended_statistics: true},
        {Benchee.Formatters.HTML, file: "benchmarks/output/#{name}.html"}
      ],
      before_scenario: fn input ->
        :telemetry.execute(
          [:prismatic, :benchmark, :scenario_start],
          %{system_time: System.system_time()},
          %{name: name, input: input}
        )
        input
      end
    )
  end

  @spec compare_to_baseline(String.t(), map()) ::
    {:ok, :within_threshold} | {:regression, map()}
  def compare_to_baseline(name, current_results) do
    case load_baseline(name) do
      {:ok, baseline} ->
        regression_check(baseline, current_results)
      {:error, :no_baseline} ->
        save_baseline(name, current_results)
        {:ok, :within_threshold}
    end
  end

  defp regression_check(baseline, current) do
    threshold = 1.15

    regressions =
      Enum.filter(current, fn {scenario, timing} ->
        baseline_timing = Map.get(baseline, scenario, timing)
        timing.median > baseline_timing.median * threshold
      end)

    case regressions do
      [] -> {:ok, :within_threshold}
      found -> {:regression, Map.new(found)}
    end
  end

  defp load_baseline(name) do
    path = "benchmarks/baselines/#{name}.json"
    case File.read(path) do
      {:ok, content} -> {:ok, Jason.decode!(content, keys: :atoms)}
      {:error, _} -> {:error, :no_baseline}
    end
  end

  defp save_baseline(name, results) do
    path = "benchmarks/baselines/#{name}.json"
    File.mkdir_p!("benchmarks/baselines")
    File.write!(path, Jason.encode!(results, pretty: true))
  end
end

LiveView Performance Assertion

LiveView pages are performance-tested with assertions that enforce the platform's hard latency limits:

defmodule PrismaticWeb.PerformanceTest do
  @moduledoc """
  Performance assertions for LiveView pages.
  Enforces the P0-absolute performance standard:
  - Total page load: < 250ms
  - Server render: < 100ms
  - LiveView mount: < 150ms
  - Event handler: < 50ms
  """

  use ExUnit.Case, async: true

  @max_render_ms 100
  @max_mount_ms 150
  @max_event_ms 50

  @spec assert_render_time(Plug.Conn.t(), String.t(), pos_integer()) :: :ok
  def assert_render_time(conn, path, max_ms \\ @max_render_ms) do
    {elapsed_us, {:ok, _conn}} = :timer.tc(fn ->
      Phoenix.ConnTest.get(conn, path)
    end)

    elapsed_ms = elapsed_us / 1_000

    assert elapsed_ms < max_ms,
      "Render time for #{path} was #{Float.round(elapsed_ms, 2)}ms, " <>
      "exceeding #{max_ms}ms limit"

    :telemetry.execute(
      [:prismatic, :performance_test, :render],
      %{duration_ms: elapsed_ms},
      %{path: path, threshold_ms: max_ms}
    )

    :ok
  end

  @spec assert_mount_time(module(), map(), pos_integer()) :: :ok
  def assert_mount_time(live_module, params, max_ms \\ @max_mount_ms) do
    {elapsed_us, {:ok, _view, _html}} = :timer.tc(fn ->
      live(build_conn(), live_module, params)
    end)

    elapsed_ms = elapsed_us / 1_000

    assert elapsed_ms < max_ms,
      "Mount time for #{inspect(live_module)} was #{Float.round(elapsed_ms, 2)}ms, " <>
      "exceeding #{max_ms}ms limit"

    :ok
  end

  @spec assert_event_time(Phoenix.LiveViewTest.View.t(), String.t(), map(), pos_integer()) :: :ok
  def assert_event_time(view, event, params, max_ms \\ @max_event_ms) do
    {elapsed_us, {:ok, _view, _html}} = :timer.tc(fn ->
      Phoenix.LiveViewTest.render_click(view, event, params)
    end)

    elapsed_ms = elapsed_us / 1_000

    assert elapsed_ms < max_ms,
      "Event #{event} took #{Float.round(elapsed_ms, 2)}ms, " <>
      "exceeding #{max_ms}ms limit"

    :ok
  end
end

Telemetry-Driven Performance Monitoring

The platform integrates performance testing with its telemetry infrastructure to provide continuous production performance validation:

defmodule Prismatic.Performance.TelemetryCollector do
  @moduledoc """
  Collects performance telemetry events and validates them
  against defined thresholds. Produces structured performance
  reports for CI gate evaluation.
  """

  use GenServer

  @type measurement :: %{
    event: [atom()],
    duration_ms: float(),
    timestamp: DateTime.t(),
    metadata: map()
  }

  @thresholds %{
    page_load: 250,
    server_render: 100,
    liveview_mount: 150,
    liveview_event: 50,
    health_check: 10,
    database_query: 50,
    ets_lookup: 1
  }

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @impl GenServer
  def init(_opts) do
    attach_handlers()
    {:ok, %{measurements: [], violations: []}}
  end

  @impl GenServer
  def handle_cast({:measurement, measurement}, state) do
    category = categorize(measurement.event)
    threshold = Map.get(@thresholds, category, 1000)

    updated_state =
      if measurement.duration_ms > threshold do
        violation = %{
          measurement: measurement,
          category: category,
          threshold_ms: threshold,
          exceeded_by_ms: measurement.duration_ms - threshold
        }
        %{state | violations: [violation | state.violations],
                  measurements: [measurement | state.measurements]}
      else
        %{state | measurements: [measurement | state.measurements]}
      end

    {:noreply, updated_state}
  end

  @impl GenServer
  def handle_call(:report, _from, state) do
    report = %{
      total_measurements: length(state.measurements),
      total_violations: length(state.violations),
      violations_by_category: group_violations(state.violations),
      p50: percentile(state.measurements, 50),
      p95: percentile(state.measurements, 95),
      p99: percentile(state.measurements, 99)
    }
    {:reply, report, state}
  end

  defp attach_handlers do
    events = [
      [:phoenix, :endpoint, :stop],
      [:phoenix, :live_view, :mount, :stop],
      [:phoenix, :live_view, :handle_event, :stop],
      [:prismatic, :query, :stop]
    ]

    :telemetry.attach_many(
      "performance-collector",
      events,
      &handle_event/4,
      nil
    )
  end

  defp handle_event(event, measurements, metadata, _config) do
    duration_ms = System.convert_time_unit(
      measurements.duration, :native, :millisecond
    )

    GenServer.cast(__MODULE__, {:measurement, %{
      event: event,
      duration_ms: duration_ms,
      timestamp: DateTime.utc_now(),
      metadata: metadata
    }})
  end

  defp categorize([:phoenix, :endpoint, :stop]), do: :page_load
  defp categorize([:phoenix, :live_view, :mount, :stop]), do: :liveview_mount
  defp categorize([:phoenix, :live_view, :handle_event, :stop]), do: :liveview_event
  defp categorize([:prismatic, :query, :stop]), do: :database_query
  defp categorize(_), do: :unknown

  defp group_violations(violations) do
    Enum.group_by(violations, & &1.category)
    |> Enum.map(fn {cat, vs} -> {cat, length(vs)} end)
    |> Map.new()
  end

  defp percentile(measurements, pct) do
    sorted = Enum.sort_by(measurements, & &1.duration_ms)
    index = round(length(sorted) * pct / 100) - 1
    Enum.at(sorted, max(index, 0), %{duration_ms: 0}).duration_ms
  end
end

Implementation

Performance testing in the Prismatic Platform is integrated at multiple stages of the development lifecycle:

Development Phase: Developers run mix performance.check locally before committing. This executes benchmark suites for changed modules and validates that no performance regressions are introduced. The tool compares current measurements against stored baselines and reports any measurements that exceed the 15% regression threshold.

Pre-Commit Phase: The .githooks/pre-commit hook includes performance regression checks for modules that have known benchmarks. If a changed module has an associated benchmark file and the benchmark shows regression, the commit is blocked.

CI Pipeline Phase: The GitLab CI pipeline runs the full performance test suite, including load tests that simulate concurrent users. Performance violations cause pipeline failures, preventing merge requests with regressions from being merged.

Production Phase: The telemetry infrastructure continuously collects performance measurements from the production deployment. P95 latency exceeding 200ms triggers alerts. Sustained violations trigger automated investigation through the AutoHeal system.

Evolution Phase: The AutoEvolve system includes performance optimization as one of its evolution dimensions. It identifies modules with degrading performance trends and generates optimization recommendations.

Comparison

Approach	Granularity	Automation	Regression Detection	Production Correlation
Manual Load Testing	Coarse (whole system)	Low (human-driven)	Inconsistent	Moderate
APM Tools Only	Fine (per-request)	High (passive collection)	Reactive (detects after deploy)	High
Prismatic Performance Testing	Multi-level (function to system)	Full (CI-integrated)	Proactive (blocks before merge)	High (telemetry-driven)
Property-Based Performance	Medium (behavioral)	High (generative)	Good (statistical)	Moderate
Chaos Engineering	System-level	High (automated injection)	Indirect (resilience, not speed)	High

The Prismatic approach combines micro-benchmarks (Benchee), macro-benchmarks (load testing), telemetry-driven monitoring, and hard CI gates into a unified pipeline. This multi-layer approach catches regressions at the earliest possible stage -- ideally during local development, certainly before merge, and as a safety net in production.

Best Practices

Benchmark the hot path first. Not all code paths warrant benchmarks. Focus on request handling, database queries, ETS lookups, and LiveView render paths -- the code that executes on every user interaction. Use telemetry data from production to identify which paths are actually hot.

Store baselines in version control. Benchmark baselines should be committed alongside the code they measure. This ensures that baseline comparisons are reproducible and that performance expectations evolve with the codebase.

Use statistical analysis, not single measurements. A single benchmark run is meaningless due to JIT warmup, garbage collection pauses, and system load variation. Benchee provides mean, median, standard deviation, and percentile distributions. Use the median for regression detection and P99 for worst-case validation.

Test with realistic data volumes. A function that performs well on 10 records may degrade catastrophically on 10,000. Benchmark with data volumes that represent production reality, including edge cases like empty collections and maximum-size inputs.

Separate performance tests from functional tests. Performance tests are inherently slower and more variable than functional tests. Run them in a dedicated CI stage rather than mixing them into the main test suite. Use mix test --only performance tagging.

Set explicit, documented thresholds. Every performance test should assert against a specific threshold, not just "faster than before." The platform's 250ms page load, 100ms render, 150ms mount, and 50ms event thresholds are documented, enforced, and non-negotiable.

Pitfalls

Benchmarking in non-representative environments. Development machines differ significantly from production infrastructure. BEAM VM performance depends on scheduler count, which correlates with CPU cores. A benchmark on a developer laptop with 4 cores may not predict behavior on a production server with 32 cores. Use consistent CI infrastructure for benchmark comparisons.

Ignoring memory performance. Latency benchmarks alone miss memory regressions. A function that becomes faster by caching more data in memory may cause ETS table bloat or increased garbage collection pressure. Always measure memory alongside time.

Over-optimizing cold paths. Code that executes once during application startup or once per deployment does not benefit meaningfully from micro-optimization. Reserve performance testing effort for code that executes on the request hot path.

Treating performance tests as flaky. Performance test variability is a signal, not noise. If a benchmark produces inconsistent results, the underlying code likely has non-deterministic performance characteristics (cache misses, lock contention, GC pauses) that warrant investigation.

Benchmarking across BEAM versions. OTP/Erlang releases can change performance characteristics of built-in functions. Ensure baselines are regenerated when upgrading the BEAM VM version to avoid false regression signals.

Use Cases

LiveView Dashboard Launch: Before deploying the Prismatic Perimeter EASM dashboard, load tests validated that the LiveView page handles 200 concurrent WebSocket connections with P95 mount time under 150ms. The benchmark revealed that initial ETS table lookups for security ratings needed batch prefetching, which was implemented before launch.

API Gateway Optimization: The Prismatic API's generic dispatch controller routes requests through module introspection. Benchmarks demonstrated that the initial implementation's Code.fetch_docs/1 call on every request added 12ms of latency. Caching the introspection results in ETS reduced per-request overhead to under 0.5ms, well within the 50ms event handler budget.

Query Performance Regression: A Credo rule change inadvertently modified a pattern-matching path that affected Ecto query construction. The performance test suite caught a 40% latency regression in the affected query before the change reached the CI merge gate.

Evolution Fitness Tracking: The AutoEvolve system uses performance benchmarks as one dimension of its fitness function. When a code evolution candidate is generated, benchmarks validate that the change does not degrade performance. Only changes that maintain or improve performance while passing all quality gates are applied.

Telemetry Events Reference

The performance testing infrastructure emits and consumes the following telemetry events:

Event	Measurements	Threshold	Gate Level
`[:phoenix, :endpoint, :stop]`	`duration`	250ms	BLOCKING
`[:phoenix, :live_view, :mount, :stop]`	`duration`	150ms	BLOCKING
`[:phoenix, :live_view, :handle_event, :stop]`	`duration`	50ms	BLOCKING
`[:phoenix, :router_dispatch, :stop]`	`duration`	100ms	WARNING
`[:prismatic, :query, :stop]`	`duration`	50ms	WARNING
`[:prismatic, :benchmark, :regression]`	`delta_pct`	15%	BLOCKING
`[:prismatic, :performance_test, :render]`	`duration_ms`	varies	BLOCKING

These events form the data backbone of the performance testing pipeline. During development, they feed into the benchmark comparison system. In CI, they feed into the merge gate evaluation. In production, they feed into the Quality Floor Guardian for continuous monitoring and alerting.

Performance Tracking -- Continuous production monitoring that complements pre-merge performance testing
Performance -- The broader concept of application performance within the platform
Testing -- The overarching testing discipline that includes performance testing as a specialized domain
Telemetry -- The instrumentation infrastructure that provides performance measurement data
Observability -- The system-wide visibility that includes performance metrics as a key signal
Latency -- The primary metric validated by performance testing
Throughput -- The capacity metric validated under load testing conditions
Quality Gates -- The enforcement pipeline that includes performance checks as blocking gates
Regression Testing -- The regression prevention methodology applied to performance baselines
Scalability -- The system property validated through load and stress testing

Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL

GitHub | GitLab | LinkedIn | Contact
Developer Portal | Architecture | Meet the Creator

Performance Testing

Definition

Interactive Performance Analytics

Quality DNA Evolution

Performance Thresholds

Platform Fitness Score Progression (Generation 19)

Historical Context

Overview

Technical Details

Benchee-Based Micro-Benchmarking

LiveView Performance Assertion

Telemetry-Driven Performance Monitoring

Implementation

Comparison

Best Practices

Pitfalls

Use Cases

Telemetry Events Reference

See Also

Connect & Contribute

Glossary

Related Terms

Performance Testing

Definition

Interactive Performance Analytics

Quality DNA Evolution

Performance Thresholds

Platform Fitness Score Progression (Generation 19)

Historical Context

Overview

Technical Details

Benchee-Based Micro-Benchmarking

LiveView Performance Assertion

Telemetry-Driven Performance Monitoring

Implementation

Comparison

Best Practices

Pitfalls

Use Cases

Telemetry Events Reference

Related Concepts

See Also

Connect & Contribute

Glossary

Related Terms