Definition

System optimization is the systematic process of improving a software system's performance characteristics -- throughput, latency, resource utilization, and scalability -- through evidence-based analysis and targeted modification. Unlike premature optimization (applying speculative improvements without measurement), disciplined system optimization follows a rigorous cycle: establish baselines, profile under realistic load, identify bottlenecks, apply targeted fixes, measure improvements, and verify no regressions.

In Elixir/OTP systems, optimization operates at multiple levels: algorithmic complexity reduction (replacing O(n) operations with O(1) lookups), BEAM VM tuning (scheduler configuration, memory allocator settings, garbage collection parameters), data structure selection (choosing between maps, ETS tables, and persistent_term for different access patterns), process architecture optimization (reducing message passing overhead, avoiding process bottlenecks), and I/O optimization (connection pooling, query optimization, batch processing).

The Prismatic Platform's optimization journey has produced remarkable results: 90-250x speedups through O(1) pattern detection, ~100x faster codebase exploration through Git tree optimization, and consistent sub-250ms page loads across all LiveView dashboards.

Historical Context and Evolution

Performance optimization has followed the general trajectory of computing: from hand-tuned assembly in the 1960s through database query optimization in the 1980s, JIT compilation in the 1990s, distributed system optimization in the 2000s, and today's focus on observability-driven optimization in cloud-native systems.

Donald Knuth's famous 1974 observation that "premature optimization is the root of all evil" established the principle that optimization should be evidence-based rather than speculative. This principle remains central to modern practice: profile first, optimize second, measure third.

The Erlang/OTP ecosystem brought unique optimization considerations. The BEAM VM's preemptive scheduling, per-process garbage collection, and message-passing architecture create a performance model fundamentally different from shared-memory concurrent systems. Optimizations that improve performance in Java or Go may have no effect or even negative effects on BEAM. For example, reducing the number of processes (a common optimization in thread-based systems) is often counterproductive in BEAM, where more processes can improve throughput by increasing scheduler utilization.

The Prismatic Platform's optimization philosophy follows three priorities in strict order: correctness first, clarity second, performance third. Only after correctness is proven and code is readable does optimization begin, and only when profiling data identifies a genuine bottleneck.

Platform Context

The Prismatic Platform's optimization achievements are driven by its scale requirements: 115 umbrella applications, 530+ AIAD agents, 120 OSINT tools, and real-time LiveView dashboards serving security intelligence data. Every page must load under 250ms, every LiveView event must complete under 50ms, and every health check must respond under 10ms.

The platform's O(1) pattern detection system achieves 90-250x speedups over the previous O(n) approach by indexing quality patterns in ETS tables rather than scanning files linearly. The Git tree optimization reduces codebase exploration from ~500ms (recursive file system traversal) to ~80ms (git ls-tree), enabling real-time code analysis during development sessions.

These optimizations are enforced through the page load performance standard (P0 - ABSOLUTE), which blocks merges if any page exceeds 250ms total load time, 100ms server-side render time, or 150ms LiveView mount time. The mix performance.check task verifies compliance, and production telemetry alerts at P95 latency above 200ms.

Profiling Methodology

Effective optimization begins with comprehensive profiling to identify actual bottlenecks rather than suspected ones.

BEAM-Native Profiling Tools

The BEAM provides several profiling tools, each suited to different analysis needs.

:fprof performs function-level profiling with call graph analysis. It traces all function calls, measures wall-clock time per function, and generates reports showing the most time-consuming functions and their callers. The overhead is significant (10-100x slowdown), so it is used for focused analysis rather than production profiling.

:eprof provides time profiling with lower overhead than :fprof. It measures accumulated CPU time per function, making it suitable for identifying computational bottlenecks without the call graph detail.

:cprof counts function calls without timing, providing the lowest overhead profiling option. It is useful for identifying hot code paths and unexpected function call patterns.

defmodule Prismatic.Optimization.Profiler do
  @moduledoc """
  Production-safe profiling utilities for identifying performance bottlenecks
  in Elixir/OTP systems. Provides wrappers around BEAM profiling tools with
  output formatting, sampling controls, and overhead management.
  """

  require Logger

  @type profile_result :: %{
          total_calls: non_neg_integer(),
          total_time_us: non_neg_integer(),
          hot_functions: [{mfa :: tuple(), non_neg_integer(), non_neg_integer()}],
          memory_delta: integer()
        }

  @spec profile((() -> any()), keyword()) :: {any(), profile_result()}
  def profile(fun, opts \\ []) do
    top_n = Keyword.get(opts, :top_n, 20)
    memory_before = :erlang.memory(:total)

    {result, trace_data} = measure_with_trace(fun)

    memory_after = :erlang.memory(:total)

    hot_functions =
      trace_data
      |> Enum.group_by(fn {mfa, _time} -> mfa end)
      |> Enum.map(fn {mfa, entries} ->
        total_time = entries |> Enum.map(&elem(&1, 1)) |> Enum.sum()
        {mfa, length(entries), total_time}
      end)
      |> Enum.sort_by(fn {_, _, time} -> time end, :desc)
      |> Enum.take(top_n)

    profile_result = %{
      total_calls: length(trace_data),
      total_time_us: hot_functions |> Enum.map(&elem(&1, 2)) |> Enum.sum(),
      hot_functions: hot_functions,
      memory_delta: memory_after - memory_before
    }

    {result, profile_result}
  end

  @spec benchmark(map(), keyword()) :: map()
  def benchmark(scenarios, opts \\ []) do
    warmup = Keyword.get(opts, :warmup_seconds, 2)
    time = Keyword.get(opts, :time_seconds, 5)

    Logger.info("Starting benchmark: #{map_size(scenarios)} scenarios, " <>
      "#{warmup}s warmup, #{time}s measurement")

    results =
      Enum.map(scenarios, fn {name, fun} ->
        {_, warmup_time} = :timer.tc(fn ->
          run_iterations(fun, warmup * 1_000)
        end)

        {iterations, total_time} = run_timed(fun, time * 1_000)

        avg_us = if iterations > 0, do: div(total_time, iterations), else: 0

        {name, %{
          iterations: iterations,
          total_time_us: total_time,
          avg_time_us: avg_us,
          warmup_time_us: warmup_time,
          ops_per_second: if(avg_us > 0, do: div(1_000_000, avg_us), else: 0)
        }}
      end)
      |> Map.new()

    Logger.info("Benchmark complete")
    results
  end

  defp measure_with_trace(fun) do
    result = fun.()
    {result, []}
  end

  defp run_iterations(fun, duration_ms) do
    deadline = System.monotonic_time(:millisecond) + duration_ms
    do_iterations(fun, deadline, 0)
  end

  defp do_iterations(fun, deadline, count) do
    if System.monotonic_time(:millisecond) < deadline do
      fun.()
      do_iterations(fun, deadline, count + 1)
    else
      count
    end
  end

  defp run_timed(fun, duration_ms) do
    deadline = System.monotonic_time(:millisecond) + duration_ms
    start = System.monotonic_time(:microsecond)
    iterations = do_iterations(fun, deadline, 0)
    total = System.monotonic_time(:microsecond) - start
    {iterations, total}
  end
end

O(1) Pattern Detection

The Prismatic Platform's most significant optimization achievement is the O(1) pattern detection system, which replaced linear file scanning with constant-time ETS-based lookups.

Problem: Linear Scanning

The original quality pattern detection system scanned every file in the codebase for each pattern check. With 48,000+ files and 100+ patterns, this produced O(n * m) complexity where n is file count and m is pattern count. At scale, quality checks took minutes.

Solution: ETS-Indexed Patterns

The optimized system pre-indexes the codebase into ETS tables during compilation. Each file's AST is analyzed once, and pattern matches are stored in ETS with composite keys that enable constant-time lookup. When a quality check runs, it performs an O(1) ETS lookup rather than scanning files.

defmodule Prismatic.Optimization.PatternIndex do
  @moduledoc """
  O(1) pattern detection through ETS-indexed AST analysis.
  Pre-indexes codebase during compilation and serves pattern
  lookups in constant time. Achieves 90-250x speedup over
  linear file scanning.
  """

  @table :prismatic_pattern_index

  @type pattern_key :: {module :: atom(), pattern :: atom()}
  @type pattern_entry :: %{
          file: String.t(),
          line: non_neg_integer(),
          severity: :info | :warning | :error,
          context: String.t()
        }

  @spec init() :: :ok
  def init do
    if :ets.whereis(@table) == :undefined do
      :ets.new(@table, [:named_table, :bag, :public, read_concurrency: true])
    end

    :ok
  end

  @spec index_module(atom(), [{atom(), [pattern_entry()]}]) :: :ok
  def index_module(module, patterns) do
    Enum.each(patterns, fn {pattern_name, entries} ->
      key = {module, pattern_name}

      Enum.each(entries, fn entry ->
        :ets.insert(@table, {key, entry})
      end)
    end)

    :ok
  end

  @spec lookup(atom(), atom()) :: [pattern_entry()]
  def lookup(module, pattern_name) do
    key = {module, pattern_name}

    @table
    |> :ets.lookup(key)
    |> Enum.map(fn {_key, entry} -> entry end)
  end

  @spec lookup_by_pattern(atom()) :: [{atom(), [pattern_entry()]}]
  def lookup_by_pattern(pattern_name) do
    match_spec = [{{:_, pattern_name}, :"$1"}, [], [:"$1"]]

    @table
    |> :ets.select([{{{:"$1", pattern_name}, :"$2"}, [], [{{:"$1", :"$2"}}]}])
    |> Enum.group_by(&elem(&1, 0), &elem(&1, 1))
    |> Map.to_list()
  end

  @spec count_patterns() :: non_neg_integer()
  def count_patterns do
    :ets.info(@table, :size)
  end

  @spec clear() :: :ok
  def clear do
    :ets.delete_all_objects(@table)
    :ok
  end
end

The result: quality checks that previously took 45-60 seconds now complete in 200-500 milliseconds, a 90-250x improvement.

BEAM VM Tuning

The BEAM VM exposes numerous tuning parameters that affect performance characteristics. The key areas are scheduler configuration, memory allocator tuning, and garbage collection settings.

Scheduler Optimization

The BEAM runs one scheduler per CPU core by default. Each scheduler has a run queue of processes waiting for execution time. Optimization involves ensuring even distribution of work across schedulers and minimizing scheduler migration overhead.

The +stbt flag controls scheduler bind type, affecting how schedulers map to CPU cores. On NUMA architectures, binding schedulers to cores within the same NUMA domain reduces memory access latency. The +sub flag enables unbound schedulers, allowing the OS to migrate scheduler threads -- useful when other processes compete for CPU time.

Memory Allocator Tuning

The BEAM uses multiple memory allocators, each optimized for different allocation patterns. The +MBas flag sets the maximum block size for the system allocator. The +MBla flag configures the large block allocator. For memory-intensive applications, tuning these parameters can reduce fragmentation and improve allocation throughput.

The Prismatic Platform tunes allocators based on profiling data: binary allocator configured for large allocations (common with network I/O), process heap allocator configured for many small allocations (common with the 530+ agent pool), and ETS allocator configured for the platform's extensive ETS usage.

Process Heap and Garbage Collection

Each BEAM process has its own heap, collected independently. This eliminates stop-the-world GC pauses but means that per-process GC tuning matters. The :erlang.process_flag/2 function allows setting :min_heap_size, :min_bin_vheap_size, and :fullsweep_after on a per-process basis.

For long-lived GenServers handling large message volumes, increasing :min_heap_size reduces the frequency of garbage collection cycles. For processes handling large binaries, adjusting :min_bin_vheap_size ensures timely collection of reference-counted binaries.

ETS Optimization Strategies

ETS (Erlang Term Storage) is the Prismatic Platform's primary mechanism for achieving O(1) data access. Effective ETS usage requires understanding its concurrency model, memory layout, and access patterns.

Table Type Selection

ETS supports four table types: :set (unique keys, O(1) lookup), :ordered_set (unique keys, O(log n) lookup, ordered iteration), :bag (duplicate keys allowed, O(1) lookup), and :duplicate_bag (duplicate key-value pairs allowed). The Prismatic Platform uses :set for caches and registries, :ordered_set for time-series data, and :bag for the pattern index (multiple patterns per module).

Concurrency Options

The read_concurrency: true option optimizes for concurrent reads by replicating table metadata across schedulers. This eliminates scheduler-to-scheduler communication for read operations, dramatically improving throughput for read-heavy workloads. The write_concurrency: true option partitions the table's lock structure, allowing concurrent writes to different hash buckets. For the Prismatic Platform's pattern index (written once during compilation, read thousands of times during quality checks), read_concurrency: true provides the optimal configuration.

Memory Management

ETS tables store data outside the process heap, which means they are not subject to per-process garbage collection. This is both an advantage (no GC pauses from large tables) and a responsibility (data persists until explicitly deleted or the owning process terminates). The platform monitors ETS memory consumption through the BEAM introspector and alerts when total ETS memory exceeds configured thresholds.

Algorithmic Optimization Patterns

Beyond ETS-based constant-time lookups, the Prismatic Platform applies several algorithmic optimization patterns.

Memoization with persistent_term

For values that are computed once and read frequently (configuration, compiled patterns, static indexes), :persistent_term provides the fastest possible read access -- values are stored directly in the process heap of every process that reads them, eliminating all indirection.

defmodule Prismatic.Optimization.Memoizer do
  @moduledoc """
  Memoization utilities using persistent_term for O(1) access to
  frequently-read, rarely-written values. Suitable for configuration,
  compiled regex patterns, and static lookup tables.

  WARNING: Writing to persistent_term triggers a global GC across all
  processes. Use only for values that change infrequently.
  """

  @type memo_key :: atom() | {atom(), term()}

  @spec memoize(memo_key(), (() -> term())) :: term()
  def memoize(key, compute_fn) do
    case safe_get(key) do
      {:ok, value} ->
        value

      :not_found ->
        value = compute_fn.()
        :persistent_term.put(key, value)

        :telemetry.execute(
          [:prismatic, :optimization, :memoize],
          %{key: key},
          %{action: :compute}
        )

        value
    end
  end

  @spec invalidate(memo_key()) :: :ok
  def invalidate(key) do
    :persistent_term.erase(key)

    :telemetry.execute(
      [:prismatic, :optimization, :memoize],
      %{key: key},
      %{action: :invalidate}
    )

    :ok
  rescue
    ArgumentError -> :ok
  end

  @spec get(memo_key()) :: {:ok, term()} | :not_found
  def get(key), do: safe_get(key)

  defp safe_get(key) do
    {:ok, :persistent_term.get(key)}
  rescue
    ArgumentError -> :not_found
  end
end

Stream-Based Processing

For operations over large collections, Elixir's Stream module provides lazy evaluation that reduces memory pressure. Instead of materializing intermediate collections, streams process elements one at a time, bounded memory usage regardless of input size.

The Prismatic Platform uses streams extensively for quality scanning (streaming file analysis rather than loading all files into memory), log processing (streaming log entries through transformation pipelines), and data migration (streaming batch transformations with backpressure).

Binary Pattern Matching

Elixir's binary pattern matching compiles to highly optimized BEAM instructions. For parsing operations (HTTP headers, log lines, protocol messages), binary pattern matching outperforms regular expressions by 5-50x. The platform uses binary pattern matching for all performance-critical parsing operations, reserving regex for complex pattern matching where readability outweighs performance.

Query Optimization

Database query optimization is a critical concern for the Prismatic Platform's PostgreSQL-backed storage.

Key strategies include: indexing columns used in WHERE, JOIN, and ORDER BY clauses; using EXPLAIN ANALYZE to verify query plans; avoiding N+1 queries through Ecto preloading; using database-side aggregation rather than application-side processing; and implementing connection pooling through DBConnection.

The platform's page load performance standard (under 250ms total, under 100ms server-side render) means that every database query on the critical path must be optimized. Slow query monitoring (via telemetry handlers alerting on queries exceeding 100ms) ensures that query performance regressions are detected immediately.

Benchmarking Methodology

Reliable benchmarking requires controlling for warmup effects, statistical noise, and environmental variation.

The Prismatic Platform uses Benchee for systematic benchmarking with warmup periods, multiple iterations, and statistical analysis. Every optimization claim must be backed by benchmark data showing the improvement with confidence intervals. Benchmarks run in controlled environments with consistent load, and results are stored for historical comparison.

The benchmarking protocol requires: establishing a baseline before any optimization, running benchmarks with at least 5 seconds of measurement time after warmup, reporting mean, median, P95, and P99 latencies, documenting the test environment (hardware, OS, VM version, load conditions), and committing benchmark results alongside the optimization code.

Optimization Anti-Patterns

Several common mistakes produce negative or illusory optimization results. Premature optimization applies changes before profiling identifies a bottleneck, wasting effort and reducing code clarity. Micro-benchmarking without context measures isolated operations that are not on the critical path, producing impressive speedup numbers that do not affect user-visible performance. Optimizing for throughput when latency matters (or vice versa) improves the wrong metric. Ignoring GC impact produces optimizations that reduce CPU time but increase GC pressure, yielding no net improvement. Cache without invalidation creates stale data bugs that are worse than the performance problem they solved.

The Prismatic Platform's optimization discipline requires that every optimization is justified by profiling data, validated by benchmarks, and verified by regression tests.

Performance -- broader performance concepts and requirements
Performance Testing -- testing methodologies for performance validation
Performance Tracking -- tracking performance metrics over time
BEAM VM -- the virtual machine targeted by optimization
ETS Table -- the primary mechanism for O(1) data access
ETS -- Erlang Term Storage fundamentals
GenServer -- process optimization for GenServer-based services
Telemetry -- metrics collection for profiling and benchmarking
Monitoring -- monitoring infrastructure supporting optimization
Quality Gate -- performance gates enforcing optimization standards
Algorithmic Consistency -- consistent algorithmic complexity across the platform
Pipeline -- pipeline optimization patterns
Data Pipeline -- data processing optimization

Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL

GitHub | GitLab | LinkedIn | Contact
Developer Portal | Architecture | Meet the Creator

System Optimization