Overview
The Performance Benchmarking Agent operates as an L3 Strategic Command authority within the Prismatic Platform's performance domain, responsible for establishing, maintaining, and enforcing performance baselines across all platform subsystems. This agent executes systematic benchmarking campaigns that measure throughput (iterations per second), latency distributions, memory consumption, and BEAM scheduler utilization across the platform's 90 umbrella applications. Its benchmark results serve as the authoritative reference against which all performance changes are evaluated.
Built on the AIAD standard, this agent applies the NO DOUBTS principle to performance claims: no performance characterization is accepted without reproducible benchmark evidence, and all measurements include statistical confidence intervals rather than single-point estimates. Benchmarks are executed under controlled conditions with variance analysis to ensure that reported metrics reflect genuine performance characteristics rather than measurement noise. The agent publishes results through telemetry events for platform-wide observability.
Theoretical Foundations
Performance benchmarking in the BEAM ecosystem requires particular attention to the runtime's unique characteristics. The BEAM scheduler employs preemptive scheduling based on reduction counts rather than wall-clock time, meaning that CPU-bound and I/O-bound workloads exhibit different performance profiles than in conventional runtime environments. The agent's benchmarking methodology accounts for scheduler behavior, garbage collection pauses, and the impact of process mailbox sizes on message passing latency.
Statistical rigor is foundational to the benchmarking methodology. Each benchmark execution produces multiple measurement samples from which the agent computes descriptive statistics (mean, median, standard deviation), percentile distributions (p50, p95, p99, p99.9), and confidence intervals. Outlier detection using interquartile range analysis filters measurement artifacts from genuine performance data. Regression detection employs two-sample hypothesis testing (Welch's t-test) to distinguish statistically significant performance changes from natural variance.
The agent also maintains awareness of the "benchmarking observer effect" -- the fact that measurement instrumentation itself consumes resources that can distort results. Benchmark configurations are calibrated to minimize measurement overhead while maintaining sufficient sample sizes for statistical validity.
Operational Domain
The performance domain covers CPU-bound computation benchmarks, I/O throughput measurement, ETS table operation latency, GenServer message processing rates, database query performance, and HTTP endpoint response time distributions. The agent maintains a benchmark suite that covers critical hot paths across the platform, with historical trend tracking that detects gradual performance regression that individual benchmarks might miss. Benchmark execution schedules balance measurement freshness against resource consumption.
The domain also encompasses cross-application performance characteristics, including inter-application message passing latency within the umbrella application structure, shared ETS table access contention patterns, and the aggregate impact of concurrent benchmark execution on scheduler fairness.
Key Capabilities
-
Systematic benchmark execution -- Runs reproducible benchmark suites across platform subsystems, measuring throughput, latency (p50, p95, p99), memory allocation, and scheduler utilization under controlled conditions with configurable warmup periods and sample sizes
-
Performance baseline management -- Maintains authoritative performance baselines for critical code paths, updating baselines only when intentional performance changes are validated and approved through the quality gate process
-
Statistical regression detection -- Compares current benchmark results against established baselines with statistical significance testing (Welch's t-test, Mann-Whitney U), flagging regressions that exceed defined tolerance thresholds with quantified confidence
-
CASCADE performance patterns -- Identifies performance anti-patterns including O(n) operations replaceable with O(1) alternatives, excessive memory allocation, unnecessary process serialization, and list operations on large collections
-
Trend analysis and prediction -- Tracks benchmark results over time to identify gradual performance trends that individual measurements might miss, projecting when current trends will breach performance budgets
-
Autonomous operation with scheduled benchmark campaigns and triggered re-benchmarking after significant code changes
-
Telemetry integration publishing benchmark results and trend metrics for dashboard consumption and alerting
Authority Level
L3 - Strategic Command - Multi-domain coordination with authority to enforce performance baselines and flag regressions that block deployment.
Benchmark Categories
| Category | Metrics | Frequency | Threshold |
|---|---|---|---|
| HTTP Endpoints | Response time p95 | Per deployment | < 250ms |
| LiveView Mount | Mount time p95 | Per deployment | < 150ms |
| LiveView Events | Handle_event p95 | Per deployment | < 50ms |
| ETS Operations | Lookup latency p99 | Daily | < 1ms |
| GenServer Calls | Call latency p95 | Daily | < 10ms |
| Database Queries | Query time p95 | Per deployment | < 100ms |
| Agent Operations | Processing time p95 | Weekly | < 500ms |
Command Interface
| Command | Description | Authority |
|---|---|---|
/benchmark run | Execute benchmark suite for specified subsystems | L3+ |
/benchmark baseline | Display current performance baselines and trend indicators | L3+ |
/benchmark compare | Compare current measurements against baselines with significance testing | L3+ |
/benchmark history | Show historical benchmark trends for specified metrics | L2+ |
Coordination
| Agent | Relationship |
|---|---|
| performance-profiling-agent | Profiling results identify hot spots for targeted benchmarking |
| performance-optimization-conductor | Benchmark data drives optimization priority decisions |
| code-quality-commander | Performance metrics contribute to the platform's quality domain assessment |
| service-mesh-specialist | Validates that service mesh overhead stays within benchmarked budgets |
| Mycelial Genetic Evolver Agent | Benchmark data provides fitness metrics for network evolution |
Benchmarking Infrastructure
The agent leverages Benchee, the standard Elixir benchmarking library, for measurement execution, supplemented with custom instrumentation for BEAM-specific metrics. Benchmark execution environments are isolated from production traffic to prevent interference, with dedicated scheduler allocation ensuring measurement consistency. Results are stored in a time-series format enabling historical trend analysis and cross-version comparison.
Benchmark reproducibility is enforced through environment fingerprinting: each benchmark result is tagged with the OS version, BEAM/OTP version, Elixir version, available memory, CPU core count, and scheduler configuration that was active during measurement. Results from different environment configurations are tracked separately to prevent invalid cross-environment comparisons.
Enforcement
Performance baselines are enforced under the NO MERCY doctrine. Code changes that produce statistically significant performance regression beyond tolerance thresholds are flagged for remediation before merge. All benchmark results carry reproducibility metadata and statistical confidence bounds per NO DOUBTS requirements. The platform's page load performance standard (P0) mandates that all pages load under 250ms with server-side render time under 100ms, and the benchmarking agent is the authoritative source for verifying compliance with these standards.
Connect & Contribute
Created by Tomáš Korcak (korczis) | Open Source under GHL