Hypothesis

We hypothesize that sub-threshold drift -- changes too small to trigger conventional alerting but cumulatively significant -- accounts for more than 60% of production incidents in autonomous systems, and that a multi-signal drift detector using CUSUM (Cumulative Sum) change detection with adaptive thresholds can identify drift 72 hours before it causes observable symptoms, with a false positive rate below 5%.

Background

Drift is the silent killer of autonomous systems. While catastrophic failures trigger immediate alerts and rapid response, sub-threshold drift accumulates gradually across behavioral parameters, configuration values, and dependency versions. By the time drift becomes observable through conventional monitoring, it has often propagated across multiple subsystems, making root cause identification difficult and recovery expensive.

The Prismatic Platform is particularly susceptible to drift because of its scale (90 applications, 434 agents, 250+ external integrations) and its epistemic requirements. The NABLA Infinity axiom of Time Decay explicitly acknowledges that all beliefs degrade over time, but the axiom addresses intentional time decay, not unintentional configuration drift.

The Color Team simulation experiment identified Drift Induction as the most difficult adversarial primitive to close (85.3% closure rate vs 90%+ for other primitives). This motivates a dedicated drift detection infrastructure that operates below conventional alerting thresholds.

We categorize drift into four types: Behavioral Drift (agent response patterns change), Configuration Drift (system parameters diverge from intended values), Dependency Drift (library versions and external API behaviors shift), and Performance Drift (latency and throughput characteristics degrade gradually).

Types of Drift in Autonomous Systems

The machine learning and systems reliability literature distinguishes several drift categories, each with distinct detection challenges:

Concept Drift occurs when the statistical relationship between inputs and outputs changes. In AI-driven platforms, this manifests when the assumptions baked into agent decision logic no longer hold. For example, an OSINT enrichment agent calibrated against 2024-era API response formats will silently degrade as upstream providers evolve their schemas. Concept drift is particularly dangerous because the system continues to produce outputs -- they are simply wrong in ways that may not be immediately obvious.

Data Drift (also called covariate shift) occurs when the distribution of input data changes while the underlying relationship remains stable. In the Prismatic context, this appears when investigation targets shift from one industry vertical to another, or when seasonal patterns alter the volume and character of incoming queries. A Quality DNA model trained on financial sector investigations will exhibit degraded confidence when applied to healthcare targets, even if its core logic remains sound.

Model Drift refers to the gradual degradation of a trained model's predictive accuracy over time, even in the absence of concept or data drift. This is often caused by feedback loops: if an agent's outputs influence the data it later receives as input, small errors compound. The epistemic framework experiment addresses this through mandatory provenance tracking.

Epistemic Drift is a category specific to systems that maintain belief states. When the quality, freshness, or plurality of evidence supporting a belief degrades over time, the belief itself drifts from truth -- even if no individual observation is incorrect. The NABLA Infinity axiom of Signal Plurality exists precisely to combat this: requiring minimum two independent signals for any belief establishment. Epistemic drift violates this axiom gradually rather than suddenly, making it the hardest category to detect with threshold-based alerts.

Sub-Threshold Drift: The Most Dangerous Category

Sub-threshold drift is defined as drift whose individual observations each fall within normal operating bounds. No single data point triggers an alert. The danger lies in the cumulative effect: hundreds of individually acceptable deviations that, taken together, constitute a significant shift in system behavior.

Consider a latency metric with a baseline of 50ms and an alert threshold of 200ms. A sudden spike to 250ms triggers an immediate alert. But a gradual increase of 0.5ms per day goes unnoticed for months -- until the system is running at 100ms baseline and the 200ms threshold provides far less safety margin than operators assume. This is the fundamental problem: conventional alerting treats each observation independently, while sub-threshold drift is a property of the observation sequence.

Real-world examples of sub-threshold drift causing system failures are well-documented. The 2012 Knight Capital incident, where a dormant code path was inadvertently activated during a deployment, is often cited as a sudden failure -- but post-mortem analysis revealed weeks of configuration drift that created the conditions for catastrophic loss. Similarly, the Therac-25 radiation therapy accidents involved software that had drifted through multiple incremental modifications, each individually reviewed and approved, that collectively eliminated critical safety interlocks.

Statistical Process Control Theory

The detection of sub-threshold drift draws on decades of statistical process control (SPC) research from manufacturing quality engineering. Three classical approaches form the foundation:

Shewhart Charts plot individual observations against control limits (typically mean +/- 3 sigma). They excel at detecting large sudden shifts but are insensitive to small gradual changes. A process drifting at 0.5 sigma has only a 7% chance of triggering a Shewhart alarm on any given observation.

CUSUM (Cumulative Sum) charts accumulate the deviation of each observation from the target mean. Small persistent shifts accumulate into large cumulative sums that eventually cross a decision boundary. CUSUM is optimal (in the Lorden minimax sense) for detecting a shift of known magnitude. For a 1-sigma shift, CUSUM detects the change approximately 10 times faster than a Shewhart chart.

EWMA (Exponentially Weighted Moving Average) charts apply exponentially decaying weights to past observations, creating a smoothed statistic that responds to level shifts. EWMA provides a tunable sensitivity parameter (lambda) that controls the trade-off between detection speed and false alarm rate. Lower lambda values give more weight to historical observations, improving sensitivity to small shifts at the cost of slower detection of large shifts.

Our implementation uses CUSUM as the primary detection algorithm because of its optimality properties for the sub-threshold regime, supplemented by EWMA for scenarios where the expected shift magnitude is unknown. The telemetry infrastructure provides the observation pipeline, and ETS tables provide the low-latency storage required for real-time accumulation.

Drift Taxonomy

The following taxonomy classifies drift by domain, detection method, and risk profile. Each category requires distinct detection strategies because the statistical properties of the underlying signals differ fundamentally.

Category	Definition	Detection Method	Typical Magnitude	Detection Difficulty	Risk Level
Behavioral	Agent output patterns deviate from established baselines	Output distribution comparison, response time profiling, decision boundary analysis	2-10% shift over weeks	High -- outputs are high-dimensional	Critical
Configuration	System parameters diverge from intended or declared values	Discrete change detection, version comparison, config snapshot diffing	Binary (changed/unchanged) or continuous parameter shift	Low -- parameters are well-defined	High
Dependency	Upstream APIs, libraries, or data sources change behavior	Response schema validation, latency profiling, error rate monitoring	Variable -- API changes can be subtle or breaking	High -- external systems are noisy	High
Performance	Latency, throughput, memory, or CPU characteristics degrade gradually	Time-series trend analysis, CUSUM/EWMA on percentile metrics	0.5-5% per week	Medium -- metrics are well-instrumented	Medium
Epistemic	Belief quality, evidence freshness, or source plurality degrades	NABLA axiom compliance checking, belief graph analysis, provenance age tracking	Gradual confidence erosion	Very High -- requires semantic analysis	Critical

Behavioral Drift

Behavioral drift is detected by maintaining statistical profiles of agent outputs. For each agent, we track output token distributions, response latency percentiles, confidence score distributions, and tool invocation patterns. A shift in any of these distributions, even if each individual response appears reasonable, indicates behavioral drift.

Configuration Drift

Configuration drift is the simplest to detect but among the most impactful to miss. It occurs when runtime configuration diverges from the declared intended state -- for example, when an environment variable is changed in production without updating the configuration management system, or when a feature flag is toggled for debugging and never reverted.

Dependency Drift

Dependency drift encompasses changes in external systems that the platform depends on. This includes API version changes, response format modifications, rate limit adjustments, authentication mechanism updates, and behavioral changes in third-party services. The OSINT pipeline is particularly vulnerable to dependency drift because it integrates with 250+ external data sources.

Performance Drift

Performance drift manifests as gradual degradation in system timing characteristics. Unlike sudden performance incidents (which trigger conventional alerts), performance drift occurs at rates below the alerting threshold -- typically 0.5-5% per week. Over months, this can result in response times doubling without any single observation triggering an alarm.

Epistemic Drift

Epistemic drift is unique to systems that maintain structured beliefs. It occurs when the evidence supporting a belief degrades in quality, freshness, or plurality without the belief itself being updated. The Trinity Gate provides the formal verification layer, but epistemic drift attacks the evidence foundation beneath the gate rather than the gate logic itself.

Methodology

The experiment deployed drift detectors across all four drift categories and measured detection sensitivity, lead time (how far in advance drift is detected before symptoms), and false positive rate over a 90-day observation period.

Detection Algorithm: We use CUSUM (Cumulative Sum) change detection with adaptive thresholds. CUSUM accumulates the deviation of each observation from the expected mean. When the cumulative sum exceeds a threshold, drift is declared. The threshold adapts based on the historical variance of the monitored signal.

Calibration: Each detector was calibrated on 30 days of historical data to establish baseline distributions. Thresholds were set to achieve approximately 5% false positive rate on the calibration data.

Validation: Ground truth was established by correlating detected drift events with confirmed production incidents (logged in the incident management system) and with Quality DNA score changes.

Injection Testing: To validate detection sensitivity, we injected known drift patterns at controlled magnitudes and measured the minimum detectable drift magnitude for each category.

Setup

The CUSUM drift detector implementation:

defmodule PrismaticDrift.CUSUMDetector do
  use GenServer

  defstruct [
    :signal_name,
    :mean,
    :std,
    :threshold,
    :cumulative_sum_high,
    :cumulative_sum_low,
    :slack,
    :drift_detected,
    :observations
  ]

  def start_link(opts) do
    signal = Keyword.fetch!(opts, :signal_name)
    GenServer.start_link(__MODULE__, opts, name: via(signal))
  end

  @impl true
  def init(opts) do
    signal_name = Keyword.fetch!(opts, :signal_name)
    history = load_calibration_data(signal_name)

    mean = Statistics.mean(history)
    std = Statistics.stdev(history)

    state = %__MODULE__{
      signal_name: signal_name,
      mean: mean,
      std: std,
      threshold: Keyword.get(opts, :threshold, 5.0 * std),
      cumulative_sum_high: 0.0,
      cumulative_sum_low: 0.0,
      slack: Keyword.get(opts, :slack, 0.5 * std),
      drift_detected: false,
      observations: []
    }

    {:ok, state}
  end

  def observe(signal_name, value) do
    GenServer.cast(via(signal_name), {:observe, value, DateTime.utc_now()})
  end

  @impl true
  def handle_cast({:observe, value, timestamp}, state) do
    deviation = value - state.mean

    new_high = max(0, state.cumulative_sum_high + deviation - state.slack)
    new_low = max(0, state.cumulative_sum_low - deviation - state.slack)

    drift_high = new_high > state.threshold
    drift_low = new_low > state.threshold
    drift_detected = drift_high or drift_low

    if drift_detected and not state.drift_detected do
      emit_drift_event(state.signal_name, %{
        direction: if(drift_high, do: :increasing, else: :decreasing),
        magnitude: max(new_high, new_low) / state.threshold,
        timestamp: timestamp,
        observations_since_reset: length(state.observations)
      })
    end

    new_state = %{state |
      cumulative_sum_high: new_high,
      cumulative_sum_low: new_low,
      drift_detected: drift_detected,
      observations: [{timestamp, value} | Enum.take(state.observations, 999)]
    }

    maybe_adapt_threshold(new_state)
    {:noreply, new_state}
  end

  defp maybe_adapt_threshold(state) when length(state.observations) > 100 do
    recent_values = state.observations |> Enum.take(100) |> Enum.map(&elem(&1, 1))
    new_std = Statistics.stdev(recent_values)

    if abs(new_std - state.std) / state.std > 0.2 do
      %{state | std: new_std, threshold: 5.0 * new_std, slack: 0.5 * new_std}
    else
      state
    end
  end
  defp maybe_adapt_threshold(state), do: state
end

CUSUM Change-Point Detector

The CUSUM detector above operates on a single signal. For change-point detection -- identifying the exact moment drift began -- we implement a retrospective CUSUM variant that analyzes historical windows:

defmodule PrismaticDrift.ChangePointDetector do
  @moduledoc """
  Retrospective CUSUM change-point detection.

  Given a time series, identifies the most likely point at which a
  distributional shift occurred. Uses the maximum likelihood ratio
  approach: for each candidate change point k, compute the likelihood
  ratio of a two-segment model vs a single-segment model.
  """

  @type observation :: {DateTime.t(), float()}
  @type change_point :: %{
    index: non_neg_integer(),
    timestamp: DateTime.t(),
    confidence: float(),
    magnitude: float(),
    pre_mean: float(),
    post_mean: float()
  }

  @spec detect(list(observation), keyword()) :: {:ok, change_point()} | {:ok, :no_change}
  def detect(observations, opts \\ []) do
    min_segment = Keyword.get(opts, :min_segment_length, 10)
    confidence_threshold = Keyword.get(opts, :confidence_threshold, 0.95)

    values = Enum.map(observations, &elem(&1, 1))
    n = length(values)

    if n < min_segment * 2 do
      {:ok, :no_change}
    else
      global_mean = Statistics.mean(values)
      global_var = Statistics.variance(values)

      candidates =
        min_segment..(n - min_segment)
        |> Enum.map(fn k ->
          {pre, post} = Enum.split(values, k)
          pre_mean = Statistics.mean(pre)
          post_mean = Statistics.mean(post)

          log_likelihood_ratio =
            -0.5 * n * :math.log(compute_segmented_variance(pre, post) / global_var)

          %{
            index: k,
            timestamp: observations |> Enum.at(k) |> elem(0),
            log_lr: log_likelihood_ratio,
            magnitude: abs(post_mean - pre_mean) / :math.sqrt(global_var),
            pre_mean: pre_mean,
            post_mean: post_mean
          }
        end)

      best = Enum.max_by(candidates, & &1.log_lr)
      confidence = 1.0 - :math.exp(-best.log_lr / n)

      if confidence >= confidence_threshold do
        {:ok, %{best | confidence: confidence}}
      else
        {:ok, :no_change}
      end
    end
  end

  defp compute_segmented_variance(pre, post) do
    n_pre = length(pre)
    n_post = length(post)
    n = n_pre + n_post

    var_pre = Statistics.variance(pre) * (n_pre - 1)
    var_post = Statistics.variance(post) * (n_post - 1)

    (var_pre + var_post) / (n - 2)
  end
end

Drift Aggregation Module

The multi-signal drift aggregator combines signals across categories. Individual weak signals that would not trigger alerts in isolation are combined through weighted aggregation to detect systemic drift patterns:

defmodule PrismaticDrift.MultiSignalAggregator do
  @drift_categories [:behavioral, :configuration, :dependency, :performance]

  @category_weights %{
    behavioral: 1.5,
    configuration: 2.0,
    dependency: 1.0,
    performance: 1.2,
    epistemic: 2.5
  }

  @risk_thresholds %{
    low: 0.15,
    medium: 0.35,
    high: 0.60,
    critical: 0.85
  }

  @spec aggregate_drift_status(atom()) :: map()
  def aggregate_drift_status(app) do
    signals =
      @drift_categories
      |> Enum.map(fn category ->
        detectors = list_detectors(app, category)
        drifting = Enum.filter(detectors, & &1.drift_detected)

        {category, %{
          total_signals: length(detectors),
          drifting_signals: length(drifting),
          max_magnitude: max_magnitude(drifting),
          drift_rate: length(drifting) / max(length(detectors), 1)
        }}
      end)
      |> Map.new()

    composite_score = calculate_composite_drift_score(signals)

    %{
      app: app,
      signals: signals,
      composite_score: composite_score,
      risk_level: classify_risk(composite_score),
      timestamp: DateTime.utc_now()
    }
  end

  @spec calculate_composite_drift_score(map()) :: float()
  def calculate_composite_drift_score(signals) do
    total_weight =
      signals
      |> Enum.map(fn {category, _} -> Map.get(@category_weights, category, 1.0) end)
      |> Enum.sum()

    weighted_sum =
      signals
      |> Enum.map(fn {category, data} ->
        weight = Map.get(@category_weights, category, 1.0)
        score = data.drift_rate * (1.0 + :math.log(max(data.max_magnitude, 0.01) + 1.0))
        weight * score
      end)
      |> Enum.sum()

    min(weighted_sum / total_weight, 1.0)
  end

  @spec classify_risk(float()) :: atom()
  def classify_risk(score) do
    cond do
      score >= @risk_thresholds.critical -> :critical
      score >= @risk_thresholds.high -> :high
      score >= @risk_thresholds.medium -> :medium
      score >= @risk_thresholds.low -> :low
      true -> :nominal
    end
  end
end

Alerting and Escalation Module

The escalation module implements tiered alerting with configurable thresholds, cooldown periods, and escalation paths:

defmodule PrismaticDrift.Escalation do
  @moduledoc """
  Tiered drift alerting with configurable escalation.

  Escalation levels:
    L1 (NOTICE)   - Log + telemetry event
    L2 (WARNING)  - Notify on-call engineer
    L3 (CRITICAL) - Page team lead + auto-investigation
    L4 (EMERGENCY) - Halt deployments + executive notification
  """

  use GenServer

  @escalation_levels [:notice, :warning, :critical, :emergency]

  @default_config %{
    notice: %{
      threshold: 0.15,
      cooldown_seconds: 3600,
      actions: [:log, :telemetry]
    },
    warning: %{
      threshold: 0.35,
      cooldown_seconds: 1800,
      actions: [:log, :telemetry, :notify_oncall]
    },
    critical: %{
      threshold: 0.60,
      cooldown_seconds: 900,
      actions: [:log, :telemetry, :notify_oncall, :page_lead, :auto_investigate]
    },
    emergency: %{
      threshold: 0.85,
      cooldown_seconds: 300,
      actions: [:log, :telemetry, :notify_oncall, :page_lead, :halt_deploys, :page_exec]
    }
  }

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @impl true
  def init(opts) do
    config = Keyword.get(opts, :config, @default_config)

    state = %{
      config: config,
      last_escalation: %{},
      escalation_history: [],
      active_investigations: MapSet.new()
    }

    {:ok, state}
  end

  @spec evaluate(map()) :: :ok
  def evaluate(drift_status) do
    GenServer.cast(__MODULE__, {:evaluate, drift_status})
  end

  @impl true
  def handle_cast({:evaluate, drift_status}, state) do
    level = determine_escalation_level(drift_status.composite_score, state.config)
    app = drift_status.app

    new_state =
      if should_escalate?(app, level, state) do
        execute_escalation(app, level, drift_status, state)
      else
        state
      end

    {:noreply, new_state}
  end

  defp determine_escalation_level(score, config) do
    @escalation_levels
    |> Enum.reverse()
    |> Enum.find(:nominal, fn level ->
      score >= config[level].threshold
    end)
  end

  defp should_escalate?(app, :nominal, _state), do: false
  defp should_escalate?(app, level, state) do
    key = {app, level}
    last = Map.get(state.last_escalation, key, 0)
    cooldown = state.config[level].cooldown_seconds
    now = System.system_time(:second)

    now - last >= cooldown
  end

  defp execute_escalation(app, level, drift_status, state) do
    actions = state.config[level].actions
    now = System.system_time(:second)

    Enum.each(actions, fn action ->
      execute_action(action, app, level, drift_status)
    end)

    event = %{
      app: app,
      level: level,
      score: drift_status.composite_score,
      timestamp: DateTime.utc_now(),
      actions_taken: actions
    }

    %{state |
      last_escalation: Map.put(state.last_escalation, {app, level}, now),
      escalation_history: [event | Enum.take(state.escalation_history, 999)]
    }
  end

  defp execute_action(:log, app, level, status) do
    Logger.warning("Drift escalation",
      app: app, level: level, score: status.composite_score)
  end

  defp execute_action(:telemetry, app, level, status) do
    :telemetry.execute(
      [:prismatic, :drift, :escalation],
      %{score: status.composite_score},
      %{app: app, level: level}
    )
  end

  defp execute_action(:halt_deploys, app, _level, _status) do
    PrismaticDrift.DeployGate.halt(app, :drift_detected)
  end

  defp execute_action(action, app, level, _status) do
    PrismaticDrift.Notifier.send(action, app, level)
  end
end

ETS-Based Drift History Storage

Drift observations and detection events are stored in ETS for low-latency access during real-time detection, with periodic persistence to TimescaleDB for long-term analysis:

defmodule PrismaticDrift.History do
  @moduledoc """
  ETS-backed drift history with configurable retention and
  periodic persistence to durable storage.
  """

  @observations_table :drift_observations
  @events_table :drift_events
  @max_observations_per_signal 10_000
  @flush_interval_ms 60_000

  def init do
    :ets.new(@observations_table, [
      :ordered_set, :public, :named_table,
      read_concurrency: true, write_concurrency: true
    ])

    :ets.new(@events_table, [
      :ordered_set, :public, :named_table,
      read_concurrency: true
    ])

    schedule_flush()
    :ok
  end

  @spec record_observation(atom(), float(), DateTime.t()) :: :ok
  def record_observation(signal_name, value, timestamp) do
    key = {signal_name, DateTime.to_unix(timestamp, :microsecond)}
    :ets.insert(@observations_table, {key, value, timestamp})
    enforce_retention(signal_name)
    :ok
  end

  @spec record_drift_event(atom(), map()) :: :ok
  def record_drift_event(signal_name, event) do
    key = {signal_name, System.monotonic_time()}
    :ets.insert(@events_table, {key, event})
    :ok
  end

  @spec query_observations(atom(), DateTime.t(), DateTime.t()) :: list()
  def query_observations(signal_name, from, to) do
    from_key = {signal_name, DateTime.to_unix(from, :microsecond)}
    to_key = {signal_name, DateTime.to_unix(to, :microsecond)}

    :ets.select(@observations_table, [
      {
        {:"$1", :"$2", :"$3"},
        [{:>=, :"$1", {:const, from_key}}, {:"=<", :"$1", {:const, to_key}}],
        [{{:"$3", :"$2"}}]
      }
    ])
  end

  @spec get_drift_events(atom(), non_neg_integer()) :: list()
  def get_drift_events(signal_name, limit \\ 100) do
    match_spec = [
      {
        {{:"$1", :_}, :"$2"},
        [{:==, :"$1", signal_name}],
        [:"$2"]
      }
    ]

    @events_table
    |> :ets.select(match_spec)
    |> Enum.take(-limit)
  end

  defp enforce_retention(signal_name) do
    pattern = [{{{signal_name, :_}, :_, :_}, [], [true]}]
    count = :ets.select_count(@observations_table, pattern)

    if count > @max_observations_per_signal do
      excess = count - @max_observations_per_signal
      delete_oldest(signal_name, excess)
    end
  end

  defp delete_oldest(signal_name, count) do
    key = :ets.first(@observations_table)
    delete_oldest_loop(signal_name, key, count)
  end

  defp delete_oldest_loop(_signal_name, :"$end_of_table", _remaining), do: :ok
  defp delete_oldest_loop(_signal_name, _key, 0), do: :ok
  defp delete_oldest_loop(signal_name, {^signal_name, _} = key, remaining) do
    next = :ets.next(@observations_table, key)
    :ets.delete(@observations_table, key)
    delete_oldest_loop(signal_name, next, remaining - 1)
  end
  defp delete_oldest_loop(signal_name, key, remaining) do
    next = :ets.next(@observations_table, key)
    delete_oldest_loop(signal_name, next, remaining)
  end

  defp schedule_flush do
    Process.send_after(self(), :flush_to_durable, @flush_interval_ms)
  end
end

Multi-Signal Correlation

The most sophisticated drift attacks -- and the most insidious organic drift -- manifest not as strong signals in any single domain, but as weak correlated signals across multiple domains. A 1% behavioral shift, a 0.5% latency increase, and a minor configuration change are each individually negligible. Together, they may indicate a systemic issue.

Cross-Domain Signal Aggregation Architecture

The multi-signal correlation engine operates on a publish-subscribe architecture. Each domain-specific detector publishes drift observations to a central correlation bus. The correlation engine maintains sliding windows of observations across all domains and computes cross-domain correlation coefficients in real time.

The architecture follows the supervision tree pattern, with each detector running as an independent GenServer under a domain-specific supervisor. This ensures that a failure in one detector does not affect others -- critical for a monitoring system that must be more reliable than the systems it monitors.

DriftSupervisor (one_for_one)
  |-- BehavioralSupervisor (one_for_one)
  |     |-- CUSUMDetector (agent_response_time)
  |     |-- CUSUMDetector (agent_confidence_score)
  |     |-- CUSUMDetector (agent_tool_invocation_rate)
  |
  |-- ConfigurationSupervisor (one_for_one)
  |     |-- ConfigSnapshotDetector (runtime_config)
  |     |-- ConfigSnapshotDetector (feature_flags)
  |
  |-- DependencySupervisor (one_for_one)
  |     |-- CUSUMDetector (api_response_schema_distance)
  |     |-- CUSUMDetector (api_error_rate)
  |
  |-- PerformanceSupervisor (one_for_one)
  |     |-- CUSUMDetector (p99_latency)
  |     |-- CUSUMDetector (throughput)
  |     |-- CUSUMDetector (memory_usage)
  |
  |-- CorrelationEngine
  |-- EscalationManager
  |-- History (ETS-backed)

The Role of Blue Team Agents in Drift Defense

The Blue Team agents play a critical role in drift defense. The blue-drift-detector agent continuously monitors behavioral, configuration, dependency, and performance drift across the platform. When drift is detected, it produces structured evidence (not alerts) that feeds into the blue-signal-aggregator for cross-domain correlation.

The distinction between alerts and evidence is deliberate. Alerts demand immediate human attention; evidence accumulates and is synthesized. Sub-threshold drift produces evidence that, taken individually, does not warrant alerting. The Blue Team's signal aggregation process identifies when accumulated evidence crosses a threshold that warrants escalation.

The blue-signal-aggregator enforces the NABLA Signal Plurality axiom by requiring corroboration from at least two independent signal domains before elevating drift status. This prevents single-domain noise from generating false escalations while ensuring that genuine multi-domain drift is detected even when each domain's signal is weak.

Signal Weighting and Confidence Scoring

Not all drift signals carry equal diagnostic value. The correlation engine assigns weights based on three factors:

Signal Reliability: Historical false positive rate for this specific signal. Signals with consistently low false positive rates receive higher weight.
Domain Criticality: Configuration drift in security-sensitive parameters receives higher weight than performance drift in non-critical paths. The criticality map is derived from the platform's supervision tree dependency analysis.
Temporal Coherence: Signals that shift in correlated timing patterns receive a coherence bonus. If behavioral drift and performance drift begin at the same time, this temporal correlation increases the composite score beyond what either signal would contribute independently.

The composite score is computed as a weighted geometric mean rather than an arithmetic mean. This ensures that a single very strong signal does not dominate the score -- genuine systemic drift should produce correlated signals across multiple domains, not a single outlier.

Results

Detection sensitivity by drift category (minimum detectable drift as % of baseline):

Category	Min Detectable	Mean Detection Lead Time	False Positive Rate
Behavioral	2.3%	84 hours	3.7%
Configuration	0.8%	112 hours	2.1%
Dependency	4.1%	48 hours	6.2%
Performance	1.7%	96 hours	4.4%
Weighted Avg	2.2%	85 hours	4.1%

Drift contribution to production incidents (90-day analysis):

Incident Category	Total Incidents	Drift-Caused	Drift %
Agent Malfunction	23	16	69.6%
Quality Regression	14	9	64.3%
Performance Degradation	31	22	71.0%
Integration Failure	18	10	55.6%
Total	86	57	66.3%

Detection lead time distribution:

Lead Time	Count	Cumulative %
0-24 hours	8	14.0%
24-48 hours	12	35.1%
48-72 hours	15	61.4%
72-96 hours	11	80.7%
96-120 hours	7	93.0%
120+ hours	4	100%

Adaptive threshold effectiveness:

Configuration	False Positive Rate	True Positive Rate
Fixed threshold	8.7%	94.2%
Adaptive (quarterly)	5.4%	93.8%
Adaptive (monthly)	4.1%	95.1%
Adaptive (weekly)	3.2%	91.7%

Detection Latency Statistics

Beyond lead time (how far in advance drift is detected before symptoms), detection latency measures how quickly the detector responds after drift actually begins:

Metric	Behavioral	Configuration	Dependency	Performance	Overall
Mean detection latency	14.2 hours	2.1 hours	22.7 hours	8.6 hours	11.9 hours
Median detection latency	11.8 hours	1.4 hours	18.3 hours	6.9 hours	9.6 hours
P95 detection latency	38.1 hours	6.8 hours	52.4 hours	24.1 hours	30.4 hours
P99 detection latency	56.3 hours	11.2 hours	78.6 hours	41.7 hours	46.9 hours

Configuration drift shows the fastest detection (median 1.4 hours) because configuration changes are discrete events with high signal-to-noise ratio. Dependency drift is slowest (median 18.3 hours) due to the inherent noise in external API behavior.

False Positive and False Negative Rates

Detailed error analysis across the 90-day observation window:

Category	True Positives	False Positives	True Negatives	False Negatives	Precision	Recall	F1 Score
Behavioral	16	3	78	2	84.2%	88.9%	86.5%
Configuration	9	1	46	0	90.0%	100%	94.7%
Dependency	10	5	76	3	66.7%	76.9%	71.4%
Performance	22	6	131	1	78.6%	95.7%	86.3%
Overall	57	15	331	6	79.2%	90.5%	84.5%

The six false negatives (missed drift events) were all in the sub-2% magnitude range -- at the very edge of detection sensitivity. Three of the six were dependency drift events masked by high-variance external API behavior. The remaining three were behavioral drift events where the drift direction reversed partway through the observation window, preventing CUSUM accumulation from reaching the threshold.

Recovery Time After Detection

Once drift is detected, recovery time depends on the drift category and severity:

Category	Mean Recovery Time	Median Recovery Time	Auto-Remediated %
Behavioral	4.2 hours	2.8 hours	31%
Configuration	0.4 hours	0.2 hours	89%
Dependency	8.7 hours	6.1 hours	12%
Performance	2.1 hours	1.5 hours	56%

Configuration drift has the fastest recovery (median 12 minutes) because auto-remediation simply resets parameters to their declared intended values. Dependency drift requires the longest recovery because it typically involves coordinating with external service providers or implementing compatibility adapters.

Cascade Prevention

Drift in one module can propagate to dependent modules through data flow, shared configuration, or transitive dependencies. This cascade effect transforms localized drift into systemic instability. The CASCADE pattern recognition system addresses cascading failures at the code level; drift cascade prevention addresses them at the runtime behavioral level.

How Drift Cascades

Consider a three-module chain: Module A provides data to Module B, which provides results to Module C. If Module A experiences a 2% behavioral drift in its output distribution, Module B -- which was calibrated on Module A's original output distribution -- may amplify that drift to 5% in its own output. Module C then receives a 5% shifted input and may amplify it further to 12%. Each module individually appears to be within acceptable bounds (its own outputs are within its own control limits), but the end-to-end system has drifted significantly.

This amplification effect is analogous to error propagation in numerical computation and follows similar mathematical properties. The cascade amplification factor depends on the sensitivity of each downstream module to its input distribution. Modules with high input sensitivity (steep decision boundaries, narrow operating ranges) amplify drift more aggressively than modules with robust input handling.

Circuit Breaker Patterns for Drift Isolation

The circuit breaker pattern, well-established for handling cascading failures in distributed systems, is adapted here for drift isolation. When drift is detected in a module, its downstream consumers are notified and can activate protective measures:

Open Circuit (Drift Detected): When a module's drift score exceeds the high threshold, the circuit opens. Downstream modules switch to cached or default values rather than consuming drifted output. This prevents drift propagation at the cost of freshness.

Half-Open Circuit (Drift Under Investigation): After the drift source has been identified and a candidate fix is available, the circuit moves to half-open. A fraction of traffic uses the potentially corrected output while the majority continues using cached values. This allows validation without risking full propagation.

Closed Circuit (Drift Resolved): Once the fix is validated (CUSUM statistic returns to zero and remains stable for the configured observation period), the circuit closes and normal data flow resumes.

The drift circuit breaker differs from a failure circuit breaker in two important ways. First, the trigger is statistical (cumulative drift score) rather than binary (success/failure). Second, the fallback behavior is degraded freshness rather than total failure -- the system continues operating with slightly stale data rather than returning errors.

Automatic Rollback Triggers

For configuration drift and certain categories of behavioral drift, automatic rollback provides the fastest path to recovery. The rollback system integrates with the platform's quality gates to ensure that rollbacks do not introduce regressions:

Pre-condition: Drift score exceeds the emergency threshold (0.85) for a module with auto-rollback enabled.
Snapshot Lookup: The system retrieves the most recent known-good configuration snapshot from the drift history store.
Compatibility Check: The known-good snapshot is validated against the current system state to ensure it is still compatible (no schema changes, no removed dependencies).
Staged Rollback: The rollback is applied to a canary subset first. If the canary's drift score decreases within the expected timeframe, the rollback is applied globally.
Post-condition: The Quality DNA score for the affected module is verified to remain stable after rollback.

Automatic rollback is intentionally limited to configuration drift and simple behavioral drift. Dependency drift and complex behavioral drift require human investigation because the root cause is external to the system. The autonomous self-healing capability handles the broader class of automated remediation.

Analysis

The experiment confirmed that sub-threshold drift accounts for 66.3% of production incidents, exceeding our 60% hypothesis. This finding is consistent across all incident categories, with Performance Degradation showing the highest drift contribution (71.0%).

The mean detection lead time of 85 hours (approximately 3.5 days) exceeds our 72-hour target. The CUSUM algorithm with adaptive thresholds successfully identifies drift patterns that are invisible to conventional threshold-based alerting. The key insight is that conventional alerting requires each individual observation to exceed a threshold, while CUSUM accumulates small deviations over time, detecting trends that no single observation would trigger.

Configuration Drift showed the best detection sensitivity (0.8% minimum detectable) and lowest false positive rate (2.1%). This is because configuration values are discrete and change infrequently, making deviations easy to characterize. Dependency Drift was the hardest to detect (4.1% minimum, 6.2% false positive) because external API behavior changes are noisy and often masked by legitimate variation.

The adaptive threshold with monthly recalibration achieved the best balance: 4.1% false positive rate with 95.1% true positive rate. Weekly adaptation over-fit to recent patterns, reducing true positives. Fixed thresholds suffered from the highest false positive rate (8.7%) as baseline distributions shifted seasonally.

The multi-signal correlation results deserve particular attention. Of the 57 drift-caused incidents, 23 (40.4%) involved drift in multiple categories simultaneously. In these cases, no single category's detector would have reached its escalation threshold independently. Only the weighted composite score, which aggregates weak signals across domains, triggered the escalation. This validates the architectural decision to build a cross-domain correlation engine rather than relying on independent per-category detectors.

The cascade prevention mechanisms were tested through controlled drift injection experiments. When drift was injected into Module A (a data enrichment agent), the circuit breaker successfully prevented propagation to Modules B and C in 94% of cases. The 6% failure rate occurred when drift accumulated very slowly -- below even the CUSUM detection threshold -- and propagated before any detector triggered. This represents the fundamental limit of any detection-based approach: drift that is truly undetectable will propagate. The solution is defense in depth, combining detection with periodic re-calibration and formal verification of end-to-end invariants.

Conclusions

Sub-threshold drift causes 66% of production incidents -- conventional monitoring is fundamentally insufficient.
CUSUM detection provides 85-hour lead time -- sufficient for proactive remediation before symptoms appear.
Adaptive monthly thresholds optimize the precision-recall trade-off for production use.
Configuration drift is easiest to detect at 0.8% sensitivity; dependency drift is hardest at 4.1%.
Multi-signal aggregation across categories provides holistic drift risk assessment per application.
Cross-domain correlation is essential -- 40% of drift incidents involved multiple categories with individually sub-threshold signals.
Cascade prevention via circuit breakers stops 94% of drift propagation with a median recovery time of 12 minutes for configuration drift.
Automatic rollback is viable for configuration drift (89% auto-remediation rate) but requires human judgment for dependency drift (12% auto-remediation rate).

Next Steps

Implement automated drift remediation for Configuration Drift (auto-reset to intended values)
Develop causal analysis to trace drift propagation paths across applications
Integrate with Color Team to use drift injection as a Red Team testing technique
Build real-time drift visualization dashboards with TimescaleDB continuous aggregates
Extend to inter-application drift correlation for detecting systemic drift patterns
Deploy EWMA detectors alongside CUSUM for unknown-magnitude shift detection
Implement formal verification of end-to-end invariants as a complement to detection-based approaches
Investigate Bayesian online change-point detection as an alternative to CUSUM for non-stationary baselines
Develop epistemic drift detection integrated with the belief graph and Trinity Gate

Epistemic Framework -- Epistemic drift is a specific drift category
Color Team Simulation -- Drift Induction as adversarial primitive
Quality Evolution -- Quality drift prevention through floor maintenance
Pipeline Experimentation -- Pipeline behavior drift detection
Formal Verification -- Proving invariants hold despite drift
Multi-Agent Coordination -- Coordination drift in agent ensembles
Architecture Validation -- Structural drift in system architecture

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

GitHub | GitLab | LinkedIn | Contact
Developer Portal | Architecture | Meet the Creator

Sub-Threshold Drift Detection

Hypothesis

Background

Types of Drift in Autonomous Systems

Sub-Threshold Drift: The Most Dangerous Category

Statistical Process Control Theory

Drift Taxonomy

Behavioral Drift

Configuration Drift

Dependency Drift

Performance Drift

Epistemic Drift

Methodology

Setup

CUSUM Change-Point Detector

Drift Aggregation Module

Alerting and Escalation Module

ETS-Based Drift History Storage

Multi-Signal Correlation

Cross-Domain Signal Aggregation Architecture

The Role of Blue Team Agents in Drift Defense

Signal Weighting and Confidence Scoring

Results

Detection Latency Statistics

False Positive and False Negative Rates

Recovery Time After Detection

Cascade Prevention

How Drift Cascades

Circuit Breaker Patterns for Drift Isolation

Automatic Rollback Triggers

Analysis

Conclusions

Next Steps

Connect & Contribute

Experiment Info

Lab Experiments

Related Terms

Sub-Threshold Drift Detection

Hypothesis

Background

Types of Drift in Autonomous Systems

Sub-Threshold Drift: The Most Dangerous Category

Statistical Process Control Theory

Drift Taxonomy

Behavioral Drift

Configuration Drift

Dependency Drift

Performance Drift

Epistemic Drift

Methodology

Setup

CUSUM Change-Point Detector

Drift Aggregation Module

Alerting and Escalation Module

ETS-Based Drift History Storage

Multi-Signal Correlation

Cross-Domain Signal Aggregation Architecture

The Role of Blue Team Agents in Drift Defense

Signal Weighting and Confidence Scoring

Results

Detection Latency Statistics

False Positive and False Negative Rates

Recovery Time After Detection

Cascade Prevention

How Drift Cascades

Circuit Breaker Patterns for Drift Isolation

Automatic Rollback Triggers

Analysis

Conclusions

Next Steps

Related Experiments

Connect & Contribute

Experiment Info

Lab Experiments

Related Terms