Definition & Overview

Variance is a statistical measure that quantifies the spread or dispersion of a dataset by calculating the average of the squared differences between each data point and the mean. A high variance indicates that data points are widely spread out from the mean, while a low variance indicates they are clustered closely together. Variance is the square of standard deviation and serves as a fundamental building block for statistical analysis, hypothesis testing, and anomaly detection.

In mathematical notation, the population variance is defined as sigma squared, the sum of squared deviations from the mean divided by the number of observations. The sample variance uses N-1 (Bessel's correction) in the denominator to provide an unbiased estimate of the population variance. This distinction matters in practice when working with finite samples, which is the typical case in platform monitoring and analytics.

The Prismatic Platform uses variance calculations across multiple domains. In performance monitoring, variance in response times reveals whether the system behaves consistently (low variance) or unpredictably (high variance). In OSINT signal analysis, variance in confidence scores across multiple sources indicates the level of agreement or disagreement between intelligence sources. In quality tracking, variance in domain scores across platform generations measures whether quality improvements are uniform or concentrated in specific areas.

Technical Deep Dive

The platform implements variance calculations as part of its statistical analysis toolkit:

defmodule PrismaticStatistics.Descriptive do
  @moduledoc """
  Descriptive statistics calculations including mean,
  variance, standard deviation, and derived measures.
  """

  @type stats :: %{
    mean: float(),
    variance: float(),
    std_dev: float(),
    min: float(),
    max: float(),
    count: non_neg_integer(),
    coefficient_of_variation: float()
  }

  @spec calculate([number()]) :: {:ok, stats()} | {:error, :insufficient_data}
  def calculate(values) when length(values) < 2, do: {:error, :insufficient_data}

  def calculate(values) do
    n = length(values)
    mean = Enum.sum(values) / n

    variance =
      values
      |> Enum.map(fn x -> (x - mean) * (x - mean) end)
      |> Enum.sum()
      |> Kernel./(n - 1)

    std_dev = :math.sqrt(variance)

    cv =
      if mean != 0 do
        Float.round(std_dev / abs(mean) * 100, 2)
      else
        0.0
      end

    {:ok, %{
      mean: Float.round(mean, 6),
      variance: Float.round(variance, 6),
      std_dev: Float.round(std_dev, 6),
      min: Enum.min(values) / 1,
      max: Enum.max(values) / 1,
      count: n,
      coefficient_of_variation: cv
    }}
  end

  @spec z_score(number(), float(), float()) :: float()
  def z_score(value, mean, std_dev) when std_dev > 0 do
    Float.round((value - mean) / std_dev, 4)
  end

  def z_score(_value, _mean, _std_dev), do: 0.0
end

For online variance calculation that processes data points incrementally without storing the entire dataset, the platform uses Welford's algorithm:

defmodule PrismaticStatistics.OnlineVariance do
  @moduledoc """
  Welford's online algorithm for computing variance
  incrementally as data points arrive.
  """

  @type t :: %__MODULE__{
    count: non_neg_integer(),
    mean: float(),
    m2: float()
  }

  defstruct count: 0, mean: 0.0, m2: 0.0

  @spec new() :: t()
  def new, do: %__MODULE__{}

  @spec update(t(), number()) :: t()
  def update(%__MODULE__{count: n, mean: mean, m2: m2}, value) do
    new_count = n + 1
    delta = value - mean
    new_mean = mean + delta / new_count
    delta2 = value - new_mean
    new_m2 = m2 + delta * delta2

    %__MODULE__{count: new_count, mean: new_mean, m2: new_m2}
  end

  @spec variance(t()) :: float()
  def variance(%__MODULE__{count: n}) when n < 2, do: 0.0

  def variance(%__MODULE__{count: n, m2: m2}) do
    m2 / (n - 1)
  end

  @spec std_dev(t()) :: float()
  def std_dev(state) do
    :math.sqrt(variance(state))
  end

  @spec merge(t(), t()) :: t()
  def merge(%__MODULE__{count: 0}, other), do: other
  def merge(state, %__MODULE__{count: 0}), do: state

  def merge(
    %__MODULE__{count: na, mean: mean_a, m2: m2a},
    %__MODULE__{count: nb, mean: mean_b, m2: m2b}
  ) do
    n = na + nb
    delta = mean_b - mean_a
    mean = (mean_a * na + mean_b * nb) / n
    m2 = m2a + m2b + delta * delta * na * nb / n

    %__MODULE__{count: n, mean: mean, m2: m2}
  end
end

Architecture & Implementation

Variance calculations in the platform serve three primary architectural purposes:

Anomaly Detection: The platform uses variance-based anomaly detection to identify unusual behavior. When a new data point falls more than 3 standard deviations from the running mean (a z-score exceeding 3), it is flagged as a potential anomaly. This approach is applied to OSINT tool response times, API request latencies, and quality metric fluctuations.

Consistency Assessment: Low variance in a set of measurements indicates consistency, while high variance indicates instability. The platform uses this to assess OSINT source reliability. If a source's confidence scores have low variance across queries, it is considered a stable source. High variance sources are treated with appropriate skepticism per the NABLA Infinity framework's signal plurality axiom.

Performance Baselining: Variance in performance metrics establishes acceptable ranges for automated alerting. Rather than using fixed thresholds, the platform dynamically adjusts alert boundaries based on running variance calculations. A sudden increase in response time variance (even if the mean remains acceptable) signals degraded stability that warrants investigation.

The online variance algorithm (Welford's) is particularly valuable for streaming telemetry data where maintaining the full history of values is impractical. It processes each telemetry event as it arrives, maintaining an accurate running variance with O(1) memory.

Usage in Prismatic Platform

The platform uses variance in performance monitoring to detect instability:

defmodule PrismaticMonitoring.StabilityAnalyzer do
  @moduledoc """
  Analyzes system stability using variance-based metrics
  to detect degradation before mean values shift.
  """

  alias PrismaticStatistics.OnlineVariance

  @max_acceptable_cv 50.0

  @spec analyze_stability(String.t(), [number()]) :: {:ok, map()} | {:alert, map()}
  def analyze_stability(metric_name, recent_values) do
    state = Enum.reduce(recent_values, OnlineVariance.new(), &OnlineVariance.update(&2, &1))

    variance = OnlineVariance.variance(state)
    std_dev = OnlineVariance.std_dev(state)
    cv = if state.mean != 0, do: std_dev / abs(state.mean) * 100, else: 0.0

    result = %{
      metric: metric_name,
      mean: Float.round(state.mean, 2),
      variance: Float.round(variance, 2),
      std_dev: Float.round(std_dev, 2),
      coefficient_of_variation: Float.round(cv, 2),
      sample_count: state.count,
      stability: classify_stability(cv)
    }

    if cv > @max_acceptable_cv do
      {:alert, Map.put(result, :reason, "High variance detected")}
    else
      {:ok, result}
    end
  end

  defp classify_stability(cv) when cv < 10, do: :excellent
  defp classify_stability(cv) when cv < 25, do: :good
  defp classify_stability(cv) when cv < 50, do: :acceptable
  defp classify_stability(_cv), do: :unstable
end

Quality variance analysis reveals whether platform quality is uniformly high or hiding weaknesses in specific domains. With the current 100/100 perfect score across all 13 domains, the variance is zero, confirming uniform quality excellence.

Cross-References

Z-Score - Standard deviation distance measure
Time Series - Temporal data analyzed for variance
Trend - Directional pattern analysis
Monitoring - Source of variance data
Anomaly Detection - Variance-based outlier identification

Created by Tomas Korcak (korczis) | GitHub

Variance

Definition & Overview

Technical Deep Dive

Architecture & Implementation

Usage in Prismatic Platform

Cross-References

See Also

Glossary

Related Terms