Regression Testing

Overview

The Mandatory Regression Test Protocol ensures that every bug fixed on the Prismatic Platform is a bug that can never return. This is not a best practice recommendation but an absolute, non-bypassable enforcement requirement: every bug fix must include a regression test that would have caught the original bug, verified to fail before the fix and pass after it. No exception, no bypass, no deferral.

Regression testing addresses one of the most persistent and costly failure modes in software engineering: the reintroduction of previously fixed bugs. Studies consistently show that 15-25% of bug fixes in complex systems inadvertently reintroduce bugs that were fixed earlier. In a platform with 2.8 million lines of code across 99 umbrella applications, even a small regression rate compounds into significant quality degradation over time. The Mandatory Regression Test Protocol eliminates this failure mode entirely.

The protocol integrates with the NO MERCY doctrine's zero-tolerance standards and is enforced through Quality Gates pre-commit hooks that reject any bug fix commit lacking the required regression test. Combined with Telemetry Integration for tracking regression test execution and Real-Time Monitoring for alerting on test failures, the protocol creates a comprehensive defense against regression.

Protocol Requirements

The Mandatory Regression Test Protocol follows a strict five-phase sequence that must be completed for every bug fix. Each phase has specific deliverables and verification criteria.

Phase 1: Root Cause Investigation

Before writing any fix code, the developer must identify the bug's root cause with sufficient precision to design a targeted regression test. This aligns with the NO DOUBTS doctrine's requirement for full investigation before action.

Phase 2: Regression Test Design

Design the regression test before implementing the fix. The test must be specific enough to catch the exact bug while general enough to detect related regressions.

defmodule Prismatic.Perimeter.AssetProcessorRegressionTest do
  @moduledoc """
  Regression tests for AssetProcessor bug fixes.
  Each test references the original issue and documents the root cause.
  """
  use ExUnit.Case, async: true

  alias Prismatic.Perimeter.AssetProcessor

  describe "regression: handle empty asset list (#1234)" do
    @tag :regression
    @tag issue: "1234"
    @tag root_cause: "Enum.map/2 called on nil instead of empty list"

    test "returns empty result for empty input" do
      # This test would have caught the original bug:
      # AssetProcessor.process([]) crashed because the function
      # pattern-matched on [head | tail] without handling []
      assert {:ok, []} = AssetProcessor.process([])
    end

    test "returns error for nil input" do
      # Related regression: nil was passed through to Enum.map/2
      assert {:error, :invalid_input} = AssetProcessor.process(nil)
    end

    test "handles single-element list correctly" do
      # Boundary case: ensure fix works for list with exactly one element
      asset = build_asset("example.com")
      assert {:ok, [processed]} = AssetProcessor.process([asset])
      assert processed.domain == "example.com"
    end
  end

  describe "regression: certificate date parsing (#1287)" do
    @tag :regression
    @tag issue: "1287"
    @tag root_cause: "DateTime.from_iso8601/1 returned error tuple unhandled"

    test "handles malformed certificate dates gracefully" do
      asset = build_asset("test.com", cert_expiry: "not-a-date")
      assert {:ok, [processed]} = AssetProcessor.process([asset])
      assert is_nil(processed.cert_expiry)
    end

    test "handles missing certificate dates" do
      asset = build_asset("test.com", cert_expiry: nil)
      assert {:ok, [processed]} = AssetProcessor.process([asset])
      assert is_nil(processed.cert_expiry)
    end

    test "parses valid ISO 8601 dates correctly" do
      asset = build_asset("test.com", cert_expiry: "2026-12-31T23:59:59Z")
      assert {:ok, [processed]} = AssetProcessor.process([asset])
      assert %DateTime{year: 2026, month: 12} = processed.cert_expiry
    end
  end

  defp build_asset(domain, opts \\ []) do
    %{
      domain: domain,
      cert_expiry: Keyword.get(opts, :cert_expiry),
      discovered_at: DateTime.utc_now()
    }
  end
end

Phase 3: Test-First Verification

The regression test must be verified to fail with the unfixed code. This critical step proves that the test actually detects the bug, rather than being a vacuous test that always passes.

Phase 4: Fix Implementation and Verification

Apply the fix and verify that the regression test now passes, along with the entire existing test suite.

Phase 5: Regression Report

Every completed bug fix must generate a regression report documenting the fix, test, and verification results.

REGRESSION TEST REPORT
Bug Fixed: AssetProcessor crashes on empty asset list (#1234)
Root Cause: Pattern match on [head | tail] without empty list clause
Test Added: test/prismatic_perimeter/asset_processor_regression_test.exs
  - "returns empty result for empty input"
  - "returns error for nil input"
  - "handles single-element list correctly"
Validation: Test fails before fix, passes after fix
Coverage: Empty list, nil input, single-element boundary case

Test Categories

The regression test protocol supports four categories of regression tests, selected based on the scope and nature of the original bug:

Property-Based Regression Tests

For bugs involving edge cases or boundary conditions, property-based testing with StreamData provides stronger guarantees than example-based tests alone:

defmodule Prismatic.Perimeter.SecurityScoreRegressionTest do
  use ExUnit.Case, async: true
  use ExUnit.Parameterized

  describe "regression: score overflow on extreme inputs (#1301)" do
    @tag :regression
    @tag issue: "1301"

    property "score never exceeds maximum (900) regardless of input" do
      check all findings <- StreamData.integer(0..100_000),
                critical <- StreamData.integer(0..findings),
                info <- StreamData.integer(0..findings) do
        evidence = %{
          findings: findings,
          critical: critical,
          info: info,
          domain: "property-test.com"
        }

        {:ok, rating} = SecurityScore.calculate(evidence)
        assert rating.score >= 300, "Score #{rating.score} below minimum 300"
        assert rating.score <= 900, "Score #{rating.score} above maximum 900"
      end
    end

    property "score decreases monotonically with more critical findings" do
      check all base <- StreamData.integer(0..100),
                delta <- StreamData.integer(1..100) do
        base_evidence = %{findings: 100, critical: base, domain: "mono-test.com"}
        worse_evidence = %{findings: 100, critical: base + delta, domain: "mono-test.com"}

        {:ok, base_rating} = SecurityScore.calculate(base_evidence)
        {:ok, worse_rating} = SecurityScore.calculate(worse_evidence)

        assert worse_rating.score <= base_rating.score
      end
    end
  end
end

Enforcement Mechanisms

The regression test protocol is enforced at multiple levels to ensure zero bypass:

Automated Detection

The pre-commit hook automatically detects bug-fix commits and verifies regression test presence:

defmodule PrismaticSafety.RegressionTestGuard do
  @moduledoc """
  Enforces the Mandatory Regression Test Protocol.
  Detects bug-fix commits and verifies regression test inclusion.
  """

  @bug_fix_indicators ["fix(", "fix:", "bugfix", "hotfix", "regression"]

  @spec validate_commit(commit_message :: String.t(), changed_files :: list(String.t())) ::
    :ok | {:error, :missing_regression_test}
  def validate_commit(commit_message, changed_files) do
    if is_bug_fix_commit?(commit_message) do
      if has_regression_test?(changed_files) do
        :ok
      else
        {:error, :missing_regression_test}
      end
    else
      :ok
    end
  end

  defp is_bug_fix_commit?(message) do
    downcased = String.downcase(message)
    Enum.any?(@bug_fix_indicators, &String.contains?(downcased, &1))
  end

  defp has_regression_test?(changed_files) do
    test_files = Enum.filter(changed_files, &String.ends_with?(&1, "_test.exs"))

    Enum.any?(test_files, fn file ->
      content = File.read!(file)
      String.contains?(content, "@tag :regression") or
        String.contains?(content, "regression:")
    end)
  end
end

Regression Test Tagging Convention

All regression tests follow a consistent tagging convention that enables filtering, reporting, and traceability:

# Run only regression tests
mix test --only regression

# Run regression tests for a specific component
mix test --only component:perimeter --only regression

# Run regression tests for a specific issue
mix test --only issue:1234

Metrics and Effectiveness

The regression test protocol produces measurable improvements in platform stability:

Integration Points

The Mandatory Regression Test Protocol integrates with the broader platform quality infrastructure:

• NO MERCY: Regression protocol is a core enforcement mechanism of the zero-tolerance doctrine
• NO DOUBTS: Root cause investigation aligns with the full investigation pillar
• Quality Gates: Regression test presence verified as a gate requirement
• Trinity Gate: Complex regressions may require formal verification through Trinity
• Session Discipline: Regression tests committed and pushed as part of session flow
• Telemetry Integration: Regression test execution tracked via telemetry events
• Real-Time Monitoring: Regression test failures trigger immediate alerts
• Autonomous Self-Healing: Regression detection can trigger auto-healing cycles
• AIAD Standard: Agent-related bug fixes follow the same regression protocol
• Color Teams: Red Team identifies potential regression vectors for proactive testing

Commands

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator