Scientific Rigor

core

Scientific Rigor

Definition

Scientific rigor in software engineering is the disciplined application of the scientific method to the design, implementation, testing, and operation of software systems. It demands that engineering decisions be grounded in evidence rather than opinion, that claims about system behavior be falsifiable and verifiable, that experiments be reproducible, and that conclusions acknowledge uncertainty and limitations. Scientific rigor rejects dogma, appeals to authority, and untested assumptions in favor of systematic observation, controlled experimentation, and transparent reasoning.

In the Prismatic Platform, scientific rigor is not an aspirational guideline but an enforced requirement. The NABLA Infinity framework codifies seven non-negotiable axioms that operationalize scientific rigor: signal plurality, contradiction preservation, absence as information, time decay, unknown as valid, source independence, and provenance tracking. The Trinity Gate requires that every significant claim pass structural, logical, and formal verification before being accepted. Together, these mechanisms ensure that the platform's 530+ agents, 115 applications, and 2.8 million lines of code evolve based on evidence, not folklore.

Historical Foundations

The relationship between scientific method and engineering practice has a long and productive history. The roots trace to Francis Bacon's Novum Organum (1620), which formalized inductive reasoning and empirical observation as the foundation of natural philosophy. Karl Popper's falsificationism (1934) added the crucial requirement that scientific claims must be testable and potentially refutable. Thomas Kuhn's Structure of Scientific Revolutions (1962) demonstrated how paradigm shifts occur when accumulated anomalies overwhelm existing frameworks.

In software engineering specifically, the evidence-based movement gained momentum through the work of Barbara Kitchenham, who adapted medical evidence-based practice to software development in the early 2000s. Her systematic review methodology brought statistical rigor to questions that the industry had previously resolved through opinion, anecdote, and appeals to authority. The Prismatic Platform builds on this tradition, extending it with automated enforcement mechanisms that prevent regression to opinion-based decision-making.

Platform Context

The Prismatic Platform operationalizes scientific rigor through multiple interconnected systems. Rather than treating rigor as a cultural aspiration, the platform encodes it into automated quality gates, measurement systems, and decision frameworks that actively prevent unsubstantiated claims from influencing system behavior.

The platform's NO MERCY, NO DOUBTS doctrine directly reflects scientific values: NO MERCY demands complete, verifiable implementations (no untested claims); NO DOUBTS requires full investigation before action (evidence before conclusion). This doctrine is not metaphorical -- it is enforced through pre-commit hooks, quality gates, and automated analysis that block code changes lacking adequate evidence of correctness.

defmodule PrismaticQuality.ScientificRigor do
  @moduledoc """
  Implements scientific rigor checks for platform decisions.
  Ensures that engineering decisions are backed by evidence,
  claims are falsifiable, and results are reproducible.

  ## Rigor Levels

  - L1: Anecdotal (single observation, lowest confidence)
  - L2: Correlational (pattern across multiple observations)
  - L3: Experimental (controlled experiment with hypothesis)
  - L4: Replicated (independently reproduced results)
  - L5: Formal (mathematically proven, highest confidence)
  """

  @type rigor_level :: :anecdotal | :correlational | :experimental | :replicated | :formal
  @type evidence :: %{
          source: String.t(),
          rigor_level: rigor_level(),
          timestamp: DateTime.t(),
          methodology: String.t(),
          reproducible: boolean(),
          confidence: float()
        }
  @type claim :: %{
          statement: String.t(),
          evidence: [evidence()],
          falsifiable: boolean(),
          tested: boolean()
        }

  @spec evaluate_claim(claim()) :: {:ok, :accepted, float()} | {:error, :insufficient_evidence}
  def evaluate_claim(%{evidence: evidence, falsifiable: falsifiable, tested: tested} = _claim) do
    with :ok <- verify_falsifiability(falsifiable),
         :ok <- verify_testing(tested),
         {:ok, confidence} <- aggregate_evidence(evidence),
         :ok <- verify_minimum_confidence(confidence) do
      {:ok, :accepted, confidence}
    else
      {:error, reason} -> {:error, reason}
    end
  end

  @spec verify_falsifiability(boolean()) :: :ok | {:error, :not_falsifiable}
  defp verify_falsifiability(true), do: :ok
  defp verify_falsifiability(false), do: {:error, :not_falsifiable}

  @spec verify_testing(boolean()) :: :ok | {:error, :untested_claim}
  defp verify_testing(true), do: :ok
  defp verify_testing(false), do: {:error, :untested_claim}

  @spec aggregate_evidence([evidence()]) :: {:ok, float()} | {:error, :insufficient_evidence}
  defp aggregate_evidence(evidence_list) when length(evidence_list) < 2 do
    {:error, :insufficient_evidence}
  end

  defp aggregate_evidence(evidence_list) do
    weighted_confidence =
      evidence_list
      |> Enum.map(fn e -> rigor_weight(e.rigor_level) * e.confidence end)
      |> Enum.sum()

    total_weight =
      evidence_list
      |> Enum.map(fn e -> rigor_weight(e.rigor_level) end)
      |> Enum.sum()

    {:ok, weighted_confidence / total_weight}
  end

  @spec rigor_weight(rigor_level()) :: float()
  defp rigor_weight(:anecdotal), do: 0.1
  defp rigor_weight(:correlational), do: 0.3
  defp rigor_weight(:experimental), do: 0.6
  defp rigor_weight(:replicated), do: 0.8
  defp rigor_weight(:formal), do: 1.0

  @spec verify_minimum_confidence(float()) :: :ok | {:error, :insufficient_evidence}
  defp verify_minimum_confidence(confidence) when confidence >= 0.80, do: :ok
  defp verify_minimum_confidence(_), do: {:error, :insufficient_evidence}
end

The Scientific Method in Software Engineering

Hypothesis Formation

Every significant engineering decision in the platform begins with a testable hypothesis. Rather than "we should use GenServer for state management" (opinion), the scientific approach demands "Using GenServer for state X will provide sub-millisecond response times under Y concurrent connections while maintaining consistency guarantees, as measurable by benchmark Z" (hypothesis). This formulation is falsifiable, measurable, and directly testable.

The platform's Quality DNA system records hypotheses alongside their test results, creating a searchable corpus of engineering decisions with their supporting evidence. This prevents the recurrence of debates that have already been settled by evidence and provides historical context for why architectural decisions were made.

Controlled Experimentation

The Prismatic Platform supports controlled experimentation through its sandbox infrastructure and benchmark framework. When evaluating competing implementations, the platform runs both under identical conditions with controlled variables, collecting statistically meaningful performance data. The AutoEvolve system uses this capability to evaluate proposed mutations against the existing codebase, accepting only changes that demonstrate measurable improvement.

Reproducibility

A hallmark of scientific rigor is that results must be reproducible by independent parties under the same conditions. In software, this translates to deterministic builds, reproducible test environments, and version-pinned dependencies. The platform enforces reproducibility through locked dependency versions, containerized test environments, and seed-controlled randomized testing.

defmodule PrismaticQuality.Reproducibility do
  @moduledoc """
  Ensures that platform measurements and test results
  are reproducible across environments and over time.
  """

  @spec verify_reproducibility(function(), pos_integer()) ::
          {:ok, :reproducible} | {:error, :non_reproducible, map()}
  def verify_reproducibility(experiment_fn, iterations \\ 10) do
    results = Enum.map(1..iterations, fn _i -> experiment_fn.() end)

    if all_consistent?(results) do
      {:ok, :reproducible}
    else
      variance = calculate_variance(results)
      {:error, :non_reproducible, %{variance: variance, results: results}}
    end
  end

  @spec all_consistent?([term()]) :: boolean()
  defp all_consistent?(results) do
    results
    |> Enum.uniq()
    |> length()
    |> Kernel.==(1)
  end

  @spec calculate_variance([number()]) :: float()
  defp calculate_variance(results) when is_list(results) do
    mean = Enum.sum(results) / length(results)
    Enum.map(results, fn x -> (x - mean) ** 2 end) |> Enum.sum() |> Kernel./(length(results))
  end
end

Falsifiability and Peer Review

Claims that cannot be proven wrong are not scientifically useful. In the platform context, this means that every assertion about system behavior must come with a way to disprove it. "This function is fast" is not falsifiable; "This function processes 10,000 requests per second on standard hardware" is falsifiable through benchmarking.

Peer review in the platform context is automated through Credo static analysis, Dialyzer type checking, property-based testing, and the multi-phase pre-commit validation pipeline. These tools serve as automated "reviewers" that catch classes of errors systematically rather than relying on human attention.

NABLA Axioms as Scientific Principles

The seven NABLA axioms directly encode scientific principles into the platform's operation:

Signal Plurality maps to the scientific requirement for multiple independent observations. A single measurement is an anecdote; multiple measurements form data. The platform requires minimum two independent signals before establishing any belief.

Contradiction Preservation reflects the scientific obligation to acknowledge anomalous results rather than discarding them. When two measurements disagree, both are preserved as data requiring explanation. The Addiction Preservation doctrine ensures that inconvenient evidence is never suppressed.

Absence as Information corresponds to the scientific principle that negative results are results. The absence of expected behavior is itself evidence that informs analysis, not merely a gap in knowledge.

Time Decay implements the scientific understanding that knowledge has a validity period. Measurements and observations must be timestamped, and older evidence carries less weight as the system evolves. The platform enforces mandatory timestamps on all beliefs.

Unknown as Valid reflects the scientific commitment to honest uncertainty. "I don't know" is a legitimate and valuable state, preferable to false certainty. The platform explicitly represents uncertainty rather than defaulting to assumptions.

Source Independence corresponds to the scientific preference for independent replication. Evidence from independent sources carries more weight than correlated observations from the same source.

Provenance Tracking implements the scientific requirement for methodological transparency. Every conclusion must be traceable back to its supporting evidence and methodology, enabling verification and critique.

Evidence Hierarchy in Practice

The platform recognizes a structured hierarchy of evidence quality, from anecdotal observations to formal mathematical proofs. This hierarchy directly influences how much confidence the system places in different claims:

LevelTypeExampleConfidence Weight
L1Anecdotal"It seemed faster after the change"0.1
L2Correlational"Performance improved in 8 of 10 measured runs"0.3
L3Experimental"Controlled benchmark shows 15% improvement (p < 0.05)"0.6
L4Replicated"Three independent teams reproduced the 15% improvement"0.8
L5Formal"Proven correct via Dialyzer types and property-based tests"1.0

The Trinity Gate requires that critical decisions be supported by evidence at L3 or higher. The quality gates system enforces this requirement automatically, blocking changes that rely solely on L1 or L2 evidence for critical paths.

Measurement and Metrics

Scientific rigor demands valid measurement instruments. The platform's measurement systems are designed with explicit attention to measurement validity -- ensuring that what is measured actually reflects what is intended to be measured. Lines of code do not measure productivity. Test count does not measure quality. Code coverage does not guarantee correctness. The platform's quality measurement system uses composite metrics that resist Goodhart's Law (when a measure becomes a target, it ceases to be a good measure).

The Quality DNA system tracks 13 quality domains, each measured through multiple independent indicators. No single metric determines quality; instead, the composite score reflects genuine system health through domain-specific analysis including Dialyzer verification, Credo compliance, compilation cleanliness, and typespec coverage.

Anti-Patterns: Pseudo-Scientific Engineering

Scientific rigor also means recognizing and rejecting pseudo-scientific practices that masquerade as evidence-based engineering:

Cargo Cult Metrics: Collecting metrics without understanding what they measure or how to act on them. The platform avoids this by tying every metric to a specific decision or action threshold.

Confirmation Bias in Testing: Writing tests that confirm expected behavior without testing for failure modes. The platform's adversarial testing approach, including Red Team simulations, explicitly seeks disconfirming evidence.

Appeal to Authority: Accepting a technical approach because an industry leader uses it, without evaluating it against the platform's specific requirements. NABLA's source independence axiom guards against this.

Survivorship Bias: Drawing conclusions from successful systems without examining failures. The platform's audit trail and post-mortem processes capture both successes and failures as evidence.

HiPPO Effect: Highest Paid Person's Opinion overriding evidence. The NO MERCY doctrine's evidence-over-opinion principle explicitly prevents this by requiring evidence for all claims regardless of who makes them.

Continuous Improvement Through Scientific Process

The AutoEvolve system embodies the scientific method in automated form. It generates hypotheses (proposed code mutations), runs controlled experiments (benchmark comparisons), evaluates results statistically, and accepts or rejects changes based on evidence. This creates a continuous improvement cycle that operates with scientific discipline at machine speed.

Each evolution cycle produces a structured report documenting the hypothesis, methodology, results, and conclusion -- a miniature scientific paper for every change. These reports accumulate into a searchable knowledge base that informs future evolution decisions.

Relationship to Quality Culture

Scientific rigor is the foundation upon which the platform's quality culture is built. The Quality Floor Guardian monitors quality metrics with statistical process control techniques borrowed from manufacturing science. Quality thresholds are set based on empirical analysis of what levels correlate with production reliability, not arbitrary round numbers. The 100/100 quality score represents genuine achievement across 13 independently measured domains, not a single inflated metric.

The Quality DNA system preserves cross-session quality context, ensuring that quality improvements are cumulative and that regressions are detected through comparison with historical baselines. This longitudinal tracking enables the platform to identify quality trends and intervene before problems become critical.

Connect & Contribute: Created by Tomas Korcak (korczis) | GitHub | GitLab | LinkedIn | Contact

Back to Glossary