Explainability

ai

Explainability

Definition

Explainability (also referred to as interpretability or transparency in AI contexts) is the property of a software system, algorithm, or decision-making process that allows human observers to understand why a particular output was produced, how internal logic was applied, and what factors influenced the result. In the context of the Prismatic Platform, explainability extends beyond traditional AI/ML interpretability to encompass the entire platform's decision-making chain -- from agent orchestration decisions through quality gate evaluations to epistemic reasoning outcomes.

Explainability is distinct from mere transparency (making code visible) or observability (collecting metrics). It specifically addresses the cognitive bridge between a system's internal state and a human's ability to form an accurate mental model of that state. A system is explainable when a qualified observer can predict its behavior given a set of inputs and can reconstruct the reasoning that led to any particular output.

Overview

The demand for explainability has grown dramatically as software systems take on increasingly consequential roles in human affairs. When an AI agent recommends a security posture change, when a quality gate rejects a commit, or when an epistemic reasoning system flags a contradiction, the humans affected by those decisions need to understand the reasoning behind them. Explainability is not a luxury feature -- it is a fundamental requirement for any system that participates in consequential decision-making.

In traditional software, explainability was often trivial: deterministic code paths could be traced through debuggers, and log files recorded the sequence of operations. The challenge emerged with the advent of machine learning models (particularly deep neural networks), probabilistic reasoning systems, and multi-agent architectures where emergent behaviors arise from the interaction of many independent components. In these systems, the relationship between inputs and outputs becomes non-linear, high-dimensional, and resistant to simple causal narratives.

The Prismatic Platform addresses explainability at multiple architectural layers. The NABLA Infinity framework enforces provenance tracking on all beliefs and decisions. The Trinity Gate requires structural, logical, and formal consistency -- each of which produces auditable evidence. The agent hierarchy documents its decision chains through explicit authority levels and escalation protocols. Together, these mechanisms create a platform where explainability is not bolted on after the fact but woven into the fundamental architecture.

Why Explainability Matters

  1. Regulatory compliance: GDPR Article 22 establishes the right to meaningful information about automated decision logic. NIS2 and ZKB frameworks require demonstrable understanding of security decision-making processes.

  2. Trust calibration: Humans cannot appropriately trust or distrust a system they do not understand. Explainability enables calibrated trust -- knowing when to rely on the system and when to override it.

  3. Debugging and improvement: When a system produces an incorrect or unexpected output, explainability is the foundation for diagnosis. Without it, debugging degrades to trial-and-error.

  4. Accountability: Organizations deploying AI systems must be able to assign responsibility for decisions. Explainability creates the audit trail that makes accountability possible.

  5. Knowledge transfer: Explainable systems teach their operators. Over time, the explanations generated by an explainable system become a knowledge base that improves the organization's collective understanding.

Technical Details

Explainability in Elixir/OTP Systems

Elixir's functional nature and OTP's process model provide natural foundations for explainability. Pure functions with explicit inputs and outputs are inherently more explainable than stateful objects with hidden side effects. The BEAM VM's process isolation ensures that the scope of any decision can be bounded to a specific process and its message history.

defmodule Prismatic.Explainability.DecisionTrace do
  @moduledoc """
  Captures and structures the reasoning chain behind platform decisions.
  Every decision point records its inputs, applied rules, confidence level,
  and the resulting output -- creating a complete audit trail.
  """

  @type trace_entry :: %{
    timestamp: DateTime.t(),
    decision_point: atom(),
    inputs: map(),
    rules_applied: [atom()],
    confidence: float(),
    output: term(),
    explanation: String.t()
  }

  @type decision_trace :: %{
    trace_id: String.t(),
    entries: [trace_entry()],
    final_decision: term(),
    total_confidence: float()
  }

  @spec record_decision(atom(), map(), [atom()], term()) :: trace_entry()
  def record_decision(decision_point, inputs, rules_applied, output) do
    confidence = calculate_confidence(rules_applied, inputs)
    explanation = generate_explanation(decision_point, inputs, rules_applied, output)

    %{
      timestamp: DateTime.utc_now(),
      decision_point: decision_point,
      inputs: sanitize_inputs(inputs),
      rules_applied: rules_applied,
      confidence: confidence,
      output: output,
      explanation: explanation
    }
  end

  @spec generate_explanation(atom(), map(), [atom()], term()) :: String.t()
  defp generate_explanation(decision_point, inputs, rules_applied, output) do
    rules_text =
      rules_applied
      |> Enum.map(&rule_description/1)
      |> Enum.join("; ")

    "Decision '#{decision_point}' evaluated #{map_size(inputs)} inputs " <>
      "against #{length(rules_applied)} rules (#{rules_text}), " <>
      "producing output: #{inspect(output)}"
  end

  defp sanitize_inputs(inputs) do
    Map.drop(inputs, [:api_key, :password, :token, :secret])
  end

  defp calculate_confidence(rules_applied, inputs) do
    rule_count = length(rules_applied)
    input_completeness = calculate_input_completeness(inputs)
    min(1.0, rule_count * 0.2 * input_completeness)
  end

  defp calculate_input_completeness(inputs) do
    total = map_size(inputs)
    non_nil = inputs |> Map.values() |> Enum.count(&(not is_nil(&1)))
    if total == 0, do: 0.0, else: non_nil / total
  end

  defp rule_description(:threshold_check), do: "threshold validation"
  defp rule_description(:pattern_match), do: "pattern matching"
  defp rule_description(:confidence_gate), do: "confidence gating"
  defp rule_description(:trinity_validation), do: "Trinity Gate validation"
  defp rule_description(rule), do: "#{rule}"
end

Explainability Layers in the Prismatic Architecture

The platform implements explainability across five distinct layers, each addressing a different aspect of system understanding:

defmodule Prismatic.Explainability.LayeredExplanation do
  @moduledoc """
  Five-layer explanation architecture that provides different levels
  of detail depending on the audience and context.
  """

  @type explanation_level :: :executive | :operational | :technical | :formal | :raw

  @spec explain(term(), explanation_level()) :: String.t()
  def explain(decision, :executive) do
    "The system #{action_verb(decision)} because #{primary_reason(decision)}."
  end

  def explain(decision, :operational) do
    factors = decision |> contributing_factors() |> Enum.join(", ")
    "Decision based on: #{factors}. " <>
      "Confidence: #{decision.confidence}. " <>
      "Override available: #{decision.overridable?}."
  end

  def explain(decision, :technical) do
    trace = decision |> decision_trace() |> format_trace()
    "Execution path: #{trace}\n" <>
      "Rules evaluated: #{length(decision.rules)}\n" <>
      "Data sources: #{inspect(decision.sources)}"
  end

  def explain(decision, :formal) do
    "Given premises: #{format_premises(decision.premises)}\n" <>
      "Applied inference: #{decision.inference_rule}\n" <>
      "Conclusion: #{decision.conclusion}\n" <>
      "Proof status: #{decision.proof_status}"
  end

  def explain(decision, :raw) do
    inspect(decision, pretty: true, limit: :infinity)
  end

  defp action_verb(%{type: :approval}), do: "approved the request"
  defp action_verb(%{type: :rejection}), do: "rejected the request"
  defp action_verb(%{type: :escalation}), do: "escalated the decision"
  defp action_verb(_), do: "made this determination"

  defp primary_reason(%{reasons: [first | _]}), do: first
  defp primary_reason(_), do: "multiple factors contributed"

  defp contributing_factors(%{factors: factors}), do: factors
  defp contributing_factors(_), do: ["insufficient data for factor analysis"]

  defp decision_trace(%{trace: trace}), do: trace
  defp decision_trace(_), do: []

  defp format_trace(trace), do: Enum.map_join(trace, " -> ", &to_string/1)
  defp format_premises(premises), do: Enum.map_join(premises, ", ", &to_string/1)
end

NABLA Infinity and Explainability

The NABLA Infinity epistemic framework enforces seven axioms that directly support explainability. The Provenance Mandatory axiom requires that every belief in the system be traceable to its source. The Signal Plurality axiom ensures that conclusions are based on multiple independent signals, each of which can be individually examined. The Contradiction Preservation axiom prevents the system from silently resolving conflicts, instead surfacing them for human review -- a critical explainability mechanism.

When the Trinity Gate evaluates a claim, it produces three independent assessments (structural, logical, and formal) that together constitute a comprehensive explanation of why a claim was accepted or rejected. This is fundamentally different from a binary pass/fail gate -- each assessment carries its own reasoning chain that can be inspected independently.

Implementation

Implementing Explainability in Practice

Effective explainability implementation follows a systematic approach:

Step 1: Identify Decision Points. Map every location in the system where a consequential choice is made. In the Prismatic Platform, these include quality gate evaluations, agent task assignments, security posture assessments, and epistemic reasoning conclusions.

Step 2: Instrument Decision Points. Each decision point must record its inputs, the rules or models applied, and the resulting output. This instrumentation must be lightweight enough to run in production without degrading performance.

Step 3: Build Explanation Generators. Raw decision data must be transformed into human-readable explanations at appropriate levels of abstraction. Different audiences need different levels of detail.

Step 4: Create Feedback Loops. Explanations should be reviewable, and reviews should feed back into the system to improve future explanations. When an operator finds an explanation insufficient or confusing, that feedback becomes training data for explanation improvement.

Step 5: Validate Explanations. An explanation is only useful if it is faithful to the system's actual reasoning. Regularly verify that explanations accurately reflect the decision process, rather than providing plausible-sounding post-hoc rationalizations.

Agent Explainability in Multi-Agent Systems

In the Prismatic Platform's 530-agent ecosystem, explainability takes on additional dimensions. Each agent must be able to explain not only its own decisions but also how it interpreted and acted upon instructions from higher-level agents. The chain-of-command architecture (L1 through L5) creates a natural hierarchy of explanations:

defmodule Prismatic.Explainability.AgentTrace do
  @moduledoc """
  Traces decision chains across the agent hierarchy,
  producing end-to-end explanations that span multiple agents.
  """

  @spec trace_decision_chain(String.t()) :: {:ok, [map()]} | {:error, term()}
  def trace_decision_chain(decision_id) do
    with {:ok, root} <- find_root_decision(decision_id),
         {:ok, chain} <- build_chain(root, []) do
      {:ok, Enum.reverse(chain)}
    end
  end

  defp build_chain(%{delegated_to: nil} = decision, acc) do
    {:ok, [format_node(decision) | acc]}
  end

  defp build_chain(%{delegated_to: child_id} = decision, acc) do
    case find_decision(child_id) do
      {:ok, child} -> build_chain(child, [format_node(decision) | acc])
      {:error, _} = error -> error
    end
  end

  defp format_node(decision) do
    %{
      agent: decision.agent_id,
      level: decision.authority_level,
      action: decision.action,
      reasoning: decision.reasoning,
      timestamp: decision.timestamp
    }
  end

  defp find_root_decision(id), do: {:ok, %{id: id, delegated_to: nil}}
  defp find_decision(id), do: {:ok, %{id: id, delegated_to: nil}}
end

Comparison

ConceptFocusRelationship to Explainability
TransparencyMaking internals visibleNecessary but not sufficient; raw visibility without interpretation is not explainability
InterpretabilityModel-specific understandingA subset of explainability focused on ML model internals
ObservabilitySystem behavior monitoringProvides the data foundation; explainability adds the "why"
AuditabilityCompliance verificationRelies on explainability for meaningful audit trails
AccountabilityResponsibility assignmentRequires explainability to determine what happened and who is responsible
DebuggabilityFault diagnosisUses explainability artifacts as primary diagnostic tools

Approaches to Explainability

ApproachStrengthsWeaknessesPrismatic Usage
Ante-hoc (built-in)Faithful to actual reasoningMay constrain model selectionPrimary approach via NABLA axioms
Post-hoc (after-the-fact)Works with any modelMay not reflect true reasoningUsed for legacy integrations
Local explanationsPrecise for individual decisionsNo global understandingPer-decision trace records
Global explanationsOverall behavior understandingMay miss edge casesQuality DNA aggregate reports
Counterfactual"What-if" understandingComputationally expensiveUsed in security scenario analysis
Example-basedIntuitive for non-expertsRequires curated example setsUsed in documentation and training

Best Practices

Designing for Explainability

  1. Prefer simpler models when performance is comparable. A decision tree that achieves 95% accuracy is often more valuable than a neural network at 97% accuracy if the additional 2% does not justify the loss of interpretability.

  2. Record inputs at decision boundaries, not everywhere. Capturing every intermediate computation creates noise. Focus instrumentation on the points where meaningful choices are made.

  3. Use structured explanation formats. Free-text explanations are difficult to aggregate and analyze. Define schemas for explanations that enable programmatic processing.

  4. Layer explanations for different audiences. The same decision may need to be explained to an executive (one sentence), an operator (one paragraph), and a developer (full trace). Build all three from the same underlying data.

  5. Test explanations with real users. An explanation that makes sense to the developer who wrote it may be incomprehensible to the operator who receives it. User testing of explanations is as important as functional testing.

  6. Treat explanation quality as a first-class metric. Track whether explanations are read, understood, and acted upon. Low engagement with explanations may indicate they are not useful.

  7. Make explanations actionable. An explanation that says "the confidence threshold was not met" is less useful than one that says "the confidence threshold of 0.95 was not met (actual: 0.87) because signal X was missing -- consider providing X to reach the threshold."

Pitfalls

Common Explainability Anti-Patterns

  1. Post-hoc rationalization. Generating plausible-sounding explanations that do not reflect the actual decision process. This is worse than no explanation at all because it creates false confidence. The Prismatic Platform's NABLA Provenance Mandatory axiom prevents this by requiring traceable reasoning chains.

  2. Explanation overload. Providing too much detail overwhelms the audience and defeats the purpose. A 500-line trace dump is technically accurate but practically useless for most stakeholders.

  3. Static explanations for dynamic systems. Explanations that describe how the system was designed to work rather than how it actually worked in a specific instance. Always generate explanations from actual execution data, not from design documentation.

  4. Ignoring uncertainty. Explanations that present probabilistic outcomes as certainties. If the system's confidence was 0.72, the explanation should communicate that uncertainty rather than presenting the output as definitive.

  5. Cherry-picking supporting evidence. Explanations that highlight factors supporting the decision while omitting factors that argued against it. The NABLA Contradiction Preservation axiom explicitly prevents this pattern.

  6. Assuming the audience. Providing developer-level explanations to business users or oversimplified explanations to technical reviewers. Always match the explanation level to the audience.

  7. Treating explainability as optional. Deferring explainability to "version 2" or treating it as a nice-to-have feature. Systems that are not explainable from the start are extremely difficult to make explainable retroactively.

Use Cases

Practical Applications

Quality Gate Decisions. When the Prismatic Platform's quality gate rejects a commit, the developer needs to understand exactly which checks failed, why they failed, and what needs to change. The quality gate's explanation includes the specific rule violated, the offending code location, and a reference to the policy that mandates the check.

Security Posture Assessment. When the Perimeter module assigns a security rating of B (780/900) to a domain, the assessment must explain which factors contributed to the score, which factors reduced it, and what specific actions would improve it. This is critical for both internal decision-making and client communication.

Agent Task Delegation. When the orchestration layer assigns a task to a specific agent rather than an alternative, the delegation decision should be explainable -- was it based on the agent's specialization, availability, authority level, or past performance? This transparency prevents opacity in the multi-agent coordination layer.

Epistemic Reasoning. When the NABLA framework flags a contradiction between two signals, the explanation must identify the specific signals, their sources, the nature of the contradiction, and the framework's recommended resolution approach. This is perhaps the most critical explainability use case because epistemic decisions directly affect the platform's relationship with truth.

Compliance Reporting. Regulatory bodies (NIS2, GDPR, ZKB) require organizations to demonstrate that their automated decision-making systems can explain their outputs. The platform's layered explanation architecture directly supports the generation of compliance-ready explanation documents.

Explainability connects deeply with several other concepts in the Prismatic Platform ecosystem:

  • Observability provides the raw telemetry data that explainability transforms into human-understandable narratives
  • Audit Trail captures the sequential record of decisions that explainability interprets and contextualizes
  • Transparency Builds Trust articulates the philosophical foundation for why explainability matters
  • NABLA Infinity enforces provenance tracking and signal plurality that make platform-wide explainability possible
  • Trinity Gate produces three-dimensional explanations through structural, logical, and formal consistency checks
  • Bias Detection relies on explainability to surface and diagnose systematic biases in decision-making
  • Epistemic Reasoning uses explainability as its primary mechanism for communicating uncertainty and reasoning chains
  • Compliance Framework requires explainability for regulatory adherence under GDPR, NIS2, and ZKB
  • Quality Gates produce structured explanations when accepting or rejecting code changes
  • Telemetry provides the instrumentation layer that captures the raw data for explanation generation

See Also

Connect & Contribute: Explainability is a living concept that evolves with the platform. Visit the Prismatic Platform repository to explore the implementation, review the NABLA Infinity doctrine for the epistemic foundations, or connect with the community through GitHub Discussions. Created by Tomas Korcak (korczis).

Back to Glossary