Prismatic Resilience

Overview

Prismatic Resilience provides the platform's fault tolerance and chaos engineering infrastructure. In a system integrating 121+ external OSINT sources, government registries, and third-party APIs, failures are not exceptional events -- they are guaranteed. Prismatic Resilience ensures the platform degrades gracefully, recovers automatically, and maintains service quality even when individual components fail. The application implements circuit breakers, bulkhead isolation, retry strategies, fallback chains, and chaos engineering tools that proactively test the platform's failure handling under controlled conditions.

The resilience architecture is built on OTP supervision principles but extends them with application-level fault tolerance patterns specifically designed for external service dependencies. While OTP supervisors handle process crashes through restart strategies, Prismatic Resilience handles a different class of failures: network timeouts, rate limiting, service degradation, and cascading failures across distributed systems. Each external service dependency is wrapped in a circuit breaker that monitors failure rates, opens the circuit when a threshold is exceeded, and periodically tests recovery by allowing probe requests through in a half-open state.

The chaos engineering subsystem enables proactive fault discovery by injecting controlled failures -- latency, errors, and resource constraints -- into specific system components during test campaigns. This approach, inspired by Netflix's Chaos Monkey philosophy, ensures that the platform's resilience mechanisms are continuously validated against realistic failure scenarios rather than waiting for production incidents to expose weaknesses.

Architecture

PrismaticResilience.Application (Supervisor)
|
+-- CircuitBreaker.Registry (GenServer)
|   Manages per-service circuit breaker state
|
+-- Bulkhead.Supervisor (DynamicSupervisor)
|   +-- :osint_queries pool (max 20 concurrent)
|   +-- :czech_registries pool (max 5 concurrent)
|   +-- :sanctions_screening pool (max 10 concurrent)
|
+-- Chaos.ExperimentRunner (GenServer)
|   +-- Chaos.LatencyInjector
|   +-- Chaos.FailureInjector
|
+-- HealthCheck.Monitor (GenServer)
|   Periodic service health assessment
|
+-- SelfHealing.Coordinator (GenServer)
    Automatic recovery orchestration

Self-Healing Flow

Service Failure Detected
        |
Circuit Breaker Opens --> Fallback Chain Activated
        |                       |
Retry Strategy Engaged    Cached Data Served
        |
Service Recovers --> Circuit Half-Open --> Test Calls --> Circuit Closes
        |
Self-Healing Report --> Telemetry Event --> Dashboard Update

Key Modules

Configuration

config :prismatic_resilience,
  # Circuit breaker defaults
  default_failure_threshold: 5,
  default_reset_timeout: 60_000,
  default_half_open_max_calls: 2,
  default_monitor_window: 120_000,

  # Bulkhead pools
  bulkheads: [
    osint_queries: [max_concurrent: 20, max_queue: 50],
    czech_registries: [max_concurrent: 5, max_queue: 10],
    sanctions_screening: [max_concurrent: 10, max_queue: 30]
  ],

  # Health check intervals
  health_check_interval: :timer.seconds(30),

  # Chaos engineering (disabled in production by default)
  chaos_enabled: Mix.env() == :test

API Reference

# Execute a function through a circuit breaker
case PrismaticResilience.call(:shodan, fn -> Shodan.host("1.2.3.4") end) do
  {:ok, result} -> process_result(result)
  {:error, :circuit_open} -> PrismaticCache.get("shodan:host:1.2.3.4")
  {:error, reason} -> handle_error(reason)
end

# Fallback chain with ordered alternatives
{:ok, result} = PrismaticResilience.with_fallback([
  fn -> Shodan.host(ip) end,
  fn -> Censys.get_host(ip) end,
  fn -> PrismaticCache.get(cache_key) end,
  fn -> {:ok, %{status: :partial, source: :none}} end
])

# Exponential backoff retry with jitter
{:ok, result} = PrismaticResilience.retry(
  fn -> ExternalApi.call(params) end,
  strategy: :exponential_backoff,
  base_delay: 1_000,
  max_delay: 30_000,
  max_retries: 5,
  jitter: :full,
  retry_on: [TimeoutError, ConnectionError]
)

# Chaos experiment with assertions
{:ok, report} = PrismaticResilience.Chaos.run_experiment(%{
  name: "osint_source_failure",
  faults: [
    %{target: :shodan, type: :failure, probability: 1.0},
    %{target: :censys, type: :latency, delay: 10_000}
  ],
  duration: :timer.minutes(15),
  assertions: [
    {:response_time, :p99, :<, 5_000},
    {:error_rate, :<, 0.01},
    {:fallback_activation, :>=, 1}
  ]
})

# System-wide health check
{:ok, health} = PrismaticResilience.health_check()

Testing

Resilience testing is inherently complex because it requires simulating failure conditions. The test suite uses the chaos engineering subsystem to inject controlled failures and verify that circuit breakers, fallbacks, and retries behave correctly under stress.

mix test apps/prismatic_resilience/test
mix test apps/prismatic_resilience/test --cover

Integration Points

• Prismatic Safety -- Quality floor recovery triggers resilience mechanisms
• Prismatic Cache -- Fallback data source when primary services are unavailable
• Prismatic Telemetry -- Resilience metrics emitted for latency and failure rate monitoring
• Prismatic OSINT Core -- External service fault tolerance for all OSINT source queries
• Prismatic Claude -- Session lifecycle circuit breakers protecting AI service calls

NABLA Compliance

Resilience decisions maintain epistemic integrity through provenance tracking. Every circuit breaker state transition is logged with timestamp, failure count, and the specific error that triggered the transition. Fallback chain activations record which alternatives were attempted and which succeeded, maintaining full Signal Plurality by preserving data from multiple source attempts rather than silently discarding failed responses. Health check results carry Time Decay metadata, ensuring that stale health assessments are weighted lower than fresh observations. The chaos engineering subsystem explicitly models uncertainty by running experiments with statistical assertions rather than deterministic pass/fail checks, aligning with the Unknown Valid axiom.

Performance

Related Resources

• Prismatic Storage Redis -- Distributed lock and cache infrastructure supporting resilience fallbacks
• Prismatic Visitor Intelligence -- Consumer of resilience patterns for OSINT enrichment queries
• Prismatic Perimeter -- EASM discovery leveraging circuit breakers for external service calls

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator