Prismatic Detection Engine

Overview

Prismatic Detection Engine combines rule-based and statistical detection methods to identify threats, anomalies, and policy violations across all platform intelligence streams. It processes events from OSINT collection, security scans, and system monitoring to generate actionable findings with severity scores, evidence links, and recommended response actions.

The engine supports two complementary detection paradigms. Rule-based detection uses YAML-defined rules with Sigma compatibility for precise, deterministic matching against known threat patterns. Statistical detection employs time-series anomaly scoring, behavioral baseline deviation, and clustering algorithms to surface previously unknown threats that evade signature-based approaches. Both paradigms feed findings into a unified management layer that handles deduplication, correlation, and MITRE ATT&CK technique mapping.

Detection rules are version-controlled and support A/B testing, allowing security teams to evaluate new detection logic against historical data before promoting rules to production. The engine integrates tightly with Prismatic Signals for real-time event ingestion and with Prismatic Traits for behavioral baseline computation. The NABLA framework's signal plurality axiom is enforced throughout the detection pipeline: findings from a single detection method are flagged as preliminary until corroborated by an independent method, preventing false positive proliferation that erodes analyst trust.

Architecture

Event Sources --> Ingestion Layer --> Detection Pipeline --> Finding Store
      |               |                  |                  |
  OSINT Data      Normalization     Rule Engine         Deduplication
  Scan Results    Schema Mapping    Anomaly Detector    Correlation
  System Events   Enrichment        Pattern Matcher     ATT&CK Mapping
                                         |
                                  Alert Generation
                                         |
                                  Response Actions

The pipeline is built on GenStage for backpressure-aware processing. Each detection stage runs as a supervised OTP process, enabling independent scaling of rule evaluation and statistical computation based on load. The separation of detection stages means that adding new detection methods requires only implementing a new stage module and registering it with the pipeline supervisor.

Key Modules

Key Features

Rule Engine

The rule engine processes YAML-defined detection rules with a rich condition language supporting boolean logic, field comparisons, regular expressions, and temporal operators. Rules are compiled into an intermediate representation at load time for efficient evaluation against event streams. The compilation step transforms YAML conditions into an optimized match tree that evaluates in constant time per event regardless of rule complexity.

• YAML-defined detection rules with multi-condition logic and boolean operators
• Sigma rule compatibility for industry-standard threat detection patterns
• Temporal operators for sequence-based and time-windowed rules
• Rule versioning, A/B testing, and effectiveness scoring

# Example detection rule in YAML
title: Suspicious DNS Resolution Pattern
id: dns-tunnel-detection-001
status: production
severity: high
mitre:
  technique: T1071.004
  tactic: command-and-control
detection:
  condition: selection AND NOT filter
  selection:
    event_type: dns_query
    query_length: ">50"
    query_entropy: ">4.0"
  filter:
    domain|endswith:
      - ".googleapis.com"
      - ".microsoft.com"
  timeframe: 5m
  threshold:
    count: 10
    field: source_ip

Rule Compilation Pipeline

Rules pass through a multi-stage compilation pipeline that transforms human-readable YAML into an optimized evaluation structure:

defmodule PrismaticDetectionEngine.RuleCompiler do
  @spec compile(String.t()) :: {:ok, CompiledRule.t()} | {:error, list(CompileError.t())}
  def compile(yaml_content) do
    with {:ok, parsed} <- YamlElixir.read_from_string(yaml_content),
         {:ok, validated} <- validate_schema(parsed),
         {:ok, conditions} <- compile_conditions(validated["detection"]),
         {:ok, mitre_map} <- resolve_mitre_mapping(validated["mitre"]) do
      {:ok, %CompiledRule{
        id: validated["id"],
        title: validated["title"],
        severity: parse_severity(validated["severity"]),
        conditions: conditions,
        mitre: mitre_map,
        compiled_at: DateTime.utc_now()
      }}
    end
  end
end

Statistical Detection

• Time-series anomaly detection using z-score and CUSUM algorithms
• Behavioral baseline computation with configurable training windows
• Clustering for pattern discovery across entity populations
• Predictive models for emerging threat trajectory analysis

# Anomaly detection configuration
defmodule PrismaticDetectionEngine.AnomalyDetector do
  @spec compute_baseline(String.t(), keyword()) :: {:ok, map()} | {:error, term()}
  def compute_baseline(entity_id, opts \\ []) do
    window = Keyword.get(opts, :window, :days_30)
    method = Keyword.get(opts, :method, :z_score)

    with {:ok, historical} <- fetch_historical_metrics(entity_id, window),
         {:ok, baseline} <- calculate_baseline(historical, method) do
      {:ok, %{
        entity: entity_id,
        baseline: baseline,
        method: method,
        window: window,
        computed_at: DateTime.utc_now()
      }}
    end
  end

  @spec score(String.t(), map()) :: {:ok, AnomalyScore.t()} | {:error, term()}
  def score(entity_id, current_metrics) do
    with {:ok, baseline} <- fetch_baseline(entity_id),
         {:ok, deviation} <- compute_deviation(current_metrics, baseline) do
      {:ok, %AnomalyScore{
        entity: entity_id,
        score: deviation.magnitude,
        method: baseline.method,
        dimensions: deviation.per_dimension,
        anomalous: deviation.magnitude > baseline.threshold
      }}
    end
  end
end

Finding Management

• Severity-scored findings with linked evidence artifacts
• Automatic deduplication across detection methods and time windows
• False positive management with feedback-driven tuning
• MITRE ATT&CK technique mapping for standardized threat classification

False Positive Management

The false positive management system uses analyst feedback to continuously improve detection accuracy. When an analyst marks a finding as a false positive, the system records the event context, rule ID, and dismissal reason. This feedback corpus is analyzed periodically to identify rules producing excessive false positives and to generate tuning recommendations:

Usage

# Evaluate an event against all active detection rules
{:ok, findings} = PrismaticDetectionEngine.evaluate(event)

# Load detection rules from YAML directory
{:ok, loaded} = PrismaticDetectionEngine.load_rules("rules/network/*.yaml")
# => %{loaded: 47, valid: 45, errors: 2}

# Compute anomaly score for entity metrics
{:ok, score} = PrismaticDetectionEngine.anomaly_score(entity, metrics)
# => %{score: 3.2, baseline: 1.1, deviation: :high}

# Query findings for a specific entity
{:ok, findings} = PrismaticDetectionEngine.findings(
  entity: "example.com",
  severity: [:high, :critical],
  from: ~U[2026-01-01 00:00:00Z]
)

# A/B test a new rule against historical data
{:ok, results} = PrismaticDetectionEngine.ab_test(
  rule: new_rule,
  dataset: :last_30_days,
  compare_against: existing_rule
)
# => %{new_detections: 12, false_positives: 1, improvement: 0.23}

NABLA Compliance

Testing

Detection rule tests use crafted event fixtures that should trigger specific rules and negative fixtures that should not match. Statistical detection tests verify baseline computation accuracy, anomaly scoring precision, and false positive rates against labeled datasets. Integration tests exercise the full pipeline from event ingestion through detection, finding storage, and alert generation.

Property-based tests generate random events to verify that no detection rule produces unbounded findings and that the pipeline maintains backpressure under load. Rule compilation tests verify YAML parsing correctness, condition tree construction, and Sigma compatibility across all supported rule formats.

Integration Points

Performance

Telemetry events: [:prismatic, :detection, :rule_match], [:prismatic, :detection, :anomaly_scored], [:prismatic, :detection, :finding_created].

Related Resources

• Prismatic Hawkeye -- Visitor behavior detection for web threat analysis
• Alert Management Specialist -- Configures detection rule alerting thresholds and false positive management
• GitLab Security Specialist Agent -- Reviews detection rule logic for security coverage gaps
• Evolution Analyzer Specialist -- Analyzes detection rule effectiveness and recommends optimization
• Intelligence Synthesis -- Correlates detection findings across methods into unified threat assessments
• Real-Time Monitoring -- Real-time event ingestion and detection pipeline monitoring
• Color Teams -- Red Team patterns feed detection rule development while Blue Team validates coverage

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator