Prismatic Compression

Overview

Prismatic Compression provides data compression and archival services for the platform. With a continuously growing OSINT data store, historical intelligence archives, and high-volume inter-node communication, efficient data encoding is essential for both cost control and performance. Compression handles three distinct concerns: reducing storage footprint for historical data, optimizing network transfer between distributed system components, and managing long-term data retention with configurable archival policies.

The module implements an adaptive algorithm selection strategy. Rather than applying a single compression algorithm uniformly, it profiles incoming data and selects the optimal algorithm based on data characteristics. Structured JSON payloads from REST API responses compress well with Zstandard's dictionary mode, while binary sensor data from Prismatic Embodiment achieves better ratios with LZ4. The selection is transparent to callers -- they compress and decompress through a unified interface while the module handles algorithm selection, dictionary management, and format versioning internally. This adapter pattern ensures new compression algorithms can be added without modifying consumer code.

For archival, Compression implements a tiered storage strategy aligned with the platform's compliance framework requirements. Recent data remains uncompressed in hot storage for fastest access. Data older than a configurable threshold is compressed and moved to warm storage. Data beyond the retention window is either purged or moved to cold storage with maximum compression, depending on regulatory requirements tracked by Prismatic CER under NIS2 and GDPR retention mandates. Across the platform's 90+ applications and millions of accumulated records, Compression saves an estimated 70% of raw storage costs while maintaining sub-millisecond decompression latency for hot data access.

Architecture

The module is organized around a Codec Registry, an Archival Scheduler, and a Transfer Optimizer.

Input Data --> Profiler --> Algorithm Selector --> Codec Engine --> Output
     |           |              |                  |            |
  Type Tag    Data Stats    Zstd/LZ4/Gzip      Parallel      Compressed
  Size Est    Entropy       Dictionary Mgmt     Streaming     + Metadata
  Priority    Pattern       Level Selection     Chunking      + Checksum
  Metadata    Classify      Fallback Chain      Pipeline      + Version

The Codec Registry maintains available compression algorithms with their performance profiles and selects the optimal codec per data type. The Archival Scheduler is a periodic GenServer that scans storage tables, identifies data eligible for archival or purging, and processes it according to configured retention policies. The Transfer Optimizer wraps inter-node Erlang distribution with optional compression for large message passing payloads, configurable per message type.

Process Topology

PrismaticCompression.Application (Supervisor, :one_for_one)
+-- PrismaticCompression.CodecRegistry (GenServer)
|     Algorithm registry with performance profiles
+-- PrismaticCompression.ArchivalScheduler (GenServer)
|     Periodic tiered archival processing
+-- PrismaticCompression.DictionaryManager (GenServer)
|     Zstandard dictionary training and management
+-- PrismaticCompression.TransferOptimizer (GenServer)
      Inter-node message compression

All operations are instrumented with Telemetry events reporting compression ratios, processing times, and storage savings for observability.

Key Modules

The Engine module handles the actual compression and decompression operations, dispatching to the appropriate algorithm implementation based on the codec selected by the registry. The Application module starts the supervision tree including the ArchivalScheduler for periodic background processing. The DictionaryManager maintains pre-trained Zstandard dictionaries for domain-specific data types, significantly improving compression ratios for small payloads (under 10KB) where standard compression algorithms have insufficient context for effective pattern exploitation.

Codec Registry and Adaptive Selection

The CodecRegistry maintains a catalog of available compression algorithms with their performance characteristics. When a compression request arrives, the Profiler analyzes the data to determine its type, size, and entropy, then the CodecRegistry selects the optimal algorithm based on these characteristics.

The adaptive selection algorithm uses a decision tree based on data size, detected content type, and target latency requirements. For example, data under 1KB with JSON content type selects Zstandard with a trained dictionary, while streaming binary data above 1MB selects LZ4 for its superior throughput.

defmodule PrismaticCompression.Profiler do
  @spec profile(binary()) :: %{
    size: non_neg_integer(),
    content_type: atom(),
    entropy: float(),
    recommended_algorithm: atom(),
    recommended_level: pos_integer()
  }
  def profile(data) do
    size = byte_size(data)
    entropy = calculate_entropy(data)
    content_type = detect_content_type(data)

    algorithm = select_algorithm(size, entropy, content_type)
    level = select_level(size, algorithm)

    %{size: size, content_type: content_type, entropy: entropy,
      recommended_algorithm: algorithm, recommended_level: level}
  end
end

Tiered Storage Architecture

The archival system implements a three-tier storage model with automatic data lifecycle management.

The ArchivalScheduler runs on a configurable interval (default: every 6 hours), scanning storage tables for data that has aged beyond the current tier threshold. Data is compressed in place with the appropriate algorithm and level for the target tier, and metadata is updated to reflect the new storage location and compression format. The scheduler processes data in batches with configurable batch sizes to prevent resource spikes during archival operations.

Cold storage uses maximum compression (Zstandard level 19) to achieve the highest possible compression ratios, trading compression time for storage savings. Since cold data is rarely accessed, the higher decompression latency is acceptable. When cold data is accessed, it is decompressed on demand and cached in warm storage for subsequent accesses within a configurable window.

Configuration

config :prismatic_compression,
  default_algorithm: :zstd,
  zstd_level: 3,
  lz4_acceleration: 1,
  adaptive_selection: true,
  archival_interval: :timer.hours(6),
  hot_threshold_days: 30,
  warm_threshold_days: 180,
  cold_threshold_days: 365,
  dictionary_path: "priv/compression/dictionaries/",
  circuit_breaker_threshold: 5,
  archival_batch_size: 1000

Configuration controls the default compression algorithm, algorithm-specific tuning parameters, adaptive selection toggle, archival scheduling interval, and tiered storage thresholds. Dictionary path specifies where trained Zstandard dictionaries are stored for domain-specific compression optimization. The circuit breaker threshold prevents compression pipeline stalls when the engine encounters persistent failures.

API Reference

# Compress data with automatic algorithm selection
{:ok, compressed} = PrismaticCompression.compress(data, profile: :api_response)
# => %Compressed{algorithm: :zstd, ratio: 0.23, original_size: 45_200, compressed_size: 10_396}

# Compress with explicit algorithm and level
{:ok, compressed} = PrismaticCompression.compress(data, algorithm: :zstd, level: 9)

# Decompress data with automatic format detection
{:ok, original} = PrismaticCompression.decompress(compressed)

# Archive old OSINT signals to warm storage
{:ok, stats} = PrismaticCompression.archive(:signals, older_than: ~D[2025-01-01])
# => %{archived: 1_245_000, compressed_size_mb: 340, original_size_mb: 2_100, ratio: 0.16}

# Read archived data with transparent decompression
{:ok, data} = PrismaticCompression.read_archived(archive_id)

# Get compression statistics for capacity planning
{:ok, stats} = PrismaticCompression.storage_stats()
# => %{hot_gb: 45, warm_gb: 120, cold_gb: 890, total_savings_pct: 72}

# Train a new dictionary from sample data
{:ok, dict} = PrismaticCompression.train_dictionary(:osint_responses, samples)

Testing

Unit tests verify round-trip compression/decompression correctness for all supported algorithms across a range of data types and sizes. Archival tests verify correct tier classification, data movement, and transparent decompression on read. Property-based tests using StreamData generate random binary data to verify that compress-then-decompress always returns the original data, regardless of content, size, or algorithm selection.

Integration tests exercise the full archival pipeline with realistic data volumes, verifying that retention policies are correctly enforced and that storage tier transitions maintain data integrity. Performance benchmarks using Benchee validate that compression and decompression meet latency targets. Dictionary training tests verify that trained dictionaries improve compression ratios for domain-specific data compared to generic compression.

Integration Points

NABLA Compliance

Compression is a low-level infrastructure module and does not produce intelligence products directly. NABLA compliance is primarily relevant for data integrity and provenance metadata preservation through compression/decompression cycles.

Performance

Storage savings across the platform average 70% with adaptive algorithm selection.

Telemetry events: [:prismatic, :compression, :compress], [:prismatic, :compression, :decompress], [:prismatic, :compression, :archive], [:prismatic, :compression, :dictionary_trained]. Metrics include compression ratio, processing time, and storage tier utilization.

Related Resources

• Prismatic Storage -- Storage adapters using Compression for transparent encoding
• Prismatic CER -- Retention policies governing archival and purge schedules
• Prismatic Embodiment -- High-volume sensor data benefits from streaming compression
• Prismatic API -- HTTP response compression for REST endpoints
• Elixir Architect -- Ensures compression implementations follow OTP patterns
• Architecture Review Specialist -- Reviews tiered storage architecture decisions
• Telemetry Integration -- Compression metrics feed into platform-wide observability
• Real-Time Monitoring -- Storage tier utilization and compression ratio monitoring
• Quality Gates -- Compression correctness verified through round-trip property tests

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator