Prismatic Storage

Abstract

Prismatic Storage provides a unified storage abstraction layer that enables application code to interact with seven heterogeneous storage backends -- ETS, PostgreSQL (via Ecto), Redis, Meilisearch, KuzuDB, DuckDB, and filesystem -- through a consistent trait-based protocol system. The architecture defines six protocol traits (Readable, Writable, Queryable, Indexable, Graphable, Cacheable) that each adapter implements according to its backend's capabilities. Application code programs against traits rather than concrete adapters, enabling runtime adapter selection, transparent failover, and multi-adapter operations without code changes. The contract testing framework ensures protocol compliance across all adapters, while adapter-specific optimizations preserve the performance characteristics of each backend. This design supports the platform's diverse storage requirements: microsecond-latency ETS for hot data, ACID-compliant PostgreSQL for durable entities, sub-millisecond Redis for distributed caching, typo-tolerant Meilisearch for full-text search, native graph traversal via KuzuDB, and columnar analytics through DuckDB.

1. Introduction

1.1 Problem Statement

An intelligence platform integrating 121+ OSINT sources, security assessments, compliance reporting, and visitor analytics generates heterogeneous data with conflicting storage requirements. Entity records need ACID durability (PostgreSQL). Hot session data needs microsecond access (ETS). Full-text search across millions of documents needs relevancy ranking and typo tolerance (Meilisearch). Entity relationship traversal needs native graph queries (KuzuDB). Analytical aggregations over time-series data need columnar storage (DuckDB). Distributed state needs sub-millisecond access across nodes (Redis).

Without a unifying abstraction, application code becomes tightly coupled to specific storage technologies, making it impossible to change backends without rewriting business logic. The trait-based protocol system solves this by defining storage operations as protocol contracts that adapters implement independently.

1.2 Design Goals

1. Trait-based protocols -- define storage operations as Elixir protocols that adapters implement, enabling polymorphic dispatch.
2. Adapter-agnostic application code -- business logic programs against traits, not concrete adapters.
3. Runtime adapter selection -- choose adapters at runtime based on data type, performance requirements, or operational conditions.
4. Contract testing -- a shared test suite that verifies every adapter correctly implements required protocols.
5. Multi-adapter operations -- support replication, federation, and migration across adapters.
6. Performance preservation -- trait dispatch introduces negligible overhead; adapter-specific optimizations are preserved.

1.3 Scope

Prismatic Storage covers the abstraction layer, protocol definitions, and adapter implementations. It does not implement application-specific schemas, migrations, or business logic. Each adapter is a separate OTP application within the umbrella.

2. Architecture

2.1 System Design

Application Code
       |
  PrismaticStorage Facade
       |
  Trait Dispatch (Protocol Resolution)
       |
  +----+----+----+----+----+----+
  |    |    |    |    |    |    |
  ETS  Ecto Redis Meili Kuzu Duck  File
  |    |    |    |    |    |    |
  ETS  PG   Redis MS   KuzuDB DDB  FS
  Tab  SQL   Srv   Srv   Embed  Embed Dir

2.2 Core Components

2.3 Process Topology

PrismaticStorage.Application (Supervisor, :one_for_one)
+-- PrismaticStorage.AdapterRegistry (GenServer)
|     Tracks available adapters and their health status
+-- PrismaticStorage.Router (GenServer)
|     Routes operations to appropriate adapters based on data type
+-- Per-adapter supervision trees (started by each adapter app)

Each adapter application manages its own supervision tree, connection pools, and health monitoring. The central Storage application coordinates routing and discovery.

2.4 Data Flow

Read operations flow through the facade, which resolves the appropriate adapter via the Router, dispatches through the trait protocol, and returns the result. Write operations follow the same path but may additionally trigger replication to secondary adapters (e.g., writing to Ecto for durability and Meilisearch for searchability). Multi-adapter operations are coordinated through the Router with eventual consistency semantics.

3. Implementation

3.1 Key Algorithms

Adapter Selection. The Router maintains a mapping from data types to preferred adapters. When no explicit adapter is specified, the Router selects based on operation type: Readable/Writable default to ETS for volatile data and Ecto for persistent data; Indexable routes to Meilisearch; Graphable routes to KuzuDB; Cacheable routes to Redis. Application code can override the default by specifying an adapter explicitly.

Contract Verification. The contract testing framework generates a comprehensive test suite from protocol definitions. Each adapter runs the same test suite, verifying that all required callbacks return correct types, handle error cases, and maintain idempotency where specified.

3.2 Data Structures

defprotocol PrismaticStorageCore.Trait.Readable do
  @spec get(adapter :: t(), key :: term()) :: {:ok, term()} | {:error, :not_found | term()}
  def get(adapter, key)

  @spec fetch(adapter :: t(), key :: term(), default :: term()) :: {:ok, term()}
  def fetch(adapter, key, default)

  @spec exists?(adapter :: t(), key :: term()) :: boolean()
  def exists?(adapter, key)
end

defprotocol PrismaticStorageCore.Trait.Writable do
  @spec put(adapter :: t(), key :: term(), value :: term()) :: :ok | {:error, term()}
  def put(adapter, key, value)

  @spec delete(adapter :: t(), key :: term()) :: :ok | {:error, term()}
  def delete(adapter, key)
end

3.3 API Surface

# Adapter-agnostic operations
@spec read(atom(), term()) :: {:ok, term()} | {:error, term()}
PrismaticStorage.read(:ets_adapter, "entity:123")

@spec write(atom(), term(), term()) :: :ok | {:error, term()}
PrismaticStorage.write(:ecto_adapter, "entity:123", entity_data)

# Search through Indexable trait
@spec search(atom(), String.t(), keyword()) :: {:ok, [term()]} | {:error, term()}
PrismaticStorage.search(:meilisearch_adapter, "Prismatic s.r.o.", limit: 20)

# Graph traversal through Graphable trait
@spec traverse(atom(), term(), term()) :: {:ok, [term()]} | {:error, term()}
PrismaticStorage.traverse(:kuzu_adapter, start_node, :OWNS)

# Multi-adapter replication
@spec replicate(atom(), atom(), term()) :: :ok | {:error, term()}
PrismaticStorage.replicate(:ecto_adapter, :meilisearch_adapter, "companies")

3.4 Configuration

config :prismatic_storage,
  default_adapter: :ets_adapter,
  persistent_adapter: :ecto_adapter,
  cache_adapter: :redis_adapter,
  search_adapter: :meilisearch_adapter,
  graph_adapter: :kuzu_adapter,
  analytics_adapter: :duckdb_adapter,

  # Routing rules
  type_routing: %{
    :entities => :ecto_adapter,
    :sessions => :ets_adapter,
    :cache => :redis_adapter,
    :search_index => :meilisearch_adapter,
    :relationships => :kuzu_adapter,
    :analytics => :duckdb_adapter
  }

4. Integration

4.1 Dependencies

4.2 Dependents

Every platform application that persists or queries data depends on Prismatic Storage. Key consumers include Prismatic Perimeter (entity and rating persistence), Prismatic OSINT Core (intelligence data storage), Prismatic Agents (agent definition registry), and Prismatic Web (session and dashboard state).

4.3 Inter-Process Communication

Adapters manage their own connection pools and process hierarchies. The Storage facade communicates with adapters through protocol dispatch (function calls), not message passing, minimizing latency. Replication between adapters uses asynchronous Task-based coordination.

4.4 External Integrations

External services: PostgreSQL (via Ecto), Redis (via Redix), Meilisearch (via HTTP client), KuzuDB (via NIF binding), DuckDB (via NIF binding). ETS and filesystem adapters have no external dependencies.

5. Performance

5.1 Benchmarks

5.2 Scalability

Each adapter scales according to its backend's characteristics. ETS scales with available memory. PostgreSQL scales with read replicas and connection pool size. Redis scales with cluster mode. Meilisearch scales with sharding. The trait abstraction allows transparent migration between adapters as scaling requirements evolve.

5.3 Resource Requirements

6. Testing Strategy

6.1 Unit Tests

Each adapter has unit tests for CRUD operations, error handling, and edge cases specific to its backend. These tests run against real backends (not mocks) to ensure correctness.

6.2 Integration Tests

The contract testing framework runs the same 121-test suite against every adapter, verifying protocol compliance. Multi-adapter integration tests verify replication, failover, and federated query behavior.

6.3 Property-Based Testing

StreamData generators produce random keys, values, and query predicates to verify adapter behavior under diverse inputs. Property tests verify that write-then-read roundtrips preserve data integrity across all adapters.

7. Security Considerations

7.1 Threat Model

Storage layer threats include unauthorized data access, SQL injection (Ecto adapter), and data corruption. Mitigations include parameterized queries (Ecto), access control through the application layer, encryption at rest (PostgreSQL), and audit logging for all write operations.

7.2 Access Control

Storage access is mediated through the application layer. The Storage facade itself does not enforce authentication; this is the responsibility of consuming applications. Connection credentials are managed through environment variables and the application configuration.

8. Operational Considerations

8.1 Deployment

Each adapter deploys with its required backend. ETS requires no external services. Ecto requires PostgreSQL. Redis requires a Redis server. Meilisearch requires a Meilisearch instance. KuzuDB and DuckDB are embedded databases requiring no external services.

8.2 Monitoring

Telemetry events for all storage operations: [:prismatic, :storage, :read], [:prismatic, :storage, :write], [:prismatic, :storage, :query]. Metrics include operation latency per adapter, error rates, connection pool utilization, and cache hit rates.

8.3 Troubleshooting

9. Future Work

Planned enhancements include automatic adapter selection based on query complexity analysis, cross-adapter transactions with two-phase commit, data lifecycle management with automatic tier migration (hot to warm to cold), and a unified query language that compiles to adapter-specific queries.

References

• Prismatic Storage Core -- Protocol definitions and contracts
• Prismatic Storage ETS -- ETS adapter
• Prismatic Storage Ecto -- PostgreSQL adapter
• Prismatic Storage Redis -- Redis adapter
• Prismatic Storage KuzuDB -- Graph database adapter
• Prismatic Storage DuckDB -- Analytical adapter
• Prismatic Meilisearch -- Full-text search adapter

Related Agents

• Adapter Pattern Specialist -- Designs the trait-based protocol system enabling 7 heterogeneous storage backends through unified interfaces
• Architecture Review Specialist -- Reviews the multi-adapter architecture for protocol compliance, performance preservation, and failover correctness
• Consolidation Architect -- Coordinates the unified storage layer consolidation across ETS, Ecto, Redis, Meilisearch, KuzuDB, and DuckDB

Related Capabilities

• Cross-Domain Flexibility -- Trait-based protocols enable runtime adapter selection across volatile, persistent, search, graph, and analytical workloads
• Quality Gates -- Contract testing framework running 121 tests against every adapter to verify protocol compliance
• Telemetry Integration -- Full operation coverage with per-adapter latency, error rates, and connection pool utilization metrics

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator