Platform Architecture and Technical Implementation

Abstract

The Prismatic DD Intelligence platform is implemented as a set of Elixir/OTP applications within the Prismatic umbrella, leveraging the BEAM virtual machine's strengths in concurrent processing, fault tolerance, and distributed state management. The architecture comprises six primary application modules coordinated through OTP supervision trees, message passing, and PubSub event distribution. This document describes the technical architecture, the role of each application module, the concurrency model, the polyglot persistence strategy, the real-time interface layer, and the deployment infrastructure that supports production DD investigations.

Introduction

Why Elixir/OTP for Due Diligence

Due diligence platforms impose specific technical requirements that align well with the Elixir/OTP technology stack:

1. Massive Concurrency: An investigation may query 122 OSINT sources simultaneously for dozens of entities. The BEAM VM supports millions of lightweight processes, enabling truly parallel collection without thread pool limitations.

2. Fault Tolerance: Individual OSINT source failures must not crash the investigation. OTP supervision trees automatically restart failed processes, and the "let it crash" philosophy enables graceful degradation when external sources are unavailable.

3. Real-Time Updates: Analysts need to see investigation progress in real-time as sources return data. Phoenix LiveView provides server-rendered real-time UI updates over WebSockets without JavaScript framework complexity.

4. Long-Running Processes: Investigations may span hours or days, with data collection happening in the background. OTP GenServer processes maintain investigation state across long-lived operations.

5. Hot Code Upgrades: Production investigations must not be interrupted by platform updates. The BEAM VM supports hot code reloading, enabling zero-downtime deployments.

Architecture Overview

flowchart TD
    LV["Phoenix LiveView<br/>(Port 4000)"]
    LV --> Case["Case Manager"]
    LV --> Entity["Entity Manager"]
    LV --> Report["Report Generator"]

    Case --> WF["Workflow Engine"]
    Case --> Access["Access Control"]
    Entity --> Graph["Graph Engine"]
    Report --> Tmpl["Template Engine"]

    WF --> PG["PostgreSQL"]
    Access --> PG
    Graph --> Kuzu["KuzuDB"]
    Tmpl --> Meili["Meilisearch"]

    PG --> Osint["OSINT Engine"]
    PG --> Val["Validation Engine"]
    PG --> Risk["Risk Scoring Engine"]

    Osint --> Sources["122 OSINT Sources<br/>(Czech · Infrastructure · Threat · Financial · Compliance)"]

    classDef ui fill:#1e293b,stroke:#a855f7,color:#fff
    classDef mgr fill:#0f172a,stroke:#3b82f6,color:#fff
    classDef engine fill:#1e293b,stroke:#10b981,color:#fff
    classDef store fill:#0f172a,stroke:#f59e0b,color:#fff
    classDef sources fill:#0f172a,stroke:#475569,color:#e2e8f0
    class LV ui
    class Case,Entity,Report mgr
    class WF,Access,Graph,Tmpl,Osint,Val,Risk engine
    class PG,Kuzu,Meili store
    class Sources sources

Application Modules

prismatic_dd -- Core Business Logic

The central application module containing the DD-specific domain logic, coordination, and workflow orchestration.

Key Responsibilities:

• Case lifecycle management and state machine
• Investigation workflow engine with task prioritization
• Entity-level business rules (validation requirements, enrichment triggers)
• Risk assessment computation and aggregation
• Report generation coordination

OTP Architecture:

prismatic_dd Application
  |
  +-- DD.Supervisor (Application supervisor)
       |
       +-- DD.CaseRegistry (Registry of active cases)
       |
       +-- DD.WorkflowEngine (Task scheduling and execution)
       |
       +-- DD.RiskEngine (Risk computation workers)
       |
       +-- DD.ReportEngine (Report generation pipeline)
       |
       +-- DD.MonitoringSupervisor (Post-deal monitoring)
            |
            +-- DD.Monitor (per-entity monitoring process)

Each active investigation case is represented by a GenServer process that maintains the case state, tracks task completion, and coordinates with other subsystems. This per-case process model enables hundreds of concurrent investigations without shared state conflicts.

prismatic_osint_core -- OSINT Framework

The OSINT integration framework providing the source behaviour contract, collection engine, and normalization pipeline.

Key Responsibilities:

• Source adapter registration and discovery
• Broadway-based parallel collection engine
• Per-source rate limiting through token bucket algorithms
• Response normalization and Finding struct construction
• Source health monitoring and degradation handling

Concurrency Model:

The OSINT collection engine operates as a Broadway pipeline with configurable concurrency per source:

defmodule PrismaticOsintCore.CollectionPipeline do
  use Broadway

  def start_link(opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {PrismaticOsintCore.QueryProducer, opts},
        concurrency: 1
      ],
      processors: [
        default: [concurrency: 50]
      ],
      batchers: [
        normalization: [concurrency: 10, batch_size: 100]
      ]
    )
  end
end

The pipeline supports up to 50 concurrent source queries, with 10 concurrent normalization workers processing results in batches. Back-pressure from slow sources automatically throttles upstream query generation, preventing memory exhaustion during large-scale collection operations.

prismatic_osint_sources -- Source Adapters

The 122 individual OSINT source adapter implementations, each in its own module within the PrismaticOsintSources namespace.

Module Organization:

prismatic_osint_sources/
  lib/
    prismatic_osint_sources/
      czech/
        ares.ex
        justice_cz.ex
        isir.ex
        rzp.ex
        cuzk.ex
        registr_smluv.ex
        ...
      infrastructure/
        shodan.ex
        censys.ex
        security_trails.ex
        ...
      threat/
        virus_total.ex
        alienvault_otx.ex
        ...
      financial/
        open_corporates.ex
        sec_edgar.ex
        ...
      compliance/
        ofac.ex
        eu_sanctions.ex
        un_sanctions.ex
        ...

Each adapter module implements the PrismaticOsintCore.Behaviours.Source behaviour and contains all source-specific logic: API client, response parser, schema mapper, and rate limit configuration.

prismatic_storage_kuzudb -- Graph Engine

The KuzuDB integration providing graph storage and traversal for entity relationships.

Key Capabilities:

• Property graph storage for entities and relationships
• Multi-hop traversal queries with variable-depth paths
• Temporal edge properties for point-in-time analysis
• Community detection and centrality algorithms
• Graph export for visualization and reporting

Integration Pattern:

defmodule PrismaticStorageKuzudb.GraphClient do
  @behaviour PrismaticStorageCore.Traits.GraphStore

  def create_relationship(from_entity, to_entity, type, properties) do
    cypher = """
    MATCH (a:Entity {id: $from_id}), (b:Entity {id: $to_id})
    CREATE (a)-[r:#{type} $props]->(b)
    RETURN r
    """
    execute_query(cypher, %{
      from_id: from_entity.id,
      to_id: to_entity.id,
      props: properties
    })
  end

  def traverse_ownership_chain(entity_id, max_depth) do
    cypher = """
    MATCH path = (start:Entity {id: $id})<-[:OWNS*1..#{max_depth}]-(owner)
    WHERE owner:Person OR NOT exists((owner)<-[:OWNS]-())
    RETURN path
    """
    execute_query(cypher, %{id: entity_id})
  end
end

prismatic_storage_meilisearch -- Search Engine

Meilisearch integration providing full-text search across entity data.

Key Capabilities:

• Typo-tolerant, real-time search across entity names, addresses, and descriptions
• Faceted filtering by entity type, case, confidence level, and risk grade
• Asynchronous index updates from PostgreSQL change events
• Sub-50ms query response for interactive search

prismatic_web -- LiveView Interface

The Phoenix LiveView interface providing real-time investigative dashboards.

Key Pages:

Real-Time Updates:

LiveView components subscribe to PubSub events from the DD subsystem, receiving real-time notifications when:

• Entity enrichment completes (new source data available)
• Cross-validation produces new confidence scores
• Graph expansion discovers new entities
• Risk scores are computed or updated
• Task status changes

This enables analysts to observe investigation progress in real-time without polling or page refreshes.

Polyglot Persistence Strategy

The platform's data storage architecture leverages three specialized storage engines, each optimized for specific access patterns:

Data flows from PostgreSQL (primary) to KuzuDB and Meilisearch (secondary) through asynchronous event propagation:

Write Path:  Application --> PostgreSQL --> PubSub Event
                                              |
                              +---------------+---------------+
                              |                               |
                         KuzuDB Sync                    Meilisearch Sync
                      (graph update)                   (index update)

This architecture ensures that PostgreSQL remains the single source of truth while enabling specialized query capabilities through secondary stores.

Concurrency and Fault Tolerance

Supervision Tree Design

The DD subsystem's supervision tree follows OTP best practices with isolated failure domains:

flowchart TD
    App["DD.Application"]
    Sup["DD.Supervisor<br/><i>one_for_one</i>"]
    App --> Sup

    Sup --> Reg["DD.CaseRegistry<br/><i>permanent</i>"]
    Sup --> WfSup["DD.WorkflowSupervisor<br/><i>one_for_one</i>"]
    Sup --> OsSup["DD.OsintSupervisor<br/><i>rest_for_one</i>"]
    Sup --> StSup["DD.StorageSupervisor<br/><i>one_for_one</i>"]
    Sup --> MnSup["DD.MonitoringSupervisor<br/><i>simple_one_for_one</i>"]

    WfSup --> Wf1["DD.WorkflowEngine<br/><i>permanent</i>"]
    WfSup --> Wf2["DD.TaskWorkerPool<br/><i>simple_one_for_one, dynamic</i>"]

    OsSup --> Os1["OsintCore.SourceRegistry"]
    OsSup --> Os2["OsintCore.CollectionPipeline"]
    OsSup --> Os3["OsintCore.HealthMonitor"]

    StSup --> St1["StorageKuzudb.Client"]
    StSup --> St2["StorageMeilisearch.Client"]

    MnSup --> Mn1["DD.Monitor<br/><i>per-entity, dynamic</i>"]

    classDef root fill:#1e293b,stroke:#a855f7,color:#fff
    classDef sup fill:#1e293b,stroke:#3b82f6,color:#fff
    classDef worker fill:#0f172a,stroke:#64748b,color:#e2e8f0
    classDef dyn fill:#0f172a,stroke:#10b981,color:#e2e8f0
    class App,Sup root
    class WfSup,OsSup,StSup,MnSup sup
    class Reg,Wf1,Os1,Os2,Os3,St1,St2 worker
    class Wf2,Mn1 dyn

Failure Isolation: If the KuzuDB client crashes, only the storage supervisor restarts it. Active OSINT collection continues, and PostgreSQL writes are unaffected. If an individual source adapter crashes during a query, only that query fails; other concurrent source queries continue.

Back-Pressure Management

The system implements back-pressure at multiple levels to prevent resource exhaustion:

1. Broadway pipeline: Automatically throttles query production when processors are saturated
2. Per-source rate limiters: Prevent exceeding external API limits
3. Database connection pools: Connection pooling prevents database connection exhaustion
4. Memory monitoring: Telemetry-based alerts when memory consumption approaches limits

Deployment Infrastructure

Production Deployment

The DD platform deploys as part of the Prismatic umbrella on Fly.io:

Performance Characteristics

Observability

The platform provides comprehensive observability through:

• Telemetry: Structured metrics for every subsystem (collection latency, validation throughput, risk computation time)
• Structured logging: JSON-formatted logs with investigation context (case ID, entity ID, source)
• Health endpoints: /health and /readiness for infrastructure monitoring
• LiveView dashboards: Real-time operational dashboards for platform operators

Security Architecture

Data Protection

Multi-Tenancy Isolation

Investigation data is isolated at the case level through:

• PostgreSQL row-level security policies
• Case-scoped Meilisearch indexes
• Case-filtered KuzuDB graph queries
• Case-scoped PubSub topics

Quality Assurance

The DD platform follows the platform's NO MERCY, NO DOUBTS quality doctrine:

• Zero compilation warnings: mix compile --warnings-as-errors across all DD modules
• Comprehensive test coverage: Unit tests, integration tests, and property-based tests for all adapters
• Dialyzer type checking: Full type specification coverage
• Credo compliance: All static analysis checks passing
• Quality gates: Automated quality gates block non-compliant code

Conclusion

The Elixir/OTP architecture provides the ideal foundation for a due diligence platform that must handle massive concurrent data collection, maintain fault tolerance across hundreds of external source integrations, and deliver real-time investigation updates through an interactive web interface. The polyglot persistence strategy ensures that each data access pattern is served by the most appropriate storage engine, while OTP supervision trees provide the fault isolation necessary for reliable production operation.

References

• Elixir Programming Language
• OTP Framework
• BEAM Virtual Machine
• Phoenix LiveView
• PostgreSQL
• KuzuDB
• Meilisearch
• Supervision Trees
• Broadway Pipeline
• Platform Architecture
• Umbrella Applications

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator