OSINT Source Integration Framework

Abstract

The Prismatic Platform integrates 122 OSINT source adapters organized into seven intelligence categories, providing comprehensive data coverage for due diligence investigations across entity types, jurisdictions, and risk dimensions. Each adapter conforms to the PrismaticOsintCore.Behaviours.Source behaviour contract, ensuring consistent error handling, rate limiting, credential management, and telemetry across all sources. This document describes the integration architecture, the intelligence category taxonomy, the adapter behaviour contract, the parallel collection engine, and the normalization pipeline that transforms heterogeneous source data into the platform's unified entity schema.

Introduction

The Breadth-Depth Trade-Off

Due diligence investigations must balance breadth of source coverage against depth of data extraction. A platform that queries only Czech registries provides deep but narrow coverage, missing global connections, cyber risk indicators, and international sanctions matches. Conversely, a platform that queries hundreds of shallow sources produces noise without actionable intelligence.

The Prismatic Platform resolves this trade-off through tiered source integration: deep, purpose-built adapters for 30+ Czech registries that extract structured data from specialized APIs, combined with 84+ global intelligence source adapters that provide breadth across infrastructure, threat, social, financial, and compliance domains. The triple-check cross-validation methodology ensures that breadth does not come at the cost of accuracy by requiring multi-source corroboration before findings are accepted.

Architecture Overview

The OSINT integration framework operates as a layered pipeline:

flowchart TD
    Trigger["Investigation Trigger"]
    Select["Source Selection Engine<br/><i>selects relevant sources per entity type</i>"]
    Collect["Parallel Collection Engine<br/><i>concurrent queries via Broadway</i>"]
    Norm["Response Normalization Pipeline<br/><i>heterogeneous data → typed Findings</i>"]
    Score["Confidence Scoring<br/><i>source-level validation</i>"]
    Merge["Entity Merger<br/><i>findings merged into entity profiles</i>"]
    Cross["Cross-Validation Engine<br/><i>triple-check methodology</i>"]

    Trigger --> Select --> Collect --> Norm --> Score --> Merge --> Cross

    classDef trigger fill:#1e293b,stroke:#dc2626,color:#fff
    classDef stage fill:#0f172a,stroke:#3b82f6,color:#fff
    classDef validate fill:#1e293b,stroke:#10b981,color:#fff
    class Trigger trigger
    class Select,Collect,Norm,Score,Merge stage
    class Cross validate

Intelligence Categories

Category Taxonomy

The 122 OSINT source adapters are organized into seven intelligence categories:

Category-Entity Type Mapping

Not all categories are relevant for all entity types. The source selection engine uses a mapping table to determine which sources to query for each entity type:

Adapter Behaviour Contract

The Source Behaviour

Every OSINT source adapter implements the PrismaticOsintCore.Behaviours.Source behaviour, which defines the contract between the integration framework and individual source implementations:

defmodule PrismaticOsintCore.Behaviours.Source do
  @callback name() :: String.t()
  @callback category() :: atom()
  @callback supported_entity_types() :: [atom()]
  @callback rate_limit() :: {count :: pos_integer(), period_ms :: pos_integer()}
  @callback query(entity :: Entity.t(), opts :: keyword()) ::
    {:ok, [Finding.t()]} | {:error, reason :: term()}
  @callback health_check() :: :ok | {:error, reason :: term()}
end

This contract ensures that:

• Every adapter declares its capabilities: name, category, supported entity types
• Rate limiting is explicit: each adapter specifies its rate limit parameters
• Query interface is uniform: all adapters accept an entity and return findings
• Health is monitorable: all adapters implement health checks for operational monitoring

The Finding Struct

All adapters return results as typed Finding structs that normalize heterogeneous source data into a common format:

defmodule PrismaticOsintCore.Finding do
  @type t :: %__MODULE__{
    source: String.t(),
    category: atom(),
    entity_type: atom(),
    entity_id: String.t(),
    attribute: String.t(),
    value: term(),
    confidence: float(),
    timestamp: DateTime.t(),
    raw_data: map(),
    metadata: map()
  }
end

The Finding struct captures both the normalized attribute value and the raw source response, enabling both automated processing and manual analyst review of original data.

Parallel Collection Engine

Broadway-Based Pipeline

The collection engine uses Broadway for concurrent, back-pressure-aware processing of OSINT queries. When an entity enters the enrichment phase, the source selection engine identifies all relevant sources and submits query tasks to the Broadway pipeline:

Entity Enrichment Request
        |
        v
Source Selection --> [Source A, Source B, Source C, ... Source N]
        |
        v
Broadway Pipeline (concurrent processing with back-pressure)
        |
        +---> Source A Worker --> Rate Limiter --> HTTP Client --> Response Parser
        +---> Source B Worker --> Rate Limiter --> HTTP Client --> Response Parser
        +---> Source C Worker --> Rate Limiter --> HTTP Client --> Response Parser
        ...
        +---> Source N Worker --> Rate Limiter --> HTTP Client --> Response Parser
        |
        v
Finding Aggregation

Rate Limiting

Each source adapter declares its rate limit, and the collection engine enforces these limits through per-source token bucket rate limiters. This prevents the platform from overwhelming external APIs while maximizing throughput within allowed limits:

Error Handling and Retry

The collection engine implements graduated retry logic for transient failures:

Failed queries after exhausting retries are recorded in the case log with the error details, ensuring that analysts are aware of any gaps in source coverage.

Collection Metrics

The platform tracks detailed collection metrics through Telemetry:

Normalization Pipeline

Data Normalization Stages

Raw source responses pass through a multi-stage normalization pipeline before entering the entity store:

Stage 1: Response Parsing Each adapter implements source-specific parsing logic that extracts relevant fields from the raw API response. This handles the heterogeneity of response formats (JSON, XML, HTML, CSV) across different sources.

Stage 2: Schema Mapping Parsed fields are mapped to the platform's unified attribute schema. For example, a company name field might be name in one source, company_name in another, and legal_name in a third. The schema mapping layer normalizes these to the canonical legal_name attribute.

Stage 3: Value Normalization Attribute values are normalized to canonical formats:

• Names: Unicode normalization, diacritical mark handling, legal form abbreviation expansion
• Addresses: Component parsing (street, number, city, postal code), geocoding
• Dates: ISO 8601 normalization from various source formats
• Numbers: Currency conversion, unit normalization
• Identifiers: Format validation (ICO check digits, IBAN validation, etc.)

Stage 4: Confidence Assignment Each normalized finding receives an initial confidence score based on the source's authority tier and data freshness, as described in the triple-check methodology.

Stage 5: Entity Merge Normalized findings are merged into existing entity profiles through the entity resolution algorithm, which determines whether new findings match existing entities or represent newly discovered entities.

Source Health Monitoring

Continuous Health Checks

The platform runs continuous health checks against all configured OSINT sources, monitoring:

• API availability: Is the source responding to queries?
• Data quality: Are responses containing expected data structures?
• Response time: Is the source meeting latency expectations?
• Authentication status: Are API credentials valid and not expired?

Source Degradation Handling

When a source fails health checks, the platform:

1. Marks the source as degraded in the source registry
2. Adjusts confidence scores for data previously collected from the source
3. Increases query frequency to alternative sources covering similar data
4. Alerts platform operators for investigation
5. Automatically re-enables the source when health checks pass again

This resilience ensures that the platform's due diligence capability degrades gracefully when individual sources experience issues, rather than failing catastrophically.

Adding New Sources

Adapter Development

New OSINT source adapters follow a standardized development process:

1. Implement the Source behaviour: Define name, category, entity types, rate limits, query logic, and health check
2. Define schema mapping: Map source response fields to the platform's unified attribute schema
3. Configure rate limits: Set appropriate rate limiting parameters based on the source's API documentation
4. Write tests: Comprehensive tests including unit tests for parsing, integration tests with recorded responses, and property-based tests for edge cases
5. Register the adapter: Add to the source registry with appropriate category and authority tier classification

The behaviour contract ensures that new adapters integrate seamlessly with the collection engine, normalization pipeline, and cross-validation framework without requiring changes to the core platform.

Performance Characteristics

Conclusion

The OSINT Source Integration Framework provides the data foundation for the Prismatic Platform's due diligence capability. By integrating 122 source adapters across seven intelligence categories through a uniform behaviour contract, the platform achieves both breadth (global coverage across entity types and risk dimensions) and depth (purpose-built Czech registry integration) while maintaining the data quality standards required by the triple-check cross-validation methodology.

References

• Czech Registry Integration
• Triple-Check Methodology
• Entity Management System
• Risk Assessment Framework
• OSINT Glossary
• Broadway Pipeline
• Telemetry System
• Adapter Pattern
• Behaviour Contract

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator