Prismatic OSINT Sources

Overview

Prismatic OSINT Sources manages the registry and lifecycle of all 121+ OSINT source adapters within the Prismatic Platform. It provides source discovery, health monitoring, capability querying, and unified access to all intelligence sources through a consistent adapter pattern protocol. Every adapter implements the same behaviour contract, ensuring that consumers can query any source -- from network intelligence to threat intelligence feeds -- through a single, uniform API without knowledge of source-specific authentication, pagination, or data format details.

The source registry maintains a capability-based catalog backed by ETS, enabling O(1) lookups when the platform needs to find all sources that support a particular query type such as IP reputation, domain intelligence, or sanctions screening. Each source entry tracks availability status, response latency percentiles, rate limiting quotas, and credential validity, providing the orchestration layer with real-time intelligence about which sources are healthy, cost-effective, and responsive for a given query.

Multi-source parallel querying enables the platform to fan out intelligence requests across multiple adapters simultaneously, collecting results through OTP supervision tree coordination and deduplicating findings with entity resolution algorithms. Source priority ordering and cost-aware selection ensure that expensive commercial APIs are queried only when free or lower-cost sources cannot satisfy the information requirement, optimizing operational costs while maintaining intelligence completeness.

Architecture

Query Request --> Source Selector --> Parallel Dispatcher --> Result Aggregator
      |                 |                     |                       |
  Capability         Priority          Per-Source Worker         Deduplication
  Requirements       Cost Model        Rate Limiter             Normalization
  Confidence         Availability      Circuit Breaker          Confidence Score
      |                 |                     |                       |
      +-----------------+---------------------+-----------------------+
                                  |
                           Source Registry (ETS) --> Health Monitor
                                  |
                           Credential Manager --> Auth Rotation

All source selection and result aggregation logic follows pure function principles. Network requests execute in isolated worker processes with circuit breaker patterns and backpressure mechanisms preventing cascade failures when individual sources become unavailable.

Key Modules

Key Features

Source Registry

The ETS-backed registry provides instant capability-based source discovery with structured metadata:

defmodule PrismaticOsintSources.Registry do
  @spec register(atom(), SourceSpec.t()) :: :ok
  def register(source_id, spec) do
    :ets.insert(@registry_table, {source_id, %{
      capabilities: spec.capabilities,
      status: :healthy,
      latency_p50: 0,
      latency_p99: 0,
      quota: spec.quota,
      cost_tier: spec.cost_tier,
      auth_type: spec.auth_type,
      registered_at: DateTime.utc_now()
    }})
    :ok
  end

  @spec with_capability(atom()) :: list(atom())
  def with_capability(capability) do
    :ets.foldl(fn {source_id, meta}, acc ->
      if capability in meta.capabilities, do: [source_id | acc], else: acc
    end, [], @registry_table)
  end

  @spec healthy_sources() :: list(atom())
  def healthy_sources do
    :ets.foldl(fn {source_id, meta}, acc ->
      if meta.status in [:healthy, :degraded], do: [source_id | acc], else: acc
    end, [], @registry_table)
  end
end

• Centralized catalog of all 121+ OSINT sources with capability metadata and health status
• Capability-based source discovery enabling queries like "find all sources supporting IP reputation lookup"
• Health monitoring and availability tracking with automatic source degradation and recovery detection
• Rate limit management per source with token bucket algorithms and quota tracking

Adapter Protocol

Every OSINT source adapter must implement the SourceAdapter behaviour, ensuring consistent interfaces:

• Unified protocol interface for all OSINT queries with structured request/response contracts
• Structured result normalization converting source-specific formats to platform-standard schemas
• Error handling with source-specific context including retry recommendations and degradation status
• Credential and authentication management supporting API keys, OAuth2, and certificate-based auth

Source Orchestration

• Multi-source parallel querying with configurable concurrency limits and timeout policies
• Result deduplication across sources using entity resolution and confidence scoring
• Source priority and fallback ordering with cost-aware selection minimizing API expenditure
• Telemetry emission for query latency, source utilization, and cache hit metrics

Cost Optimization

The cost model ensures that expensive commercial APIs are queried only when necessary:

Source Categories

Usage

# List all available sources with health status
{:ok, sources} = PrismaticOsintSources.list(include_health: true)
# => [%{name: "shodan", status: :healthy, latency_p99: 230}, ...]

# Query by capability
{:ok, sources} = PrismaticOsintSources.with_capability(:ip_lookup)
# => [%{name: "shodan"}, %{name: "censys"}, %{name: "binary_edge"}]

# Unified multi-source query with parallel execution
{:ok, results} = PrismaticOsintSources.query("example.com",
  sources: [:shodan, :censys, :crtsh],
  parallel: true,
  deduplicate: true
)
# => %{findings: [...], sources_queried: 3, confidence: 0.89}

# Check source health across all adapters
{:ok, health} = PrismaticOsintSources.health_report()
# => %{healthy: 98, degraded: 3, unavailable: 2, total: 103}

# Register a new source adapter
:ok = PrismaticOsintSources.register(:new_source, %{
  capabilities: [:ip_lookup, :domain_lookup],
  cost_tier: :low,
  auth_type: :api_key
})

NABLA Compliance

Testing

Registry tests verify ETS-backed source registration, capability indexing, and O(1) lookup performance. Source selection tests verify cost optimization, quota awareness, and plurality enforcement. Parallel dispatcher tests verify concurrent query execution, timeout handling, and error isolation across sources.

Health monitoring tests verify degradation detection, recovery triggering, and alert emission. Integration tests exercise the full pipeline from capability query through parallel dispatch to deduplicated result aggregation. Adapter contract tests verify behaviour compliance across all 121+ source adapters.

Integration Points

Performance

Telemetry events: [:prismatic, :osint_sources, :query_dispatched], [:prismatic, :osint_sources, :source_health_changed], [:prismatic, :osint_sources, :quota_consumed].

Related Resources

• Prismatic Czech Autocrawler -- Czech registry adapters consuming the source protocol
• Cross Pollination Specialist -- Cross-domain intelligence synthesis leveraging source diversity
• Business Financial Intelligence Specialist -- Business source selection and analysis coordination
• Crawler Development Specialist -- Source adapter development and maintenance
• Intelligence Synthesis -- Multi-source evidence fusion and deduplication
• NABLA Axioms -- Source independence and signal plurality enforcement
• Real-Time Monitoring -- Source health monitoring and availability tracking

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator