Prismatic Crawler

Overview

Prismatic Crawler provides the platform's web crawling and data extraction infrastructure. It powers the automated data collection from Czech government registries, OSINT web sources, and target websites for the Prismatic Perimeter attack surface discovery and HAWKEYE visitor intelligence systems.

The crawler implements strict politeness policies, respects robots.txt, enforces rate limits per domain, and manages a supervised pool of concurrent crawl workers. It is designed for intelligence gathering, not mass scraping -- quality and accuracy of extracted data take priority over volume. The application builds on Prismatic Crawler Core primitives while adding higher-level orchestration, specialized Czech registry crawlers, and pluggable extractor modules for technology detection, metadata extraction, email discovery, and form analysis.

The module contains a rich set of specialized crawlers including JusticeCz for the Czech Commercial Register, RZP for the Trade Register, Insolvency Register monitoring, InfoDeska Justice crawlers (with both standard and v2 implementations), and a Podnikatel crawler for business entity data. Each registry crawler handles the specific authentication patterns, anti-bot protections, HTML structures, and data formats used by its target portal. A pipeline orchestrator coordinates multi-step extraction jobs, and an LLM extractor and validator leverage AI for complex unstructured content extraction.

Architecture

PrismaticCrawler
+-- Orchestrator      # Job management and scheduling
+-- WorkerPool        # Supervised crawl workers (poolboy)
+-- Queue             # URL frontier with priority queue
+-- PolitenessEngine  # Rate limiting, robots.txt, Crawl-delay
+-- Extractors        # Pluggable data extraction modules
|   +-- LinkExtractor
|   +-- TechnologyDetector
|   +-- MetadataExtractor
|   +-- EmailExtractor
|   +-- FormExtractor
+-- Registries        # Specialized Czech registry crawlers
|   +-- JusticeCz
|   +-- Rzp
|   +-- VrCz
|   +-- InsolvencniRejstrik
+-- Pipeline          # Multi-step extraction orchestration
+-- Cache             # Crawl result caching
+-- Storage           # Persistent crawl data storage

The worker pool pattern uses supervised processes under a pool supervisor:

Orchestrator --> Queue (URL Frontier)
                   |
          WorkerPool (10 workers)
          |    |    |    |    |
        W1   W2   W3   W4   W5 ...
          |    |    |    |    |
    PolitenessEngine (rate limiting)
          |
    HTTP Client --> Target
          |
    Extractors --> Results

Key Modules

Configuration

config :prismatic_crawler,
  global_rate_limit: {10, :second},
  per_domain_rate_limit: {1, :second},
  respect_robots_txt: true,
  respect_crawl_delay: true,
  user_agent: "PrismaticBot/1.0 (+https://prismatic-prod.fly.dev/bot)",
  max_concurrent_domains: 5,
  request_timeout: 30_000,
  retry_on_failure: true,
  max_retries: 3,
  backoff_strategy: :exponential,
  worker_pool_size: 10

API Reference

# Start a crawl job for a target domain
{:ok, job} = PrismaticCrawler.crawl("example.com",
  depth: 3, max_pages: 500, rate_limit: {1, :second},
  respect_robots: true, follow_external: false,
  extractors: [:links, :emails, :technologies, :metadata])

# Monitor crawl progress
{:ok, status} = PrismaticCrawler.status(job.job_id)

# Get crawl results with extracted data
{:ok, results} = PrismaticCrawler.results(job.job_id)

# Czech registry crawling
{:ok, data} = PrismaticCrawler.Registries.JusticeCz.crawl_company(ico)

# Stop a running crawl
:ok = PrismaticCrawler.stop(job.job_id)

Testing

Worker pool tests verify correct allocation, release, and timeout handling. Rate limiter tests verify token bucket behavior including burst handling and per-domain isolation. Registry crawler tests use HTML fixtures from Czech government portals to verify extraction accuracy against known data.

Integration tests exercise the full crawl lifecycle from seed URL through extraction and storage. Politeness tests verify robots.txt parsing and Crawl-delay directive compliance. Pipeline orchestrator tests verify multi-step extraction coordination with simulated registry responses.

Integration Points

The crawler feeds into the EASM pipeline for web-based asset discovery, complementing data from Shodan, Censys, and crt.sh. Specialized crawlers extract structured data from Czech registries (ARES, Justice.cz, RZP, VR.cz, Insolvency Register).

NABLA Compliance

Performance

Telemetry events: [:prismatic, :crawler, :fetch], [:prismatic, :crawler, :extract], [:prismatic, :crawler, :job_complete].

Related Resources

• Prismatic Crawler Core -- Low-level crawling primitives
• Prismatic Browser -- JavaScript rendering support
• Prismatic Czech Autocrawler -- Higher-level Czech registry automation
• Competitor Researcher -- Competitive intelligence gathering
• Adapter Pattern Specialist -- Extractor pattern compliance
• Architecture Review Specialist -- Worker pool and politeness design review
• Real-Time Monitoring -- Crawl job health and throughput monitoring
• Autonomous Self-Healing -- Automatic retry and circuit breaker recovery
• Intelligence Synthesis -- Crawled data feeds into multi-source intelligence fusion

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator