Prismatic Crawler Core

Overview

Prismatic Crawler Core is the platform's web crawling engine, providing HTTP-based data collection with comprehensive rate limiting, robots.txt compliance, and structured content extraction. It manages prioritized crawl queues, handles authentication for protected resources, respects politeness policies, and extracts structured data from HTML pages using configurable CSS and XPath selectors.

The crawler serves as the foundational collection mechanism for several platform subsystems. OSINT monitoring uses it to periodically check web-accessible intelligence sources. The Czech autocrawler leverages its extraction capabilities for registry data collection. Prismatic Perimeter relies on it for web technology fingerprinting and content analysis during attack surface discovery.

Resilience is a core design principle. The crawler implements retry with exponential backoff, proxy rotation with health checking, circuit breakers for persistently failing domains, and a dead letter queue for pages that cannot be successfully fetched after all retry attempts. This ensures that transient failures do not prevent data collection while persistent failures are surfaced for operator attention. The architecture follows OTP supervision patterns with a configurable fetcher pool, per-domain rate limiting enforced through token bucket counters stored in ETS, and content extraction through pluggable CSS, XPath, JSON-LD, and microdata parsers.

Architecture

The architecture follows a pipeline design from seed URL ingestion through HTTP fetching, content parsing, data extraction, and storage persistence. Each stage is independently supervised and horizontally scalable.

Seed URLs --> Queue Manager --> Fetcher Pool --> Parser --> Extractor --> Storage
                |                |              |          |
          Priority/Dedup   Rate Limiter     Encoding    CSS/XPath
          Domain Queue     Robots.txt       Detection   JSON-LD
          Depth Control    Proxy Rotation   HTML Parse  Microdata
                |                |              |
          Dead Letter      Circuit Breaker  Schema Map

The fetcher pool uses a configurable number of worker processes managed under a pool supervisor, with each worker handling one HTTP request at a time. Per-domain rate limiting is enforced through token bucket counters preventing any single crawl job from overwhelming target servers. The queue manager maintains separate per-domain queues with priority ordering, ensuring that high-priority pages are fetched before lower-priority discovery links.

Process Topology

PrismaticCrawlerCore.Application (Supervisor, :one_for_one)
+-- PrismaticCrawlerCore.CrawlCoordinator (GenServer)
|     Crawl job orchestration and worker dispatch
+-- PrismaticCrawlerCore.QueueManager (GenServer)
|     Per-domain priority queues with deduplication
+-- PrismaticCrawlerCore.FetcherPool (PoolSupervisor)
|     Configurable worker pool for HTTP requests
+-- PrismaticCrawlerCore.RateLimiter (GenServer)
|     ETS-backed token bucket per domain
+-- PrismaticCrawlerCore.DeadLetterQueue (GenServer)
      Failed pages awaiting manual review

Requests enter the QueueManager where they are deduplicated and assigned to per-domain queues. The CrawlCoordinator checks the RateLimiter before dispatching work to the FetcherPool. Workers perform the HTTP request, feed the response through the Parser for encoding detection and HTML normalization, then pass the DOM to the Extractor for structured data retrieval. Results flow into the storage layer while the Coordinator updates queue state and schedules follow-up URLs discovered in the page.

Key Modules

The CrawlerBehaviour defines the interface that pluggable crawler implementations must satisfy. This enables specialized crawlers (such as the Czech registry crawlers in Prismatic Crawler) to reuse the core infrastructure while providing domain-specific fetch logic, authentication handling, and extraction rules.

defmodule PrismaticCrawlerCore.CrawlerBehaviour do
  @callback crawl(seed_url :: String.t(), opts :: keyword()) ::
    {:ok, CrawlResult.t()} | {:error, term()}

  @callback extract(html :: String.t(), selectors :: map()) ::
    {:ok, map()} | {:error, term()}

  @callback should_follow?(url :: String.t(), depth :: non_neg_integer()) ::
    boolean()
end

Crawl Strategies

Prismatic Crawler Core supports three crawl strategies that determine the order in which URLs are processed from the queue. The choice of strategy significantly affects both the data collected and the resource consumption pattern.

Breadth-first crawling processes all pages at depth N before moving to depth N+1. This strategy provides the broadest coverage of a site's top-level structure and is preferred for attack surface discovery where understanding the site's overall architecture matters more than deep content analysis.

Depth-first crawling follows link chains to maximum depth before backtracking. This strategy is preferred for specific content extraction tasks where the target data is likely located deep within a site's navigation hierarchy, such as Czech registry detail pages accessed through multi-step form submissions.

Priority-based crawling assigns scores to discovered URLs based on configurable heuristics (URL pattern matching, content type hints, link anchor text analysis) and processes highest-priority URLs first regardless of depth. This strategy is used for intelligence-directed crawling where specific page types are more valuable than others.

Configuration

config :prismatic_crawler_core,
  fetcher_pool_size: 10,
  per_domain_rate_limit: {1, :second},
  global_rate_limit: {10, :second},
  respect_robots_txt: true,
  max_depth: 5,
  request_timeout: 30_000,
  retry_attempts: 3,
  backoff_strategy: :exponential,
  proxy_rotation: false,
  user_agent: "PrismaticBot/1.0 (+https://prismatic-prod.fly.dev/bot)",
  dead_letter_max_age: :timer.hours(72),
  max_queue_size_per_domain: 1000

Configuration controls pool size, rate limiting at both domain and global levels, robots.txt compliance, maximum crawl depth, request timeouts, retry behavior, and user agent identification. Proxy rotation can be enabled for distributed crawling through rotating proxy pools. The dead letter maximum age determines how long failed pages are retained before automatic cleanup.

API Reference

# Start a crawl job with structured extraction
{:ok, job} = PrismaticCrawlerCore.crawl(%{
  seed_urls: ["https://example.com"],
  max_depth: 3,
  selectors: %{title: "h1", content: "article", links: "a[href]"},
  rate_limit: {1, :second},
  strategy: :breadth_first
})

# Check crawl progress
{:ok, status} = PrismaticCrawlerCore.status(job.id)
# => %{pages_crawled: 145, pages_queued: 302, errors: 3, elapsed: "2m 34s"}

# Fetch a single page with extraction
{:ok, result} = PrismaticCrawlerCore.fetch("https://example.com/page",
  selectors: %{title: "h1", meta: "meta[name=description]"},
  follow_redirects: true)

# List dead letter queue entries for review
{:ok, failed} = PrismaticCrawlerCore.dead_letters(job.id)
# => [%{url: "https://...", attempts: 3, last_error: :timeout, last_attempt: ~U[...]}]

# Retry specific dead letter entries
:ok = PrismaticCrawlerCore.retry_dead_letters(job.id, urls: ["https://..."])

Robots.txt and Politeness

The crawler implements full robots.txt parsing and compliance. On first contact with a domain, the crawler fetches and parses the robots.txt file, caching the parsed rules for subsequent requests. The parser handles standard directives (User-agent, Disallow, Allow, Crawl-delay) and common extensions (Sitemap). When a Crawl-delay directive is present for the crawler's user agent, the per-domain rate limiter is automatically adjusted to respect the specified delay, even if it is slower than the configured default.

Politeness extends beyond robots.txt compliance. The crawler monitors HTTP response headers for rate limiting signals (429 status codes, Retry-After headers, X-RateLimit-Remaining) and dynamically adjusts request frequency. A 429 response triggers immediate backoff with the delay specified by the Retry-After header, or a default exponential backoff if no header is present.

Testing

Crawl coordination tests verify correct worker dispatch, rate limiting enforcement, and depth control. Extraction tests validate CSS selector and XPath parsing against known HTML fixtures. Resilience tests verify retry behavior, circuit breaker state transitions, and dead letter queue management.

Integration tests exercise the full crawl pipeline from seed URL through page fetch, parsing, extraction, and storage. Property-based tests use StreamData generators to produce random URL patterns and extraction selectors, verifying that the pipeline handles all valid inputs without crashes. Dead letter queue tests verify correct retention, retry, and cleanup behavior.

Integration Points

NABLA Compliance

Performance

Telemetry events: [:prismatic, :crawler_core, :fetch], [:prismatic, :crawler_core, :extract], [:prismatic, :crawler_core, :dead_letter], [:prismatic, :crawler_core, :rate_limited]. Metrics include pages per second, error rate, queue depth, and dead letter queue size.

Related Resources

• Prismatic Browser -- Headless browser for JavaScript-rendered pages
• Prismatic Crawler -- Higher-level crawl orchestration and job management
• Prismatic Compression -- Response decompression for crawled content
• Competitor Researcher -- Leverages crawler infrastructure for competitive intelligence collection
• Adapter Pattern Specialist -- Ensures crawler extractors follow consistent adapter patterns
• Architecture Review Specialist -- Reviews crawler pipeline architecture for resilience and scalability
• Real-Time Monitoring -- Continuous monitoring of crawl job health and throughput metrics
• Autonomous Self-Healing -- Automatic recovery from crawler failures via circuit breakers and retry
• Telemetry Integration -- Crawl performance metrics emitted through the telemetry pipeline

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator