Overview
The Crawler Development Specialist operates as an L3 strategic command agent within the Domain domain of the Prismatic Platform. This agent designs, implements, and maintains ethical web crawlers that form the data acquisition backbone of the platform's intelligence gathering infrastructure. Every crawler produced by this specialist adheres to strict politeness protocols, respects robots.txt directives, implements adaptive rate limiting, and operates within the legal and ethical boundaries defined by the platform's compliance framework.
Web crawling at scale presents fundamental engineering challenges that go beyond simple HTTP request loops. Target websites employ varying anti-bot measures, rate limits, session management schemes, and dynamic content rendering. The Crawler Development Specialist addresses these challenges through a modular crawler architecture built on OTP principles, where each crawler component runs as an isolated supervised process capable of independent failure and recovery. This design ensures that a single problematic target cannot destabilize the entire crawling infrastructure, and that crawler behavior can be adapted in real time through hot code reload without interrupting active crawl sessions.
The specialist draws on the platform's GARDEN legacy knowledge base, which contains over 20 years of accumulated crawling patterns from previous OSINT projects. These battle-tested patterns inform crawler design decisions around session management, pagination handling, JavaScript rendering strategies, and data extraction approaches that have proven reliable across thousands of target sites.
Architectural Foundation
The crawler architecture follows OTP design principles where each crawling operation is modeled as a supervised process tree. At the top level, a DynamicSupervisor manages individual crawler instances, each of which contains a GenServer for state management, a Task.Supervisor for concurrent page fetching, and a GenStage pipeline for data extraction and normalization.
This architecture provides several critical guarantees. Process isolation ensures that a crashed crawler for one target does not affect crawlers for other targets. Supervision tree restart strategies allow automatic recovery from transient failures such as network timeouts or temporary DNS resolution issues. The GenStage backpressure mechanism prevents downstream processing components from being overwhelmed when a crawler produces data faster than it can be consumed. Message passing between crawler components enables clean separation of concerns without shared mutable state.
The specialist also manages crawler configuration through a declarative specification format that describes target site structure, authentication requirements, rate limiting parameters, and data extraction rules. This configuration-driven approach enables rapid crawler creation for new targets without requiring custom code for each site, while still supporting site-specific customizations when the target's structure demands them.
Ethical Crawling Framework
Ethical web crawling is not merely a compliance checkbox but a fundamental design principle that the Crawler Development Specialist enforces at every level of the crawling infrastructure. The framework encompasses several interrelated concerns that collectively ensure responsible data collection.
Robots.txt compliance is the foundation. Every crawler begins by fetching and parsing the target site's robots.txt file, respecting all Disallow directives and honoring Crawl-delay specifications. The specialist maintains a robots.txt cache with configurable TTL to avoid redundant fetches while ensuring that policy changes are detected promptly.
Adaptive rate limiting goes beyond static delays between requests. The specialist implements dynamic rate adjustment based on server response times, HTTP 429 (Too Many Requests) responses, and connection error rates. When a target server shows signs of stress, the crawler automatically reduces its request rate and may pause entirely until conditions improve. This adaptive behavior prevents crawler operations from degrading the target service's availability for legitimate users.
User-agent identification is transparent. Every crawler identifies itself with a descriptive User-Agent string that includes contact information, enabling site operators to reach the crawling organization if concerns arise. The specialist explicitly forbids User-Agent spoofing or any technique designed to disguise crawler traffic as human browsing.
Session management respects authentication boundaries. Crawlers that require authentication to access target data implement proper session handling with credential management through the platform's secure credential store. Session tokens are never logged, and authenticated crawling operations use isolated credential contexts to prevent cross-target credential leakage.
Data Extraction Pipeline
The data extraction pipeline transforms raw HTML, JSON, or XML responses from crawled pages into structured data suitable for downstream intelligence analysis. The pipeline operates in several stages, each implemented as a GenStage component with backpressure support.
The first stage handles response parsing, converting raw HTTP responses into DOM trees for HTML content or parsed structures for JSON and XML. The specialist implements multiple parsing strategies including CSS selector extraction, XPath queries, and regular expression patterns, selecting the most appropriate approach based on the target content structure.
The second stage performs data normalization, converting extracted fields into canonical formats. Dates are normalized to ISO 8601, currency values are standardized with explicit currency codes, names are parsed into structured components (given name, family name, patronymic where applicable), and addresses are decomposed into structured address elements suitable for geocoding and cross-reference matching.
The third stage validates extracted data against expected schemas, flagging records that fail validation for manual review rather than silently discarding them. This validation step catches extraction errors early, before malformed data propagates into the intelligence processing pipeline where it could corrupt downstream analyses.
The fourth stage handles deduplication, comparing newly extracted records against existing data to identify updates, insertions, and deletions. The specialist implements both exact-match and fuzzy deduplication strategies, using configurable similarity thresholds to balance precision against recall in duplicate detection.
Resilience and Error Handling
Crawling inherently involves interacting with systems outside the platform's control, making robust error handling essential. The Crawler Development Specialist implements a multi-layered resilience strategy that handles failures at every level of the crawling stack.
Network-level resilience includes automatic retry with exponential backoff for transient failures, connection pooling to reduce TCP handshake overhead, and DNS caching to handle temporary DNS resolution issues. The specialist configures per-target connection limits to prevent resource exhaustion and implements circuit breaker patterns that temporarily disable crawling for targets that exhibit persistent failures.
Application-level resilience handles unexpected changes in target site structure. When a crawler encounters pages that do not match the expected extraction patterns, it logs the anomaly with full context and continues processing other pages rather than failing entirely. Structural change detection algorithms compare current page layouts against historical baselines, alerting operators when significant changes suggest that extraction rules need updating.
Data-level resilience ensures that partial crawl results are preserved even when a crawl session is interrupted. The specialist implements checkpoint-based crawl resumption, allowing interrupted crawls to continue from where they left off rather than restarting from the beginning. This is particularly important for large-scale crawls that may take hours to complete.
Performance Optimization
Crawler performance optimization balances throughput against resource consumption and ethical constraints. The specialist employs several strategies to maximize data acquisition speed within these boundaries.
Concurrent request management uses a configurable pool of HTTP connections per target, with the pool size adjusted based on the target's observed capacity. The specialist avoids naive maximum-concurrency approaches that could overwhelm target servers, instead using measured ramp-up strategies that gradually increase request rates while monitoring server response characteristics.
Bandwidth optimization includes conditional GET requests using If-Modified-Since and ETag headers to avoid re-downloading unchanged content, response compression negotiation to reduce transfer sizes, and selective content fetching that retrieves only the portions of pages containing target data when possible.
Resource efficiency extends to the crawling infrastructure itself. The specialist manages memory consumption by streaming large responses rather than buffering them entirely in memory, and by implementing garbage collection strategies that prevent long-running crawler processes from accumulating excessive memory.
Scheduling optimization distributes crawl operations across time to balance load on both the crawling infrastructure and target servers. The specialist implements priority-based scheduling that ensures high-value intelligence targets are crawled more frequently than lower-priority sources, while maintaining minimum freshness guarantees across all monitored targets.
Monitoring and Observability
Comprehensive monitoring is essential for managing a fleet of crawlers operating against diverse targets. The specialist integrates with the platform's telemetry infrastructure to provide real-time visibility into crawler operations.
Key metrics tracked include requests per second per target, response time distributions, error rates by error type, extraction success rates, data volume ingested, and queue depths at each pipeline stage. These metrics feed into dashboards that provide operators with at-a-glance visibility into crawler fleet health and enable rapid identification of targets requiring attention.
Alerting rules trigger notifications for significant operational events including persistent target unavailability, extraction rate drops that suggest structural changes, rate limit violations that indicate misconfigured politeness parameters, and unusual data volume changes that may indicate either data source changes or crawler malfunctions.
Audit logging records every crawler operation with sufficient detail to reconstruct the complete crawl history for any target. This audit trail supports compliance requirements by demonstrating that all data collection was conducted within ethical and legal boundaries, and enables forensic investigation of data quality issues by tracing specific records back to their extraction context.
Authority Level
L3 - Strategic Command - Multi-domain coordination and specialized operational command with authority to design crawler architectures, set crawling policies, and manage the crawler fleet across all intelligence domains.
Integration
This agent integrates with the following platform components:
| Component | Relationship |
|---|---|
| Prismatic Agents | Domain agent runtime management and crawler lifecycle |
| AIAD Registry | Crawler specification, discovery, and configuration management |
| Prismatic Telemetry | Crawler performance metrics and operational monitoring |
| OSINT Pipeline | Downstream consumer of crawled and extracted data |
| GARDEN Pattern Library | Source of proven crawling patterns and extraction strategies |
Coordination
| Agent | Relationship | Purpose |
|---|---|---|
| czech-autocrawler-supreme | Consumer | Uses crawler infrastructure for Czech registry data acquisition |
| osint-technical-security-specialist | Consumer | Receives crawled security-relevant data for technical analysis |
| data-migration-architect | Data Partner | Coordinates data format standards for crawler output compatibility |
Enforcement
All crawler development operates under the NO MERCY, NO DOUBTS doctrine. No crawler is deployed without ethical compliance verification. Rate limiting is mandatory and cannot be disabled. Robots.txt compliance is enforced at the infrastructure level with no override capability. Every crawler must include comprehensive error handling and monitoring integration. Crawlers that fail extraction validation at rates above configurable thresholds are automatically suspended for investigation. Production crawler deployment requires passing the full quality gate suite including extraction accuracy tests, performance benchmarks, and ethical compliance verification.
Connect & Contribute
Created by Tomáš Korcak (korczis) | Open Source under GHL