Prismatic Browser

Overview

Prismatic Browser provides headless browser automation capabilities for web content collection that requires JavaScript rendering, managing a supervised pool of Chromium instances for parallel page loading, screenshot capture, DOM extraction, and interaction automation. The system is essential for OSINT collection from modern JavaScript-heavy web applications where HTTP-only crawling returns empty or incomplete content.

Modern web applications render content dynamically through JavaScript execution. A standard HTTP client retrieving the HTML source of such applications receives only the application shell without rendered content. Government portals, financial services, and corporate websites increasingly use single-page application architectures that load data asynchronously after initial page load. OSINT collection from these sources requires a full browser environment that executes JavaScript, waits for content rendering, and then extracts the final DOM state.

The architecture implements a pool manager that maintains configurable browser instances with automatic recycling, health monitoring, and memory-aware scaling. Anti-detection features include user-agent rotation, fingerprint management, proxy rotation with health checking, and request timing randomization to avoid triggering anti-bot protections on target sites. Each browser instance runs as an isolated supervised process, ensuring that a crash or memory leak in one instance does not affect the pool or other rendering operations. After each request, instances are recycled by clearing cookies, local storage, and navigation history, with full restarts after a configurable number of requests to prevent JavaScript heap accumulation.

Architecture

The architecture follows a pool-based design with priority-ordered request queuing, managed instance allocation, and health monitoring.

Request Queue (priority-based)
       |
  Pool Manager
  (instance allocation, health monitoring)
       |
  +----+----+----+----+----+
  |    |    |    |    |    |
  Browser Browser Browser Browser Browser
  Instance Instance Instance Instance Instance
       |
  Page Load --> JavaScript Execution --> Wait for Selector
       |
  Content Extraction (CSS/XPath selectors)
  Screenshot Capture (full page/element)
  PDF Generation
       |
  Result Return --> Instance Recycle

The process topology uses three supervised GenServers managing the pool, request queue, and health monitoring:

PrismaticBrowser.Application (Supervisor, :one_for_one)
+-- PrismaticBrowser.PoolManager (GenServer)
|     Manages browser instance pool, allocation, recycling
+-- PrismaticBrowser.RequestQueue (GenServer)
|     Priority queue for pending browser requests
+-- PrismaticBrowser.HealthMonitor (GenServer)
      Periodic instance health checks and memory monitoring

Requests enter the RequestQueue with priority ordering. The PoolManager allocates an available browser instance or queues the request if all instances are busy. The instance navigates to the target URL, executes JavaScript, waits for the specified selector or timeout, and performs the requested operation. Results are returned to the caller, and the instance is recycled for reuse.

Key Modules

defmodule PrismaticBrowser.RenderResult do
  @type t :: %__MODULE__{
    url: String.t(),
    status: pos_integer(),
    content: String.t(),
    extracted_data: map() | nil,
    screenshot: binary() | nil,
    render_time_ms: pos_integer(),
    javascript_errors: [String.t()]
  }
end

Configuration

config :prismatic_browser,
  pool_size: 5,
  max_requests_per_instance: 100,
  request_timeout: 30_000,
  page_load_timeout: 15_000,
  memory_limit_mb: 512,
  user_agent_pool_size: 50,
  proxy_rotation: true,
  anti_detection: true

Pool size determines concurrent rendering capacity with each instance consuming approximately 100-200MB of memory. The max requests per instance setting controls how many pages an instance renders before a full restart to prevent memory leaks. Request and page load timeouts protect against unresponsive target sites. The memory limit triggers instance recycling when heap usage exceeds the threshold.

API Reference

# Render page and extract content
@spec render(String.t(), keyword()) :: {:ok, RenderResult.t()} | {:error, term()}
PrismaticBrowser.render("https://example.com",
  wait_for: "#main-content", extract: :text)

# Capture screenshot at specified resolution
@spec screenshot(String.t(), keyword()) :: {:ok, binary()} | {:error, term()}
PrismaticBrowser.screenshot("https://example.com",
  width: 1920, height: 1080, format: :png)

# Extract structured data using selector map
@spec extract(String.t(), map()) :: {:ok, map()} | {:error, term()}
PrismaticBrowser.extract("https://example.com", %{
  title: "h1", items: "ul.results li", links: "a[href]"})

Testing

Extractor tests verify CSS selector and XPath extraction against known HTML structures. Anti-detection tests verify user-agent rotation and fingerprint randomization produce expected distributions. Full rendering pipeline integration tests load known web pages and verify content extraction, screenshot capture, and error handling.

Property-based tests use StreamData generators to produce random CSS selectors and URLs, verifying that the browser pool handles all inputs without crashing or leaking instances. Pool management tests verify correct allocation, recycling, and exhaustion handling under concurrent load.

Integration Points

External dependency: Chromium browser binary (headless mode) with Chrome DevTools Protocol for browser control. No cloud rendering services are used.

NABLA Compliance

Malicious web pages could attempt to exploit browser vulnerabilities. Sandboxed Chromium execution with restricted permissions mitigates this risk. No file system access or network access beyond the target page is permitted from the browser context.

Performance

Telemetry events: [:prismatic, :browser, :render], [:prismatic, :browser, :instance_recycled], [:prismatic, :browser, :pool_exhausted]. Key metrics include render latency, pool utilization, and memory consumption per instance.

Related Resources

• Prismatic Crawler -- HTTP-based crawling infrastructure
• Prismatic OSINT Core -- Intelligence source layer
• Chrome DevTools Protocol -- Browser automation protocol
• Adapter Pattern Specialist -- Designs adapter interfaces for browser pool management and content extraction
• Alert Management Specialist -- Configures alerting for pool exhaustion and instance health degradation
• Competitor Researcher -- Leverages browser automation for competitive intelligence collection
• Intelligence Synthesis -- Combines browser-rendered content with other OSINT sources
• Real-Time Monitoring -- Monitors browser pool utilization and render latency in real time
• Quality Gates -- Validates content extraction accuracy and pool management reliability

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator