Definition

Token Bucket is a rate-limiting algorithm that governs the flow of operations through the Prismatic Platform by maintaining a finite pool of tokens that are replenished at a fixed rate. Each incoming request or operation must acquire one or more tokens from the bucket before proceeding. When the bucket is empty, operations are either queued for later execution, rejected immediately, or routed through a degraded service path depending on the configured policy. The algorithm provides a natural mechanism for burst tolerance -- the bucket can accumulate tokens up to a maximum capacity during idle periods, allowing short bursts of activity that exceed the steady-state rate without triggering rejection.

Within the Prismatic Platform's 115-application umbrella architecture, token bucket rate limiting operates at multiple levels: per-endpoint in the API Gateway, per-agent in the AIAD agent orchestration layer, per-adapter in the OSINT data collection pipeline, and per-tenant in the multi-user web interface. The algorithm is implemented as an OTP-compliant GenServer backed by ETS tables for microsecond-level token acquisition, supervised within the platform's Supervision Tree to guarantee availability even under extreme load conditions.

The token bucket is a foundational component of the platform's resilience strategy, working in concert with Circuit Breakers, Backpressure mechanisms, and Fault Tolerance patterns to ensure that no single client, agent, or subsystem can destabilize the entire platform through excessive resource consumption.

Mathematical Foundation

The token bucket algorithm is defined by four parameters that determine its behavior:

Rate (r): The number of tokens added to the bucket per unit time. In the Prismatic Platform, this is expressed as tokens per second. A rate of 100 means 100 tokens are added every second, or equivalently, one token every 10 milliseconds.

Capacity (C): The maximum number of tokens the bucket can hold. Once the bucket is full, additional tokens generated by the rate are discarded. The capacity determines the maximum burst size -- the number of operations that can be processed instantaneously before rate limiting takes effect.

Token Cost (k): The number of tokens consumed by a single operation. Not all operations cost the same: a simple health check might cost 1 token, while a complex OSINT investigation might cost 10 tokens. This allows proportional rate limiting based on operation complexity.

Current Fill Level (n): The number of tokens currently in the bucket, where 0 <= n <= C at all times.

The steady-state throughput of a token bucket is exactly r/k operations per second when k tokens are consumed per operation. The maximum burst size is C/k operations. The time to refill from empty to full is C/r seconds. These relationships govern how the platform configures buckets for different use cases:

Use Case	Rate (r)	Capacity (C)	Token Cost (k)	Steady Throughput	Max Burst
API endpoint	100/s	500	1	100 req/s	500 req
OSINT adapter	10/s	30	1	10 req/s	30 req
Agent task	50/s	200	1-10	5-50 tasks/s	20-200 tasks
Web session	30/s	100	1	30 req/s	100 req
Perimeter scan	5/s	15	3	1.67 scan/s	5 scans

Token Bucket vs. Leaky Bucket

The token bucket is frequently compared with the leaky bucket algorithm, and understanding the distinction is critical for selecting the appropriate rate-limiting strategy:

Token Bucket permits bursts. When the bucket has accumulated tokens during a quiet period, a client can send a burst of requests up to the bucket capacity. The algorithm smooths traffic over time but allows instantaneous spikes. This is ideal for OSINT operations where an adapter might need to issue 30 requests in rapid succession to enumerate a company's subsidiaries, then remain idle for several seconds.

Leaky Bucket enforces a strict constant output rate. Regardless of input patterns, the output is always exactly r operations per second. Bursts are absorbed into a queue and drained at the fixed rate. This is ideal for external API calls where the target service enforces strict per-second rate limits (e.g., Shodan API at 1 request per second).

The Prismatic Platform uses both algorithms in different contexts:

defmodule PrismaticRateLimiter.TokenBucket do
  @moduledoc """
  Implements the token bucket rate-limiting algorithm using ETS for
  microsecond-resolution token state. Supports configurable rate,
  capacity, and per-operation token costs.

  The bucket state is stored in an ETS table for lock-free concurrent
  reads. Token acquisition uses `:ets.update_counter/4` with atomic
  operations to prevent race conditions under concurrent access.

  Designed for use cases that benefit from burst tolerance:
  API endpoints, agent task scheduling, and web session management.
  """

  use GenServer

  alias PrismaticRateLimiter.{BucketConfig, TelemetryEmitter}

  @type bucket_id :: atom() | String.t()
  @type acquire_result :: {:ok, remaining :: non_neg_integer()} | {:error, :rate_limited}

  @ets_table :prismatic_token_buckets

  @spec start_link(config :: BucketConfig.t()) :: GenServer.on_start()
  def start_link(%BucketConfig{} = config) do
    GenServer.start_link(__MODULE__, config, name: via_tuple(config.id))
  end

  @spec acquire(bucket_id(), cost :: pos_integer()) :: acquire_result()
  def acquire(bucket_id, cost \\ 1) do
    refill(bucket_id)

    case :ets.update_counter(@ets_table, bucket_id, {3, -cost, 0, 0}, {bucket_id, 0, 0, 0}) do
      0 ->
        TelemetryEmitter.emit_rejected(bucket_id, cost)
        {:error, :rate_limited}

      remaining when remaining >= 0 ->
        TelemetryEmitter.emit_acquired(bucket_id, cost, remaining)
        {:ok, remaining}
    end
  end

  @spec inspect_bucket(bucket_id()) :: {:ok, map()} | {:error, :not_found}
  def inspect_bucket(bucket_id) do
    case :ets.lookup(@ets_table, bucket_id) do
      [{^bucket_id, last_refill, tokens, capacity}] ->
        {:ok, %{
          id: bucket_id,
          tokens: tokens,
          capacity: capacity,
          last_refill: last_refill,
          fill_percentage: Float.round(tokens / max(capacity, 1) * 100, 1)
        }}

      [] ->
        {:error, :not_found}
    end
  end

  @impl GenServer
  def init(%BucketConfig{} = config) do
    ensure_ets_table()
    :ets.insert(@ets_table, {config.id, now_ms(), config.capacity, config.capacity})
    schedule_refill(config.refill_interval_ms)
    {:ok, config}
  end

  @impl GenServer
  def handle_info(:refill, %BucketConfig{} = config) do
    refill(config.id)
    schedule_refill(config.refill_interval_ms)
    {:noreply, config}
  end

  @spec refill(bucket_id()) :: :ok
  defp refill(bucket_id) do
    case :ets.lookup(@ets_table, bucket_id) do
      [{^bucket_id, last_refill, _tokens, capacity}] ->
        elapsed_ms = now_ms() - last_refill
        config = lookup_config(bucket_id)
        tokens_to_add = div(elapsed_ms * config.rate, 1_000)

        if tokens_to_add > 0 do
          :ets.update_counter(@ets_table, bucket_id, [
            {2, 0, 0, now_ms()},
            {3, tokens_to_add, capacity, capacity}
          ])
        end

        :ok

      [] ->
        :ok
    end
  end

  defp now_ms, do: System.monotonic_time(:millisecond)
  defp schedule_refill(interval), do: Process.send_after(self(), :refill, interval)
  defp via_tuple(id), do: {:via, Registry, {PrismaticRateLimiter.Registry, id}}
  defp ensure_ets_table do
    if :ets.whereis(@ets_table) == :undefined do
      :ets.new(@ets_table, [:named_table, :public, :set, {:write_concurrency, true}])
    end
  end
  defp lookup_config(bucket_id), do: GenServer.call(via_tuple(bucket_id), :get_config)
end

Hierarchical Rate Limiting Architecture

The Prismatic Platform implements rate limiting in a hierarchical structure that mirrors the platform's supervision tree. Each level has its own token bucket with distinct parameters, and a request must acquire tokens at every level before proceeding:

Level 1 -- Global Platform Bucket: A single bucket that governs the total throughput of the entire platform. This protects against cascading failures where multiple subsystems simultaneously spike in resource consumption. The global bucket is configured with generous capacity (10,000 tokens) and high rate (1,000/s), serving as an absolute ceiling rather than a day-to-day limiter.

Level 2 -- Application Bucket: Each of the 115 umbrella applications has its own token bucket. The API application (prismatic_api) might have a rate of 500/s, while a background processing application might have 50/s. Application buckets prevent any single app from monopolizing platform resources.

Level 3 -- Feature Bucket: Within each application, specific features have dedicated buckets. The Perimeter EASM scanning feature has strict limits to avoid overwhelming target networks. The OSINT toolbox has per-category buckets aligned with external API rate limits.

Level 4 -- Client/Tenant Bucket: Individual clients or tenants receive their own buckets, ensuring fair resource distribution. A single tenant running intensive OSINT investigations cannot degrade service for other tenants.

defmodule PrismaticRateLimiter.HierarchicalLimiter do
  @moduledoc """
  Implements hierarchical rate limiting where requests must acquire
  tokens at multiple levels before proceeding. If any level rejects
  the request, the tokens acquired at higher levels are returned.

  The hierarchy mirrors the platform supervision tree:
  Global -> Application -> Feature -> Client/Tenant
  """

  alias PrismaticRateLimiter.TokenBucket

  @type level :: :global | :application | :feature | :client
  @type hierarchy_result :: {:ok, map()} | {:error, :rate_limited, level()}

  @spec acquire_hierarchical(levels :: [{level(), TokenBucket.bucket_id(), pos_integer()}]) ::
          hierarchy_result()
  def acquire_hierarchical(levels) do
    acquire_recursive(levels, [])
  end

  defp acquire_recursive([], acquired) do
    {:ok, %{levels_acquired: length(acquired), tokens: acquired}}
  end

  defp acquire_recursive([{level, bucket_id, cost} | rest], acquired) do
    case TokenBucket.acquire(bucket_id, cost) do
      {:ok, remaining} ->
        acquire_recursive(rest, [{level, bucket_id, cost, remaining} | acquired])

      {:error, :rate_limited} ->
        rollback(acquired)
        {:error, :rate_limited, level}
    end
  end

  @spec rollback(acquired :: list()) :: :ok
  defp rollback(acquired) do
    Enum.each(acquired, fn {_level, bucket_id, cost, _remaining} ->
      TokenBucket.return(bucket_id, cost)
    end)
  end
end

Adaptive Rate Limiting

Static rate limits are insufficient for a platform that operates across varying load conditions, time zones, and external API availability windows. The Prismatic Platform implements adaptive rate limiting that adjusts bucket parameters based on real-time system health metrics:

Load-Based Adaptation: When CPU utilization exceeds 80% or memory pressure is detected, the adaptive policy reduces bucket capacities by 25-50%, protecting system stability at the cost of throughput. When load drops below 50%, capacities are gradually restored.

Error-Rate Adaptation: When an external API begins returning 429 (Too Many Requests) or 503 (Service Unavailable) responses, the corresponding feature bucket's rate is automatically reduced. The reduction follows an exponential backoff pattern: 50% reduction on first trigger, 75% on second, 90% on third, with gradual recovery after a cool-down period.

Time-Based Adaptation: Some external services have time-based quotas (e.g., 1,000 requests per day). The adaptive policy distributes these quotas evenly across the operational window, adjusting the bucket rate dynamically as the remaining quota decreases relative to the remaining time window.

defmodule PrismaticRateLimiter.AdaptivePolicy do
  @moduledoc """
  Dynamically adjusts token bucket parameters based on real-time
  system health metrics, external API response patterns, and
  time-based quota management.

  Policies are evaluated every 5 seconds and adjustments are applied
  smoothly using exponential moving averages to prevent oscillation.
  """

  @spec evaluate_and_adjust(bucket_id :: atom(), metrics :: map()) ::
          {:ok, :no_change} | {:ok, :adjusted, map()}
  def evaluate_and_adjust(bucket_id, metrics) do
    current_config = PrismaticRateLimiter.TokenBucket.get_config(bucket_id)

    adjustments =
      []
      |> maybe_load_adjustment(metrics.cpu_percent, metrics.memory_percent)
      |> maybe_error_rate_adjustment(metrics.error_rate_5m)
      |> maybe_quota_adjustment(metrics.remaining_quota, metrics.remaining_window_seconds)

    case adjustments do
      [] ->
        {:ok, :no_change}

      adjustments ->
        new_config = apply_adjustments(current_config, adjustments)
        PrismaticRateLimiter.TokenBucket.update_config(bucket_id, new_config)
        {:ok, :adjusted, new_config}
    end
  end
end

Integration with Circuit Breaker Pattern

Token buckets and circuit breakers serve complementary but distinct purposes. The token bucket controls the rate of requests to a service, while the circuit breaker controls whether requests should be sent at all based on failure patterns. In the Prismatic Platform, these two patterns are composed into a unified flow control pipeline:

Token Bucket Check: Does the caller have available rate budget?
Circuit Breaker Check: Is the target service healthy enough to receive requests?
Execution: If both checks pass, the request proceeds.
Result Feedback: The response outcome feeds back into both systems -- a failure increments the circuit breaker's failure counter, while a timeout or rate-limit response from the target triggers adaptive policy adjustments on the token bucket.

This composition prevents two common failure modes: (a) a token bucket allowing requests through to a service that is already failing (solved by the circuit breaker stopping requests), and (b) a circuit breaker allowing requests through at a rate that overwhelms the recovering service (solved by the token bucket limiting the rate of requests as the circuit breaker transitions from open to half-open to closed).

ETS-Backed Performance

The choice of ETS as the storage backend for token bucket state is driven by performance requirements. Rate limiting is a hot path -- every request to the platform passes through at least one token bucket check. The implementation must add negligible latency:

Operation	Average Latency	P99 Latency	Mechanism
Token acquire	~1 microsecond	~5 microseconds	`:ets.update_counter/4` atomic
Token refill	~2 microseconds	~8 microseconds	`:ets.update_counter/4` batch
Bucket inspect	~0.5 microseconds	~2 microseconds	`:ets.lookup/2`
Config update	~3 microseconds	~10 microseconds	`:ets.insert/2` overwrite

The :write_concurrency option on the ETS table allows concurrent write operations to different keys (different bucket IDs) without contention. Since each bucket ID maps to a single row in the ETS table, concurrent requests to different buckets experience zero contention. Concurrent requests to the same bucket use the atomic :ets.update_counter/4 operation, which provides serialization at the ETS level without requiring a GenServer bottleneck.

Supervision and Fault Recovery

Each token bucket GenServer is supervised within a DynamicSupervisor that can create and destroy buckets at runtime as new clients connect or features are enabled:

defmodule PrismaticRateLimiter.BucketSupervisor do
  @moduledoc """
  Dynamically supervises token bucket processes. Creates new buckets
  on demand and handles bucket process crashes by restarting with
  the last known configuration and a full token allocation.

  Uses :one_for_one strategy -- a single bucket crash does not
  affect other buckets, maintaining rate limiting for all other
  clients and features during recovery.
  """

  use DynamicSupervisor

  @spec start_link(opts :: keyword()) :: Supervisor.on_start()
  def start_link(opts) do
    DynamicSupervisor.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @impl DynamicSupervisor
  def init(_opts) do
    DynamicSupervisor.init(strategy: :one_for_one)
  end

  @spec create_bucket(config :: BucketConfig.t()) :: {:ok, pid()} | {:error, term()}
  def create_bucket(%BucketConfig{} = config) do
    DynamicSupervisor.start_child(__MODULE__, {PrismaticRateLimiter.TokenBucket, config})
  end

  @spec remove_bucket(bucket_id :: atom()) :: :ok | {:error, :not_found}
  def remove_bucket(bucket_id) do
    case Registry.lookup(PrismaticRateLimiter.Registry, bucket_id) do
      [{pid, _}] -> DynamicSupervisor.terminate_child(__MODULE__, pid)
      [] -> {:error, :not_found}
    end
  end
end

When a bucket GenServer crashes (due to unexpected conditions in adaptive policy evaluation, ETS table corruption, or any other fault), the supervisor restarts it with the last known configuration. The restarted bucket begins with a full token allocation -- a deliberate choice that favors availability over strictness. The alternative (restarting with zero tokens) would punish all clients during recovery, which violates the platform's principle that infrastructure faults should not cascade into user-visible degradation.

Telemetry and Observability

Token bucket operations emit telemetry events that feed into the platform's monitoring infrastructure, enabling real-time dashboards, alerting, and capacity planning:

defmodule PrismaticRateLimiter.TelemetryEmitter do
  @moduledoc """
  Emits telemetry events for all token bucket operations.
  Events are consumed by the platform's monitoring infrastructure
  for dashboards, alerts, and adaptive policy feedback loops.
  """

  @spec emit_acquired(bucket_id :: atom(), cost :: pos_integer(), remaining :: non_neg_integer()) ::
          :ok
  def emit_acquired(bucket_id, cost, remaining) do
    :telemetry.execute(
      [:prismatic, :rate_limiter, :token, :acquired],
      %{cost: cost, remaining: remaining, fill_ratio: remaining / lookup_capacity(bucket_id)},
      %{bucket_id: bucket_id, timestamp: DateTime.utc_now()}
    )
  end

  @spec emit_rejected(bucket_id :: atom(), cost :: pos_integer()) :: :ok
  def emit_rejected(bucket_id, cost) do
    :telemetry.execute(
      [:prismatic, :rate_limiter, :token, :rejected],
      %{cost: cost},
      %{bucket_id: bucket_id, timestamp: DateTime.utc_now()}
    )
  end
end

Key metrics tracked across all token buckets include:

Metric	Description	Alert Threshold
Rejection rate	Percentage of requests rejected per bucket	> 5% sustained
Fill ratio	Current tokens / capacity	< 10% sustained
Burst frequency	Number of burst events per minute	> 20/min
Adaptive adjustments	Number of policy adjustments per hour	> 10/hour
Recovery time	Time from empty bucket to 50% fill	> 30 seconds

OSINT Adapter Rate Limiting

The platform's 120 OSINT tools present a unique rate-limiting challenge: each external data source has its own rate limits, quotas, and throttling behavior. The token bucket implementation includes adapter-specific configurations that align platform-side rate limiting with external service constraints:

Czech Registry Adapters (ARES, Justice, Commercial Register): Relatively generous limits of 10-30 requests per second, with daily quotas of 10,000-50,000 requests. Token buckets are configured with matching rates and time-based adaptive policies for quota management.

Global Intelligence Adapters (Shodan, VirusTotal, Censys): Strict per-second limits (1-5 requests/second) with monthly quotas tied to API key tiers. Token buckets enforce the per-second limit while a separate quota tracker manages the monthly allocation.

Sanctions Databases (EU, OFAC, UN): Low-frequency, high-value queries with no strict rate limits but significant latency. Token buckets prevent concurrent queries that might timeout, limiting to 2-3 concurrent requests with generous burst capacity for batch screening operations.

Distributed Token Bucket Considerations

In a multi-node deployment on Fly.io, token bucket state must be coordinated across nodes to maintain accurate global rate limits. The Prismatic Platform supports two distribution strategies:

Node-Local Buckets (default): Each node maintains its own token bucket with rate limits divided by the number of active nodes. If the global limit is 1,000/s and 4 nodes are active, each node's bucket rate is 250/s. This approach has zero coordination overhead but requires accurate node counting and can be slightly imprecise during node scaling events.

Distributed Buckets (for critical limits): A CRDTs-based distributed counter using Horde provides globally consistent rate limiting. Token acquisitions are coordinated across nodes through eventually-consistent counters. This adds 1-5ms of coordination overhead but ensures that the global rate limit is respected exactly, even during scaling events.

The choice between local and distributed buckets is configured per bucket based on the precision requirements of the use case.

Testing Token Buckets

The platform's Quality Gate requirements demand comprehensive testing of rate-limiting behavior, including timing-sensitive tests that verify burst tolerance, refill accuracy, and adaptive policy responses:

defmodule PrismaticRateLimiter.TokenBucketTest do
  use ExUnit.Case, async: true

  alias PrismaticRateLimiter.TokenBucket

  describe "acquire/2" do
    test "allows requests within rate limit" do
      config = %BucketConfig{id: :test_basic, rate: 10, capacity: 10, refill_interval_ms: 100}
      {:ok, _pid} = TokenBucket.start_link(config)

      for _ <- 1..10 do
        assert {:ok, _remaining} = TokenBucket.acquire(:test_basic)
      end
    end

    test "rejects requests when bucket is empty" do
      config = %BucketConfig{id: :test_reject, rate: 1, capacity: 1, refill_interval_ms: 1_000}
      {:ok, _pid} = TokenBucket.start_link(config)

      assert {:ok, 0} = TokenBucket.acquire(:test_reject)
      assert {:error, :rate_limited} = TokenBucket.acquire(:test_reject)
    end

    test "refills tokens over time" do
      config = %BucketConfig{id: :test_refill, rate: 100, capacity: 10, refill_interval_ms: 50}
      {:ok, _pid} = TokenBucket.start_link(config)

      # Drain the bucket
      for _ <- 1..10, do: TokenBucket.acquire(:test_refill)
      assert {:error, :rate_limited} = TokenBucket.acquire(:test_refill)

      # Wait for refill
      Process.sleep(150)

      assert {:ok, _remaining} = TokenBucket.acquire(:test_refill)
    end
  end
end

Rate Limiting -- The broader concept of controlling request throughput
Backpressure -- Flow control mechanism that signals upstream to slow down
Circuit Breaker -- Pattern for failing fast when a service is unhealthy
GenServer -- OTP behavior used to implement the token bucket process
ETS -- In-memory storage providing microsecond token state access
Fault Tolerance -- System property ensuring operation despite component failures
Supervision Tree -- Hierarchy governing token bucket process lifecycle
API Gateway -- Entry point where platform-level rate limiting is enforced
Quality Gate -- Verification checkpoint that includes rate-limit testing
Process Isolation -- BEAM property enabling independent bucket processes

Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL

GitHub | GitLab | LinkedIn | Contact
Developer Portal | Architecture | Meet the Creator

Token Bucket