Orchestration

agents

Orchestration

Definition

Orchestration is the coordinated management and execution of multiple processes, agents, or services to accomplish a complex goal that no single component could achieve independently. An orchestrator decomposes high-level objectives into subtasks, assigns them to appropriate agents or services, manages dependencies between tasks, handles failures and retries, and synthesizes results into a unified outcome. In the Prismatic Platform, orchestration operates at multiple levels: the /orchestrate command provides 10x efficiency gains for complex operations, the AIAD hierarchy coordinates 530+ agents across 5 tiers, and multi-agent pipelines execute structured sequences of agent actions with fault tolerance and backpressure management.

Overview

Orchestration is the difference between having many capable agents and having an effective system. A platform with 530+ agents that cannot coordinate them is like an orchestra with 530+ musicians and no conductor -- each musician may be excellent individually, but the collective output is noise rather than music. The orchestrator provides the score, sets the tempo, cues entries, and ensures that the final performance is greater than the sum of its parts.

The Prismatic Platform's approach to orchestration is deeply rooted in Erlang/OTP principles. Rather than adopting external orchestration frameworks (Kubernetes operators, Apache Airflow, Temporal), the platform builds orchestration natively on OTP's supervision trees, GenServer state machines, and Task concurrency primitives. This means orchestration benefits from the same fault tolerance, hot code reloading, and process isolation that makes the BEAM VM uniquely suited for concurrent systems.

The orchestration architecture follows a hierarchical pattern mirroring military command structures. The Archer Supreme operates at L5 (Supreme Authority), strategic commanders at L3, tactical specialists at L2, and operational units at L1. Each level orchestrates the level below it, creating a fractal coordination structure where complex operations cascade through the hierarchy until they reach agents capable of executing concrete actions.

This hierarchical approach avoids the two failure modes of flat orchestration: bottleneck orchestrators (single point of failure processing all coordination) and peer-to-peer chaos (agents negotiating directly with unpredictable emergent behavior). Instead, the hierarchy provides structured delegation, clear authority boundaries, and predictable escalation paths.

Technical Details

Orchestration Patterns

Orchestration in distributed systems follows several well-established patterns, each with different trade-offs:

Centralized Orchestration uses a single coordinator that dispatches tasks and aggregates results. Simple to reason about but creates a single point of failure and potential bottleneck. The Prismatic Platform uses this pattern for simple workflows where reliability requirements are moderate.

Hierarchical Orchestration distributes coordination across a command hierarchy. Each level orchestrates its children, providing fault isolation and scalability. This is the platform's primary orchestration pattern, implemented through the AIAD tier system.

Choreography is the antithesis of orchestration -- each agent independently reacts to events without central coordination. While useful for loosely coupled systems, choreography produces emergent behavior that is difficult to reason about, test, and debug. The platform avoids pure choreography.

Saga Pattern orchestrates long-running transactions across multiple services with compensating actions for rollback. Used in the platform for operations that span multiple storage backends or external APIs.

The AIAD Tier System

The AIAD hierarchy defines five tiers of agent authority, each with specific orchestration responsibilities:

TierLevelRoleOrchestration Scope
L5SupremeArcher SupremePlatform-wide strategic orchestration
L4Safety-CriticalGuards and EnforcersSafety-scoped orchestration with override authority
L3StrategicCommanders (per domain)Domain-level coordination of L2 specialists
L2TacticalSpecialistsTask execution with limited sub-orchestration
L1OperationalUnitsAtomic task execution, no orchestration authority

Task Decomposition

The orchestrator's primary function is breaking complex objectives into executable subtasks. This follows a recursive decomposition pattern:

Objective: "Assess security posture of target.com"
├── L3: Security Commander
│   ├── L2: DNS Enumerator → discover subdomains
│   ├── L2: Port Scanner → identify open services
│   ├── L2: Certificate Analyzer → check TLS configuration
│   ├── L2: Vulnerability Scanner → assess known CVEs
│   └── L2: Compliance Assessor → evaluate NIS2/ZKB compliance
│       ├── L1: NIS2 Checker → specific compliance rules
│       └── L1: ZKB Checker → specific compliance rules
└── L3: Synthesis Commander
    ├── L2: Evidence Aggregator → combine findings
    ├── L2: Risk Scorer → compute security rating
    └── L2: Report Generator → produce assessment report

Implementation in Prismatic Platform

Orchestrator GenServer

The core orchestrator is implemented as a GenServer that manages task lifecycle, agent dispatch, and result aggregation:

defmodule PrismaticAgents.Orchestrator do
  @moduledoc """
  Central orchestration engine for multi-agent task coordination.
  Decomposes objectives into subtasks, dispatches to appropriate agents,
  manages dependencies, and aggregates results.
  """

  use GenServer

  @type objective :: %{
    id: String.t(),
    description: String.t(),
    requester: String.t(),
    priority: :low | :normal | :high | :critical,
    constraints: map(),
    deadline: DateTime.t() | nil
  }

  @type task :: %{
    id: String.t(),
    objective_id: String.t(),
    agent_id: String.t(),
    status: :pending | :dispatched | :running | :completed | :failed | :cancelled,
    input: term(),
    output: term() | nil,
    dependencies: [String.t()],
    started_at: DateTime.t() | nil,
    completed_at: DateTime.t() | nil,
    timeout_ms: pos_integer()
  }

  @type state :: %{
    objectives: %{String.t() => objective()},
    tasks: %{String.t() => task()},
    agent_registry: module(),
    max_concurrent: pos_integer(),
    active_count: non_neg_integer()
  }

  @spec start_link(keyword()) :: GenServer.on_start()
  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @spec orchestrate(objective()) :: {:ok, String.t()} | {:error, term()}
  def orchestrate(objective) do
    GenServer.call(__MODULE__, {:orchestrate, objective}, :infinity)
  end

  @spec status(String.t()) :: {:ok, map()} | {:error, :not_found}
  def status(objective_id) do
    GenServer.call(__MODULE__, {:status, objective_id})
  end

  @spec cancel(String.t()) :: :ok | {:error, :not_found}
  def cancel(objective_id) do
    GenServer.call(__MODULE__, {:cancel, objective_id})
  end

  @impl GenServer
  @spec init(keyword()) :: {:ok, state()}
  def init(opts) do
    {:ok,
     %{
       objectives: %{},
       tasks: %{},
       agent_registry: Keyword.get(opts, :registry, PrismaticAgents.Registry),
       max_concurrent: Keyword.get(opts, :max_concurrent, 50),
       active_count: 0
     }}
  end

  @impl GenServer
  def handle_call({:orchestrate, objective}, _from, state) do
    objective = Map.put(objective, :id, generate_id())
    tasks = decompose_objective(objective, state.agent_registry)

    :telemetry.execute(
      [:prismatic, :orchestrator, :objective_started],
      %{task_count: length(tasks)},
      %{objective_id: objective.id, priority: objective.priority}
    )

    new_state =
      state
      |> Map.update!(:objectives, &Map.put(&1, objective.id, objective))
      |> register_tasks(tasks)
      |> dispatch_ready_tasks()

    {:reply, {:ok, objective.id}, new_state}
  end

  def handle_call({:status, objective_id}, _from, state) do
    case Map.get(state.objectives, objective_id) do
      nil ->
        {:reply, {:error, :not_found}, state}

      objective ->
        tasks =
          state.tasks
          |> Map.values()
          |> Enum.filter(&(&1.objective_id == objective_id))

        completed = Enum.count(tasks, &(&1.status == :completed))
        failed = Enum.count(tasks, &(&1.status == :failed))
        total = length(tasks)

        status = %{
          objective: objective,
          progress: %{completed: completed, failed: failed, total: total},
          tasks: tasks
        }

        {:reply, {:ok, status}, state}
    end
  end

  def handle_call({:cancel, objective_id}, _from, state) do
    case Map.get(state.objectives, objective_id) do
      nil ->
        {:reply, {:error, :not_found}, state}

      _objective ->
        new_tasks =
          state.tasks
          |> Enum.map(fn {id, task} ->
            if task.objective_id == objective_id and
                 task.status in [:pending, :dispatched, :running] do
              {id, %{task | status: :cancelled}}
            else
              {id, task}
            end
          end)
          |> Map.new()

        {:reply, :ok, %{state | tasks: new_tasks}}
    end
  end

  @impl GenServer
  def handle_info({:task_completed, task_id, result}, state) do
    case Map.get(state.tasks, task_id) do
      nil ->
        {:noreply, state}

      task ->
        updated_task = %{
          task
          | status: :completed,
            output: result,
            completed_at: DateTime.utc_now()
        }

        :telemetry.execute(
          [:prismatic, :orchestrator, :task_completed],
          %{
            duration_ms:
              DateTime.diff(updated_task.completed_at, updated_task.started_at, :millisecond)
          },
          %{task_id: task_id, agent_id: task.agent_id}
        )

        new_state =
          state
          |> Map.update!(:tasks, &Map.put(&1, task_id, updated_task))
          |> Map.update!(:active_count, &max(&1 - 1, 0))
          |> maybe_complete_objective(task.objective_id)
          |> dispatch_ready_tasks()

        {:noreply, new_state}
    end
  end

  def handle_info({:task_failed, task_id, reason}, state) do
    case Map.get(state.tasks, task_id) do
      nil ->
        {:noreply, state}

      task ->
        updated_task = %{
          task
          | status: :failed,
            output: {:error, reason},
            completed_at: DateTime.utc_now()
        }

        :telemetry.execute(
          [:prismatic, :orchestrator, :task_failed],
          %{},
          %{task_id: task_id, agent_id: task.agent_id, reason: reason}
        )

        new_state =
          state
          |> Map.update!(:tasks, &Map.put(&1, task_id, updated_task))
          |> Map.update!(:active_count, &max(&1 - 1, 0))
          |> dispatch_ready_tasks()

        {:noreply, new_state}
    end
  end

  @spec decompose_objective(objective(), module()) :: [task()]
  defp decompose_objective(objective, registry) do
    # Query the agent registry for agents capable of handling
    # subtasks derived from the objective
    capabilities = extract_required_capabilities(objective)

    Enum.flat_map(capabilities, fn capability ->
      case registry.find_agents(capability) do
        [] ->
          []

        agents ->
          agent = select_best_agent(agents, objective.priority)

          [
            %{
              id: generate_id(),
              objective_id: objective.id,
              agent_id: agent.id,
              status: :pending,
              input: %{capability: capability, context: objective.constraints},
              output: nil,
              dependencies: [],
              started_at: nil,
              completed_at: nil,
              timeout_ms: 30_000
            }
          ]
      end
    end)
  end

  @spec dispatch_ready_tasks(state()) :: state()
  defp dispatch_ready_tasks(state) do
    available_slots = state.max_concurrent - state.active_count

    ready_tasks =
      state.tasks
      |> Map.values()
      |> Enum.filter(&(&1.status == :pending))
      |> Enum.filter(&dependencies_met?(&1, state.tasks))
      |> Enum.take(available_slots)

    Enum.reduce(ready_tasks, state, fn task, acc ->
      dispatch_task(task, acc)
    end)
  end

  @spec dispatch_task(task(), state()) :: state()
  defp dispatch_task(task, state) do
    orchestrator_pid = self()

    Task.start(fn ->
      result = execute_agent_task(task)

      case result do
        {:ok, output} ->
          send(orchestrator_pid, {:task_completed, task.id, output})

        {:error, reason} ->
          send(orchestrator_pid, {:task_failed, task.id, reason})
      end
    end)

    updated_task = %{task | status: :dispatched, started_at: DateTime.utc_now()}

    state
    |> Map.update!(:tasks, &Map.put(&1, task.id, updated_task))
    |> Map.update!(:active_count, &(&1 + 1))
  end

  @spec dependencies_met?(task(), %{String.t() => task()}) :: boolean()
  defp dependencies_met?(%{dependencies: []}, _tasks), do: true

  defp dependencies_met?(%{dependencies: deps}, tasks) do
    Enum.all?(deps, fn dep_id ->
      case Map.get(tasks, dep_id) do
        %{status: :completed} -> true
        _ -> false
      end
    end)
  end

  @spec maybe_complete_objective(state(), String.t()) :: state()
  defp maybe_complete_objective(state, objective_id) do
    objective_tasks =
      state.tasks
      |> Map.values()
      |> Enum.filter(&(&1.objective_id == objective_id))

    all_done = Enum.all?(objective_tasks, &(&1.status in [:completed, :failed, :cancelled]))

    if all_done do
      results = Enum.map(objective_tasks, fn t -> {t.agent_id, t.output} end)

      :telemetry.execute(
        [:prismatic, :orchestrator, :objective_completed],
        %{task_count: length(objective_tasks)},
        %{objective_id: objective_id, results: results}
      )
    end

    state
  end

  @spec execute_agent_task(task()) :: {:ok, term()} | {:error, term()}
  defp execute_agent_task(task) do
    # Dispatch to the assigned agent and await result
    case PrismaticAgents.Registry.get_agent(task.agent_id) do
      {:ok, agent_pid} ->
        GenServer.call(agent_pid, {:execute, task.input}, task.timeout_ms)

      {:error, reason} ->
        {:error, {:agent_unavailable, reason}}
    end
  end

  @spec extract_required_capabilities(objective()) :: [atom()]
  defp extract_required_capabilities(_objective), do: []

  @spec select_best_agent([map()], atom()) :: map()
  defp select_best_agent([agent | _], _priority), do: agent

  @spec generate_id() :: String.t()
  defp generate_id, do: :crypto.strong_rand_bytes(16) |> Base.encode16(case: :lower)

  @spec register_tasks(state(), [task()]) :: state()
  defp register_tasks(state, tasks) do
    new_tasks =
      Enum.reduce(tasks, state.tasks, fn task, acc ->
        Map.put(acc, task.id, task)
      end)

    %{state | tasks: new_tasks}
  end
end

Pipeline Orchestration

For structured, repeatable workflows, the platform provides a pipeline orchestration mechanism that composes agent actions into declarative sequences:

defmodule PrismaticAgents.Pipeline do
  @moduledoc """
  Declarative pipeline orchestration for structured multi-agent workflows.
  Pipelines define ordered sequences of agent actions with dependency
  management, error handling, and backpressure control.
  """

  @type step :: %{
    name: atom(),
    agent: String.t(),
    input_fn: (map() -> map()),
    timeout_ms: pos_integer(),
    retry_count: non_neg_integer(),
    on_failure: :halt | :skip | :retry
  }

  @type pipeline_def :: %{
    name: String.t(),
    steps: [step()],
    context: map(),
    max_retries: non_neg_integer()
  }

  @type pipeline_result :: %{
    status: :completed | :failed | :partial,
    steps_completed: non_neg_integer(),
    steps_failed: non_neg_integer(),
    results: %{atom() => term()},
    duration_ms: non_neg_integer()
  }

  @spec execute(pipeline_def()) :: {:ok, pipeline_result()} | {:error, term()}
  def execute(%{steps: steps, context: initial_context} = pipeline) do
    start_time = System.monotonic_time(:millisecond)

    :telemetry.execute(
      [:prismatic, :pipeline, :started],
      %{step_count: length(steps)},
      %{pipeline: pipeline.name}
    )

    result =
      Enum.reduce_while(steps, {:ok, initial_context, %{}, 0, 0}, fn step, {:ok, ctx, results, completed, failed} ->
        input = step.input_fn.(ctx)

        case execute_step(step, input) do
          {:ok, output} ->
            new_ctx = Map.merge(ctx, %{step.name => output})
            new_results = Map.put(results, step.name, output)
            {:cont, {:ok, new_ctx, new_results, completed + 1, failed}}

          {:error, reason} when step.on_failure == :skip ->
            new_results = Map.put(results, step.name, {:skipped, reason})
            {:cont, {:ok, ctx, new_results, completed, failed + 1}}

          {:error, reason} when step.on_failure == :halt ->
            new_results = Map.put(results, step.name, {:failed, reason})
            {:halt, {:error, ctx, new_results, completed, failed + 1}}

          {:error, _reason} ->
            # Default: retry
            case retry_step(step, input, step.retry_count) do
              {:ok, output} ->
                new_ctx = Map.merge(ctx, %{step.name => output})
                new_results = Map.put(results, step.name, output)
                {:cont, {:ok, new_ctx, new_results, completed + 1, failed}}

              {:error, final_reason} ->
                new_results = Map.put(results, step.name, {:failed, final_reason})
                {:halt, {:error, ctx, new_results, completed, failed + 1}}
            end
        end
      end)

    duration = System.monotonic_time(:millisecond) - start_time

    case result do
      {:ok, _ctx, results, completed, failed} ->
        pipeline_result = %{
          status: if(failed == 0, do: :completed, else: :partial),
          steps_completed: completed,
          steps_failed: failed,
          results: results,
          duration_ms: duration
        }

        {:ok, pipeline_result}

      {:error, _ctx, results, completed, failed} ->
        pipeline_result = %{
          status: :failed,
          steps_completed: completed,
          steps_failed: failed,
          results: results,
          duration_ms: duration
        }

        {:ok, pipeline_result}
    end
  end

  @spec execute_step(step(), map()) :: {:ok, term()} | {:error, term()}
  defp execute_step(step, input) do
    task =
      Task.async(fn ->
        PrismaticAgents.Registry.dispatch(step.agent, input)
      end)

    case Task.yield(task, step.timeout_ms) || Task.shutdown(task) do
      {:ok, result} -> result
      nil -> {:error, :timeout}
    end
  end

  @spec retry_step(step(), map(), non_neg_integer()) ::
          {:ok, term()} | {:error, term()}
  defp retry_step(_step, _input, 0), do: {:error, :retries_exhausted}

  defp retry_step(step, input, remaining) do
    # Exponential backoff
    delay = :math.pow(2, step.retry_count - remaining) |> round() |> min(30_000)
    Process.sleep(delay)

    case execute_step(step, input) do
      {:ok, output} -> {:ok, output}
      {:error, _} -> retry_step(step, input, remaining - 1)
    end
  end
end

Comparison with Alternatives

Orchestration vs. Choreography

AspectOrchestration (Prismatic)Choreography
ControlCentralized coordinatorDecentralized, event-driven
VisibilityFull workflow state in one placeState distributed across agents
DebuggingStraightforward (follow the orchestrator)Difficult (trace through events)
CouplingAgents coupled to orchestratorAgents coupled to event schema
ScalabilityOrchestrator can bottleneckScales naturally
Error handlingCentralized retry/compensationEach agent handles independently
PredictabilityHigh (deterministic flow)Low (emergent behavior)

Orchestration vs. External Workflow Engines

AspectNative OTP (Prismatic)Temporal/Airflow/Argo
InfrastructureSame BEAM nodeSeparate service cluster
LatencyMicroseconds (function calls)Milliseconds (network + serialization)
Fault toleranceOTP supervision treesExternal HA mechanisms
Hot code reloadYes (OTP release upgrades)No (requires redeployment)
LanguageElixir (same as application)DSL or separate language
Operational overheadNone (part of the application)Significant (separate system to manage)

Orchestration vs. Simple Task Parallelism

Simple Task.async_stream or parallel map operations execute independent tasks concurrently but lack dependency management, error recovery, and result aggregation. The Prismatic orchestrator adds these capabilities while still leveraging OTP's Task primitives for the actual concurrent execution.

Best Practices

1. Design Orchestration Hierarchically

Avoid flat orchestration where a single coordinator manages hundreds of tasks directly. Instead, delegate to sub-orchestrators that manage their own domains. The AIAD tier system naturally produces this hierarchy, with L3 commanders orchestrating L2 specialists.

2. Make Tasks Idempotent

Orchestrators retry failed tasks. If tasks have side effects that are not idempotent (e.g., sending an email twice), retries cause duplicate actions. Design all orchestrated tasks to be safely re-executable.

3. Set Meaningful Timeouts

Every dispatched task must have a timeout. Without timeouts, a single stuck agent can block the entire orchestration pipeline. The Prismatic orchestrator enforces timeouts on every task and escalates timeout failures to the appropriate handler.

4. Emit Telemetry at Every State Transition

Orchestration is opaque without observability. Emit Telemetry events when objectives start, tasks dispatch, results arrive, and pipelines complete. This enables monitoring dashboards to show orchestration progress in real time.

5. Separate Orchestration Logic from Business Logic

The orchestrator should know how to coordinate tasks, not how to execute them. Business logic belongs in the agents. The orchestrator's responsibility is decomposition, dispatch, dependency management, and result aggregation -- never domain computation.

6. Plan for Partial Success

Not every task in a complex orchestration will succeed. Design orchestration flows to handle partial success: some tasks may be skippable, others may have fallbacks, and some failures should halt the entire pipeline. The pipeline's on_failure: :halt | :skip | :retry configuration enables this explicitly.

Common Pitfalls

1. Orchestrator as God Object

An orchestrator that accumulates too much responsibility -- task execution, result transformation, error handling, logging, notification -- becomes unmaintainable. Keep the orchestrator focused on coordination; delegate everything else.

2. Ignoring Backpressure

Dispatching 500 tasks simultaneously to a system that can handle 50 concurrently causes resource exhaustion. The Prismatic orchestrator enforces max_concurrent limits and queues tasks beyond this threshold, providing natural backpressure.

3. Synchronous Orchestration

Blocking the caller until the entire orchestration completes ties up resources and prevents progress updates. The Prismatic orchestrator returns immediately with an objective ID, enabling the caller to poll for status or receive async notifications.

4. Missing Compensation Logic

When a multi-step orchestration fails partway through, the completed steps may need to be rolled back. Without compensation actions (the saga pattern), partial failures leave the system in an inconsistent state. For critical operations, define compensation logic for every step.

5. Circular Task Dependencies

If task A depends on task B and task B depends on task A, the orchestrator deadlocks. Dependency validation must detect and reject circular dependencies before dispatching any tasks -- the same principle enforced by the platform's DependencyResolver for umbrella applications.

6. Insufficient Observability

Orchestration without monitoring is flying blind. When a 50-step pipeline stalls, operators need to know which step, which agent, and what error. Every state transition in the orchestration must emit observable events.

Use Cases

1. The /orchestrate Command

The platform's primary orchestration interface is the /orchestrate command, which coordinates multiple agents to execute complex operations with claimed 10x efficiency. A single /orchestrate invocation can trigger security assessments, quality checks, code generation, and reporting as a unified workflow.

2. OSINT Intelligence Gathering

When querying a target across 120 OSINT sources, orchestration manages the parallel dispatch of queries, rate limiting per source, result aggregation, contradiction detection (via NABLA axioms), and final intelligence report generation.

3. Security Assessment Pipeline

The Prismatic Perimeter security assessment orchestrates DNS enumeration, port scanning, certificate analysis, vulnerability checking, and compliance evaluation as a structured pipeline where each step's output feeds the next.

4. Quality Gate Execution

The mix quality.gates command orchestrates 13 quality domain checks in parallel, aggregates results, and produces a unified pass/fail decision. Each check is an independent task that can be individually retried on transient failure.

5. Color Team Security Exercises

Color team operations orchestrate coordinated activities across Red (attack simulation), Blue (defense), Purple (synthesis), Gray (boundary exploration), White (verification), and Black (threat modeling) teams, with the Purple coordinator synthesizing findings from all teams.

  • Agent Orchestration -- specific application of orchestration to multi-agent systems
  • Multi-Agent System -- the agent ecosystem that orchestration coordinates
  • Workflow -- structured sequences of actions that orchestration executes
  • AIAD -- the agent specification standard defining orchestration hierarchy
  • Archer Supreme -- the L5 supreme authority orchestrating platform-wide operations
  • Pipeline -- linear sequence of data transformations managed by orchestration
  • Agent Tier -- the hierarchical authority levels governing orchestration scope
  • Supervision Tree -- OTP's fault tolerance mechanism underlying orchestration
  • GenServer -- the process abstraction implementing orchestrator state management
  • Backpressure -- flow control mechanism preventing orchestration overload
  • Command -- the invocation interface through which orchestration is triggered

See Also


Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL