Definition
Orchestration is the coordinated management and execution of multiple processes, agents, or services to accomplish a complex goal that no single component could achieve independently. An orchestrator decomposes high-level objectives into subtasks, assigns them to appropriate agents or services, manages dependencies between tasks, handles failures and retries, and synthesizes results into a unified outcome. In the Prismatic Platform, orchestration operates at multiple levels: the /orchestrate command provides 10x efficiency gains for complex operations, the AIAD hierarchy coordinates 530+ agents across 5 tiers, and multi-agent pipelines execute structured sequences of agent actions with fault tolerance and backpressure management.
Overview
Orchestration is the difference between having many capable agents and having an effective system. A platform with 530+ agents that cannot coordinate them is like an orchestra with 530+ musicians and no conductor -- each musician may be excellent individually, but the collective output is noise rather than music. The orchestrator provides the score, sets the tempo, cues entries, and ensures that the final performance is greater than the sum of its parts.
The Prismatic Platform's approach to orchestration is deeply rooted in Erlang/OTP principles. Rather than adopting external orchestration frameworks (Kubernetes operators, Apache Airflow, Temporal), the platform builds orchestration natively on OTP's supervision trees, GenServer state machines, and Task concurrency primitives. This means orchestration benefits from the same fault tolerance, hot code reloading, and process isolation that makes the BEAM VM uniquely suited for concurrent systems.
The orchestration architecture follows a hierarchical pattern mirroring military command structures. The Archer Supreme operates at L5 (Supreme Authority), strategic commanders at L3, tactical specialists at L2, and operational units at L1. Each level orchestrates the level below it, creating a fractal coordination structure where complex operations cascade through the hierarchy until they reach agents capable of executing concrete actions.
This hierarchical approach avoids the two failure modes of flat orchestration: bottleneck orchestrators (single point of failure processing all coordination) and peer-to-peer chaos (agents negotiating directly with unpredictable emergent behavior). Instead, the hierarchy provides structured delegation, clear authority boundaries, and predictable escalation paths.
Technical Details
Orchestration Patterns
Orchestration in distributed systems follows several well-established patterns, each with different trade-offs:
Centralized Orchestration uses a single coordinator that dispatches tasks and aggregates results. Simple to reason about but creates a single point of failure and potential bottleneck. The Prismatic Platform uses this pattern for simple workflows where reliability requirements are moderate.
Hierarchical Orchestration distributes coordination across a command hierarchy. Each level orchestrates its children, providing fault isolation and scalability. This is the platform's primary orchestration pattern, implemented through the AIAD tier system.
Choreography is the antithesis of orchestration -- each agent independently reacts to events without central coordination. While useful for loosely coupled systems, choreography produces emergent behavior that is difficult to reason about, test, and debug. The platform avoids pure choreography.
Saga Pattern orchestrates long-running transactions across multiple services with compensating actions for rollback. Used in the platform for operations that span multiple storage backends or external APIs.
The AIAD Tier System
The AIAD hierarchy defines five tiers of agent authority, each with specific orchestration responsibilities:
| Tier | Level | Role | Orchestration Scope |
|---|---|---|---|
| L5 | Supreme | Archer Supreme | Platform-wide strategic orchestration |
| L4 | Safety-Critical | Guards and Enforcers | Safety-scoped orchestration with override authority |
| L3 | Strategic | Commanders (per domain) | Domain-level coordination of L2 specialists |
| L2 | Tactical | Specialists | Task execution with limited sub-orchestration |
| L1 | Operational | Units | Atomic task execution, no orchestration authority |
Task Decomposition
The orchestrator's primary function is breaking complex objectives into executable subtasks. This follows a recursive decomposition pattern:
Objective: "Assess security posture of target.com"
├── L3: Security Commander
│ ├── L2: DNS Enumerator → discover subdomains
│ ├── L2: Port Scanner → identify open services
│ ├── L2: Certificate Analyzer → check TLS configuration
│ ├── L2: Vulnerability Scanner → assess known CVEs
│ └── L2: Compliance Assessor → evaluate NIS2/ZKB compliance
│ ├── L1: NIS2 Checker → specific compliance rules
│ └── L1: ZKB Checker → specific compliance rules
└── L3: Synthesis Commander
├── L2: Evidence Aggregator → combine findings
├── L2: Risk Scorer → compute security rating
└── L2: Report Generator → produce assessment reportImplementation in Prismatic Platform
Orchestrator GenServer
The core orchestrator is implemented as a GenServer that manages task lifecycle, agent dispatch, and result aggregation:
defmodule PrismaticAgents.Orchestrator do
@moduledoc """
Central orchestration engine for multi-agent task coordination.
Decomposes objectives into subtasks, dispatches to appropriate agents,
manages dependencies, and aggregates results.
"""
use GenServer
@type objective :: %{
id: String.t(),
description: String.t(),
requester: String.t(),
priority: :low | :normal | :high | :critical,
constraints: map(),
deadline: DateTime.t() | nil
}
@type task :: %{
id: String.t(),
objective_id: String.t(),
agent_id: String.t(),
status: :pending | :dispatched | :running | :completed | :failed | :cancelled,
input: term(),
output: term() | nil,
dependencies: [String.t()],
started_at: DateTime.t() | nil,
completed_at: DateTime.t() | nil,
timeout_ms: pos_integer()
}
@type state :: %{
objectives: %{String.t() => objective()},
tasks: %{String.t() => task()},
agent_registry: module(),
max_concurrent: pos_integer(),
active_count: non_neg_integer()
}
@spec start_link(keyword()) :: GenServer.on_start()
def start_link(opts \\ []) do
GenServer.start_link(__MODULE__, opts, name: __MODULE__)
end
@spec orchestrate(objective()) :: {:ok, String.t()} | {:error, term()}
def orchestrate(objective) do
GenServer.call(__MODULE__, {:orchestrate, objective}, :infinity)
end
@spec status(String.t()) :: {:ok, map()} | {:error, :not_found}
def status(objective_id) do
GenServer.call(__MODULE__, {:status, objective_id})
end
@spec cancel(String.t()) :: :ok | {:error, :not_found}
def cancel(objective_id) do
GenServer.call(__MODULE__, {:cancel, objective_id})
end
@impl GenServer
@spec init(keyword()) :: {:ok, state()}
def init(opts) do
{:ok,
%{
objectives: %{},
tasks: %{},
agent_registry: Keyword.get(opts, :registry, PrismaticAgents.Registry),
max_concurrent: Keyword.get(opts, :max_concurrent, 50),
active_count: 0
}}
end
@impl GenServer
def handle_call({:orchestrate, objective}, _from, state) do
objective = Map.put(objective, :id, generate_id())
tasks = decompose_objective(objective, state.agent_registry)
:telemetry.execute(
[:prismatic, :orchestrator, :objective_started],
%{task_count: length(tasks)},
%{objective_id: objective.id, priority: objective.priority}
)
new_state =
state
|> Map.update!(:objectives, &Map.put(&1, objective.id, objective))
|> register_tasks(tasks)
|> dispatch_ready_tasks()
{:reply, {:ok, objective.id}, new_state}
end
def handle_call({:status, objective_id}, _from, state) do
case Map.get(state.objectives, objective_id) do
nil ->
{:reply, {:error, :not_found}, state}
objective ->
tasks =
state.tasks
|> Map.values()
|> Enum.filter(&(&1.objective_id == objective_id))
completed = Enum.count(tasks, &(&1.status == :completed))
failed = Enum.count(tasks, &(&1.status == :failed))
total = length(tasks)
status = %{
objective: objective,
progress: %{completed: completed, failed: failed, total: total},
tasks: tasks
}
{:reply, {:ok, status}, state}
end
end
def handle_call({:cancel, objective_id}, _from, state) do
case Map.get(state.objectives, objective_id) do
nil ->
{:reply, {:error, :not_found}, state}
_objective ->
new_tasks =
state.tasks
|> Enum.map(fn {id, task} ->
if task.objective_id == objective_id and
task.status in [:pending, :dispatched, :running] do
{id, %{task | status: :cancelled}}
else
{id, task}
end
end)
|> Map.new()
{:reply, :ok, %{state | tasks: new_tasks}}
end
end
@impl GenServer
def handle_info({:task_completed, task_id, result}, state) do
case Map.get(state.tasks, task_id) do
nil ->
{:noreply, state}
task ->
updated_task = %{
task
| status: :completed,
output: result,
completed_at: DateTime.utc_now()
}
:telemetry.execute(
[:prismatic, :orchestrator, :task_completed],
%{
duration_ms:
DateTime.diff(updated_task.completed_at, updated_task.started_at, :millisecond)
},
%{task_id: task_id, agent_id: task.agent_id}
)
new_state =
state
|> Map.update!(:tasks, &Map.put(&1, task_id, updated_task))
|> Map.update!(:active_count, &max(&1 - 1, 0))
|> maybe_complete_objective(task.objective_id)
|> dispatch_ready_tasks()
{:noreply, new_state}
end
end
def handle_info({:task_failed, task_id, reason}, state) do
case Map.get(state.tasks, task_id) do
nil ->
{:noreply, state}
task ->
updated_task = %{
task
| status: :failed,
output: {:error, reason},
completed_at: DateTime.utc_now()
}
:telemetry.execute(
[:prismatic, :orchestrator, :task_failed],
%{},
%{task_id: task_id, agent_id: task.agent_id, reason: reason}
)
new_state =
state
|> Map.update!(:tasks, &Map.put(&1, task_id, updated_task))
|> Map.update!(:active_count, &max(&1 - 1, 0))
|> dispatch_ready_tasks()
{:noreply, new_state}
end
end
@spec decompose_objective(objective(), module()) :: [task()]
defp decompose_objective(objective, registry) do
# Query the agent registry for agents capable of handling
# subtasks derived from the objective
capabilities = extract_required_capabilities(objective)
Enum.flat_map(capabilities, fn capability ->
case registry.find_agents(capability) do
[] ->
[]
agents ->
agent = select_best_agent(agents, objective.priority)
[
%{
id: generate_id(),
objective_id: objective.id,
agent_id: agent.id,
status: :pending,
input: %{capability: capability, context: objective.constraints},
output: nil,
dependencies: [],
started_at: nil,
completed_at: nil,
timeout_ms: 30_000
}
]
end
end)
end
@spec dispatch_ready_tasks(state()) :: state()
defp dispatch_ready_tasks(state) do
available_slots = state.max_concurrent - state.active_count
ready_tasks =
state.tasks
|> Map.values()
|> Enum.filter(&(&1.status == :pending))
|> Enum.filter(&dependencies_met?(&1, state.tasks))
|> Enum.take(available_slots)
Enum.reduce(ready_tasks, state, fn task, acc ->
dispatch_task(task, acc)
end)
end
@spec dispatch_task(task(), state()) :: state()
defp dispatch_task(task, state) do
orchestrator_pid = self()
Task.start(fn ->
result = execute_agent_task(task)
case result do
{:ok, output} ->
send(orchestrator_pid, {:task_completed, task.id, output})
{:error, reason} ->
send(orchestrator_pid, {:task_failed, task.id, reason})
end
end)
updated_task = %{task | status: :dispatched, started_at: DateTime.utc_now()}
state
|> Map.update!(:tasks, &Map.put(&1, task.id, updated_task))
|> Map.update!(:active_count, &(&1 + 1))
end
@spec dependencies_met?(task(), %{String.t() => task()}) :: boolean()
defp dependencies_met?(%{dependencies: []}, _tasks), do: true
defp dependencies_met?(%{dependencies: deps}, tasks) do
Enum.all?(deps, fn dep_id ->
case Map.get(tasks, dep_id) do
%{status: :completed} -> true
_ -> false
end
end)
end
@spec maybe_complete_objective(state(), String.t()) :: state()
defp maybe_complete_objective(state, objective_id) do
objective_tasks =
state.tasks
|> Map.values()
|> Enum.filter(&(&1.objective_id == objective_id))
all_done = Enum.all?(objective_tasks, &(&1.status in [:completed, :failed, :cancelled]))
if all_done do
results = Enum.map(objective_tasks, fn t -> {t.agent_id, t.output} end)
:telemetry.execute(
[:prismatic, :orchestrator, :objective_completed],
%{task_count: length(objective_tasks)},
%{objective_id: objective_id, results: results}
)
end
state
end
@spec execute_agent_task(task()) :: {:ok, term()} | {:error, term()}
defp execute_agent_task(task) do
# Dispatch to the assigned agent and await result
case PrismaticAgents.Registry.get_agent(task.agent_id) do
{:ok, agent_pid} ->
GenServer.call(agent_pid, {:execute, task.input}, task.timeout_ms)
{:error, reason} ->
{:error, {:agent_unavailable, reason}}
end
end
@spec extract_required_capabilities(objective()) :: [atom()]
defp extract_required_capabilities(_objective), do: []
@spec select_best_agent([map()], atom()) :: map()
defp select_best_agent([agent | _], _priority), do: agent
@spec generate_id() :: String.t()
defp generate_id, do: :crypto.strong_rand_bytes(16) |> Base.encode16(case: :lower)
@spec register_tasks(state(), [task()]) :: state()
defp register_tasks(state, tasks) do
new_tasks =
Enum.reduce(tasks, state.tasks, fn task, acc ->
Map.put(acc, task.id, task)
end)
%{state | tasks: new_tasks}
end
endPipeline Orchestration
For structured, repeatable workflows, the platform provides a pipeline orchestration mechanism that composes agent actions into declarative sequences:
defmodule PrismaticAgents.Pipeline do
@moduledoc """
Declarative pipeline orchestration for structured multi-agent workflows.
Pipelines define ordered sequences of agent actions with dependency
management, error handling, and backpressure control.
"""
@type step :: %{
name: atom(),
agent: String.t(),
input_fn: (map() -> map()),
timeout_ms: pos_integer(),
retry_count: non_neg_integer(),
on_failure: :halt | :skip | :retry
}
@type pipeline_def :: %{
name: String.t(),
steps: [step()],
context: map(),
max_retries: non_neg_integer()
}
@type pipeline_result :: %{
status: :completed | :failed | :partial,
steps_completed: non_neg_integer(),
steps_failed: non_neg_integer(),
results: %{atom() => term()},
duration_ms: non_neg_integer()
}
@spec execute(pipeline_def()) :: {:ok, pipeline_result()} | {:error, term()}
def execute(%{steps: steps, context: initial_context} = pipeline) do
start_time = System.monotonic_time(:millisecond)
:telemetry.execute(
[:prismatic, :pipeline, :started],
%{step_count: length(steps)},
%{pipeline: pipeline.name}
)
result =
Enum.reduce_while(steps, {:ok, initial_context, %{}, 0, 0}, fn step, {:ok, ctx, results, completed, failed} ->
input = step.input_fn.(ctx)
case execute_step(step, input) do
{:ok, output} ->
new_ctx = Map.merge(ctx, %{step.name => output})
new_results = Map.put(results, step.name, output)
{:cont, {:ok, new_ctx, new_results, completed + 1, failed}}
{:error, reason} when step.on_failure == :skip ->
new_results = Map.put(results, step.name, {:skipped, reason})
{:cont, {:ok, ctx, new_results, completed, failed + 1}}
{:error, reason} when step.on_failure == :halt ->
new_results = Map.put(results, step.name, {:failed, reason})
{:halt, {:error, ctx, new_results, completed, failed + 1}}
{:error, _reason} ->
# Default: retry
case retry_step(step, input, step.retry_count) do
{:ok, output} ->
new_ctx = Map.merge(ctx, %{step.name => output})
new_results = Map.put(results, step.name, output)
{:cont, {:ok, new_ctx, new_results, completed + 1, failed}}
{:error, final_reason} ->
new_results = Map.put(results, step.name, {:failed, final_reason})
{:halt, {:error, ctx, new_results, completed, failed + 1}}
end
end
end)
duration = System.monotonic_time(:millisecond) - start_time
case result do
{:ok, _ctx, results, completed, failed} ->
pipeline_result = %{
status: if(failed == 0, do: :completed, else: :partial),
steps_completed: completed,
steps_failed: failed,
results: results,
duration_ms: duration
}
{:ok, pipeline_result}
{:error, _ctx, results, completed, failed} ->
pipeline_result = %{
status: :failed,
steps_completed: completed,
steps_failed: failed,
results: results,
duration_ms: duration
}
{:ok, pipeline_result}
end
end
@spec execute_step(step(), map()) :: {:ok, term()} | {:error, term()}
defp execute_step(step, input) do
task =
Task.async(fn ->
PrismaticAgents.Registry.dispatch(step.agent, input)
end)
case Task.yield(task, step.timeout_ms) || Task.shutdown(task) do
{:ok, result} -> result
nil -> {:error, :timeout}
end
end
@spec retry_step(step(), map(), non_neg_integer()) ::
{:ok, term()} | {:error, term()}
defp retry_step(_step, _input, 0), do: {:error, :retries_exhausted}
defp retry_step(step, input, remaining) do
# Exponential backoff
delay = :math.pow(2, step.retry_count - remaining) |> round() |> min(30_000)
Process.sleep(delay)
case execute_step(step, input) do
{:ok, output} -> {:ok, output}
{:error, _} -> retry_step(step, input, remaining - 1)
end
end
endComparison with Alternatives
Orchestration vs. Choreography
| Aspect | Orchestration (Prismatic) | Choreography |
|---|---|---|
| Control | Centralized coordinator | Decentralized, event-driven |
| Visibility | Full workflow state in one place | State distributed across agents |
| Debugging | Straightforward (follow the orchestrator) | Difficult (trace through events) |
| Coupling | Agents coupled to orchestrator | Agents coupled to event schema |
| Scalability | Orchestrator can bottleneck | Scales naturally |
| Error handling | Centralized retry/compensation | Each agent handles independently |
| Predictability | High (deterministic flow) | Low (emergent behavior) |
Orchestration vs. External Workflow Engines
| Aspect | Native OTP (Prismatic) | Temporal/Airflow/Argo |
|---|---|---|
| Infrastructure | Same BEAM node | Separate service cluster |
| Latency | Microseconds (function calls) | Milliseconds (network + serialization) |
| Fault tolerance | OTP supervision trees | External HA mechanisms |
| Hot code reload | Yes (OTP release upgrades) | No (requires redeployment) |
| Language | Elixir (same as application) | DSL or separate language |
| Operational overhead | None (part of the application) | Significant (separate system to manage) |
Orchestration vs. Simple Task Parallelism
Simple Task.async_stream or parallel map operations execute independent tasks concurrently but lack dependency management, error recovery, and result aggregation. The Prismatic orchestrator adds these capabilities while still leveraging OTP's Task primitives for the actual concurrent execution.
Best Practices
1. Design Orchestration Hierarchically
Avoid flat orchestration where a single coordinator manages hundreds of tasks directly. Instead, delegate to sub-orchestrators that manage their own domains. The AIAD tier system naturally produces this hierarchy, with L3 commanders orchestrating L2 specialists.
2. Make Tasks Idempotent
Orchestrators retry failed tasks. If tasks have side effects that are not idempotent (e.g., sending an email twice), retries cause duplicate actions. Design all orchestrated tasks to be safely re-executable.
3. Set Meaningful Timeouts
Every dispatched task must have a timeout. Without timeouts, a single stuck agent can block the entire orchestration pipeline. The Prismatic orchestrator enforces timeouts on every task and escalates timeout failures to the appropriate handler.
4. Emit Telemetry at Every State Transition
Orchestration is opaque without observability. Emit Telemetry events when objectives start, tasks dispatch, results arrive, and pipelines complete. This enables monitoring dashboards to show orchestration progress in real time.
5. Separate Orchestration Logic from Business Logic
The orchestrator should know how to coordinate tasks, not how to execute them. Business logic belongs in the agents. The orchestrator's responsibility is decomposition, dispatch, dependency management, and result aggregation -- never domain computation.
6. Plan for Partial Success
Not every task in a complex orchestration will succeed. Design orchestration flows to handle partial success: some tasks may be skippable, others may have fallbacks, and some failures should halt the entire pipeline. The pipeline's on_failure: :halt | :skip | :retry configuration enables this explicitly.
Common Pitfalls
1. Orchestrator as God Object
An orchestrator that accumulates too much responsibility -- task execution, result transformation, error handling, logging, notification -- becomes unmaintainable. Keep the orchestrator focused on coordination; delegate everything else.
2. Ignoring Backpressure
Dispatching 500 tasks simultaneously to a system that can handle 50 concurrently causes resource exhaustion. The Prismatic orchestrator enforces max_concurrent limits and queues tasks beyond this threshold, providing natural backpressure.
3. Synchronous Orchestration
Blocking the caller until the entire orchestration completes ties up resources and prevents progress updates. The Prismatic orchestrator returns immediately with an objective ID, enabling the caller to poll for status or receive async notifications.
4. Missing Compensation Logic
When a multi-step orchestration fails partway through, the completed steps may need to be rolled back. Without compensation actions (the saga pattern), partial failures leave the system in an inconsistent state. For critical operations, define compensation logic for every step.
5. Circular Task Dependencies
If task A depends on task B and task B depends on task A, the orchestrator deadlocks. Dependency validation must detect and reject circular dependencies before dispatching any tasks -- the same principle enforced by the platform's DependencyResolver for umbrella applications.
6. Insufficient Observability
Orchestration without monitoring is flying blind. When a 50-step pipeline stalls, operators need to know which step, which agent, and what error. Every state transition in the orchestration must emit observable events.
Use Cases
1. The /orchestrate Command
The platform's primary orchestration interface is the /orchestrate command, which coordinates multiple agents to execute complex operations with claimed 10x efficiency. A single /orchestrate invocation can trigger security assessments, quality checks, code generation, and reporting as a unified workflow.
2. OSINT Intelligence Gathering
When querying a target across 120 OSINT sources, orchestration manages the parallel dispatch of queries, rate limiting per source, result aggregation, contradiction detection (via NABLA axioms), and final intelligence report generation.
3. Security Assessment Pipeline
The Prismatic Perimeter security assessment orchestrates DNS enumeration, port scanning, certificate analysis, vulnerability checking, and compliance evaluation as a structured pipeline where each step's output feeds the next.
4. Quality Gate Execution
The mix quality.gates command orchestrates 13 quality domain checks in parallel, aggregates results, and produces a unified pass/fail decision. Each check is an independent task that can be individually retried on transient failure.
5. Color Team Security Exercises
Color team operations orchestrate coordinated activities across Red (attack simulation), Blue (defense), Purple (synthesis), Gray (boundary exploration), White (verification), and Black (threat modeling) teams, with the Purple coordinator synthesizing findings from all teams.
Related Concepts
- Agent Orchestration -- specific application of orchestration to multi-agent systems
- Multi-Agent System -- the agent ecosystem that orchestration coordinates
- Workflow -- structured sequences of actions that orchestration executes
- AIAD -- the agent specification standard defining orchestration hierarchy
- Archer Supreme -- the L5 supreme authority orchestrating platform-wide operations
- Pipeline -- linear sequence of data transformations managed by orchestration
- Agent Tier -- the hierarchical authority levels governing orchestration scope
- Supervision Tree -- OTP's fault tolerance mechanism underlying orchestration
- GenServer -- the process abstraction implementing orchestrator state management
- Backpressure -- flow control mechanism preventing orchestration overload
- Command -- the invocation interface through which orchestration is triggered
See Also
- Strategic Command -- the decision-making layer that initiates orchestration
- Concurrent Programming -- the execution model enabling parallel task dispatch
- Fault Tolerance -- how orchestration handles failures gracefully
- Circuit Breaker -- protection mechanism preventing cascade failures in orchestration
Connect & Contribute
Created by Tomas Korcak (korczis) | Open Source under GHL