Multi-Agent Systems

architecture

Multi-Agent Systems

Definition

Multi-Agent Systems (MAS) are computational architectures composed of multiple autonomous agents that interact, collaborate, and coordinate to solve problems that exceed the capabilities of any individual agent. Each agent in a MAS operates with its own knowledge, goals, and capabilities, making independent decisions while communicating with other agents through defined protocols. The system's emergent behavior -- the collective intelligence that arises from agent interactions -- is greater than the sum of its parts. Multi-agent systems draw from distributed computing, artificial intelligence, and game theory to address complex, dynamic, and large-scale problems where centralized control is impractical, insufficient, or undesirable.

In the context of platform engineering, multi-agent systems provide a natural architecture for decomposing complex operations into specialized, independently evolving components. Rather than building monolithic systems that attempt to handle every concern, a MAS delegates specific domains to specialized agents (security agents, quality agents, performance agents, intelligence agents) that collaborate through well-defined interfaces. This decomposition mirrors the OTP philosophy of process isolation and supervised concurrency, making Elixir and the BEAM virtual machine particularly well-suited for implementing multi-agent architectures.

Overview

The field of multi-agent systems emerged from the intersection of distributed artificial intelligence and software engineering in the 1990s. Early MAS research focused on theoretical models of agent communication, negotiation, and cooperation. Modern MAS implementations have moved from academic theory to production infrastructure, powering everything from autonomous vehicle fleets to financial trading systems to platform engineering tools.

The fundamental insight behind MAS is that complex problems are often better solved by multiple specialized entities working together than by a single generalist entity working alone. This parallels the biological principle of division of labor: an ant colony solves complex problems (foraging, nest building, defense) not through central planning but through thousands of simple agents following local rules that produce sophisticated global behavior.

Key Properties of Multi-Agent Systems

PropertyDescription
AutonomyEach agent operates independently, making decisions based on its own knowledge and goals
ReactivityAgents perceive their environment and respond to changes in real time
ProactivityAgents take initiative to achieve their goals, not just responding passively
Social abilityAgents communicate and coordinate with other agents through defined protocols
HeterogeneityDifferent agents can have different architectures, capabilities, and implementations
ScalabilityNew agents can be added without redesigning existing agents
Fault toleranceIndividual agent failures do not bring down the entire system

The Prismatic Agent Ecosystem

The Prismatic Platform operates one of the largest production multi-agent systems in the Elixir ecosystem, with 530+ AIAD agents organized across 16 domains and 5 authority tiers. Each agent is a fully autonomous entity with defined capabilities, communication protocols, and governance policies. The system demonstrates that multi-agent architecture scales from simple automation tasks to complex epistemic operations like adversarial security testing (Red/Blue/Purple teams) and autonomous quality evolution.

Technical Details

Agent Architecture in Elixir/OTP

The BEAM virtual machine provides a natural foundation for multi-agent systems. Each OTP process can represent an agent, with message passing providing the communication infrastructure, supervision trees providing fault tolerance, and GenServer providing the state management framework.

defmodule PrismaticAgents.Agent do
  @moduledoc """
  Base agent behaviour for the Prismatic multi-agent system.
  Every agent implements this behaviour, gaining standard
  lifecycle management, communication protocols, and
  telemetry instrumentation.
  """

  @type agent_id :: String.t()
  @type agent_tier :: :l1_operational | :l2_tactical | :l3_strategic | :l4_specialist | :l5_supreme
  @type agent_state :: %{
    id: agent_id(),
    tier: agent_tier(),
    domain: atom(),
    capabilities: [atom()],
    status: :idle | :active | :suspended | :terminated,
    message_count: non_neg_integer(),
    started_at: DateTime.t()
  }
  @type message :: %{
    from: agent_id(),
    to: agent_id(),
    type: :request | :response | :broadcast | :directive,
    payload: term(),
    timestamp: DateTime.t(),
    correlation_id: String.t()
  }

  @callback init(keyword()) :: {:ok, agent_state()} | {:error, term()}
  @callback handle_message(message(), agent_state()) :: {:reply, term(), agent_state()} | {:noreply, agent_state()}
  @callback capabilities() :: [atom()]
  @callback tier() :: agent_tier()
  @callback domain() :: atom()

  defmacro __using__(opts) do
    quote do
      use GenServer
      @behaviour PrismaticAgents.Agent

      @agent_domain Keyword.get(unquote(opts), :domain, :general)
      @agent_tier Keyword.get(unquote(opts), :tier, :l1_operational)

      def start_link(opts) do
        id = Keyword.get(opts, :id, generate_id())
        GenServer.start_link(__MODULE__, Keyword.put(opts, :id, id), name: via_tuple(id))
      end

      @impl GenServer
      def init(opts) do
        state = %{
          id: Keyword.fetch!(opts, :id),
          tier: @agent_tier,
          domain: @agent_domain,
          capabilities: capabilities(),
          status: :idle,
          message_count: 0,
          started_at: DateTime.utc_now()
        }

        :telemetry.execute(
          [:prismatic, :agent, :started],
          %{},
          %{id: state.id, tier: state.tier, domain: state.domain}
        )

        __MODULE__.init(opts)
      end

      @impl GenServer
      def handle_call({:message, message}, _from, state) do
        new_state = %{state | message_count: state.message_count + 1, status: :active}

        :telemetry.execute(
          [:prismatic, :agent, :message_received],
          %{message_count: new_state.message_count},
          %{id: state.id, type: message.type, from: message.from}
        )

        case __MODULE__.handle_message(message, new_state) do
          {:reply, response, final_state} ->
            {:reply, {:ok, response}, %{final_state | status: :idle}}
          {:noreply, final_state} ->
            {:reply, :ok, %{final_state | status: :idle}}
        end
      end

      defp generate_id do
        "#{@agent_domain}-#{Base.encode16(:crypto.strong_rand_bytes(4), case: :lower)}"
      end

      defp via_tuple(id) do
        {:via, Registry, {PrismaticAgents.Registry, id}}
      end

      defoverridable init: 1
    end
  end
end

Agent Communication Protocols

Agents in a MAS communicate through structured protocols. The Prismatic Platform uses four communication patterns:

defmodule PrismaticAgents.Communication do
  @moduledoc """
  Communication protocols for inter-agent messaging.
  Supports direct request-response, broadcast to domain,
  chain-of-command directives, and publish-subscribe events.
  """

  alias PrismaticAgents.Agent

  @type routing :: :direct | :broadcast | :chain_of_command | :pubsub

  @doc """
  Sends a direct request to a specific agent and waits for response.
  Used for targeted queries and task delegation.
  """
  @spec request(Agent.agent_id(), Agent.agent_id(), term()) ::
    {:ok, term()} | {:error, term()}
  def request(from_id, to_id, payload) do
    message = build_message(from_id, to_id, :request, payload)

    case Registry.lookup(PrismaticAgents.Registry, to_id) do
      [{pid, _}] -> GenServer.call(pid, {:message, message}, :timer.seconds(30))
      [] -> {:error, :agent_not_found}
    end
  end

  @doc """
  Broadcasts a message to all agents in a domain.
  Used for announcements, status updates, and event notifications.
  """
  @spec broadcast(Agent.agent_id(), atom(), term()) :: :ok
  def broadcast(from_id, domain, payload) do
    message = build_message(from_id, "broadcast:#{domain}", :broadcast, payload)

    Registry.dispatch(PrismaticAgents.Registry, {:domain, domain}, fn entries ->
      for {pid, _} <- entries do
        GenServer.cast(pid, {:broadcast, message})
      end
    end)

    :ok
  end

  @doc """
  Sends a directive down the chain of command.
  Higher-tier agents can issue directives to lower-tier agents.
  Used for orchestration, task assignment, and policy enforcement.
  """
  @spec directive(Agent.agent_id(), Agent.agent_tier(), atom(), term()) :: :ok
  def directive(from_id, target_tier, domain, payload) do
    message = build_message(from_id, "directive:#{target_tier}:#{domain}", :directive, payload)

    Registry.dispatch(PrismaticAgents.Registry, {:tier_domain, target_tier, domain}, fn entries ->
      for {pid, _} <- entries do
        GenServer.cast(pid, {:directive, message})
      end
    end)

    :ok
  end

  defp build_message(from, to, type, payload) do
    %{
      from: from,
      to: to,
      type: type,
      payload: payload,
      timestamp: DateTime.utc_now(),
      correlation_id: Base.encode16(:crypto.strong_rand_bytes(8))
    }
  end
end

Agent Orchestration and Coordination

Orchestration manages the lifecycle and coordination of multiple agents working on a shared task:

defmodule PrismaticAgents.Orchestrator do
  @moduledoc """
  Orchestrates multi-agent workflows by decomposing complex
  tasks into subtasks, assigning them to capable agents,
  monitoring execution, and aggregating results.

  The orchestrator implements the scatter-gather pattern:
  distribute work to specialized agents in parallel,
  collect results, and synthesize a unified response.
  """

  use GenServer

  @type task_spec :: %{
    id: String.t(),
    description: String.t(),
    required_capabilities: [atom()],
    priority: :low | :medium | :high | :critical,
    timeout: pos_integer()
  }

  @type orchestration_result :: %{
    task_id: String.t(),
    status: :completed | :partial | :failed,
    agent_results: [%{agent_id: String.t(), result: term(), duration: non_neg_integer()}],
    total_duration: non_neg_integer()
  }

  @spec execute(String.t(), [task_spec()]) :: {:ok, orchestration_result()} | {:error, term()}
  def execute(workflow_id, tasks) when is_list(tasks) do
    GenServer.call(__MODULE__, {:execute, workflow_id, tasks}, :timer.minutes(5))
  end

  @impl GenServer
  def init(_opts) do
    {:ok, %{active_workflows: %{}, completed: 0}}
  end

  @impl GenServer
  def handle_call({:execute, workflow_id, tasks}, _from, state) do
    start_time = System.monotonic_time(:millisecond)

    results =
      tasks
      |> Enum.map(fn task ->
        Task.async(fn -> execute_task(task) end)
      end)
      |> Task.await_many(:timer.minutes(2))

    duration = System.monotonic_time(:millisecond) - start_time

    orchestration_result = %{
      task_id: workflow_id,
      status: determine_status(results),
      agent_results: results,
      total_duration: duration
    }

    :telemetry.execute(
      [:prismatic, :orchestrator, :workflow_completed],
      %{duration: duration, task_count: length(tasks)},
      %{workflow_id: workflow_id, status: orchestration_result.status}
    )

    {:reply, {:ok, orchestration_result}, %{state | completed: state.completed + 1}}
  end

  defp execute_task(task) do
    case find_capable_agent(task.required_capabilities) do
      {:ok, agent_id} ->
        start = System.monotonic_time(:millisecond)
        result = PrismaticAgents.Communication.request("orchestrator", agent_id, task)
        duration = System.monotonic_time(:millisecond) - start
        %{agent_id: agent_id, result: result, duration: duration}

      {:error, :no_capable_agent} ->
        %{agent_id: "none", result: {:error, :no_capable_agent}, duration: 0}
    end
  end

  defp find_capable_agent(required_capabilities) do
    PrismaticAgents.Registry
    |> Registry.select([{{:"$1", :"$2", :"$3"}, [], [{{:"$1", :"$2", :"$3"}}]}])
    |> Enum.find_value({:error, :no_capable_agent}, fn {id, _pid, meta} ->
      if MapSet.subset?(MapSet.new(required_capabilities), MapSet.new(meta[:capabilities] || [])) do
        {:ok, id}
      end
    end)
  end

  defp determine_status(results) do
    error_count = Enum.count(results, fn r -> match?({:error, _}, r.result) end)

    cond do
      error_count == 0 -> :completed
      error_count < length(results) -> :partial
      true -> :failed
    end
  end
end

Implementation

Implementing a production multi-agent system requires careful attention to several architectural concerns that go beyond individual agent design.

Agent Registration and Discovery

Every agent in the system must be discoverable by other agents. The Prismatic Platform uses OTP's Registry for local agent discovery and Horde for distributed discovery across cluster nodes. Agents register with their ID, domain, tier, and capabilities, enabling dynamic routing based on required capabilities rather than hardcoded agent addresses.

Supervision Strategy

Multi-agent systems require robust supervision strategies because the system must continue operating when individual agents fail. The Prismatic Platform uses a domain-based supervision hierarchy: each domain (security, quality, intelligence, etc.) has its own supervisor that manages agents within that domain. Domain supervisors are themselves supervised by a top-level platform supervisor. This hierarchy ensures that a failing security agent does not impact quality agents, and a domain-wide failure is contained and recovered automatically.

Conflict Resolution

When multiple agents have overlapping capabilities or conflicting recommendations, the system needs a conflict resolution mechanism. The Prismatic Platform uses a tier-based authority model: higher-tier agents (L3 Strategic, L5 Supreme) have authority over lower-tier agents (L1 Operational, L2 Tactical). When agents at the same tier conflict, the orchestrator applies domain-specific resolution rules or escalates to a higher-tier agent.

State Isolation

Each agent maintains its own state, isolated from other agents through OTP process boundaries. This isolation prevents cascading state corruption -- if one agent's state becomes inconsistent, it does not affect other agents. The BEAM VM's per-process garbage collection further ensures that memory management in one agent does not create latency spikes for others.

Observability

In a system with 530+ agents, observability is critical. Every agent emits telemetry events for message receipt, processing duration, state transitions, and errors. The orchestrator emits workflow-level telemetry. These events feed into dashboards that display agent status, communication patterns, processing latency, and error rates across the entire system.

Comparison

Multi-Agent Systems vs. Microservices

DimensionMulti-Agent SystemsMicroservices
GranularityFine-grained (individual agents)Coarse-grained (service boundaries)
CommunicationStructured protocols, typed messagesREST, gRPC, message queues
AutonomyHigh -- agents make independent decisionsMedium -- services follow request-response
StatePer-agent state, process-isolatedPer-service state, database-backed
CoordinationOrchestration, negotiation, votingAPI calls, saga patterns
IntelligenceBuilt-in decision-making capabilityExternal to service logic
DeploymentSame runtime, process-level isolationSeparate deployments, network-level isolation

Multi-Agent Systems vs. Actor Model

The actor model (as implemented by Erlang/BEAM processes) is a subset of multi-agent systems. All actors share the core properties of message passing, state isolation, and concurrent execution. Multi-agent systems add higher-level abstractions: goal-oriented behavior, communication protocols, negotiation mechanisms, and coordination strategies. Every MAS is built on some form of actor model, but not every actor system is a MAS.

Multi-Agent Systems vs. Workflow Engines

Workflow engines (Temporal, Airflow, Broadway) execute predefined sequences of steps. Multi-agent systems enable dynamic, adaptive behavior where agents decide their actions based on current conditions rather than following fixed scripts. Workflow engines are appropriate when the process is well-understood and stable; MAS are appropriate when the problem space is dynamic and requires adaptive responses.

Best Practices

Design agents for single responsibility. Each agent should have a clearly defined domain and capability set. An agent that handles both security scanning and performance monitoring is harder to test, harder to evolve, and harder to reason about than two specialized agents that communicate when needed.

Use structured communication protocols. Agents must communicate through well-defined message types with clear semantics. Ad-hoc message passing leads to brittle inter-agent dependencies. The Prismatic Platform uses four standardized message types (request, response, broadcast, directive) with typed payloads and correlation IDs for request tracing.

Implement graceful degradation. The system should continue operating with reduced capability when agents fail, rather than failing entirely. If the security scanning agent is down, the orchestrator should proceed with available agents and flag the gap, rather than aborting the entire workflow.

Separate agent logic from infrastructure. Agent business logic (decision-making, domain expertise) should be separated from infrastructure concerns (message routing, state persistence, telemetry). The PrismaticAgents.Agent behaviour demonstrates this: infrastructure is in the macro-generated code; business logic is in the callback implementations.

Version agent protocols. As the system evolves, communication protocols change. Version message formats so that agents can be upgraded independently without breaking inter-agent communication.

Monitor agent population health. Track the number of running agents per domain, message queue depths, processing latencies, and error rates. Set alerts for anomalies: a sudden drop in agent count, a spike in message queue depth, or increasing error rates indicate system health issues.

Common Pitfalls

Emergent complexity. As agent count grows, the number of potential interactions grows combinatorially. A system with 100 agents has nearly 10,000 possible pairwise interactions. Without clear communication protocols and authority hierarchies, the system becomes unpredictable. The Prismatic Platform mitigates this through domain isolation (agents primarily communicate within their domain) and tier-based authority (higher tiers coordinate lower tiers).

Distributed deadlocks. Agents waiting for responses from each other create deadlocks that are harder to detect and resolve than single-process deadlocks. Use timeouts on all inter-agent communications, implement deadlock detection through correlation ID tracking, and design protocols to avoid circular dependencies.

Message storms. Broadcast messages in a large MAS can create cascading message storms where each agent's response triggers further messages. Use rate limiting on broadcasts, implement back-pressure through mailbox size monitoring, and design protocols that minimize broadcast fan-out.

Over-anthropomorphizing agents. Calling software components "agents" can lead to over-attributing intelligence and autonomy. Most production agents are deterministic state machines that follow programmed rules, not truly autonomous reasoning entities. Design for predictability and testability rather than artificial intelligence.

Insufficient testing of agent interactions. Testing individual agents in isolation is necessary but insufficient. The most critical bugs in MAS emerge from agent interactions -- race conditions, protocol mismatches, and emergent behaviors that no individual agent exhibits. Implement integration tests that exercise multi-agent workflows end-to-end.

Use Cases

Prismatic AIAD Ecosystem

The Prismatic Platform's 530+ AIAD agents represent a production multi-agent system operating across 16 domains: security (Red/Blue/Purple/White/Black/Gray teams), quality (Quality Floor Guardian, AutoEvolve, Credo agents), intelligence (OSINT, HAWKEYE), infrastructure (supervisor, health monitoring), and more. Agents communicate through structured AIAD protocols, are organized in a 5-tier authority hierarchy, and are governed by documented policies. This ecosystem demonstrates that MAS can scale to hundreds of agents while maintaining operational coherence.

Color-Team Security Operations

The platform's 6 color-team security groups (Gray, Red, Blue, Purple, White, Black) exemplify MAS coordination in practice. Gray agents explore boundary conditions and pass findings to Red agents, who simulate adversarial scenarios. Blue agents develop defensive responses. Purple agents synthesize Red-Blue findings into closure assessments. White agents verify defenses formally. Black agents model theoretical threats in complete isolation. This pipeline demonstrates how specialized agents collaborate to achieve a capability (comprehensive security assessment) that no single agent could provide.

Autonomous Quality Evolution

The AutoEvolve agent system continuously scans the platform for improvement opportunities, generates fixes, validates them through quality gates, and applies them. This requires coordination between scanner agents (identifying issues), analyzer agents (assessing priority and feasibility), generator agents (producing fixes), and validator agents (testing fixes). The multi-agent architecture ensures that each concern is handled by a specialized agent while the orchestrator maintains overall coherence.

Distributed Intelligence Gathering

The OSINT (Open Source Intelligence) toolbox uses multiple agents to gather intelligence from 120+ sources simultaneously -- Czech business registries, global security databases, sanctions lists, and more. Each source adapter operates as an independent agent with its own rate limiting, error handling, and data normalization. The orchestrator coordinates parallel queries, aggregates results, and resolves conflicts between sources.

Self-Healing Infrastructure

The AutoHeal system uses multiple monitoring agents that observe different aspects of system health (process counts, memory usage, message queue depths, error rates). When anomalies are detected, healing agents coordinate to diagnose the issue and apply corrective actions. The multi-agent approach allows different healing strategies (restart, scale, reconfigure, rollback) to be tried in parallel with the first successful strategy winning.

  • Agent -- Individual autonomous entity that serves as the building block of multi-agent systems
  • Agent Orchestration -- Coordination patterns for managing multi-agent workflows
  • Autonomous Agent -- Self-governing agent capable of independent decision-making
  • Actor Model -- Computational model of concurrent message-passing processes underlying MAS
  • Message Passing -- Communication mechanism enabling inter-agent interaction
  • Supervision Tree -- OTP fault tolerance structure managing agent process lifecycles
  • Distributed System -- Computing paradigm for agents operating across multiple nodes
  • Fault Tolerance -- System property ensuring continued operation despite agent failures
  • GenServer -- OTP behaviour providing the process infrastructure for individual agents
  • AIAD -- The Prismatic Platform's agent framework defining standards for 530+ agents

See Also

  • Prismatic Agents -- Runtime infrastructure for the platform's multi-agent system
  • Color Teams -- Security-focused multi-agent coordination across 6 color-coded teams
  • Architecture -- Platform architecture with multi-agent system as core design principle
  • Agent Registry -- Discovery and metadata system for the 530+ agent population

Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL

Back to Glossary