Cooperative Systems

architecture

Cooperative Systems

Definition

Cooperative systems are computational architectures in which multiple autonomous components -- whether processes, agents, services, or nodes -- work together through shared protocols, negotiated interactions, and coordinated behavior to achieve collective outcomes that exceed the capabilities of any individual component. Unlike centrally controlled systems where a single orchestrator dictates all behavior, cooperative systems distribute decision-making authority among participants who voluntarily adhere to shared rules and communication protocols.

In the Prismatic Platform, cooperative systems manifest at multiple architectural levels: at the BEAM VM level through cooperative process scheduling, at the OTP level through supervision trees and inter-process message passing, and at the AIAD level through multi-agent systems where 530+ agents coordinate their activities to perform complex intelligence, security, and quality operations. The platform's entire design philosophy rests on the principle that sophisticated behavior emerges from the cooperation of simple, well-defined components.

Theoretical Foundations

Cooperative vs. Competitive Systems

System design exists on a spectrum between pure cooperation and pure competition. In cooperative systems, components share information freely, align their objectives, and sacrifice local optimization for global benefit. In competitive systems, components pursue independent objectives and interact through market-like mechanisms. Most real-world systems blend both approaches.

CharacteristicCooperativeCompetitivePrismatic Approach
Information sharingOpen, broadcastStrategic, selectiveOpen within teams, controlled between tiers
Objective alignmentShared global goalsIndependent local goalsAligned through doctrine and policies
Resource allocationCoordinatedMarket-basedSupervised allocation with backpressure
Failure handlingMutual aid (supervision)Individual responsibilityCooperative supervision trees
Decision makingConsensus or delegationAutonomousHierarchical with cooperative input

The Actor Model as Cooperative Foundation

The Actor Model, which underpins the BEAM virtual machine, provides the theoretical foundation for cooperative systems in the Prismatic Platform. Each actor (process) is an autonomous entity that communicates exclusively through asynchronous message passing. Cooperation emerges from the protocols that processes follow when exchanging messages, not from shared memory or global state.

This shared-nothing architecture is fundamentally cooperative: processes cannot forcefully access another process's state. All interaction requires the willing participation of both parties through message send and receive. This constraint ensures that cooperation is explicit and observable, eliminating entire categories of concurrency bugs that plague shared-memory systems.

BEAM VM Cooperative Scheduling

The BEAM virtual machine implements a cooperative-preemptive hybrid scheduling model that is central to understanding cooperative systems in Elixir/OTP. Each BEAM process receives a budget of approximately 4,000 reductions (roughly equivalent to function calls), after which it yields control to the scheduler. This is cooperative in spirit -- processes are expected to perform reasonable units of work -- but preemptive in mechanism, as the scheduler enforces the yield after the reduction budget is exhausted.

defmodule Prismatic.Scheduling.CooperativeDemo do
  @moduledoc """
  Demonstrates cooperative scheduling properties of the BEAM VM.
  Processes yield after their reduction budget, ensuring fair
  scheduling across all platform processes regardless of workload.
  """

  @spec demonstrate_fair_scheduling(pos_integer()) :: [map()]
  def demonstrate_fair_scheduling(process_count) do
    parent = self()

    processes =
      for i <- 1..process_count do
        spawn(fn ->
          start = System.monotonic_time(:microsecond)
          result = do_work(1_000_000)
          elapsed = System.monotonic_time(:microsecond) - start
          send(parent, {:done, i, elapsed, result})
        end)
      end

    results =
      for _ <- processes do
        receive do
          {:done, id, elapsed, _result} -> %{process: id, elapsed_us: elapsed}
        end
      end

    # All processes complete in roughly similar time due to
    # cooperative scheduling - no single process starves others
    Enum.sort_by(results, & &1.elapsed_us)
  end

  @spec do_work(pos_integer()) :: non_neg_integer()
  defp do_work(iterations) do
    Enum.reduce(1..iterations, 0, fn i, acc -> acc + rem(i, 7) end)
  end
end

Scheduler Fairness Guarantees

The BEAM's cooperative scheduling provides several guarantees critical for cooperative systems:

  1. No starvation: Every runnable process will eventually receive CPU time, regardless of how many processes are running.
  2. Bounded latency: The reduction limit ensures that no single process can monopolize a scheduler for more than a few milliseconds.
  3. IO fairness: Processes waiting on IO are handled through a separate mechanism that does not consume reduction budgets.
  4. Dirty schedulers: Long-running NIFs and BIFs execute on separate dirty schedulers to avoid disrupting cooperative scheduling of normal processes.

OTP Cooperation Patterns

OTP provides several battle-tested patterns for building cooperative systems that go beyond simple message passing.

Supervision as Cooperative Infrastructure

Supervision trees are the primary cooperative infrastructure in OTP. A supervisor cooperates with its children by monitoring their health, restarting them when they fail, and enforcing lifecycle contracts. Children cooperate by adhering to the child_spec contract, implementing proper init/1 and terminate/2 callbacks, and respecting shutdown signals.

defmodule Prismatic.Cooperation.TeamSupervisor do
  @moduledoc """
  Supervises a team of cooperative agent processes that work together
  on a shared objective. Implements the :rest_for_one strategy to
  ensure dependent agents restart together, maintaining team coherence.
  """

  use Supervisor

  @spec start_link(keyword()) :: Supervisor.on_start()
  def start_link(opts) do
    team_name = Keyword.fetch!(opts, :team_name)
    Supervisor.start_link(__MODULE__, opts, name: via_tuple(team_name))
  end

  @impl Supervisor
  @spec init(keyword()) :: {:ok, {Supervisor.sup_flags(), [Supervisor.child_spec()]}}
  def init(opts) do
    team_config = Keyword.fetch!(opts, :config)

    children = [
      {Prismatic.Cooperation.SharedState, team_config},
      {Prismatic.Cooperation.Coordinator, team_config},
      {Prismatic.Cooperation.WorkerPool, team_config}
    ]

    # :rest_for_one ensures that if SharedState crashes,
    # Coordinator and WorkerPool also restart with fresh state
    Supervisor.init(children, strategy: :rest_for_one)
  end

  defp via_tuple(name), do: {:via, Registry, {Prismatic.TeamRegistry, name}}
end

GenServer Cooperation Protocols

GenServer processes cooperate through well-defined message protocols. The call/cast/info trichotomy provides different cooperation semantics:

  • call: Synchronous cooperation where the caller waits for a response. Used when the calling process needs the result before continuing.
  • cast: Asynchronous cooperation where the sender fires and forgets. Used for notifications, updates, and non-blocking commands.
  • info: Asynchronous system messages for timers, monitors, and inter-process signals.
defmodule Prismatic.Cooperation.WorkDistributor do
  @moduledoc """
  Distributes work items across a pool of cooperative worker processes.
  Workers register their availability, receive assignments, and report
  completion back to the distributor, forming a cooperative work loop.
  """

  use GenServer

  @type state :: %{
    pending: :queue.queue(work_item()),
    active: %{pid() => work_item()},
    available: [pid()],
    completed: non_neg_integer()
  }

  @type work_item :: %{id: binary(), payload: term(), assigned_at: DateTime.t() | nil}

  @spec submit_work(GenServer.server(), term()) :: :ok
  def submit_work(server, payload) do
    GenServer.cast(server, {:submit, payload})
  end

  @spec register_worker(GenServer.server()) :: :ok
  def register_worker(server) do
    GenServer.cast(server, {:register, self()})
  end

  @spec report_completion(GenServer.server(), binary(), term()) :: :ok
  def report_completion(server, work_id, result) do
    GenServer.cast(server, {:complete, work_id, self(), result})
  end

  @impl GenServer
  def handle_cast({:submit, payload}, state) do
    item = %{id: generate_id(), payload: payload, assigned_at: nil}

    case state.available do
      [worker | rest] ->
        assign_work(worker, item)
        {:noreply, %{state | active: Map.put(state.active, worker, item), available: rest}}

      [] ->
        {:noreply, %{state | pending: :queue.in(item, state.pending)}}
    end
  end

  @impl GenServer
  def handle_cast({:complete, _work_id, worker, _result}, state) do
    state = %{state |
      active: Map.delete(state.active, worker),
      completed: state.completed + 1
    }

    case :queue.out(state.pending) do
      {{:value, item}, remaining} ->
        assign_work(worker, item)
        {:noreply, %{state | active: Map.put(state.active, worker, item), pending: remaining}}

      {:empty, _} ->
        {:noreply, %{state | available: [worker | state.available]}}
    end
  end

  defp assign_work(worker, item) do
    send(worker, {:work_assignment, %{item | assigned_at: DateTime.utc_now()}})
  end

  defp generate_id, do: :crypto.strong_rand_bytes(16) |> Base.encode16(case: :lower)
end

PubSub for Broadcast Cooperation

The PubSub pattern enables one-to-many cooperative communication where publishers share information with all interested subscribers without needing to know who they are.

defmodule Prismatic.Cooperation.EventBus do
  @moduledoc """
  Platform-wide event bus enabling cooperative communication between
  loosely coupled system components. Events propagate through topic
  subscriptions, allowing components to react to changes without
  direct coupling.
  """

  @spec broadcast(binary(), term()) :: :ok
  def broadcast(topic, event) do
    Phoenix.PubSub.broadcast(Prismatic.PubSub, topic, event)
  end

  @spec subscribe(binary()) :: :ok
  def subscribe(topic) do
    Phoenix.PubSub.subscribe(Prismatic.PubSub, topic)
  end

  @spec cooperative_event_topics() :: [binary()]
  def cooperative_event_topics do
    [
      "agent:status_change",
      "quality:gate_result",
      "security:threat_detected",
      "osint:entity_updated",
      "boot:domain_ready"
    ]
  end
end

AIAD Multi-Agent Cooperation

The Prismatic Platform's AIAD (AI Agent Definition) framework implements sophisticated multi-agent cooperation across 530+ agents organized into hierarchical tiers with cooperative protocols.

Agent Hierarchy and Cooperation Levels

TierRoleCooperation Model
L1 OperationalTask executionCooperates with peers via message passing
L2 TacticalDomain coordinationCooperates with L1 agents through task delegation
L3 StrategicCross-domain orchestrationCooperates with L2 agents through strategic directives
L4 Safety-CriticalOverride authorityCooperates through veto and escalation protocols
L5 SupremePlatform-wide authorityCooperates through doctrine and policy enforcement

Cooperative Agent Patterns

defmodule Prismatic.Agents.CooperativeProtocol do
  @moduledoc """
  Defines the cooperative protocol that all AIAD agents must implement.
  Ensures consistent cooperation semantics across the 530+ agent ecosystem.
  """

  @callback accept_task(task :: map()) :: {:ok, task_id :: binary()} | {:reject, reason :: term()}
  @callback report_progress(task_id :: binary(), progress :: float()) :: :ok
  @callback request_assistance(task_id :: binary(), capability :: atom()) :: {:ok, helper_pid :: pid()} | {:error, :no_helper}
  @callback share_finding(finding :: map()) :: :ok
  @callback yield_to_authority(authority_level :: pos_integer()) :: :ok

  @spec cooperation_contract() :: map()
  def cooperation_contract do
    %{
      max_response_time_ms: 5_000,
      must_acknowledge_tasks: true,
      must_report_failures: true,
      must_share_findings: true,
      must_respect_authority: true,
      must_yield_on_request: true
    }
  end
end

Color-Team Cooperative Security

The Color Teams architecture represents a specialized cooperative system where teams with deliberately different objectives cooperate to improve platform security. The Red Team attacks while the Blue Team defends, but both cooperate within the larger goal of identifying and closing security gaps. The Purple Team serves as the cooperative synthesis layer, ensuring that adversarial findings translate into defensive improvements.

This adversarial cooperation pattern -- where agents cooperate at the meta-level while competing at the operational level -- is one of the most sophisticated cooperation models in the platform.

Backpressure as Cooperative Flow Control

Backpressure is a cooperative flow control mechanism where downstream consumers signal upstream producers to slow down when they cannot process messages fast enough. Without backpressure, cooperative systems degrade into overwhelmed systems where message queues grow unbounded.

defmodule Prismatic.Cooperation.BackpressureAware do
  @moduledoc """
  Implements cooperative backpressure between producer and consumer
  processes. The consumer advertises its capacity, and the producer
  respects the limit, creating a cooperative flow control loop.
  """

  use GenServer

  @type state :: %{
    demand: non_neg_integer(),
    buffer: :queue.queue(term()),
    consumers: [pid()]
  }

  @spec request_demand(GenServer.server(), non_neg_integer()) :: :ok
  def request_demand(server, amount) do
    GenServer.cast(server, {:demand, self(), amount})
  end

  @impl GenServer
  def handle_cast({:produce, item}, state) do
    case state.demand do
      0 ->
        # No demand - buffer cooperatively instead of dropping
        {:noreply, %{state | buffer: :queue.in(item, state.buffer)}}

      demand when demand > 0 ->
        dispatch_to_consumer(item, state.consumers)
        {:noreply, %{state | demand: demand - 1}}
    end
  end

  @impl GenServer
  def handle_cast({:demand, consumer, amount}, state) do
    state = add_consumer(state, consumer)
    {items, remaining_buffer} = drain_buffer(state.buffer, amount)

    for item <- items do
      dispatch_to_consumer(item, [consumer])
    end

    remaining_demand = amount - length(items)
    {:noreply, %{state | demand: state.demand + remaining_demand, buffer: remaining_buffer}}
  end
end

Distributed Cooperation Across Nodes

When the Prismatic Platform runs in a distributed configuration across multiple BEAM nodes, cooperation extends across network boundaries. Erlang's distribution protocol makes remote message passing syntactically identical to local message passing, enabling cooperative patterns to scale transparently across nodes.

Cluster Cooperation Patterns

  • Leader election: Nodes cooperate to select a single leader for global coordination tasks, with automatic re-election on leader failure.
  • Consistent hashing: Work is distributed across nodes using consistent hashing, with cooperative handoff when nodes join or leave.
  • CRDT synchronization: Conflict-free replicated data types allow cooperative state sharing without coordination overhead.
  • Gossip protocols: Nodes cooperatively disseminate cluster metadata through epidemic-style gossip.

Cooperation Failure Modes

Cooperative systems can fail in ways that non-cooperative systems cannot. Understanding these failure modes is essential for building resilient cooperative architectures.

Byzantine Failures

A cooperating component that begins sending incorrect or malicious messages can corrupt the entire cooperative system. OTP's supervision model addresses this through crash-and-restart semantics: a misbehaving process is terminated and replaced with a fresh instance.

Cooperation Deadlocks

Two processes waiting for each other's response create a cooperation deadlock. OTP mitigates this through timeout-based calls (GenServer.call/3 with timeout) and the general preference for asynchronous cast operations.

Cascading Cooperation Failures

When a critical cooperative component fails, all components depending on it may also fail. Supervision trees address this through restart strategies (:one_for_all, :rest_for_one) that ensure dependent components restart together in a consistent state.

Split-Brain in Distributed Cooperation

Network partitions can split a cooperative cluster into isolated groups, each believing the other has failed. The platform uses CRDT-based state and configurable partition handling strategies to maintain cooperative consistency during network splits.

Measuring Cooperation Effectiveness

MetricDescriptionHealthy Range
Message ThroughputMessages exchanged per second between cooperating components10K-100K/s
Cooperation LatencyTime from cooperative request to response< 5 ms (local), < 50 ms (remote)
Utilization BalanceStandard deviation of work distribution across cooperating agents< 15%
Failure Recovery TimeTime from cooperative failure to restored operation< 500 ms
Backpressure EventsFrequency of backpressure activation< 1% of total operations

Comparison with Other Cooperation Models

ModelCommunicationStateSchedulingExample
BEAM/OTPMessage passingShared-nothingCooperative-preemptivePrismatic Platform
Akka (JVM)Message passingShared-nothingThread-basedReactive applications
Go (goroutines)ChannelsShared memory possibleCooperative-preemptiveCloud infrastructure
KubernetesAPI callsExternal state storesContainer schedulingMicroservices
MPIMessage passingDistributed memoryCooperativeScientific computing

The BEAM/OTP model provides the strongest guarantees for cooperative systems due to its process isolation, lightweight processes (supporting millions of concurrent actors), and built-in distribution protocol. These properties make it uniquely suited for the Prismatic Platform's cooperative agent architecture.

Best Practices

  1. Define cooperation contracts explicitly: Every cooperative interaction should have a documented contract specifying message formats, timeouts, error handling, and backpressure semantics.

  2. Prefer asynchronous cooperation: Use cast and info messages for cooperation that does not require an immediate response. This reduces coupling and eliminates deadlock risk.

  3. Implement backpressure at every boundary: Every producer-consumer relationship should include backpressure to prevent unbounded queue growth.

  4. Monitor cooperation health: Track message latencies, queue depths, and cooperation failure rates to detect degradation before it becomes critical.

  5. Design for partial cooperation failure: Assume that any cooperating component can fail at any time. Use supervision trees, circuit breakers, and graceful degradation to maintain system availability.

  6. Test cooperation under adversarial conditions: Use the Red Team to inject cooperation failures, delays, and Byzantine behavior to validate system resilience.

  7. Keep cooperation protocols simple: Complex multi-step cooperation protocols are fragile. Prefer simple request-response or event-driven patterns.

Further Reading

  • Armstrong, Joe. "A History of Erlang." Proceedings of the Third ACM SIGPLAN Conference on History of Programming Languages, 2007.
  • Hewitt, Carl. "A Universal Modular ACTOR Formalism for Artificial Intelligence." IJCAI, 1973.
  • Wooldridge, Michael. "An Introduction to MultiAgent Systems." John Wiley & Sons, 2009.
  • Prismatic Platform AIAD Standard: .aiad/README.md

Connect & Contribute: Created by Tomas Korcak (korczis) | GitHub | GitLab | LinkedIn | Open Source under GHL