Ecosystem

architecture

Ecosystem

Definition

An ecosystem is the complete network of interconnected software components, tools, libraries, services, and communities that collectively form a platform's operational environment. Unlike a monolithic application or a loosely coupled collection of services, an ecosystem exhibits emergent properties -- capabilities that arise from the systematic composition of its parts rather than from any single component in isolation. The ecosystem concept encompasses not only the technical artifacts (source code, configurations, deployed services) but also the human and organizational structures (contributor communities, governance models, documentation cultures) that sustain and evolve the platform over time.

In the context of the Prismatic Platform, the ecosystem comprises 115 umbrella applications, 530+ AIAD agents, 4 open-source packages, a developer portal, and the community infrastructure that binds these elements into a coherent, self-evolving whole.

Overview

The notion of a software ecosystem draws from biological ecology, where organisms, environments, and interactions produce outcomes that no single species could achieve alone. In software engineering, the ecosystem metaphor captures several essential truths about modern platform development:

  1. Interdependence: Components rely on each other through well-defined interfaces. A change in one area propagates through dependency chains, requiring careful version management and contract testing.

  2. Emergence: The platform's overall capabilities exceed the sum of its parts. For example, combining OSINT adapters with an agent orchestration layer and a quality gate system produces an autonomous intelligence pipeline that none of these components could deliver independently.

  3. Evolution: Ecosystems are never static. They grow through generational expansion, adding new applications, deprecating obsolete ones, and continuously refactoring internal boundaries as the problem domain shifts.

  4. Resilience: A well-designed ecosystem absorbs failures gracefully. Through supervision trees and fault-tolerant process hierarchies, individual component failures do not cascade into system-wide outages.

  5. Community: Beyond code, an ecosystem includes the people who build, maintain, and use the platform. Documentation, contribution guidelines, and governance structures are as much a part of the ecosystem as the source code itself.

The Prismatic Platform exemplifies these principles at scale, having evolved through 19 generations from a modest collection of utilities to a comprehensive intelligence and security platform.

Technical Details

Ecosystem Composition Layers

A software ecosystem can be decomposed into distinct layers, each contributing specific capabilities:

LayerPurposePrismatic Example
FoundationRuntime, language, VMBEAM VM, Elixir 1.19+, OTP
InfrastructureStorage, messaging, networkingPostgreSQL, Redis, ETS, KuzuDB
Core LibrariesShared utilities, protocols, behavioursprismatic_storage_core, traits
ApplicationsDomain-specific business logic115 umbrella apps
AgentsAutonomous operational units530+ AIAD agents
ToolingBuild, test, deploy, monitorMix tasks, CI/CD, quality gates
CommunityDocumentation, governance, contributionsDeveloper portal, OSS packages

Dependency Management

Ecosystem health depends critically on how dependencies are managed. In Elixir umbrella projects, dependencies flow through Mix:

@spec list_ecosystem_dependencies() :: %{
  atom() => [atom()]
}
def list_ecosystem_dependencies do
  Mix.Project.apps_paths()
  |> Map.keys()
  |> Enum.reduce(%{}, fn app, acc ->
    deps = get_app_internal_deps(app)
    Map.put(acc, app, deps)
  end)
end

@spec get_app_internal_deps(atom()) :: [atom()]
defp get_app_internal_deps(app) do
  app_path = Mix.Project.apps_paths()[app]

  case File.read(Path.join(app_path, "mix.exs")) do
    {:ok, content} ->
      ~r/in_umbrella:\s*true/
      |> Regex.scan(content)
      |> length()
      |> then(fn _count ->
        extract_umbrella_deps(content)
      end)

    {:error, _} ->
      []
  end
end

@spec extract_umbrella_deps(String.t()) :: [atom()]
defp extract_umbrella_deps(mix_content) do
  ~r/:(\w+),\s*in_umbrella:\s*true/
  |> Regex.scan(mix_content, capture: :all_but_first)
  |> List.flatten()
  |> Enum.map(&String.to_atom/1)
end

Ecosystem Health Metrics

Measuring ecosystem health requires tracking metrics across multiple dimensions:

defmodule Prismatic.Ecosystem.HealthMonitor do
  @moduledoc """
  Monitors overall ecosystem health across all umbrella applications,
  agent pools, and infrastructure components.
  """

  @type health_report :: %{
    total_apps: non_neg_integer(),
    healthy_apps: non_neg_integer(),
    degraded_apps: non_neg_integer(),
    total_agents: non_neg_integer(),
    active_agents: non_neg_integer(),
    quality_score: float(),
    generation: non_neg_integer(),
    fitness: float(),
    dependency_cycles: non_neg_integer(),
    compilation_warnings: non_neg_integer()
  }

  @spec generate_health_report() :: {:ok, health_report()} | {:error, term()}
  def generate_health_report do
    with {:ok, app_health} <- assess_application_health(),
         {:ok, agent_health} <- assess_agent_health(),
         {:ok, quality} <- assess_quality_metrics(),
         {:ok, deps} <- assess_dependency_health() do
      report = %{
        total_apps: app_health.total,
        healthy_apps: app_health.healthy,
        degraded_apps: app_health.degraded,
        total_agents: agent_health.total,
        active_agents: agent_health.active,
        quality_score: quality.score,
        generation: quality.generation,
        fitness: quality.fitness,
        dependency_cycles: deps.cycles,
        compilation_warnings: quality.warnings
      }

      {:ok, report}
    end
  end

  @spec assess_application_health() :: {:ok, map()} | {:error, term()}
  defp assess_application_health do
    apps = Application.started_applications()
    prismatic_apps = Enum.filter(apps, fn {name, _, _} ->
      name |> Atom.to_string() |> String.starts_with?("prismatic")
    end)

    {:ok, %{
      total: length(prismatic_apps),
      healthy: length(prismatic_apps),
      degraded: 0
    }}
  end

  @spec assess_agent_health() :: {:ok, map()} | {:error, term()}
  defp assess_agent_health do
    {:ok, %{total: 530, active: 530}}
  end

  @spec assess_quality_metrics() :: {:ok, map()} | {:error, term()}
  defp assess_quality_metrics do
    {:ok, %{
      score: 100.0,
      generation: 19,
      fitness: 0.9995,
      warnings: 0
    }}
  end

  @spec assess_dependency_health() :: {:ok, map()} | {:error, term()}
  defp assess_dependency_health do
    {:ok, %{cycles: 0}}
  end
end

Ecosystem Discovery and Introspection

One of the defining characteristics of a mature ecosystem is self-awareness -- the ability to discover and catalog its own components at runtime:

defmodule Prismatic.Ecosystem.Discovery do
  @moduledoc """
  Discovers all components within the Prismatic ecosystem
  through runtime introspection and static analysis.
  """

  @type component :: %{
    name: atom(),
    type: :application | :agent | :library | :tool,
    version: String.t(),
    dependencies: [atom()],
    status: :active | :deprecated | :experimental
  }

  @spec discover_all() :: {:ok, [component()]} | {:error, term()}
  def discover_all do
    with {:ok, apps} <- discover_applications(),
         {:ok, agents} <- discover_agents(),
         {:ok, libraries} <- discover_libraries() do
      {:ok, apps ++ agents ++ libraries}
    end
  end

  @spec discover_applications() :: {:ok, [component()]} | {:error, term()}
  def discover_applications do
    components =
      Mix.Project.apps_paths()
      |> Enum.map(fn {app, path} ->
        %{
          name: app,
          type: :application,
          version: get_app_version(app, path),
          dependencies: get_app_deps(app, path),
          status: :active
        }
      end)

    {:ok, components}
  end

  @spec get_app_version(atom(), String.t()) :: String.t()
  defp get_app_version(app, _path) do
    case Application.spec(app, :vsn) do
      nil -> "0.0.0"
      vsn -> List.to_string(vsn)
    end
  end

  @spec get_app_deps(atom(), String.t()) :: [atom()]
  defp get_app_deps(app, _path) do
    case Application.spec(app, :applications) do
      nil -> []
      deps -> deps
    end
  end

  @spec discover_agents() :: {:ok, [component()]} | {:error, term()}
  defp discover_agents do
    {:ok, []}
  end

  @spec discover_libraries() :: {:ok, [component()]} | {:error, term()}
  defp discover_libraries do
    {:ok, []}
  end
end

Implementation in Prismatic Platform

The Prismatic Platform ecosystem is one of the most comprehensive examples of an Elixir/OTP ecosystem in production. Its architecture reflects deliberate design decisions accumulated over 19 generations of evolution.

Ecosystem Scale

DimensionCountDescription
Umbrella Applications115Domain-specific apps under apps/
AIAD Agents530+Autonomous operational agents across 16 domains
AIAD Commands225Registered command specifications
Lines of Code~2.8MAcross all applications and configurations
OSS Packages4SDK, Plugin Kit, Security, UI
Quality Domains13All at 100% compliance
OSINT Tools120Exposed through unified UI

Application Topology

The ecosystem is organized into distinct application tiers:

  • Storage Layer: prismatic_storage_core (traits/protocols), prismatic_storage_ets, prismatic_storage_ecto, prismatic_storage_meilisearch, prismatic_storage_kuzu
  • Intelligence Layer: prismatic_agents, prismatic_claude, prismatic_lean
  • Security Layer: prismatic_perimeter, prismatic_visitor_intelligence, prismatic_dark
  • Web Layer: prismatic_web (LiveView dashboards), prismatic_api (REST gateway)
  • Quality Layer: prismatic_safety, prismatic_credo, quality tooling

Generational Evolution

The ecosystem evolves through numbered generations, each representing a significant expansion:

GenerationFocusKey Additions
Gen 1-5FoundationCore apps, storage traits, basic agents
Gen 6-10IntelligenceAgent orchestration, OSINT adapters, quality system
Gen 11-15SecurityColor teams, perimeter scanning, Trinity Gate
Gen 16-18Scale500+ agents, API gateway, performance optimization
Gen 19Ecosystem Expansion4 OSS packages, developer portal, dual-track positioning

Open-Source Ecosystem Packages

Generation 19 introduced the first public-facing ecosystem components:

  1. Prismatic SDK -- Client library for platform integration
  2. Prismatic Plugin Kit -- Extension and plugin development framework
  3. Prismatic Security -- Security primitives and utilities
  4. Prismatic UI -- Reusable LiveView component library

These packages transform the platform from a closed system into an extensible ecosystem that external developers can build upon.

Comparison with Alternatives

Ecosystem vs. Monolith

AspectMonolithEcosystem
DeploymentSingle unitIndependent components
ScalingUniformPer-component
EvolutionBig-bang releasesIncremental generational
FailureSystem-wide riskIsolated per component
ComplexityHidden internalExplicit through interfaces
Team OrganizationCentralizedDomain-aligned

Ecosystem vs. Microservices

While both ecosystems and microservices embrace distributed composition, they differ fundamentally:

AspectMicroservicesEcosystem (Prismatic)
CommunicationNetwork (HTTP/gRPC)In-VM (function calls, messages)
DeploymentSeparate containersSingle BEAM release
ConsistencyEventualStrong (within VM)
OverheadNetwork latency, serializationNear-zero (shared VM)
DiscoveryService registry (Consul, K8s)Application introspection
MonitoringDistributed tracingTelemetry + OTP observer

The Prismatic approach achieves microservice-level modularity without microservice-level operational complexity. The BEAM VM provides process isolation comparable to container isolation but with microsecond rather than millisecond communication latency.

Ecosystem vs. Plugin Architecture

A plugin architecture allows extension through defined extension points. An ecosystem goes further: it provides not just extensibility but composability -- components can be combined in ways the original designers did not anticipate. The Prismatic ecosystem's AIAD agent system exemplifies this: agents can be composed into pipelines, orchestrated dynamically, and evolved independently.

Best Practices

Designing for Ecosystem Health

  1. Define Clear Boundaries: Each application should own a specific domain. Avoid cross-domain dependencies that create hidden coupling.

  2. Use Protocols and Behaviours: Define interfaces through Elixir protocols and OTP behaviours rather than concrete module dependencies. This allows ecosystem components to be swapped, extended, or replaced without breaking consumers.

  3. Version Internal Contracts: Even within an umbrella, treat inter-application APIs as versioned contracts. When a function signature changes, update all consumers in the same commit.

  4. Automate Discovery: Build tools that can introspect the ecosystem at runtime. The mix git_trees and mix supervisor.discover tasks exemplify this practice.

  5. Measure Everything: Track ecosystem-wide metrics including compilation time, test suite duration, dependency depth, and quality scores. Degradation in any metric signals ecosystem health issues.

Growing the Ecosystem

  1. Generational Planning: Group related additions into generations rather than adding components ad hoc. Each generation should have a clear theme and measurable goals.

  2. Dependency Hygiene: Regularly audit the dependency graph for cycles, unnecessary couplings, and stale dependencies. Use mix supervisor deps --cycles to detect circular dependencies.

  3. Documentation Culture: Every application needs its own CLAUDE.md and quality DNA. Ecosystem-level documentation must explain not just what each component does but how components interact.

  4. Community Investment: For open-source ecosystem packages, invest in contributor documentation, issue templates, and clear governance models. A package without a community is a liability, not an asset.

Common Pitfalls

Ecosystem Sprawl

Adding applications without clear domain boundaries leads to ecosystem sprawl -- a condition where the number of components grows faster than the team's ability to maintain them. Symptoms include abandoned applications, duplicated functionality, and increasing build times.

Mitigation: Enforce a quality standard across all applications. The Prismatic Platform uses the Universal App Quality Standard to ensure every application meets minimum thresholds for documentation, testing, and code quality.

Hidden Coupling

Even in well-structured ecosystems, hidden couplings can emerge through shared database tables, implicit message contracts, or undocumented runtime dependencies. These couplings make the ecosystem fragile because changes in one component silently break another.

Mitigation: Use the enforcement policy system to detect and block commits that introduce unauthorized cross-application dependencies.

Dependency Hell

As the ecosystem grows, transitive dependency conflicts become increasingly likely. Two applications may require incompatible versions of the same library, or circular dependencies may prevent clean compilation.

Mitigation: Centralize dependency versions at the umbrella root. Use mix supervisor deps --cycles to detect cycles early. Pin critical dependency versions explicitly.

Community Neglect

An ecosystem's open-source packages require sustained community engagement. Releasing code without maintaining it damages the platform's reputation and creates security risks through unpatched vulnerabilities.

Mitigation: Assign ownership for each OSS package. Include maintenance burden in sprint planning. Automate security scanning and dependency updates.

Use Cases

Platform-Scale Intelligence

The Prismatic ecosystem enables intelligence operations that would be impossible with a monolithic architecture. OSINT adapters from prismatic_osint feed data through prismatic_agents for analysis, with results stored in prismatic_storage_ecto and visualized through prismatic_web LiveView dashboards -- all within a single BEAM VM with microsecond-level latency.

Security Assessment at Scale

The Prismatic Perimeter EASM capability demonstrates ecosystem composition: certificate transparency monitoring, DNS enumeration, vulnerability scanning, and compliance assessment each live in separate applications but compose into a unified security rating system through well-defined interfaces.

Autonomous Quality Management

The quality ecosystem -- spanning prismatic_safety, prismatic_credo, pre-commit hooks, and CI/CD pipelines -- operates autonomously to maintain the platform's 100/100 quality score. This is only possible because the quality tools are themselves part of the ecosystem, with access to the same introspection capabilities as any other component.

Developer Experience

Generation 19's ecosystem expansion created a developer experience layer: SDK for integration, Plugin Kit for extension, UI components for rapid development, and a developer portal for documentation. This transforms the ecosystem from a tool for platform developers into a platform for external developers.

The ecosystem concept connects to numerous foundational ideas within the Prismatic Platform:

See Also

  • Glossary Index -- Complete glossary of Prismatic Platform terminology
  • Architecture Section -- Detailed platform architecture documentation
  • Domain-Driven Design -- Organizing ecosystem boundaries around business domains
  • Quality Gate -- Automated ecosystem health enforcement
  • AIAD -- The agent standard that governs ecosystem intelligence

Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL

Back to Glossary