Overview
The replication-specialist operates as an L3 Strategic Command authority within the Prismatic Platform's infrastructure domain, serving as the expert agent for database replication setup, configuration, monitoring, and management across the platform's PostgreSQL infrastructure. In a platform processing intelligence data across multiple storage backends, replication is not merely a backup mechanism -- it provides read scaling, geographic distribution, disaster recovery, and zero-downtime maintenance capabilities essential for production reliability.
The replication-specialist manages PostgreSQL streaming replication, logical replication for selective data distribution, and ETS table replication for in-memory state synchronization across distributed BEAM nodes. Each replication topology serves different operational requirements: streaming replication provides byte-level consistency for disaster recovery, logical replication enables selective data distribution to analytical systems, and ETS replication ensures in-memory cache coherence across cluster nodes.
Built on the AIAD standard and governed by the NO MERCY, NO DOUBTS doctrine, the replication-specialist enforces strict data consistency guarantees. Replication lag monitoring is continuous, and the agent ensures that no client reads stale data without explicit acknowledgment of eventual consistency semantics. The NO DOUBTS principle mandates that replication health claims are backed by measurable metrics, not assumed from configuration correctness.
Replication Architecture
The platform's replication infrastructure operates at three levels, each serving distinct operational requirements.
PostgreSQL streaming replication provides physical replication of the primary database to one or more standby servers. The standby servers maintain byte-level copies of the primary, enabling rapid failover with minimal data loss (bounded by the synchronous_commit configuration). Streaming replication supports both synchronous mode (zero data loss, higher latency) and asynchronous mode (minimal latency, bounded data loss window). The replication-specialist manages the recovery.conf configuration, monitors replication lag through pg_stat_replication, and orchestrates failover procedures when primary database unavailability is detected.
PostgreSQL logical replication enables selective table-level replication from the primary database to subscriber databases. This supports scenarios including analytical database synchronization, cross-region data distribution, and major version upgrade testing. Logical replication operates through WAL decoding and publication/subscription mechanisms, allowing the specialist to configure precisely which tables and operations (INSERT, UPDATE, DELETE) are replicated.
ETS table replication synchronizes in-memory state across distributed BEAM nodes. The platform uses ETS extensively for caching (API catalog, agent registry, quality state), and the replication-specialist ensures cache coherence when multiple nodes serve the same data. ETS replication uses the platform's GenServer-based synchronization protocol with conflict resolution strategies appropriate to each cache type.
Key Capabilities
- Streaming replication management -- Configures and monitors PostgreSQL streaming replication including WAL shipping, standby promotion, and automatic failover with bounded data loss guarantees
- Logical replication orchestration -- Manages publication/subscription configurations for selective table replication, supporting analytical synchronization and cross-region distribution
- ETS cache coherence -- Implements distributed ETS table synchronization across BEAM cluster nodes with configurable conflict resolution strategies
- Replication lag monitoring -- Provides real-time replication lag measurement across all replication streams with configurable alerting thresholds
- Failover automation -- Orchestrates automatic failover from primary to standby with connection draining, standby promotion, and application connection reconfiguration
- Circuit breaker integration -- Applies circuit breaker patterns to replication streams, preventing cascading failures when replication targets become unavailable
- Autonomous operation with continuous replication health monitoring and automatic recovery
- Telemetry integration for replication lag, throughput, and health metric tracking
Replication Topology
| Stream Type | Source | Target | Consistency | Use Case |
|---|---|---|---|---|
| Streaming Sync | Primary PG | Standby PG | Strong | Disaster recovery |
| Streaming Async | Primary PG | Read replica | Eventual | Read scaling |
| Logical | Primary PG | Analytics DB | Eventual | Analytical queries |
| ETS Sync | Leader node | Follower nodes | Eventual | Cache coherence |
| KuzuDB Sync | Primary | Mirror | Eventual | Graph query scaling |
Implementation Architecture
defmodule PrismaticInfra.ReplicationSpecialist do
@moduledoc """
Database replication management agent handling PostgreSQL
streaming/logical replication and ETS cache synchronization.
"""
use GenServer
alias PrismaticInfra.{StreamingReplication, LogicalReplication, ETSSync}
@type replication_status :: %{
streaming: %{lag_bytes: non_neg_integer(), state: atom()},
logical: %{subscriptions: non_neg_integer(), state: atom()},
ets: %{synced_tables: non_neg_integer(), conflicts: non_neg_integer()}
}
@spec health_check() :: {:ok, replication_status()} | {:error, term()}
def health_check do
with {:ok, streaming} <- StreamingReplication.status(),
{:ok, logical} <- LogicalReplication.status(),
{:ok, ets} <- ETSSync.status() do
{:ok, %{streaming: streaming, logical: logical, ets: ets}}
end
end
@spec failover(atom()) :: {:ok, atom()} | {:error, term()}
def failover(target_standby) do
with :ok <- drain_connections(),
:ok <- StreamingReplication.promote(target_standby),
:ok <- reconfigure_connections(target_standby) do
{:ok, target_standby}
end
end
endReplication Monitoring Metrics
| Metric | Warning Threshold | Critical Threshold | Measurement |
|---|---|---|---|
| Streaming Lag (bytes) | >1 MB | >10 MB | pg_stat_replication |
| Streaming Lag (time) | >5 seconds | >30 seconds | WAL timestamp delta |
| Logical Lag (LSN) | >100 MB | >1 GB | Subscription state |
| ETS Sync Conflicts | >10/minute | >100/minute | Conflict counter |
| Replication Slot Inactive | >5 minutes | >30 minutes | Slot activity check |
Failover Protocol
| Phase | Duration | Actions | Data Loss Risk |
|---|---|---|---|
| Detection | <10 seconds | Health check failure, confirmed by retry | None |
| Drainage | <30 seconds | Connection draining, new connections refused | None |
| Promotion | <5 seconds | Standby promoted to primary | Bounded by async lag |
| Reconfiguration | <15 seconds | Application connections redirected | None |
| Verification | <30 seconds | Health checks, replication re-establishment | None |
| Total | <90 seconds | End-to-end automatic failover | Minimal |
Authority Level
L3 - Strategic Command - Multi-domain coordination with authority to configure replication topologies, execute failover procedures, and manage cross-node data synchronization.
Command Interface
| Command | Description | Authority |
|---|---|---|
/replication status | Display replication health across all streams | L3+ |
/replication failover | Execute failover to specified standby | L3+ |
/replication configure | Configure replication topology for a database | L3+ |
/replication lag | Display real-time replication lag metrics | L3+ |
Coordination
| Agent | Relationship |
|---|---|
| prismatic-supreme-commander | Failover decisions escalated for supreme approval |
| quality-enforcement-commander | Replication health is a quality metric |
| prismatic-api-introspector | API catalog ETS replication across cluster nodes |
| route-testing-supreme | Route testing verifies failover does not break endpoints |
Enforcement
Replication management operates under the NO MERCY doctrine: replication lag exceeding critical thresholds triggers immediate automated response, and no deployment is permitted when replication health is degraded. The NO DOUBTS principle requires that replication status claims are backed by actual lag measurements, not configuration assumptions. The Trinity Gate validates replication topology configurations for structural consistency before application.
Connect & Contribute
Created by Tomáš Korcak (korczis) | Open Source under GHL