Continuous Deployment

quality

Continuous Deployment

Definition and Overview

Continuous Deployment (CD) is the practice of automatically deploying code changes to production after they pass all automated quality checks in the CI pipeline. Every validated commit is a potential release, eliminating manual release processes and reducing the time from code change to production availability. CD requires high confidence in automated testing and quality enforcement to safely release without human gatekeeping.

Continuous Deployment represents the final stage of the deployment automation spectrum. While Continuous Integration ensures code is built and tested automatically, and Continuous Delivery ensures code is always in a deployable state, Continuous Deployment takes the additional step of automatically releasing every validated change to production. This eliminates the "last mile" of manual intervention that often becomes a bottleneck in software delivery.

The practice is grounded in the principle that smaller, more frequent deployments are inherently safer than large, infrequent releases. Each individual deployment carries less risk because it contains fewer changes, making it easier to identify the cause of any regression. Combined with automated rollback capabilities, CD creates a deployment cadence where the cost of a failed deployment is measured in minutes, not hours or days.

Technical Deep Dive

CD Pipeline Stages

A typical Continuous Deployment pipeline follows a linear progression from code change to production:

StagePurposeBlocking?Duration
Source ControlTrigger on commit/mergeYesInstant
BuildCompile code, resolve dependenciesYes1-5 min
Unit TestsTest individual componentsYes2-10 min
Integration TestsTest component interactionsYes5-15 min
Static AnalysisLint, type check, security scanYes2-10 min
Staging DeployDeploy to staging environmentYes2-5 min
Staging ValidationSmoke tests, health checksYes5-15 min
Production DeployDeploy to productionYes2-5 min
Production ValidationHealth checks, monitoringYes5-15 min

Deployment Frequency Spectrum

PracticeDeployment FrequencyHuman Gatekeeping
Manual DeployMonthly to quarterlyFull manual process
Continuous IntegrationMultiple times daily (build/test only)Manual deploy decision
Continuous DeliveryAlways deployable, manual triggerHuman approval required
Continuous DeploymentEvery commit that passes CIFully automated

Prerequisites for CD

Continuous Deployment requires several foundational capabilities:

PrerequisitePurposePrismatic Implementation
Comprehensive test suiteCatch regressions automatically5,864 test files, 100% coverage target
Quality gatesPrevent quality degradation13 quality domains, zero-tolerance
Feature flagsDecouple deploy from releaseRuntime feature toggles
Automated rollbackRecover from failed deploymentsFly.io automatic rollback
Infrastructure as codeReproducible environmentsDocker + fly.toml
Monitoring and alertingDetect production issues quicklyTelemetry, Quality Floor Guardian
Database migration safetyPrevent data loss during deploysEcto migrations, expand-contract

Rollback Strategies

StrategySpeedData RiskComplexity
Revert deploy (previous version)Fast (seconds)LowLow
Database rollbackSlow (minutes)HighHigh
Feature flag toggleInstantNoneModerate
Blue-green switchInstantLowModerate
Forward fixVariableLowModerate

Architecture and Implementation

GitLab CI/CD Pipeline Configuration

# .gitlab-ci.yml - Prismatic Platform CD pipeline
stages:
  - build
  - test
  - quality
  - staging
  - production

build:
  stage: build
  script:
    - mix deps.get
    - mix compile --warnings-as-errors

test:
  stage: test
  script:
    - mix test --cover
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: cover/cobertura.xml

quality:
  stage: quality
  script:
    - mix credo --strict
    - mix dialyzer
    - mix quality.gates

deploy_staging:
  stage: staging
  script:
    - flyctl deploy --app prismatic-staging
    - flyctl checks wait --app prismatic-staging
  environment:
    name: staging
    url: https://prismatic-staging.fly.dev

deploy_production:
  stage: production
  script:
    - flyctl deploy --app prismatic-prod
    - flyctl checks wait --app prismatic-prod
  environment:
    name: production
    url: https://prismatic-prod.fly.dev
  when: on_success

Deployment Orchestration in Elixir

defmodule PrismaticDeploy.Pipeline do
  @moduledoc """
  Orchestrates the continuous deployment pipeline stages.
  Each stage must pass before the next begins.
  """

  @stages [
    :compile,
    :test,
    :quality_gates,
    :staging_deploy,
    :staging_validate,
    :production_deploy,
    :production_validate
  ]

  @spec execute(map()) :: {:ok, map()} | {:error, atom(), term()}
  def execute(context) do
    Enum.reduce_while(@stages, {:ok, context}, fn stage, {:ok, ctx} ->
      case execute_stage(stage, ctx) do
        {:ok, updated_ctx} ->
          Logger.info("Stage #{stage}: PASSED")
          {:cont, {:ok, updated_ctx}}

        {:error, reason} ->
          Logger.error("Stage #{stage}: FAILED - #{inspect(reason)}")
          trigger_rollback(stage, ctx)
          {:halt, {:error, stage, reason}}
      end
    end)
  end

  defp execute_stage(:quality_gates, ctx) do
    results = for domain <- quality_domains() do
      {domain, check_domain(domain)}
    end

    failures = Enum.filter(results, fn {_, status} -> status != :pass end)

    case failures do
      [] -> {:ok, Map.put(ctx, :quality_results, results)}
      _ -> {:error, {:quality_gate_failures, failures}}
    end
  end

  defp execute_stage(:production_validate, ctx) do
    health_url = "#{ctx.production_url}/health"

    case HTTPoison.get(health_url, [], recv_timeout: 10_000) do
      {:ok, %{status_code: 200}} -> {:ok, ctx}
      {:ok, %{status_code: status}} -> {:error, {:health_check_failed, status}}
      {:error, reason} -> {:error, {:health_check_unreachable, reason}}
    end
  end
end

Quality Gate Integration

defmodule PrismaticDeploy.QualityGate do
  @moduledoc """
  Enforces quality gates as a blocking stage in the CD pipeline.
  All 13 quality domains must pass for deployment to proceed.
  """

  @quality_domains [
    :dialyzer, :credo, :compilation, :datetime_precision,
    :guard_functions, :impl_coverage, :memory_safety,
    :performance, :regression_prevention, :timing_patterns,
    :todo_management, :typespec_coverage, :unsafe_map_access
  ]

  @spec check_all() :: :pass | {:fail, [atom()]}
  def check_all do
    failures = Enum.filter(@quality_domains, fn domain ->
      check_domain(domain) != :pass
    end)

    case failures do
      [] -> :pass
      domains -> {:fail, domains}
    end
  end
end

Automated Rollback

defmodule PrismaticDeploy.Rollback do
  @moduledoc """
  Handles automated rollback when production deployments fail.
  """

  @spec rollback(String.t(), map()) :: {:ok, String.t()} | {:error, term()}
  def rollback(app_name, context) do
    Logger.warning("Initiating rollback for #{app_name}")

    with {:ok, previous_version} <- get_previous_version(app_name),
         {:ok, _} <- deploy_version(app_name, previous_version),
         :ok <- validate_health(app_name) do
      Logger.info("Rollback to #{previous_version} successful")
      notify_team(:rollback_complete, app_name, context)
      {:ok, previous_version}
    else
      {:error, reason} ->
        Logger.error("Rollback failed: #{inspect(reason)}")
        escalate_to_on_call(app_name, reason)
        {:error, reason}
    end
  end
end

Usage in Prismatic Platform

The Prismatic Platform implements continuous deployment through GitLab CI/CD pipelines deploying to Fly.io infrastructure.

Pipeline Flow

Developer commits -> GitLab CI triggers
  |
  +-- Build: mix compile --warnings-as-errors (ZERO warnings)
  |
  +-- Test: mix test --cover (5,864 test files)
  |
  +-- Quality: 13 domains checked (0 violations)
  |     - Dialyzer, Credo, Compilation
  |     - DateTime, Guard Functions, @impl
  |     - Memory Safety, Performance, Regression
  |     - Timing, TODO, Typespec, Unsafe Map
  |
  +-- Stage: Deploy to prismatic-staging.fly.dev
  |     - Health check validation
  |     - Smoke tests
  |
  +-- Production: Deploy to prismatic-prod.fly.dev
        - Blue-green deployment
        - Health check validation
        - Automatic rollback on failure

Pre-Commit Quality Enforcement

The platform implements a two-layer quality system that catches issues before they reach the CD pipeline:

# .githooks/pre-commit runs:
# 1. Compile with warnings-as-errors
# 2. Credo strict check on changed files
# 3. Quality gate fast check
# 4. Template validation (promo site)
# 5. Design consistency check

Deployment Metrics

MetricTargetCurrent
Deploy frequencyMultiple dailyAchieved
Lead time (commit to production)< 30 minutes~20 minutes
Change failure rate< 5%< 2%
Mean time to recovery< 15 minutes< 10 minutes
Quality gate pass rate100%100%

Best Practices

  1. Invest in test quality, not just coverage -- High test coverage with weak assertions provides false confidence. Tests must catch real regressions to make CD safe.

  2. Make deployments boring -- The best CD pipeline is one nobody thinks about because it just works. Automate everything, including rollback.

  3. Monitor continuously -- CD moves the quality assurance focus from pre-deployment gates to post-deployment monitoring. Invest heavily in observability.

  4. Keep deployments small -- Smaller deployments are easier to reason about, faster to roll back, and less likely to cause issues.

  5. Feature flags for incomplete work -- Use feature flags to deploy code that is not yet ready for users. This allows trunk-based development with CD.

  6. Maintain environment parity -- Staging must closely mirror production to catch environment-specific issues before they reach users.

Advanced CD Patterns

Progressive Delivery Strategies

Continuous Deployment can be enhanced with progressive delivery techniques that reduce risk:

defmodule PrismaticDeploy.ProgressiveDelivery do
  @moduledoc """
  Implements progressive delivery patterns for risk reduction.
  """

  @deployment_strategies [
    canary: %{traffic_percentage: 5, duration_minutes: 15},
    blue_green: %{cutover_strategy: :instant, validation_minutes: 10},
    rolling: %{batch_size: 2, wait_between_batches: 300},
    feature_flag: %{rollout_percentage: 10, increment_interval: 3600}
  ]

  def deploy_with_strategy(strategy, deployment_config) do
    strategy_config = @deployment_strategies[strategy]

    case strategy do
      :canary -> deploy_canary(deployment_config, strategy_config)
      :blue_green -> deploy_blue_green(deployment_config, strategy_config)
      :rolling -> deploy_rolling(deployment_config, strategy_config)
      :feature_flag -> deploy_with_feature_flag(deployment_config, strategy_config)
    end
  end

  defp deploy_canary(config, strategy_config) do
    # Deploy to small percentage of traffic
    with {:ok, _} <- deploy_canary_version(config.version, strategy_config.traffic_percentage),
         :ok <- monitor_canary_health(config.version, strategy_config.duration_minutes),
         :ok <- validate_canary_metrics(config.version) do

      # Full rollout if canary succeeds
      deploy_full_version(config.version)
    else
      {:error, reason} ->
        Logger.error("Canary deployment failed: #{inspect(reason)}")
        rollback_canary(config.previous_version)
        {:error, :canary_failed}
    end
  end

  defp monitor_canary_health(version, duration_minutes) do
    end_time = System.system_time(:second) + duration_minutes * 60

    monitor_loop(version, end_time)
  end

  defp monitor_loop(version, end_time) do
    if System.system_time(:second) >= end_time do
      :ok
    else
      case check_canary_metrics(version) do
        :healthy ->
          :timer.sleep(30_000)  # Check every 30 seconds
          monitor_loop(version, end_time)

        :unhealthy ->
          {:error, :canary_unhealthy}

        :degraded ->
          Logger.warning("Canary showing degraded performance")
          :timer.sleep(30_000)
          monitor_loop(version, end_time)
      end
    end
  end

  defp check_canary_metrics(version) do
    metrics = %{
      error_rate: get_error_rate(version),
      response_time_p95: get_response_time_p95(version),
      throughput: get_throughput(version)
    }

    cond do
      metrics.error_rate > 0.05 -> :unhealthy
      metrics.response_time_p95 > 500 -> :unhealthy
      metrics.throughput < 0.8 -> :degraded
      true -> :healthy
    end
  end
end

Database Migration Strategies

Safe database migrations are crucial for CD:

defmodule PrismaticDeploy.DatabaseMigration do
  @moduledoc """
  Handles database migrations in CD pipelines using expand-contract pattern.
  """

  def execute_migration_strategy(migration, strategy \\ :expand_contract) do
    case strategy do
      :expand_contract -> execute_expand_contract(migration)
      :blue_green_db -> execute_blue_green_db(migration)
      :online_ddl -> execute_online_ddl(migration)
      :maintenance_window -> execute_maintenance_window(migration)
    end
  end

  defp execute_expand_contract(migration) do
    # Phase 1: Expand - Add new structures without breaking old code
    with {:ok, _} <- expand_schema(migration),
         {:ok, _} <- deploy_code_supporting_both_schemas(),
         {:ok, _} <- migrate_data_to_new_schema(),
         {:ok, _} <- validate_data_consistency() do

      # Phase 2: Contract - Remove old structures after code fully migrated
      schedule_contract_phase(migration)
      {:ok, :expand_complete}
    else
      {:error, reason} ->
        Logger.error("Expand phase failed: #{inspect(reason)}")
        rollback_expand(migration)
        {:error, reason}
    end
  end

  defp expand_schema(migration) do
    case migration.operation do
      :add_column -> add_column_safely(migration)
      :rename_column -> add_new_column_for_rename(migration)
      :change_column_type -> add_new_typed_column(migration)
      :add_index -> add_index_concurrently(migration)
      :add_table -> create_table(migration)
    end
  end

  defp add_column_safely(migration) do
    # Add column with default value to avoid blocking
    sql = """
    ALTER TABLE #{migration.table}
    ADD COLUMN #{migration.column_name} #{migration.column_type}
    DEFAULT #{migration.default_value};
    """

    Ecto.Adapters.SQL.query(PrismaticRepo, sql)
  end

  defp add_index_concurrently(migration) do
    # Use CONCURRENTLY to avoid table locks
    sql = """
    CREATE INDEX CONCURRENTLY #{migration.index_name}
    ON #{migration.table} (#{Enum.join(migration.columns, ", ")});
    """

    Ecto.Adapters.SQL.query(PrismaticRepo, sql)
  end

  defp validate_data_consistency do
    # Check that old and new schemas contain same data
    inconsistencies = run_consistency_checks()

    if Enum.empty?(inconsistencies) do
      {:ok, :consistent}
    else
      {:error, {:data_inconsistent, inconsistencies}}
    end
  end

  defp schedule_contract_phase(migration) do
    # Schedule the contract phase for later (e.g., next deploy cycle)
    ContractPhaseScheduler.schedule(migration, delay_hours: 24)
  end
end

Environment Parity Management

Ensuring staging mirrors production:

defmodule PrismaticDeploy.EnvironmentParity do
  @moduledoc """
  Ensures staging environment closely mirrors production for reliable testing.
  """

  @parity_checks [
    :elixir_version,
    :erlang_version,
    :dependency_versions,
    :environment_variables,
    :database_schema,
    :infrastructure_config,
    :resource_allocation,
    :network_topology
  ]

  def verify_environment_parity(staging_env, production_env) do
    results = Enum.map(@parity_checks, fn check ->
      {check, perform_parity_check(check, staging_env, production_env)}
    end)

    mismatches = Enum.filter(results, fn {_check, result} -> result != :match end)

    case mismatches do
      [] -> {:ok, :environments_match}
      _ -> {:error, {:parity_mismatches, mismatches}}
    end
  end

  defp perform_parity_check(:elixir_version, staging, production) do
    staging_version = get_elixir_version(staging)
    production_version = get_elixir_version(production)

    if Version.match?(staging_version, production_version) do
      :match
    else
      {:mismatch, staging: staging_version, production: production_version}
    end
  end

  defp perform_parity_check(:dependency_versions, staging, production) do
    staging_lock = get_mix_lock(staging)
    production_lock = get_mix_lock(production)

    differences = find_dependency_differences(staging_lock, production_lock)

    if Enum.empty?(differences) do
      :match
    else
      {:mismatch, differences: differences}
    end
  end

  defp perform_parity_check(:database_schema, staging, production) do
    staging_schema = get_database_schema(staging)
    production_schema = get_database_schema(production)

    schema_diff = compare_schemas(staging_schema, production_schema)

    if Enum.empty?(schema_diff) do
      :match
    else
      {:mismatch, schema_differences: schema_diff}
    end
  end

  def synchronize_environments(source_env, target_env, options \\ []) do
    sync_operations = [
      {:dependencies, &sync_dependencies/2},
      {:environment_variables, &sync_env_vars/2},
      {:database_schema, &sync_database_schema/2},
      {:configuration, &sync_configuration/2}
    ]

    Enum.reduce_while(sync_operations, {:ok, []}, fn {operation, sync_fn}, {:ok, acc} ->
      case sync_fn.(source_env, target_env) do
        :ok ->
          {:cont, {:ok, [operation | acc]}}

        {:error, reason} ->
          {:halt, {:error, {operation, reason}}}
      end
    end)
  end
end

Deployment Orchestration at Scale

Managing deployments across multiple services:

defmodule PrismaticDeploy.OrchestrationEngine do
  use GenStateMachine, callback_mode: :handle_event_function

  @services [
    :prismatic_storage_core,
    :prismatic_web,
    :prismatic_api,
    :prismatic_perimeter,
    :prismatic_agents
  ]

  defstruct [
    :deployment_id,
    :target_version,
    :services_status,
    :rollback_versions,
    :start_time,
    :timeout_timer
  ]

  def start_link(deployment_config) do
    GenStateMachine.start_link(__MODULE__, deployment_config, name: __MODULE__)
  end

  def init(config) do
    state_data = %__MODULE__{
      deployment_id: config.deployment_id,
      target_version: config.target_version,
      services_status: initialize_services_status(),
      rollback_versions: config.rollback_versions,
      start_time: System.system_time(),
      timeout_timer: nil
    }

    {:ok, :planning, state_data}
  end

  def handle_event(:enter, :planning, state_data) do
    # Plan deployment order based on dependencies
    deployment_plan = create_deployment_plan(@services, state_data.target_version)

    # Start with the first service
    timer = :erlang.start_timer(30_000, self(), :deployment_timeout)
    new_state_data = %{state_data | timeout_timer: timer}

    {:next_state, :deploying, new_state_data, [{:next_event, :internal, {:deploy_next, deployment_plan}}]}
  end

  def handle_event(:internal, {:deploy_next, [service | remaining_services]}, state_data) do
    case deploy_service(service, state_data.target_version) do
      {:ok, deployment_info} ->
        updated_status = Map.put(state_data.services_status, service, :deployed)
        new_state_data = %{state_data | services_status: updated_status}

        if Enum.empty?(remaining_services) do
          {:next_state, :validating, new_state_data, [{:next_event, :internal, :validate_deployment}]}
        else
          actions = [
            {:next_event, :internal, {:deploy_next, remaining_services}},
            {:state_timeout, 5000, :check_service_health}
          ]
          {:keep_state, new_state_data, actions}
        end

      {:error, reason} ->
        Logger.error("Failed to deploy #{service}: #{inspect(reason)}")
        {:next_state, :rolling_back, state_data, [{:next_event, :internal, :initiate_rollback}]}
    end
  end

  def handle_event(:internal, :validate_deployment, state_data) do
    # Validate all services are healthy
    validation_results = Enum.map(@services, fn service ->
      {service, validate_service_health(service)}
    end)

    failed_validations = Enum.filter(validation_results, fn {_service, result} -> result != :healthy end)

    if Enum.empty?(failed_validations) do
      {:next_state, :completed, state_data}
    else
      Logger.error("Validation failed for services: #{inspect(failed_validations)}")
      {:next_state, :rolling_back, state_data, [{:next_event, :internal, :initiate_rollback}]}
    end
  end

  def handle_event(:internal, :initiate_rollback, state_data) do
    # Rollback all deployed services
    rollback_tasks = Enum.map(@services, fn service ->
      case Map.get(state_data.services_status, service) do
        :deployed ->
          rollback_version = Map.get(state_data.rollback_versions, service)
          Task.async(fn -> rollback_service(service, rollback_version) end)

        _ ->
          nil
      end
    end)
    |> Enum.reject(&is_nil/1)

    # Wait for all rollbacks to complete
    rollback_results = Task.await_many(rollback_tasks)

    failed_rollbacks = Enum.filter(rollback_results, fn
      {:ok, _} -> false
      _ -> true
    end)

    if Enum.empty?(failed_rollbacks) do
      {:next_state, :rolled_back, state_data}
    else
      {:next_state, :failed, state_data}
    end
  end

  def handle_event({:timeout, timer_ref}, :deployment_timeout, %{timeout_timer: timer_ref} = state_data) do
    Logger.error("Deployment timeout exceeded")
    {:next_state, :rolling_back, state_data, [{:next_event, :internal, :initiate_rollback}]}
  end

  defp deploy_service(service, version) do
    Logger.info("Deploying #{service} to version #{version}")

    # Service-specific deployment logic
    case service do
      :prismatic_storage_core -> deploy_storage_core(version)
      :prismatic_web -> deploy_web_service(version)
      :prismatic_api -> deploy_api_service(version)
      :prismatic_perimeter -> deploy_perimeter_service(version)
      :prismatic_agents -> deploy_agents_service(version)
    end
  end

  defp validate_service_health(service) do
    health_endpoint = get_health_endpoint(service)

    case HTTPoison.get(health_endpoint, [], recv_timeout: 10_000) do
      {:ok, %{status_code: 200, body: body}} ->
        case Jason.decode(body) do
          {:ok, %{"status" => "healthy"}} -> :healthy
          {:ok, %{"status" => "degraded"}} -> :degraded
          _ -> :unhealthy
        end

      {:ok, %{status_code: _}} -> :unhealthy
      {:error, _} -> :unreachable
    end
  end
end

Monitoring and Observability Integration

Integrating CD with comprehensive monitoring:

defmodule PrismaticDeploy.DeploymentObservability do
  @moduledoc """
  Provides comprehensive observability for continuous deployment pipelines.
  """

  def track_deployment_metrics(deployment_id, stage, metrics) do
    :telemetry.execute(
      [:prismatic_deploy, :stage, :complete],
      metrics,
      %{deployment_id: deployment_id, stage: stage}
    )

    # Store metrics for analysis
    PrismaticTelemetry.record_deployment_metric(deployment_id, stage, metrics)
  end

  def calculate_deployment_success_rate(time_window_hours \\ 24) do
    since = System.system_time() - time_window_hours * 3600

    deployments = PrismaticTelemetry.get_deployments_since(since)

    total_deployments = length(deployments)
    successful_deployments = Enum.count(deployments, &(&1.status == :successful))

    if total_deployments > 0 do
      success_rate = successful_deployments / total_deployments
      {:ok, %{success_rate: success_rate, total: total_deployments, successful: successful_deployments}}
    else
      {:ok, %{success_rate: 0.0, total: 0, successful: 0}}
    end
  end

  def analyze_deployment_trends(days \\ 7) do
    end_time = System.system_time()
    start_time = end_time - days * 24 * 3600

    deployments = PrismaticTelemetry.get_deployments_in_range(start_time, end_time)

    trends = %{
      daily_frequency: calculate_daily_frequency(deployments, days),
      average_duration: calculate_average_duration(deployments),
      failure_patterns: analyze_failure_patterns(deployments),
      performance_trends: analyze_performance_trends(deployments)
    }

    {:ok, trends}
  end

  defp calculate_daily_frequency(deployments, days) do
    deployment_count = length(deployments)
    deployment_count / days
  end

  defp analyze_failure_patterns(deployments) do
    failed_deployments = Enum.filter(deployments, &(&1.status == :failed))

    failure_reasons = Enum.frequencies_by(failed_deployments, & &1.failure_reason)
    failure_stages = Enum.frequencies_by(failed_deployments, & &1.failed_stage)

    %{
      most_common_reasons: Enum.take(Enum.sort_by(failure_reasons, &elem(&1, 1), :desc), 5),
      most_problematic_stages: Enum.take(Enum.sort_by(failure_stages, &elem(&1, 1), :desc), 3),
      total_failures: length(failed_deployments)
    }
  end
end

Security Integration in CD Pipeline

Implementing security checks in the deployment pipeline:

defmodule PrismaticDeploy.SecurityGate do
  @moduledoc """
  Implements security checks as part of the CD pipeline.
  """

  @security_checks [
    :dependency_vulnerabilities,
    :secrets_scanning,
    :container_scanning,
    :sast_analysis,
    :compliance_validation
  ]

  def run_security_gate(deployment_context) do
    results = Enum.map(@security_checks, fn check ->
      {check, run_security_check(check, deployment_context)}
    end)

    failures = Enum.filter(results, fn {_check, result} -> not success?(result) end)

    case failures do
      [] ->
        {:ok, :security_gate_passed}

      _ ->
        Logger.error("Security gate failures: #{inspect(failures)}")
        {:error, {:security_violations, failures}}
    end
  end

  defp run_security_check(:dependency_vulnerabilities, context) do
    # Check for known vulnerabilities in dependencies
    case System.cmd("mix", ["deps.audit"]) do
      {output, 0} -> {:ok, :no_vulnerabilities}
      {output, _} -> {:error, {:vulnerabilities_found, parse_audit_output(output)}}
    end
  end

  defp run_security_check(:secrets_scanning, context) do
    # Scan for accidentally committed secrets
    secrets_patterns = [
      ~r/PRIVATE[_-]KEY/,
      ~r/-----BEGIN.+PRIVATE.+KEY-----/,
      ~r/password\s*[:=]\s*["'][^"']+["']/i,
      ~r/secret\s*[:=]\s*["'][^"']+["']/i,
      ~r/token\s*[:=]\s*["'][^"']+["']/i
    ]

    files_to_scan = get_changed_files(context)

    violations = Enum.flat_map(files_to_scan, fn file ->
      content = File.read!(file)
      find_secret_violations(file, content, secrets_patterns)
    end)

    case violations do
      [] -> {:ok, :no_secrets_found}
      _ -> {:error, {:secrets_detected, violations}}
    end
  end

  defp run_security_check(:container_scanning, context) do
    case System.cmd("trivy", ["image", context.container_image]) do
      {output, 0} ->
        vulnerabilities = parse_trivy_output(output)
        high_severity = Enum.filter(vulnerabilities, &(&1.severity in ["HIGH", "CRITICAL"]))

        case high_severity do
          [] -> {:ok, :no_critical_vulnerabilities}
          _ -> {:error, {:critical_vulnerabilities, high_severity}}
        end

      {output, _} ->
        {:error, {:scan_failed, output}}
    end
  end

  defp run_security_check(:compliance_validation, context) do
    compliance_checks = [
      check_gdpr_compliance(context),
      check_data_encryption(context),
      check_access_controls(context),
      check_audit_logging(context)
    ]

    failed_checks = Enum.filter(compliance_checks, fn
      {:ok, _} -> false
      _ -> true
    end)

    case failed_checks do
      [] -> {:ok, :compliance_validated}
      _ -> {:error, {:compliance_failures, failed_checks}}
    end
  end

  defp success?({:ok, _}), do: true
  defp success?(_), do: false
end

Common Pitfalls

  • Insufficient test coverage: Deploying automatically without comprehensive tests turns CD into "Continuous Disaster". Invest in testing before enabling CD.

  • Flaky tests: Tests that randomly fail cause pipeline fatigue and erode trust. Fix or remove flaky tests immediately.

  • Missing rollback automation: A CD pipeline without automated rollback is dangerous. Always have a tested rollback path.

  • Database migration coupling: Tightly coupling application deployments with database migrations creates rollback challenges. Use expand-contract patterns.

  • Alert fatigue: Too many alerts from production monitoring lead to ignored alerts. Tune thresholds and focus on actionable signals.

Further Reading


Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL