ADR-008: AWS ALB Decommission Workflow - Enterprise Refactoring¶

Note: This ADR was originally numbered ADR-001 but was renumbered to ADR-008 during the 2026-01-28 namespace consolidation to resolve collision with ADR-001-cost-metric-selection.md

Status: Proposed Date: 2026-01-22 Deciders: @cloud-architect, @product-owner, HITL/Manager JIRA: AWSO-80 Constitutional Principle: IV. Hybrid Deployment (3-Tier Testing)

Context¶

The /aws:alb-decommission skill execution revealed architectural anti-patterns that violate ADLC v3.0.0 enterprise framework principles:

Current Anti-Patterns Identified¶

❌ NO 3-Tier Testing Strategy - Went straight to Tier 3 (AWS production) without Tier 1 (Docker) or Tier 2 (LocalStack/Sandbox) - No progressive validation ($0 → $25/mo → $50/mo → production) - Risk: Expensive failures, no local debugging capability
❌ NO MCP Server Integration - awslabs-elbv2 configured in .mcp-aws.json but completely unused - No cross-validation between CLI and MCP - Anti-pattern: Single source of truth (fragile)
❌ Fragile Bash Scripts - JSON parsing errors (jq failures) - Exit code failures (non-zero on grep no match) - No error handling or retries - Example: verify_awso80_phase6.sh fails silently
❌ NO Cross-Validation - Single data source (AWS CLI only) - No accuracy target (ADLC requires ≥99.5%) - No 4-way validation (Console/CLI/MCP/Runbooks)
❌ NO Docker-First Development - Execution requires AWS credentials immediately - No consistent local environment - Violates Local-First Development Pattern (ADLC Principle IV)
❌ NO Agent Coordination - Skill executed standalone without product-owner or cloud-architect approval - Violates Enterprise Coordination Protocol (BLOCKING rule) - No evidence logging to tmp/cloud-infrastructure/coordination-logs/
❌ NATO Violations - Claims completion without evidence paths - No structured artifacts in tmp/ directories - Example: "Investigation complete" without investigation-report-*.json reference

Decision¶

Refactor /aws:alb-decommission skill to ADLC v3.0.0 compliant enterprise architecture with:

1. 3-Tier Testing Strategy (BLOCKING)¶

┌─────────────────────────────────────────────────────────────────┐
│  TIER 1: Docker Compose ($0, 2-3 sec)                           │
│  ├── Python unit tests (pytest)                                 │
│  ├── JSON schema validation                                     │
│  ├── Evidence file structure validation                         │
│  └── PASS REQUIRED before Tier 2                                │
│                          ↓                                       │
│  TIER 2: LocalStack + AWS CLI Mock (~$0, 30-60 sec)            │
│  ├── alb_decommission_investigation.py (mocked boto3)           │
│  ├── Evidence JSON generation                                   │
│  ├── 4-way validation simulation                                │
│  └── PASS REQUIRED before Tier 3                                │
│                          ↓                                       │
│  TIER 3: AWS Sandbox Account (~$25/mo, 5-10 min)               │
│  ├── Real ALB discovery (boto3 + MCP cross-validation)          │
│  ├── CloudWatch metrics collection                              │
│  ├── 4-way validation (Console/CLI/MCP/Runbooks)                │
│  ├── HITL APPROVAL REQUIRED for production                      │
│  └── Evidence: tmp/cloud-infrastructure/alb-investigation/      │
│                          ↓                                       │
│  PRODUCTION: AWS Production Account (ap-southeast-2)            │
│  ├── Read-only investigation (no mutations)                     │
│  ├── Manager approval gate                                      │
│  ├── Decommission plan generation                               │
│  └── HITL EXECUTION for actual decommissioning                  │
└─────────────────────────────────────────────────────────────────┘

2. MCP Integration Architecture (4-Way Validation)¶

# Cross-validation pattern (≥99.5% accuracy target)
class ALBInvestigator:
    def __init__(self, alb_name: str, profile: str):
        self.boto3_client = boto3.Session(profile_name=profile).client("elbv2")
        self.mcp_client = MCPClient("awslabs-elbv2")  # NEW
        self.cli_executor = AWSCLIExecutor(profile)

    def validate_4way(self) -> ValidationReport:
        """4-Way Validation: Console/CLI/MCP/Runbooks"""
        results = {
            "console": self.get_console_export(),      # Method 1: Manual CSV
            "boto3": self.get_boto3_data(),            # Method 2: SDK
            "mcp": self.get_mcp_data(),                # Method 3: MCP Server (NEW)
            "runbooks": self.get_runbooks_data(),      # Method 4: Runbooks CLI
        }

        # Cross-validate all methods (≥99.5% match required)
        accuracy = self.compare_methods(results)

        if accuracy < 99.5:
            raise ValidationError(f"Accuracy {accuracy}% < 99.5% threshold")

        return ValidationReport(
            accuracy=accuracy,
            methods=results,
            evidence_path="tmp/cloud-infrastructure/alb-investigation/<alb>/validation/"
        )

3. Agent Communication Protocol (Enterprise Coordination)¶

sequenceDiagram
    participant User
    participant PO as product-owner
    participant CA as cloud-architect
    participant Skill as /aws:alb-decommission
    participant Docker
    participant LocalStack
    participant AWS

    User->>PO: Request: "Investigate ALB for decommission"
    PO->>PO: Business validation (priority, impact, stakeholders)
    PO->>User: Log: tmp/coordination-logs/product-owner-YYYY-MM-DD.json

    PO->>CA: Handoff with requirements
    CA->>CA: Technical design (3-tier, MCP, evidence)
    CA->>CA: Architecture review (ADR-001)
    CA->>User: Log: tmp/coordination-logs/cloud-architect-YYYY-MM-DD.json

    CA->>Skill: Execute with approved design

    Skill->>Docker: Tier 1 - Unit tests (2-3s)
    Docker-->>Skill: PASS ✓

    Skill->>LocalStack: Tier 2 - Mocked AWS (30-60s)
    LocalStack-->>Skill: PASS ✓

    Skill->>AWS: Tier 3 - Real AWS Sandbox
    AWS-->>Skill: Evidence JSON + 4-way validation

    Skill->>User: Evidence: tmp/cloud-infrastructure/alb-investigation/<alb>/
    Skill->>User: HITL approval required for production

4. Evidence Architecture (Structured Paths)¶

tmp/cloud-infrastructure/alb-investigation/<alb-name>/
├── coordination-logs/               # NEW - Enterprise coordination
│   ├── product-owner-2026-01-22.json
│   └── cloud-architect-2026-01-22.json
├── dependencies/                    # Existing - ALB topology
│   ├── alb-config.json
│   ├── alb-target-health.json
│   ├── nlb-alb-target-health.json
│   ├── nlb-smtp-target-health.json
│   └── exchange-instances.json
├── metrics/                         # Existing - CloudWatch data
│   └── cloudwatch-metrics-YYYY-MM-DD.json
├── diagrams/                        # Existing - Visual evidence
│   ├── exchange-topology-v2-YYYY-MM-DD.png
│   ├── health-summary-v2-YYYY-MM-DD.png
│   └── diagram-evidence-v2-YYYY-MM-DD.json
├── validation/                      # NEW - 4-way validation
│   ├── 4way-validation-YYYY-MM-DD.json
│   ├── mcp-validation.json
│   ├── cli-validation.json
│   └── accuracy-report.json
├── test-results/                    # NEW - 3-tier test logs
│   ├── tier1-unit-tests-YYYY-MM-DD.log
│   ├── tier2-localstack-YYYY-MM-DD.log
│   └── tier3-aws-sandbox-YYYY-MM-DD.log
└── investigation-report-YYYY-MM-DD.json  # Existing - Final report

5. HITL Gates (Manager Approval Points)¶

Tier	Gate	Approval Required	Auto-Proceed
Tier 1	Unit tests pass	None	✅ Yes
Tier 2	LocalStack validation	None	✅ Yes
Tier 3	AWS Sandbox	Platform Team Lead	❌ No - HITL required
Production	Decommission execution	Application Team + Exchange Admin	❌ No - HITL required

Consequences¶

Positive¶

✅ Cost Reduction: Tier 1/2 testing catches 90%+ issues at $0 cost (no AWS calls)
✅ Validation Accuracy: 4-way cross-validation achieves ≥99.5% accuracy target
✅ Local Development: Docker-first enables debugging without AWS credentials
✅ Constitutional Compliance: Follows ADLC Principle IV (Hybrid Deployment)
✅ Evidence-Based: Structured tmp/ artifacts prevent NATO violations
✅ Audit Trail: Full coordination logs for manager review

Negative¶

⚠️ Initial Complexity: Refactoring requires ~3-5 days of engineering effort
⚠️ MCP Learning Curve: Team needs training on MCP Server usage
⚠️ Test Infrastructure: LocalStack setup requires initial configuration

Risks¶

Risk	Probability	Impact	Mitigation
MCP Server unavailable	Medium	High	Fallback to CLI-only with reduced accuracy
LocalStack drift from AWS	Low	Medium	Tier 3 sandbox validation catches differences
Evidence directory bloat	Medium	Low	Automated cleanup of >90-day old evidence

Alternatives Considered¶

Alternative 1: Keep Current Bash Scripts (REJECTED)¶

Reasoning: - Violates ADLC v3.0.0 constitutional principles - No 3-tier testing strategy - Fragile error handling - Single source of truth (no cross-validation) - Rejection Criteria: Architecture validation failed (circular dependencies on AWS credentials)

Alternative 2: Use Only MCP Servers (REJECTED)¶

Reasoning: - MCP servers are still experimental (uvx installation can fail) - No fallback mechanism if MCP unavailable - Over-reliance on single technology - Rejection Criteria: Missing sandboxing strategy (no Tier 1/2 testing)

Alternative 3: Hybrid Python + MCP with 3-Tier Testing (SELECTED)¶

Reasoning: - ✅ ADLC v3.0.0 compliant (Principle IV: Hybrid Deployment) - ✅ Progressive validation ($0 → $25 → production) - ✅ Cross-validation (4-way: Console/CLI/MCP/Runbooks) - ✅ Docker-first local development - ✅ Evidence-based completion (no NATO) - ✅ Enterprise coordination (product-owner → cloud-architect → execution)

Implementation Roadmap¶

Phase 1: Tier 1 Testing (Docker-First) - 1 day¶

Deliverables: - tests/alb/test_alb_investigation_unit.py - Unit tests for Python scripts - tests/alb/test_evidence_structure.py - JSON schema validation - Docker Compose service: diagrams (already exists, enhance for testing) - Evidence: tmp/cloud-infrastructure/test-results/tier1-unit-tests-*.log

Acceptance Criteria: - ✅ All unit tests pass in <3 seconds - ✅ No AWS credentials required - ✅ JSON evidence matches schema

Phase 2: Tier 2 LocalStack Integration - 1-2 days¶

Deliverables: - LocalStack service in docker-compose.yml (add to existing) - tests/alb/test_alb_investigation_localstack.py - Mocked boto3 tests - Evidence: tmp/cloud-infrastructure/test-results/tier2-localstack-*.log

Acceptance Criteria: - ✅ Tests pass against LocalStack in 30-60 seconds - ✅ Evidence JSON structure validated - ✅ Cost: $0 (no AWS API calls)

Phase 3: MCP Cross-Validation - 1 day¶

Deliverables: - scripts/alb_mcp_validator.py - MCP cross-validation script - Update .mcp-aws.json with awslabs-elbv2 server (already configured) - Evidence: tmp/cloud-infrastructure/alb-investigation/<alb>/validation/mcp-validation.json

Acceptance Criteria: - ✅ MCP data matches boto3 data (≥99.5% accuracy) - ✅ Fallback to CLI-only if MCP unavailable - ✅ 4-way validation report generated

Phase 4: Enterprise Coordination Integration - 0.5 day¶

Deliverables: - Update /aws:alb-decommission skill with coordination checkpoint - Auto-logging to tmp/coordination-logs/ - Evidence: cloud-architect-YYYY-MM-DD.json, product-owner-YYYY-MM-DD.json

Acceptance Criteria: - ✅ Skill BLOCKS without product-owner + cloud-architect approval - ✅ Coordination logs include PDCA cycle count - ✅ HITL escalation after 3 PDCA cycles

Phase 5: Documentation & Runbooks - 0.5 day¶

Deliverables: - docs/cloudops/alb-decommission-workflow.md - User guide - Update JIRA/AWSO-80-*/AWSO-80-plan.md with new architecture - HITL runbook: JIRA/AWSO-80-*/AWSO-80-phase6-hitl-runbook.md

Acceptance Criteria: - ✅ Documentation includes architecture diagrams - ✅ HITL approval gates clearly defined - ✅ Rollback procedures documented (RTO ≤30min)

Technical Design Specifications¶

Python Script Refactoring¶

Current: scripts/alb_decommission_investigation.py (Phase 6.8) - ✅ CLI parameterization (argparse + env vars) - ✅ Evidence logging to tmp/ - ❌ No MCP integration - ❌ No 3-tier testing

Target: scripts/alb_investigator.py (ADR-001 compliant)

#!/usr/bin/env python3
"""
ALB Investigator v3.0 - ADLC v3.0.0 Compliant

3-Tier Testing Strategy + MCP Cross-Validation + Enterprise Coordination
"""

import argparse
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol

@dataclass
class InvestigationConfig:
    alb_name: str
    profile: str
    region: str
    tier: int  # 1=Docker, 2=LocalStack, 3=AWS Sandbox
    evidence_dir: Path

class DataSource(Protocol):
    """Abstract interface for data sources (CLI/MCP/Runbooks)"""
    def get_alb_details(self, alb_name: str) -> dict: ...
    def get_target_health(self, alb_arn: str) -> list: ...
    def get_cloudwatch_metrics(self, alb_name: str) -> dict: ...

class Boto3DataSource:
    """AWS SDK implementation"""
    def __init__(self, session: boto3.Session):
        self.elbv2 = session.client("elbv2")
        self.cloudwatch = session.client("cloudwatch")

    def get_alb_details(self, alb_name: str) -> dict:
        response = self.elbv2.describe_load_balancers(Names=[alb_name])
        return response["LoadBalancers"][0]

class MCPDataSource:
    """MCP Server implementation (awslabs-elbv2)"""
    def __init__(self, mcp_client):
        self.mcp = mcp_client

    def get_alb_details(self, alb_name: str) -> dict:
        # Call MCP server via Claude Code context
        return self.mcp.call("elbv2:describe-load-balancers", {"Names": [alb_name]})

class ALBInvestigator:
    def __init__(self, config: InvestigationConfig):
        self.config = config
        self.sources: list[DataSource] = []

        # Initialize data sources based on tier
        if config.tier >= 2:
            self.sources.append(Boto3DataSource(boto3.Session(profile_name=config.profile)))

        if config.tier >= 3:
            # MCP only available in Tier 3 (real AWS)
            try:
                mcp_client = MCPClient("awslabs-elbv2")
                self.sources.append(MCPDataSource(mcp_client))
            except Exception as e:
                logger.warning(f"MCP server unavailable: {e}")

    def validate_4way(self) -> ValidationReport:
        """Cross-validate all data sources (≥99.5% accuracy)"""
        results = {}
        for source in self.sources:
            results[source.__class__.__name__] = source.get_alb_details(self.config.alb_name)

        # Compare results
        accuracy = self.calculate_accuracy(results)

        return ValidationReport(
            accuracy=accuracy,
            sources=results,
            evidence_path=self.config.evidence_dir / "validation" / f"4way-validation-{datetime.now():%Y-%m-%d}.json"
        )

    def run_tier1_tests(self) -> bool:
        """Tier 1: Unit tests (Docker, no AWS)"""
        # pytest tests/alb/test_alb_investigation_unit.py
        pass

    def run_tier2_tests(self) -> bool:
        """Tier 2: LocalStack integration tests"""
        # pytest tests/alb/test_alb_investigation_localstack.py
        pass

    def run_tier3_investigation(self) -> InvestigationReport:
        """Tier 3: Real AWS investigation with 4-way validation"""
        validation = self.validate_4way()

        if validation.accuracy < 99.5:
            raise ValidationError(f"Accuracy {validation.accuracy}% < 99.5%")

        # Collect metrics, generate diagrams, etc.
        # ...

        return InvestigationReport(...)

Docker Compose Service Enhancement¶

Add to docker-compose.yml:

# LocalStack for Tier 2 testing
localstack:
  image: localstack/localstack:latest
  container_name: cloud-infra-localstack
  environment:
    - SERVICES=elbv2,cloudwatch,ec2
    - DEFAULT_REGION=ap-southeast-2
  ports:
    - "4566:4566"
  volumes:
    - ./tmp/localstack:/var/lib/localstack
  networks:
    - cloud-infra-network
  profiles:
    - tier2

# ALB Investigation Runner (3-tier capable)
alb-investigator:
  <<: *runbooks-cloudops-image
  container_name: cloud-infra-alb-investigator
  working_dir: /workspace
  volumes:
    - .:/workspace:cached
    - ${HOME}/.aws:/home/os/.aws:cached
  environment:
    <<: *cloudops-env
    TIER: ${TIER:-1}  # 1=Docker, 2=LocalStack, 3=AWS
    LOCALSTACK_ENDPOINT: http://localstack:4566
  depends_on:
    - localstack
  networks:
    - cloud-infra-network
  command: ["python", "/workspace/scripts/alb_investigator.py"]
  profiles:
    - alb-investigation

Taskfile Commands¶

Add to Taskfile.yml:

# ALB Investigation Tasks
alb:investigate:tier1:
  desc: "Tier 1 - Unit tests (Docker, no AWS, 2-3s)"
  cmds:
    - docker compose run --rm alb-investigator pytest tests/alb/test_alb_investigation_unit.py

alb:investigate:tier2:
  desc: "Tier 2 - LocalStack integration (30-60s, $0)"
  cmds:
    - docker compose --profile tier2 up -d localstack
    - docker compose --profile tier2 run --rm alb-investigator pytest tests/alb/test_alb_investigation_localstack.py

alb:investigate:tier3:
  desc: "Tier 3 - AWS Sandbox (5-10min, requires creds)"
  cmds:
    - docker compose --profile alb-investigation run alb-investigator python /workspace/scripts/alb_investigator.py --alb-name={{.ALB_NAME}} --profile={{.AWS_PROFILE}} --tier=3

alb:investigate:
  desc: "Full 3-tier investigation (Tier 1 → 2 → 3)"
  cmds:
    - task: alb:investigate:tier1
    - task: alb:investigate:tier2
    - task: alb:investigate:tier3
      vars:
        ALB_NAME: '{{.ALB_NAME | default "ams-exchange-alb-internal"}}'
        AWS_PROFILE: '{{.AWS_PROFILE | default "ams-shared-services-ReadOnlyAccess"}}'

Quality Gate Enforcement¶

Quantitative Thresholds (BLOCKING)¶

Metric	Target	Measurement	Evidence
Tier 1 Pass Rate	100%	pytest exit code	`tmp/test-results/tier1-*.log`
Tier 2 Pass Rate	≥95%	pytest exit code	`tmp/test-results/tier2-*.log`
4-Way Validation Accuracy	≥99.5%	Cross-validation report	`tmp/validation/4way-validation-*.json`
Evidence Completeness	100%	All required files exist	`scripts/validate_evidence.py`
MCP Availability	≥90%	MCP server uptime	Fallback to CLI if <90%

Rejection Criteria (BLOCKING)¶

❌ Tier 1 tests fail → REJECT ("fix unit tests before AWS deployment")
❌ 4-way accuracy <99.5% → REJECT ("data sources inconsistent")
❌ Missing evidence files → REJECT ("incomplete investigation")
❌ No coordination logs → REJECT ("enterprise protocol violated")

References¶

ADLC Constitution: .specify/memory/constitution.md (Principle IV: Hybrid Deployment)
AWS Well-Architected: Operational Excellence, Reliability, Cost Optimization
Cloud-Architect Agent: .claude/agents/cloud-architect.md
JIRA Ticket: AWSO-80 (ALB Decommission Investigation)
Python Scripts: scripts/alb_decommission_investigation.py (current), scripts/alb_investigator.py (target)
MCP Configuration: .mcp-aws.json (awslabs-elbv2 server)

Evidence¶

Architecture Diagram: tmp/cloud-infrastructure/architecture-decisions/ADR-001-architecture-diagram.png (pending)
Cost Estimate: tmp/cloud-infrastructure/cost-reports/ADR-001-cost-estimate.json (pending)
Coordination Log: tmp/cloud-infrastructure/coordination-logs/cloud-architect-2026-01-22.json

Version: 1.0.0 Status: Proposed (awaiting HITL approval) Next Steps: Phase 1 implementation (Tier 1 testing)