Skip to content

ADR-007: ITSM Dashboard Deployment ArchitectureΒΆ

Status: Approved Date: 2026-01-24 Deciders: @cloud-architect, @observability-engineer Sprint: Sprint 7 (Session 2) Constitutional Compliance: CHK021 (Hybrid Deployment), CHK027 (Observability), CHK022 (Sandboxing)


ContextΒΆ

Sprint 6 delivered a functional ITSM Dashboard with: - Data Layer: Pydantic models + PII masking (398 LOC) - Visualization Layer: 4 KPIs + 8 charts (522 LOC) - Application Layer: Multi-page Vizro dashboard (264 LOC) - Test Coverage: 100% (33/33 tests passing in Tier 1)

Current Gap: The dashboard runs only via manual CLI (uv run python dashboards/itsm/src/app.py). To enable production usage, we need: 1. Docker Compose service definition (local-first deployment) 2. Observability Stack (Prometheus metrics + Grafana dashboards) 3. Operational Runbooks (startup, shutdown, incident response, data refresh) 4. SLO Monitoring (availability, latency, error budget)

This ADR defines Phases 4-6 of ADLC: Deploy β†’ Monitor β†’ Operate.


DecisionΒΆ

Implement local-first Docker Compose deployment with integrated observability stack, following ADLC Principle IV (Hybrid Deployment) and Principle V (Observability & Resilience).

Deployment Target: Local Docker Compose (bare-metal + DevContainer compatible) Rationale: KISS principle - start local-first, defer Kubernetes/Cloud until business value proven (Phase 6 validation first).


ArchitectureΒΆ

1. Docker Compose Service DefinitionΒΆ

File: docker-compose.yml (extend existing root compose file)

services:
  # ═══════════════════════════════════════════════════════════════════════════
  # ITSM Dashboard Service
  # ═══════════════════════════════════════════════════════════════════════════
  itsm-dashboard:
    build:
      context: ./dashboards/itsm
      dockerfile: Dockerfile
    container_name: cloud-infrastructure-itsm-dashboard
    ports:
      - "8050:8050"   # Vizro dashboard
      - "9090:9090"   # Prometheus metrics
    environment:
      - PYTHONUNBUFFERED=1
      - ITSM_AZURE_FILE=/data/Azure-Tickets.xlsx
      - ITSM_AWS_FILE=/data/AWS-Tickets.xlsx
      - PROMETHEUS_PORT=9090
      - LOG_LEVEL=INFO
    volumes:
      - ./dashboards/itsm/architecture:/data:ro  # Read-only data mount
      - ./tmp/cloud-infrastructure/itsm-logs:/app/logs  # Evidence logs
      - ./tmp/cloud-infrastructure/observability/itsm:/app/metrics  # Metrics
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8050/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    restart: unless-stopped
    networks:
      - cloud-infrastructure
    labels:
      - "com.adlc.project=cloud-infrastructure"
      - "com.adlc.component=itsm-dashboard"
      - "com.adlc.sprint=sprint-7"

  # ═══════════════════════════════════════════════════════════════════════════
  # Prometheus (Observability)
  # ═══════════════════════════════════════════════════════════════════════════
  prometheus:
    image: prom/prometheus:v2.48.0
    container_name: cloud-infrastructure-prometheus
    ports:
      - "9091:9090"   # Avoid conflict with itsm-dashboard:9090
    volumes:
      - ./observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./tmp/cloud-infrastructure/observability/prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
      - '--web.enable-lifecycle'
    networks:
      - cloud-infrastructure
    restart: unless-stopped

  # ═══════════════════════════════════════════════════════════════════════════
  # Grafana (Dashboards)
  # ═══════════════════════════════════════════════════════════════════════════
  grafana:
    image: grafana/grafana:10.2.0
    container_name: cloud-infrastructure-grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin  # Change in production
      - GF_INSTALL_PLUGINS=
    volumes:
      - ./observability/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./observability/grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro
      - ./tmp/cloud-infrastructure/observability/grafana-data:/var/lib/grafana
    networks:
      - cloud-infrastructure
    restart: unless-stopped
    depends_on:
      - prometheus

networks:
  cloud-infrastructure:
    name: cloud-infrastructure
    driver: bridge

Total LOC: ~100 LOC (Docker Compose extension)


2. Dockerfile for ITSM DashboardΒΆ

File: dashboards/itsm/Dockerfile (120 LOC)

# Stage 1: Builder
FROM python:3.13-slim AS builder

WORKDIR /build

# Install UV
RUN pip install --no-cache-dir uv

# Copy dependency files
COPY pyproject.toml uv.lock ./

# Create virtual environment and install dependencies
RUN uv venv && \
    . .venv/bin/activate && \
    uv sync --frozen

# Stage 2: Runtime
FROM python:3.13-slim

WORKDIR /app

# Copy virtual environment from builder
COPY --from=builder /build/.venv /app/.venv

# Copy application code
COPY src/ /app/src/
COPY architecture/ /app/architecture/

# Install curl for healthcheck
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

# Create non-root user
RUN useradd -m -u 1000 dashboard && \
    chown -R dashboard:dashboard /app

USER dashboard

# Health check endpoint
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
  CMD curl -f http://localhost:8050/health || exit 1

# Expose ports
EXPOSE 8050 9090

# Set Python path
ENV PYTHONPATH=/app
ENV PATH="/app/.venv/bin:$PATH"

# Entry point
CMD ["python", "src/app.py"]

Total LOC: ~120 LOC


3. Observability Stack ArchitectureΒΆ

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  ITSM Dashboard (Port 8050)                                     β”‚
β”‚  β”œβ”€β”€ Application Metrics: /metrics (Prometheus format)          β”‚
β”‚  β”œβ”€β”€ Health Check: /health (200 OK / 503 Service Unavailable)  β”‚
β”‚  └── Logs: /app/logs (JSON structured logs)                     β”‚
β”‚                          ↓                                       β”‚
β”‚  Prometheus (Port 9091)                                          β”‚
β”‚  β”œβ”€β”€ Scrape Interval: 15s                                       β”‚
β”‚  β”œβ”€β”€ Retention: 90 days                                         β”‚
β”‚  β”œβ”€β”€ Metrics Stored:                                            β”‚
β”‚  β”‚   - itsm_dashboard_requests_total                            β”‚
β”‚  β”‚   - itsm_dashboard_request_duration_seconds                  β”‚
β”‚  β”‚   - itsm_dashboard_data_load_duration_seconds                β”‚
β”‚  β”‚   - itsm_dashboard_tickets_loaded_total                      β”‚
β”‚  β”‚   - itsm_dashboard_chart_render_duration_seconds             β”‚
β”‚  β”‚   - itsm_dashboard_up (service availability)                 β”‚
β”‚  └── Alerting: Alert when itsm_dashboard_up == 0 for >5min     β”‚
β”‚                          ↓                                       β”‚
β”‚  Grafana (Port 3000)                                             β”‚
β”‚  β”œβ”€β”€ Dashboard: ITSM Dashboard SLO Monitoring                   β”‚
β”‚  β”œβ”€β”€ Panels:                                                    β”‚
β”‚  β”‚   - Availability: uptime percentage (30-day rolling)         β”‚
β”‚  β”‚   - Latency: p50, p95, p99 request duration                  β”‚
β”‚  β”‚   - Error Rate: 4xx/5xx responses per minute                 β”‚
β”‚  β”‚   - Data Load Time: Trend chart (last 7 days)               β”‚
β”‚  β”‚   - Ticket Load Count: Gauge (Azure + AWS)                   β”‚
β”‚  └── Alerts: Slack webhook when SLO breached                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

4. SLO DefinitionsΒΆ

Per ADLC Principle V (Observability & Resilience):

SLO Metric Target Measurement Alert Threshold
Availability β‰₯99.0% itsm_dashboard_up (30-day rolling) <95% for 5min
Latency (p95) ≀2000ms itsm_dashboard_request_duration_seconds >3000ms for 5min
Latency (p99) ≀5000ms itsm_dashboard_request_duration_seconds >7000ms for 5min
Error Rate ≀1% itsm_dashboard_requests_total{status=~"4..|5.."} >5% for 5min
Data Load Time ≀10s itsm_dashboard_data_load_duration_seconds >30s for 2 consecutive loads

Error Budget: 1% of requests over 30-day period = ~432 minutes of downtime allowed.


5. Prometheus Metrics ImplementationΒΆ

File: dashboards/itsm/src/metrics.py (150 LOC)

"""
ITSM Dashboard Prometheus Metrics Exporter

Exposes Prometheus metrics on port 9090.
Sprint 7 | ADLC v3.0.0 | 2026-01-24
"""

from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time
import functools

# ═══════════════════════════════════════════════════════════════════════════
# Metrics Definitions
# ═══════════════════════════════════════════════════════════════════════════

# Request metrics
REQUEST_COUNT = Counter(
    'itsm_dashboard_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'itsm_dashboard_request_duration_seconds',
    'Request duration in seconds',
    ['method', 'endpoint'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
)

# Data loading metrics
DATA_LOAD_DURATION = Histogram(
    'itsm_dashboard_data_load_duration_seconds',
    'Data loading duration in seconds',
    buckets=[1.0, 3.0, 5.0, 10.0, 30.0]
)

TICKETS_LOADED = Gauge(
    'itsm_dashboard_tickets_loaded_total',
    'Total tickets loaded',
    ['provider']
)

# Chart rendering metrics
CHART_RENDER_DURATION = Histogram(
    'itsm_dashboard_chart_render_duration_seconds',
    'Chart rendering duration in seconds',
    ['chart_name'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0]
)

# Service availability
SERVICE_UP = Gauge(
    'itsm_dashboard_up',
    'Service availability (1=up, 0=down)'
)

# ═══════════════════════════════════════════════════════════════════════════
# Decorator for Request Tracking
# ═══════════════════════════════════════════════════════════════════════════

def track_request(endpoint: str):
    """Decorator to track request metrics."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            start_time = time.time()
            status = "200"
            try:
                result = func(*args, **kwargs)
                return result
            except Exception as e:
                status = "500"
                raise
            finally:
                duration = time.time() - start_time
                REQUEST_COUNT.labels(method="GET", endpoint=endpoint, status=status).inc()
                REQUEST_DURATION.labels(method="GET", endpoint=endpoint).observe(duration)
        return wrapper
    return decorator

# ═══════════════════════════════════════════════════════════════════════════
# Metrics Server
# ═══════════════════════════════════════════════════════════════════════════

def start_metrics_server(port: int = 9090):
    """Start Prometheus metrics HTTP server."""
    SERVICE_UP.set(1)
    start_http_server(port)
    print(f"Metrics server started on port {port}")

Total LOC: ~150 LOC


6. Grafana Dashboard ConfigurationΒΆ

File: observability/grafana/dashboards/itsm-dashboard-slo.json (500 LOC)

{
  "dashboard": {
    "title": "ITSM Dashboard SLO Monitoring",
    "panels": [
      {
        "id": 1,
        "title": "Availability (30-day rolling)",
        "type": "gauge",
        "targets": [
          {
            "expr": "avg_over_time(itsm_dashboard_up[30d]) * 100",
            "legendFormat": "Uptime %"
          }
        ],
        "thresholds": {
          "mode": "absolute",
          "steps": [
            { "value": 0, "color": "red" },
            { "value": 95, "color": "yellow" },
            { "value": 99, "color": "green" }
          ]
        }
      },
      {
        "id": 2,
        "title": "Request Latency (p50, p95, p99)",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.50, rate(itsm_dashboard_request_duration_seconds_bucket[5m]))",
            "legendFormat": "p50"
          },
          {
            "expr": "histogram_quantile(0.95, rate(itsm_dashboard_request_duration_seconds_bucket[5m]))",
            "legendFormat": "p95"
          },
          {
            "expr": "histogram_quantile(0.99, rate(itsm_dashboard_request_duration_seconds_bucket[5m]))",
            "legendFormat": "p99"
          }
        ]
      },
      {
        "id": 3,
        "title": "Error Rate (%)",
        "type": "graph",
        "targets": [
          {
            "expr": "(sum(rate(itsm_dashboard_requests_total{status=~\"4..|5..\"}[5m])) / sum(rate(itsm_dashboard_requests_total[5m]))) * 100",
            "legendFormat": "Error Rate"
          }
        ],
        "thresholds": [
          { "value": 1, "color": "yellow" },
          { "value": 5, "color": "red" }
        ]
      },
      {
        "id": 4,
        "title": "Data Load Duration (seconds)",
        "type": "graph",
        "targets": [
          {
            "expr": "itsm_dashboard_data_load_duration_seconds",
            "legendFormat": "Load Time"
          }
        ]
      },
      {
        "id": 5,
        "title": "Tickets Loaded by Provider",
        "type": "stat",
        "targets": [
          {
            "expr": "itsm_dashboard_tickets_loaded_total{provider=\"Azure\"}",
            "legendFormat": "Azure"
          },
          {
            "expr": "itsm_dashboard_tickets_loaded_total{provider=\"AWS\"}",
            "legendFormat": "AWS"
          }
        ]
      }
    ]
  }
}

Total LOC: ~500 LOC (JSON configuration)


7. Operational RunbooksΒΆ

Runbook 1: Startup ProcedureΒΆ

File: docs/runbooks/itsm-dashboard-startup.md (50 LOC)

# ITSM Dashboard Startup Procedure

**Duration**: 2-3 minutes
**Prerequisites**: Docker Compose installed, data files present

## Steps

1. **Verify Data Files**
   ```bash
   ls -lh dashboards/itsm/architecture/Azure-Tickets.xlsx
   ls -lh dashboards/itsm/architecture/AWS-Tickets.xlsx
   ```

2. **Start Services**
   ```bash
   docker compose up -d itsm-dashboard prometheus grafana
   ```

3. **Verify Health**
   ```bash
   # Wait 60s for startup
   sleep 60

   # Check health endpoint
   curl -f http://localhost:8050/health
   # Expected: {"status": "healthy", "tickets_loaded": 2998}
   ```

4. **Verify Metrics**
   ```bash
   curl -s http://localhost:9090/metrics | grep itsm_dashboard_up
   # Expected: itsm_dashboard_up 1.0
   ```

5. **Access Dashboard**
   - ITSM Dashboard: http://localhost:8050
   - Grafana: http://localhost:3000 (admin/admin)
   - Prometheus: http://localhost:9091

## Troubleshooting

- **Error: Port 8050 already in use**
  - Solution: `docker compose down && docker compose up -d`

- **Error: Data files not found**
  - Solution: Place Excel files in `dashboards/itsm/architecture/`

- **Error: Health check failing**
  - Solution: Check logs with `docker compose logs itsm-dashboard`

Runbook 2: Shutdown ProcedureΒΆ

File: docs/runbooks/itsm-dashboard-shutdown.md (30 LOC)

# ITSM Dashboard Shutdown Procedure

**Duration**: 30 seconds
**Safe**: Yes (no data loss)

## Steps

1. **Graceful Shutdown**
   ```bash
   docker compose stop itsm-dashboard
   ```

2. **Stop Observability Stack** (optional)
   ```bash
   docker compose stop prometheus grafana
   ```

3. **Full Cleanup** (removes containers)
   ```bash
   docker compose down
   ```

4. **Preserve Data** (keep volumes)
   ```bash
   # Volumes in tmp/ are preserved automatically
   ls -lh tmp/cloud-infrastructure/observability/
   ```

## Post-Shutdown Verification

```bash
# Verify no containers running
docker ps | grep itsm-dashboard
# Expected: No output
#### Runbook 3: Incident Response

**File**: `docs/runbooks/itsm-dashboard-incident-response.md` (100 LOC)

```markdown
# ITSM Dashboard Incident Response

## Incident: Dashboard Not Loading

**Symptoms**: HTTP 503 Service Unavailable

### Diagnosis

1. **Check Health Endpoint**
   ```bash
   curl -i http://localhost:8050/health
   ```

2. **Check Container Status**
   ```bash
   docker compose ps itsm-dashboard
   ```

3. **Check Logs**
   ```bash
   docker compose logs --tail=100 itsm-dashboard
   ```

### Resolution

**Scenario A: Data Load Failure**
```bash
# Symptom: "FileNotFoundError: Azure-Tickets.xlsx"
# Solution: Verify data files exist
ls dashboards/itsm/architecture/*.xlsx

# Restart with correct volume mount
docker compose restart itsm-dashboard

Scenario B: Memory Limit

# Symptom: "MemoryError" or OOMKilled
# Solution: Increase Docker memory limit to 2GB
# docker-compose.yml: Add `mem_limit: 2g` to itsm-dashboard service
docker compose up -d itsm-dashboard

Scenario C: Port Conflict

# Symptom: "Address already in use"
# Solution: Find conflicting process
lsof -i :8050
kill -9 <PID>
docker compose up -d itsm-dashboard

Incident: High Latency (p95 >3000ms)ΒΆ

DiagnosisΒΆ

  1. Check Prometheus Metrics

    curl -s http://localhost:9091/api/v1/query?query=histogram_quantile\(0.95,rate\(itsm_dashboard_request_duration_seconds_bucket\[5m\]\)\)
    

  2. Check Data Load Time

    curl -s http://localhost:9090/metrics | grep data_load_duration
    

ResolutionΒΆ

Scenario A: Large Dataset

# Solution: Implement data pagination or filtering
# Code change required in data_loader.py
# Implement: load_itsm_data(limit=1000, offset=0)

Scenario B: Chart Rendering Bottleneck

# Solution: Enable chart caching
# Code change required in charts.py
# Implement: @lru_cache(maxsize=128) on chart functions

EscalationΒΆ

SLO Breach: Alert observability-engineer via Slack Data Corruption: Alert security-compliance-engineer Infrastructure Failure: Alert devops-engineer

#### Runbook 4: Data Refresh Procedure

**File**: `docs/runbooks/itsm-dashboard-data-refresh.md` (40 LOC)

```markdown
# ITSM Dashboard Data Refresh Procedure

**Frequency**: Monthly (after JIRA export)
**Downtime**: ~30 seconds

## Steps

1. **Export Fresh Data from JIRA**
   - JIRA Filter: Azure tickets (last 12 months)
   - Export as: Azure-Tickets.xlsx
   - JIRA Filter: AWS tickets (last 12 months)
   - Export as: AWS-Tickets.xlsx

2. **Backup Current Data**
   ```bash
   cp dashboards/itsm/architecture/Azure-Tickets.xlsx \
      dashboards/itsm/architecture/backups/Azure-Tickets-$(date +%Y-%m-%d).xlsx
   cp dashboards/itsm/architecture/AWS-Tickets.xlsx \
      dashboards/itsm/architecture/backups/AWS-Tickets-$(date +%Y-%m-%d).xlsx
   ```

3. **Replace Data Files**
   ```bash
   cp ~/Downloads/Azure-Tickets.xlsx dashboards/itsm/architecture/
   cp ~/Downloads/AWS-Tickets.xlsx dashboards/itsm/architecture/
   ```

4. **Restart Dashboard**
   ```bash
   docker compose restart itsm-dashboard
   ```

5. **Verify Data Loaded**
   ```bash
   curl http://localhost:8050/health
   # Expected: New ticket count
   ```

## Rollback

```bash
# Restore from backup if data refresh fails
cp dashboards/itsm/architecture/backups/Azure-Tickets-2026-01-24.xlsx \
   dashboards/itsm/architecture/Azure-Tickets.xlsx
docker compose restart itsm-dashboard
**Total Runbook LOC**: ~220 LOC (4 runbooks)

---

## 3-Tier Testing Strategy

### Tier 1: Functional Tests ($0, 2-3s)

Already covered in Sprint 6 (`test_tier1.py` - 33/33 tests passing).

### Tier 2: Integration Tests ($0, 30-60s)

**File**: `dashboards/itsm/tests/test_tier2_docker.py` (150 LOC)

```python
"""
ITSM Dashboard Tier 2 Tests - Docker Integration

Tests Docker Compose deployment with health checks.
Run with: uv run pytest tests/test_tier2_docker.py -v
"""

import subprocess
import time
import requests
import pytest

def test_docker_compose_builds():
    """Verify Docker image builds successfully."""
    result = subprocess.run(
        ["docker", "compose", "build", "itsm-dashboard"],
        cwd="dashboards/itsm",
        capture_output=True
    )
    assert result.returncode == 0

def test_docker_compose_starts():
    """Verify services start successfully."""
    subprocess.run(["docker", "compose", "up", "-d", "itsm-dashboard"])
    time.sleep(60)  # Wait for startup

    # Check container running
    result = subprocess.run(
        ["docker", "compose", "ps", "itsm-dashboard"],
        capture_output=True,
        text=True
    )
    assert "running" in result.stdout.lower()

def test_health_endpoint():
    """Verify health endpoint returns 200."""
    response = requests.get("http://localhost:8050/health", timeout=10)
    assert response.status_code == 200
    assert response.json()["status"] == "healthy"

def test_prometheus_metrics_available():
    """Verify Prometheus metrics endpoint."""
    response = requests.get("http://localhost:9090/metrics", timeout=10)
    assert response.status_code == 200
    assert "itsm_dashboard_up" in response.text

@pytest.fixture(scope="module", autouse=True)
def teardown():
    """Cleanup after tests."""
    yield
    subprocess.run(["docker", "compose", "down"])

Coverage Target: 5/5 integration tests passing

Tier 3: E2E Tests (Deferred to Sprint 8)ΒΆ

Full user journey testing with Playwright (future).


Quality GatesΒΆ

QG-DEPLOY-001: Docker Build SuccessΒΆ

Metric: Docker image builds without errors Command: docker compose build itsm-dashboard Threshold: Exit code 0 Evidence: tmp/cloud-infrastructure/deployment-logs/docker-build-YYYY-MM-DD.log

QG-DEPLOY-002: Health Check PassΒΆ

Metric: Health endpoint returns 200 OK within 60s of startup Command: curl -f http://localhost:8050/health Threshold: 200 OK + {"status": "healthy"} Evidence: tmp/cloud-infrastructure/deployment-logs/health-check-YYYY-MM-DD.log

QG-DEPLOY-003: SLO Baseline EstablishedΒΆ

Metric: Prometheus scrapes metrics successfully for 1 hour Command: curl http://localhost:9091/api/v1/query?query=up{job="itsm-dashboard"} Threshold: "value": [timestamp, "1"] (service up) Evidence: tmp/cloud-infrastructure/observability/slo-baseline-YYYY-MM-DD.json

QG-DEPLOY-004: Grafana Dashboard RendersΒΆ

Metric: Grafana dashboard displays 5 panels Command: Manual verification at http://localhost:3000 Threshold: 5/5 panels render with data Evidence: Screenshot in tmp/cloud-infrastructure/observability/grafana-dashboard-YYYY-MM-DD.png


File Structure with LOC EstimatesΒΆ

Cloud-Infrastructure/
β”œβ”€β”€ docker-compose.yml                     # +100 LOC - Extended with itsm-dashboard service
β”œβ”€β”€ dashboards/itsm/
β”‚   β”œβ”€β”€ Dockerfile                         # 120 LOC - Multi-stage build
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ metrics.py                     # 150 LOC - Prometheus metrics
β”‚   β”‚   └── app.py                         # +50 LOC - Health endpoint + metrics integration
β”‚   └── tests/
β”‚       └── test_tier2_docker.py           # 150 LOC - Docker integration tests
β”œβ”€β”€ observability/
β”‚   β”œβ”€β”€ prometheus/
β”‚   β”‚   └── prometheus.yml                 # 50 LOC - Scrape config
β”‚   β”œβ”€β”€ grafana/
β”‚   β”‚   β”œβ”€β”€ dashboards/
β”‚   β”‚   β”‚   └── itsm-dashboard-slo.json    # 500 LOC - Grafana dashboard
β”‚   β”‚   └── datasources.yml                # 20 LOC - Prometheus datasource
β”œβ”€β”€ docs/runbooks/
β”‚   β”œβ”€β”€ itsm-dashboard-startup.md          # 50 LOC
β”‚   β”œβ”€β”€ itsm-dashboard-shutdown.md         # 30 LOC
β”‚   β”œβ”€β”€ itsm-dashboard-incident-response.md  # 100 LOC
β”‚   └── itsm-dashboard-data-refresh.md     # 40 LOC
└── tmp/cloud-infrastructure/
    β”œβ”€β”€ deployment-logs/
    β”‚   β”œβ”€β”€ docker-build-YYYY-MM-DD.log
    β”‚   └── health-check-YYYY-MM-DD.log
    └── observability/
        β”œβ”€β”€ itsm/                          # Dashboard logs
        β”œβ”€β”€ prometheus-data/               # Prometheus TSDB
        β”œβ”€β”€ grafana-data/                  # Grafana config
        β”œβ”€β”€ slo-baseline-YYYY-MM-DD.json
        └── grafana-dashboard-YYYY-MM-DD.png

Total New LOC: ~1360 LOC (Docker + Observability + Runbooks)


Constitutional ComplianceΒΆ

CHK021: Hybrid Deployment Target SpecifiedΒΆ

Compliance: βœ… PASS - Deployment target: Local Docker Compose (local-first pattern) - Rationale: KISS principle - prove business value before Kubernetes/Cloud - Migration path: Kubernetes Helm chart (deferred to Sprint 8+)

CHK022: Sandboxing StrategyΒΆ

Compliance: βœ… PASS - Docker container isolation (separate network namespace) - Read-only volume mounts for data files (:ro flag) - Non-root user (UID 1000) - Healthcheck enforcement (restart on failure)

CHK027: Observability InstrumentationΒΆ

Compliance: βœ… PASS - MELT telemetry: Metrics (Prometheus), Events (N/A), Logs (JSON), Traces (deferred) - Agent-specific observability: Data load time, chart render time, ticket counts - Real-time monitoring: 15s scrape interval, 90-day retention - SLO dashboard: Grafana with 5 panels (availability, latency, error rate, data load, ticket count)

CHK028: SLO DefinitionsΒΆ

Compliance: βœ… PASS - Availability: β‰₯99.0% (30-day rolling) - Latency p95: ≀2000ms, p99: ≀5000ms - Error rate: ≀1% - Error budget: 432 minutes/month


ConsequencesΒΆ

PositiveΒΆ

  1. Local-First: $0 deployment cost, instant feedback loop
  2. Production-Ready: Prometheus + Grafana observability from day 1
  3. Operational Excellence: 4 runbooks cover all operational scenarios
  4. KISS Compliance: Docker Compose avoids Kubernetes complexity until needed
  5. Evidence-Driven: All metrics logged to tmp/ for ADLC compliance

NegativeΒΆ

  1. Manual Scaling: Docker Compose doesn't auto-scale (acceptable for local deployment)
  2. No High Availability: Single container (acceptable for MVP)
  3. Local Storage: Data in tmp/ not backed up (acceptable for dashboards)

RisksΒΆ

  1. Port Conflicts: Risk = MEDIUM, Mitigation = Configurable ports in .env file
  2. Memory Limits: Risk = LOW, Mitigation = Docker mem_limit: 2g in compose
  3. Data Refresh Downtime: Risk = LOW, Mitigation = ~30s downtime acceptable for monthly refresh

Alternatives ConsideredΒΆ

Alternative 1: Kubernetes DeploymentΒΆ

Rejected: Overkill for MVP, violates KISS principle. Deferred to Sprint 8+ when business value proven.

Alternative 2: Serverless (AWS Lambda + API Gateway)ΒΆ

Rejected: Violates local-first principle, introduces cloud costs (~$50/month).

Alternative 3: Manual Deployment (No Docker)ΒΆ

Rejected: Not reproducible, violates ADLC Principle IV (Hybrid Deployment).


ReferencesΒΆ


EvidenceΒΆ

  • Architecture Diagram: tmp/cloud-infrastructure/architecture-decisions/ADR-007-diagram.png (to be generated)
  • Cost Estimate: $0 (local Docker Compose deployment)
  • LOC Estimate: 1360 LOC (Docker + Observability + Runbooks)
  • SLO Baseline: To be established in Sprint 7 deployment

Version: 1.0.0 Sprint: Sprint 7 (Session 2) Next Review: Post-deployment monitoring in Sprint 7 Supersedes: None (new architecture decision)