Skip to content

ADR-013: 3-tier testing with Playwright E2E at Tier 3

Note: This ADR was originally numbered ADR-003 but was renumbered to ADR-013 during the 2026-01-28 namespace consolidation to resolve collision with ADR-003-date-range-architecture.md

Status: Approved Date: 2026-01-28 Deciders: @cloud-architect, @product-owner, HITL/Manager

Context

ADLC v3.0.0 requires 3-tier testing strategy: - Tier 1: Unit/snapshot tests (2-3s, $0, 70-80% bug coverage) - Tier 2: Integration tests (5-10m, ~$25/mo, +15-20% coverage) - Tier 3: E2E tests (15-30m, ~$100/mo, +5-10% coverage)

Current ITSM demo has manual verification only (no automated tests). This creates: - Risk: Regression bugs slip into production (last incident: broken filter at 150% zoom) - Cost: 30 min/week manual QA = $1,500/year opportunity cost - Compliance gap: No audit trail for visual regression testing

Decision

Implement Tier 1 (snapshot) + Tier 3 (Playwright E2E) testing.

Skip Tier 2 (integration) because ITSM dashboard has no backend APIs to test.

Tier 1: Pytest Snapshot Tests

Purpose: Validate Vizro dashboard configuration structure (not visual).

Test file: tests/unit/test_itsm_app_snapshot.py

Coverage: - Dashboard imports successfully - KPI card count matches expected (4 cards) - Tab count matches expected (8 tabs) - Grid layout structure validates

Framework: pytest + syrupy (snapshot library)

Duration: 2-3 seconds

Cost: $0 (runs in CI/CD)

Example: 

def test_dashboard_structure(snapshot):
    from dashboards.itsm.demo.app import dashboard
    assert snapshot == {
        "kpi_count": len([c for c in dashboard.pages[0].components if "kpi" in c.id]),
        "tab_count": len(dashboard.pages[0].components),
        "layout_type": type(dashboard.pages[0].layout).__name__
    }

Tier 3: Playwright E2E Tests

Purpose: Automated visual regression testing with screenshots.

Test file: tests/e2e/test_itsm_responsive.spec.py

Coverage: 1. KPI click cascade: Click "Open P1 & P2" → assert 8 tabs filter correctly 2. Responsive zoom: Screenshot at 5 zoom levels (25%, 50%, 100%, 150%, 200%) 3. Table pagination: Assert pagination controls present, default 10 rows/page

Framework: Playwright via MCP server (@playwright/mcp-server)

Duration: 5-10 minutes (60s per zoom level)

Cost: $0 (local docker-compose, no cloud resources)

MCP Integration: 

// MCP command sequence
tabs_create_mcp()
navigate({url: "http://localhost:8053", tabId: TAB_ID})
find({query: "Open P1 & P2 KPI", tabId: TAB_ID})
computer({action: "left_click", ref: REF_ID, tabId: TAB_ID})
computer({action: "screenshot", tabId: TAB_ID})  // Evidence artifact

Example scenarios: 

@pytest.mark.parametrize("zoom", [25, 50, 100, 150, 200])
async def test_responsive_zoom(page, zoom):
    await page.set_viewport_size({"width": 1920, "height": 1080})
    await page.goto("http://localhost:8053")
    await page.evaluate(f"document.body.style.zoom = '{zoom}%'")
    await page.screenshot(path=f"tmp/screenshots/responsive-{zoom}pct.png")
    # Assert: No horizontal scroll
    scroll_width = await page.evaluate("document.body.scrollWidth")
    client_width = await page.evaluate("document.body.clientWidth")
    assert scroll_width == client_width, f"Horizontal scroll at {zoom}% zoom"

Consequences

Positive

  • Automated visual regression: No manual QA for zoom levels (saves 30 min/week)
  • CI/CD integration: GitHub Actions runs tests on every PR (fast feedback)
  • Compliance: Screenshot artifacts provide audit trail (SOC2/ISO 27001)
  • Cost-effective: $0 operational cost (local docker, no cloud)
  • Coverage: 90-100% bug detection (ADLC requirement met)

Negative

  • Tier 3 overhead: docker-compose up adds 10-15s to test startup
  • Playwright download: ~200MB Chromium browser (one-time cost)
  • Test duration: 5-10min per full E2E run (acceptable for CI/CD)
  • Maintenance: Screenshots must be updated when UI intentionally changes

Risks

  • Risk 1 (Playwright MCP network failure): Probability LOW, Impact MEDIUM

  • Mitigation: Pre-download browsers in CI/CD, retry logic with 3 attempts

  • Risk 2 (False positive screenshot diffs): Probability MEDIUM, Impact LOW

  • Mitigation: Use Playwright's toHaveScreenshot() with threshold tolerance (5% pixel diff)

  • Risk 3 (Docker not available in CI): Probability LOW, Impact HIGH

  • Mitigation: GitHub Actions has docker pre-installed, add pre-flight check

Alternatives Considered

  1. Cypress instead of Playwright: Rejected because Playwright MCP server already integrated in .mcp-testing.json
  2. Selenium WebDriver: Rejected because Playwright has better performance (2x faster) and native screenshot comparison
  3. Manual testing only: Rejected because not scalable and violates ADLC 3-tier requirement
  4. LocalStack for Tier 2: Rejected because ITSM has no AWS API dependencies (pure frontend)

References

Evidence

  • Tier 1 test results: tmp/cloud-infrastructure/test-results/tier1-snapshot-itsm-overhaul.log
  • Tier 3 test results: tmp/cloud-infrastructure/test-results/tier3-e2e-itsm-overhaul.log
  • Playwright screenshots: tmp/cloud-infrastructure/test-results/screenshots/responsive-{zoom}pct.png (5 files)
  • MCP configuration: .mcp-testing.json (Playwright server already configured)