MCP Cross-Validation ≥99.5%

What is MCP Validation?

The Model Context Protocol (MCP) validation layer ensures that runbooks finops output matches authoritative cloud provider APIs (AWS Cost Explorer, Azure Cost Management) within ≥99.5% accuracy.

This quality gate prevents: - Silent drift between runbooks output and actual billing data - Rounding errors accumulating across thousands of resources - API schema changes causing output fields to diverge from reality - FinOps leaders making decisions on outdated or incorrect cost data

---

3-Way Validation Methodology

All FOCUS-compliant cost data produced by runbooks must pass a 3-way validation before publication to FinOps tools, ServiceNow, or executive dashboards.

Level 1: Notebook Calculation

• Tool: jupyter notebooks in cloudops/finops/notebooks/
• Method: Manual FOCUS calculation on sample data (10-100 line items)
• Result: Base cost values without aggregation
• Gate: Sample passes internal business-logic review

Level 2: MCP Cross-Validation

• Tool: awslabs.cost-explorer-mcp for AWS; awslabs.core-mcp for Azure
• Method: Fetch same period/account/service via MCP, compare field-by-field
• Result: Accuracy % = (matching_rows / total_rows) × 100
• Gate: ≥99.5% match required

Example (AWS):

# 1. Export runbooks FOCUS data
runbooks finops export-focus --profile $AWS_BILLING_PROFILE \
  --month 2026-05 --format csv > /tmp/runbooks-focus-2026-05.csv

# 2. Fetch Cost Explorer data via MCP
curl -X POST http://localhost:3000/mcp/awslabs.cost-explorer-mcp \
  -d '{"method":"GetCostAndUsage","params":{"TimePeriod":{"Start":"2026-05-01","End":"2026-05-31"},"Granularity":"MONTHLY","Metrics":["UnblendedCost"],"GroupBy":[{"Type":"DIMENSION","Key":"SERVICE"}]}}' \
  > /tmp/ce-2026-05.json

# 3. Cross-validate
runbooks validation cross-validate \
  --file /tmp/runbooks-focus-2026-05.csv \
  --method focus-vs-ce \
  --profile $AWS_BILLING_PROFILE \
  --output /tmp/validation-report-2026-05-21.json

Level 3: Native AWS CLI Validation

• Tool: AWS CLI or Azure CLI (direct API, no MCP)
• Method: Spot-check key fields (total cost, service breakdown, region distribution)
• Result: Manual inspection of high-variance rows
• Gate: No unexplained discrepancies >0.5% of period total

Example:

# Validate total monthly cost matches AWS CLI
RUNBOOKS_TOTAL=$(tail -n +2 /tmp/runbooks-focus-2026-05.csv | awk -F, '{sum+=$NF} END {print sum}')
AWS_TOTAL=$(aws ce get-cost-and-usage \
  --time-period Start=2026-05-01,End=2026-05-31 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --profile $AWS_BILLING_PROFILE \
  --query 'ResultsByTime[0].Total.UnblendedCost.Amount' \
  --output text)

# Compute percentage difference
DIFF=$(echo "scale=4; (($RUNBOOKS_TOTAL - $AWS_TOTAL) / $AWS_TOTAL) * 100" | bc)
if (( $(echo "$DIFF < 0.5" | bc -l) )); then
  echo "PASS: Discrepancy $DIFF% is within ≤0.5% tolerance"
else
  echo "FAIL: Discrepancy $DIFF% exceeds tolerance"
fi

---

MCP Servers Used

awslabs.cost-explorer-mcp

• Purpose: AWS-native cost retrieval and aggregation
• APIs Covered: GetCostAndUsage, GetDimensionValues, GetReservationUtilization
• Validation Target: Billing rows, cost aggregation by service/region/account
• Accuracy Threshold: ≥99.5% match with Cost Explorer console

awslabs.core-mcp

• Purpose: General AWS API operations and metadata
• APIs Covered: sts:GetCallerIdentity, ce:*, organizations:*
• Validation Target: Account ID resolution, org hierarchy verification
• Accuracy Threshold: 100% match (metadata operations)

---

Quality Gate Rules

Rule	Metric	Pass Criteria	Evidence
Data Completeness	Rows in runbooks vs MCP	≥95% parity (runbooks may have more detail rows)	validation-report.json completeness_pct
Cost Accuracy	Total cost match	≥99.5% within 0.5% of MCP total	validation-report.json cost_discrepancy_pct
Field Match	Per-field accuracy	≥99.5% of fields match exactly or within rounding tolerance	validation-report.json field_accuracy_matrix
Service Breakdown	Service-level costs	≥99.5% for top 10 services by cost	validation-report.json service_breakdown_variance
Timeliness	Validation currency	Validation run within 3 days of month-end	validation-report.json timestamp

---

Validation Workflow

Before publishing FOCUS data to production:

1. Schedule validation immediately after month-end close (typically 3-5 days after month-end)
2. Run 3-way validation (Notebook → MCP → CLI)
3. Document results in tmp/command-center/mcp-validation/validation-report-YYYY-MM-DD.json
4. Review high-variance rows (>1% difference) manually
5. Publish signed report to Confluence with pass/fail stamp
6. Export to ServiceNow/Atlassian only AFTER ≥99.5% gate passes

Script: runbooks validation finops-monthly --profile $AWS_BILLING_PROFILE --month YYYY-MM (orchestrates all 3 levels)

---

Common Variance Causes & Resolution

Cause	Impact	Resolution
Timing skew (charge period vs data fetch)	0-2% variance	Re-run validation 24h later; document as timing noise
Tax differences (state/region specific)	0.1-0.5% variance	Verify tax calculation method; acceptable if consistent
Rounding in intermediate aggregations	<0.1% variance	Use Decimal type for all cost calculations; no floats
Committed discount application timing	0-2% variance in EffectiveCost	Document discount periods in validation report; flag as known
Reserved Instance amortization	0-3% variance	Use --amortized flag consistently; disclose methodology
API limit hits (incomplete response)	>2% variance	Implement retry loop with exponential backoff; mark incomplete

---

Skill Reference

Full methodology documentation: See .adlc/.claude/skills/quality/mcp-validation/SKILL.md for: - Detailed 3-way validation procedure - Error handling and recovery - Scoring methodology (how ≥99.5% is calculated) - False-positive detection and suppression - Regression testing between versions

---

CI/CD Integration

The validation is automated in CloudOps pipelines:

# .github/workflows/finops-validation.yml
- name: FOCUS Data Validation
  run: |
    runbooks validation finops-monthly \
      --profile $AWS_BILLING_PROFILE \
      --month $(date -d "last month" +%Y-%m) \
      --output-json /tmp/validation-report.json \
      --fail-on-accuracy-below 99.5

- name: Upload Validation Evidence
  if: always()
  uses: actions/upload-artifact@v3
  with:
    name: mcp-validation-report-${{ github.run_id }}
    path: /tmp/validation-report.json

---

When Validation Fails

If the accuracy gate falls below 99.5%:

1. Stop publication — do not export to ServiceNow/Atlassian
2. Investigate high-variance rows — validation-report.json identifies them
3. Check for breaking changes: - AWS API schema changes (new charge categories) - Billing system updates (tax handling, discount logic) - runbooks code changes (mapping logic drift)
4. Document root cause in ticket with remediation
5. Re-run after fix — gate must pass before publication

Example Investigation:

# See top variance rows
jq '.variance_by_row | sort_by(-.discrepancy_pct) | .[0:10]' \
  /tmp/validation-report.json

---

Success Criteria

✅ All 4 pages authored
✅ FOCUS 1.2 column count documented (60 columns)
✅ runbooks finops coverage measured (73% = 44 of 60)
✅ 3-way validation methodology documented with CLI examples
✅ MCP servers cited (awslabs.cost-explorer-mcp, awslabs.core-mcp)
✅ CSV evidence file created: /tmp/command-center/finops-focus-mapping-2026-05-21.csv
✅ mkdocs build passes (4 pages in nav)

---

Last Updated: 2026-05-21
Next Review: Post-S2 delivery (2026-06-15)
Maintained By: finops-engineer + product-owner