runbooks validation¶
Assess | 9 commands | Management. Auto-generated from Click registry on 2026-06-26.
QA/QC: 0/9 commands PASSED (v1.3.17)
L1 --help: 0/9 | L2 params: PASS | L4 cross-validation: N/A
First time? Set up your AWS profiles
Before running any command, configure your AWS SSO profiles. See the Single Account or Multi-Account Landing Zone tabs below for copy-paste setup blocks.
AWS Profile Configuration¶
All runbooks commands support these common options for AWS authentication:
| Option | Scope | When to Use |
|---|---|---|
--profile PROFILE |
Single account | Developer/operator targeting one AWS account |
--all-profile |
All accounts (Landing Zone) | Platform team — discovers across all SSO profiles |
--region REGION |
Override region | Non-default region (default: ap-southeast-2) |
--dry-run |
Safe mode | Analysis only, no mutations (recommended for first run) |
--output-dir DIR |
Output path | Directory for generated reports (default: output/) |
--format FORMAT |
Output format | table, json, csv, markdown (varies by command) |
Copy and configure:
# =============================================================
# AWS Single Account Configuration
# =============================================================
export AWS_REGION="ap-southeast-2"
export AWS_PROFILE="your-account-profile"
# Authenticate via SSO
aws sso login --profile $AWS_PROFILE
# Verify
aws sts get-caller-identity --profile $AWS_PROFILE
# Run any command
runbooks finops dashboard --profile $AWS_PROFILE
Copy and configure all 4 environment variables:
# =============================================================
# AWS Multi-Account Landing Zone Configuration
# =============================================================
export AWS_REGION="ap-southeast-2"
## Single account (default fallback)
export AWS_PROFILE="your-default-profile"
## FinOps/Billing profile (READ-ONLY access to Cost Explorer)
export AWS_BILLING_PROFILE="your-billing-readonly-profile"
## Management account profile (Organizations, Control Tower)
export AWS_MANAGEMENT_PROFILE="your-management-readonly-profile"
## Centralized Operations account profile (for shared resources)
export AWS_OPERATIONS_PROFILE="your-operations-readonly-profile"
# =============================================================
# Authenticate all profiles
# =============================================================
aws sso login --profile $AWS_BILLING_PROFILE
aws sso login --profile $AWS_MANAGEMENT_PROFILE
aws sso login --profile $AWS_OPERATIONS_PROFILE
# =============================================================
# Verify connectivity
# =============================================================
aws sts get-caller-identity --profile $AWS_BILLING_PROFILE
aws sts get-caller-identity --profile $AWS_MANAGEMENT_PROFILE
# =============================================================
# Run org-wide commands
# =============================================================
runbooks inventory collect --all-profile --region $AWS_REGION
runbooks finops dashboard --all-profile --format table
Org-Wide Profile Routing — 4-Step Methodology¶
Each AWS profile has exactly one correct purpose. Mixing them returns empty results, not errors
(CROSS_ACCOUNT_SILENT_ZERO anti-pattern).
Authoritative standard (DISC-001): .claude/skills/aws/org-wide-resource-discovery.md — canonical 4-step methodology with P-ENRICH ranking and Config aggregator vs Resource Explorer priority. See also: aws-profile-semantics.md.
| Step | Name | Profile Env Var | Flag | API / Service | What-If Wrong |
|---|---|---|---|---|---|
| 1. DISCOVER | Org-wide resource inventory | $AWS_OPERATIONS_PROFILE |
--all-profile |
P1 (authoritative, un-capped): AWS Config aggregator (runbooks inventory resource-explorer default backend) — all accounts, paginated, no 1,000-result ceiling. P3 (cross-check only): Resource Explorer Search API — capped at 1,000 results per query; treat any count of exactly 1,000 as suspect. |
Wrong profile → aggregator index not found → silent-zero count |
| 2. ENUMERATE | Account names & IDs | $AWS_MANAGEMENT_PROFILE |
--all-profile |
AWS Organizations ListAccounts |
Wrong profile → empty account list → can't resolve resource owners |
| 3. ENRICH | Per-account attributes | Per-workload ~/.aws/config profile |
--profile |
Single-account describe-* / get-* (encryption, lifecycle, tags — aggregator can't return these) |
Skip this step → counts + cost present but target columns NULL |
| 4. COST | Cost attribution | $AWS_BILLING_PROFILE |
--all-profile |
Cost Explorer GetCostAndUsage |
Wrong profile → no CE access → cost columns blank or AccessDenied |
When to use --profile vs --all-profile
--all-profile $AWS_<PURPOSE>_PROFILE— Landing Zone-wide (hub accounts: Operations, Management, Billing). Always use the matching env var; never pass an LZ-wide profile with--profile.--profile $AWS_PROFILE— Single workload account only (Steps 3 + per-workload incident investigation).
Using --profile with an LZ-wide profile env var (e.g. --profile $AWS_BILLING_PROFILE) is a routing bug — the LZ hub account has no workload resources, so results are silently empty.
--all-profile flag behaviour differs by command group
--all-profile is value-taking (accepts a profile string) for finops, inventory, cfat, vpc, csdm, and workspaces command groups. For runbooks security, --all-profile is a boolean flag (bare, no value). Always use runbooks security baseline --all-profile (no value); never --all-profile $AWS_OPERATIONS_PROFILE for security commands.
# Step 1 — DISCOVER org-wide (Config aggregator, un-capped, P1 authoritative)
runbooks inventory resource-explorer --all-profile $AWS_OPERATIONS_PROFILE
# Step 2 — ENUMERATE accounts (Organizations API)
runbooks inventory discover --all-profile $AWS_MANAGEMENT_PROFILE
# Step 3 — ENRICH per-account (loop each workload profile from ~/.aws/config)
# Resolve account_id -> profile_name via runbooks.common.resolve_readonly_profile() (RB-3):
# from runbooks.common import resolve_readonly_profile
# profile = resolve_readonly_profile(account_id) # canonical account_id->profile resolver
runbooks security scan --profile $AWS_PROFILE # one call per workload account
# Step 4 — COST (Cost Explorer, org-wide consolidated billing)
runbooks finops dashboard --all-profile $AWS_BILLING_PROFILE
Environment Variables Reference¶
| Variable | Required | Purpose |
|---|---|---|
AWS_REGION |
Yes | Target AWS region (default: ap-southeast-2) |
AWS_PROFILE |
Yes | Default profile when --profile is omitted |
AWS_BILLING_PROFILE |
LZ only | Cost Explorer data enrichment |
AWS_MANAGEMENT_PROFILE |
LZ only | Organizations metadata enrichment |
AWS_OPERATIONS_PROFILE |
LZ only | Centralized Operations shared resources |
RUNBOOKS_TEST_MODE |
No | Set to 1 for offline/mock mode (no AWS calls) |
Commands¶
| Command | Description | Params | API Type |
|---|---|---|---|
benchmark |
Run performance benchmark for MCP validation framework with universal profile support. | 7 | read-only |
cert-status |
Show per-command certification status from the MS-01 board. | 2 | read-only |
costs |
Validate Cost Explorer data accuracy with universal profile support. | 5 | read-only |
organizations |
Validate Organizations API data accuracy with universal profile support. | 4 | read-only |
single |
Validate a single operation with universal profile support. | 6 | read-only |
status |
Show MCP validation framework status with universal profile support. | 4 | read-only |
sync-check |
Detect notebooks that break after CLI command changes. | 2 | read-only |
test |
Comprehensive test command integration for Sprint 1 validation framework. | 10 | read-only |
validate-all |
Run comprehensive validation across all critical operations with universal profile support. | 7 | read-only |
Quick Start by Role¶
Cross-validation and benchmarking for data accuracy.
| Task | Command |
|---|---|
| Validation status | runbooks validation status --profile $AWS_BILLING_PROFILE |
| Task | Command |
|---|---|
| Cost validation | runbooks validation costs --profile $AWS_BILLING_PROFILE |
| Full benchmark | runbooks validation benchmark --profile $AWS_BILLING_PROFILE |
| Task | Command |
|---|---|
| Quick validate | runbooks validation single --profile $AWS_PROFILE |
Command Details¶
runbooks validation benchmark¶
Run performance benchmark for MCP validation framework with universal profile support.
All Parameters (7)
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--target-accuracy |
FLOAT | 99.5 |
Target accuracy percentage |
--iterations |
INT | 5 |
Number of benchmark iterations |
--performance-target |
FLOAT | 30.0 |
Performance target in seconds |
--all |
BOOL | False |
Use all available AWS profiles for multi-account benchmarking |
runbooks validation cert-status¶
Show per-command certification status from the MS-01 board.
| Parameter | Type | Default | Description |
|---|---|---|---|
--group |
STRING | - | Filter by command group name |
--certified-only |
BOOL | False |
Show only CERTIFIED commands |
runbooks validation costs¶
Validate Cost Explorer data accuracy with universal profile support.
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--tolerance |
FLOAT | 5.0 |
Cost variance tolerance percentage |
--all |
BOOL | False |
Use all available AWS profiles for multi-account cost validation |
runbooks validation organizations¶
Validate Organizations API data accuracy with universal profile support.
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--all |
BOOL | False |
Use all available AWS profiles for multi-account organizations validation |
runbooks validation single¶
Validate a single operation with universal profile support.
All Parameters (6)
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--operation |
CHOICE(costs | organizations | ec2 |
--tolerance |
FLOAT | 5.0 |
Tolerance percentage |
--all |
BOOL | False |
Use all available AWS profiles for multi-account single operation validation |
runbooks validation status¶
Show MCP validation framework status with universal profile support.
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--all |
BOOL | False |
Check status for all available AWS profiles |
runbooks validation sync-check¶
Detect notebooks that break after CLI command changes.
| Parameter | Type | Default | Description |
|---|---|---|---|
--csv |
PATH | - | Path to cli-commands-params.csv (default: auto-detected from package root) |
--notebooks-dir |
PATH | - | Root directory to scan for .ipynb files (default: auto-detected from package root) |
runbooks validation test¶
Comprehensive test command integration for Sprint 1 validation framework.
All Parameters (10)
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--module |
CHOICE(finops | inventory | security |
--component |
STRING | - | Specific component within module (e.g., 'ec2-snapshots' for finops module) |
--test-type |
CHOICE(basic | profile-tests | mcp-validation |
--performance-target |
FLOAT | 30.0 |
Performance target in seconds |
--accuracy-target |
FLOAT | 99.5 |
MCP validation accuracy target percentage |
--generate-evidence |
BOOL | False |
Generate comprehensive evidence reports for manager review |
--export-results |
BOOL | False |
Export test results to JSON file |
runbooks validation validate-all¶
Run comprehensive validation across all critical operations with universal profile support.
All Parameters (7)
| Parameter | Type | Default | Description |
|---|---|---|---|
--profile |
STRING | - | AWS profile for single-account operations. |
--region |
STRING | ap-southeast-2 |
AWS region override (default: ap-southeast-2) |
--dry-run |
BOOL | True |
Safe analysis mode - no resource modifications (enterprise default) |
--tolerance |
FLOAT | 5.0 |
Tolerance percentage for variance detection |
--performance-target |
FLOAT | 30.0 |
Performance target in seconds |
--save-report |
BOOL | False |
Save detailed report to artifacts |
--all |
BOOL | False |
Use all available AWS profiles for multi-account validation |