fix: [nomad-step-2] S2.6 — CI: vault policy fmt + validate + roles.yaml check (#884)
This commit is contained in:
Agent
parent
88e49b9e9d
commit
108b928cfc
5 changed files with 725 additions and 9 deletions
|
|
@ -48,12 +48,43 @@ validation.
|
|||
1. Drop a file matching one of the four naming patterns above. Use an
|
||||
existing file in the same family as the template — comment header,
|
||||
capability list, and KV path layout should match the family.
|
||||
2. Run `tools/vault-apply-policies.sh --dry-run` to confirm the new
|
||||
2. Run `vault policy fmt -write <file>` to ensure consistent formatting.
|
||||
3. Run `vault policy validate <file>` locally to check syntax + semantics.
|
||||
4. Run `tools/vault-apply-policies.sh --dry-run` to confirm the new
|
||||
basename appears in the planned-work list with the expected SHA.
|
||||
3. Run `tools/vault-apply-policies.sh` against a Vault instance to
|
||||
5. Run `tools/vault-apply-policies.sh` against a Vault instance to
|
||||
create it; re-run to confirm it reports `unchanged`.
|
||||
4. The CI fmt + validate step lands in S2.6 (#884). Until then
|
||||
`vault policy fmt <file>` locally is the fastest sanity check.
|
||||
|
||||
## Policy lifecycle
|
||||
|
||||
Adding a new policy is a three-step process:
|
||||
|
||||
1. **Add policy HCL** — Drop a file in `vault/policies/` matching one of the
|
||||
naming patterns. Run `vault policy fmt <file>` locally to ensure consistent
|
||||
formatting.
|
||||
2. **Update roles.yaml** — Add a JWT auth role in `vault/roles.yaml` that
|
||||
references the new policy name (basename without `.hcl`).
|
||||
3. **Attach to Nomad job** — In S2.4, add the policy to a jobspec's
|
||||
`template { vault { policies = ["<policy-name>"] } }` stanza.
|
||||
|
||||
CI enforces:
|
||||
|
||||
- `vault policy fmt -check` — all `.hcl` files must be formatted
|
||||
- `vault policy validate` — syntax + semantic check (no unknown stanzas,
|
||||
valid capabilities)
|
||||
- `roles.yaml` validator — each role must reference a policy that exists
|
||||
in `vault/policies/`
|
||||
- secret-scan gate — no literal secrets in policy files (rare but
|
||||
dangerous copy-paste mistake)
|
||||
|
||||
## Common failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `vault policy fmt -check` fails | HCL not formatted (wrong indentation, trailing spaces) | Run `vault policy fmt -write <file>` |
|
||||
| `vault policy validate` fails | Unknown stanza, invalid capability, missing required field | Check Vault docs; valid capabilities: `read`, `list`, `create`, `update`, `delete`, `sudo` |
|
||||
| `roles.yaml` validator fails | Policy name in role doesn't match any `.hcl` basename | Ensure policy name = filename without `.hcl` |
|
||||
| secret-scan fails | Literal secret value embedded (e.g., `token = "abc123..."`) | Use env var reference (`$TOKEN`) or sops/age-encrypted secret |
|
||||
|
||||
## What this directory does NOT own
|
||||
|
||||
|
|
@ -63,4 +94,3 @@ validation.
|
|||
(#881).
|
||||
- **Writing the secret values themselves.** That's S2.2 (#880) via
|
||||
`tools/vault-import.sh`.
|
||||
- **CI policy fmt + validate + roles.yaml check.** That's S2.6 (#884).
|
||||
|
|
|
|||
224
vault/policies/validate.sh
Executable file
224
vault/policies/validate.sh
Executable file
|
|
@ -0,0 +1,224 @@
|
|||
#!/usr/bin/env bash
|
||||
# vault/policies/validate.sh — Validate Vault policy HCL files
|
||||
#
|
||||
# Usage: vault/policies/validate.sh [--check-exists]
|
||||
#
|
||||
# This script provides CI validation for Vault policy files:
|
||||
# 1. `vault policy fmt -check` — ensures consistent formatting (non-destructive)
|
||||
# 2. `vault policy validate` — syntax + semantic validation (requires Vault dev mode)
|
||||
# 3. Optional: check that referenced policies exist in roles.yaml
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — all checks pass
|
||||
# 1 — formatting or validation error
|
||||
# 2 — policy reference validation error (roles.yaml check)
|
||||
#
|
||||
# Environment:
|
||||
# VAULT_ADDR — Vault server URL (defaults to http://127.0.0.1:8200 for dev mode)
|
||||
# VAULT_TOKEN — Dev mode token (defaults to "root" for CI)
|
||||
#
|
||||
# CI usage:
|
||||
# vault/policies/validate.sh
|
||||
# vault/policies/validate.sh --check-exists # when roles.yaml exists
|
||||
#
|
||||
# Notes:
|
||||
# - fmt -check is non-destructive; it only reports diff
|
||||
# - validate requires a running Vault instance (dev mode is sufficient for CI)
|
||||
# - Exit 2 is tolerated for advisory warnings (TLS-disabled listeners)
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROLES_YAML="${SCRIPT_DIR}/../roles.yaml"
|
||||
VAULT_ADDR="${VAULT_ADDR:-http://127.0.0.1:8200}"
|
||||
VAULT_TOKEN="${VAULT_TOKEN:-root}"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "$0") [OPTIONS]
|
||||
|
||||
Validate Vault policy HCL files.
|
||||
|
||||
Options:
|
||||
--check-exists Also validate that roles.yaml references exist policies
|
||||
--help Show this help message
|
||||
|
||||
Environment:
|
||||
VAULT_ADDR Vault server URL (default: http://127.0.0.1:8200)
|
||||
VAULT_TOKEN Dev mode token (default: root)
|
||||
|
||||
Exit codes:
|
||||
0 All checks pass
|
||||
1 Formatting or validation error
|
||||
2 Policy reference validation error (roles.yaml check)
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Check if Vault is available (dev mode is sufficient for CI)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
check_vault_available() {
|
||||
local rc=0
|
||||
# Try to fetch version; exit 2 is advisory (TLS warning), which is OK for CI
|
||||
if ! vault status > /dev/null 2>&1; then
|
||||
rc=$?
|
||||
case "$rc" in
|
||||
0|2) return 0 ;; # OK for CI
|
||||
*) echo "vault/policies/validate.sh: Vault not available (exit $rc)" >&2; return 1 ;;
|
||||
esac
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 1: vault policy fmt -check
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
fmt_check() {
|
||||
local failed=0
|
||||
local hcl_files
|
||||
hcl_files=$(find "$SCRIPT_DIR" -maxdepth 1 -name '*.hcl' -type f 2>/dev/null || true)
|
||||
|
||||
if [ -z "$hcl_files" ]; then
|
||||
echo "vault/policies/validate.sh: no .hcl files found in $SCRIPT_DIR" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
for f in $hcl_files; do
|
||||
echo "fmt-check: $f"
|
||||
if ! vault policy fmt -check "$f" > /dev/null 2>&1; then
|
||||
echo " ERROR: file not formatted correctly" >&2
|
||||
vault policy fmt -check "$f" 2>&1 | head -20 >&2 || true
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
|
||||
return $failed
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 2: vault policy validate (syntax + semantic)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
validate_syntax() {
|
||||
local failed=0
|
||||
local hcl_files
|
||||
hcl_files=$(find "$SCRIPT_DIR" -maxdepth 1 -name '*.hcl' -type f 2>/dev/null || true)
|
||||
|
||||
if [ -z "$hcl_files" ]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Check Vault is available first
|
||||
if ! check_vault_available; then
|
||||
echo "vault/policies/validate.sh: skipping validation (Vault unavailable)" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
for f in $hcl_files; do
|
||||
echo "validate: $f"
|
||||
local rc=0
|
||||
if ! vault policy validate "$f" > /dev/null 2>&1; then
|
||||
rc=$?
|
||||
case "$rc" in
|
||||
0) ;; # Should not happen, but be safe
|
||||
1|2)
|
||||
echo " ERROR: validation failed (exit $rc)" >&2
|
||||
vault policy validate "$f" 2>&1 | head -20 >&2 || true
|
||||
failed=1
|
||||
;;
|
||||
*)
|
||||
echo " ERROR: unexpected exit code $rc" >&2
|
||||
failed=1
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
done
|
||||
|
||||
return $failed
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 3: Check that roles.yaml references exist
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
check_policy_references() {
|
||||
if [ ! -f "$ROLES_YAML" ]; then
|
||||
echo "vault/policies/validate.sh: roles.yaml not found, skipping reference check" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
local failed=0
|
||||
local policy_names
|
||||
|
||||
# Get list of policy names (basenames without .hcl)
|
||||
policy_names=$(find "$SCRIPT_DIR" -maxdepth 1 -name '*.hcl' -type f -exec basename {} .hcl \; | sort)
|
||||
|
||||
# Extract policy names from roles.yaml using yq or grep+sed
|
||||
local referenced_policies
|
||||
if command -v yq > /dev/null 2>&1; then
|
||||
# yq is available, use it
|
||||
referenced_policies=$(yq -r '.roles[].policies[]?' "$ROLES_YAML" 2>/dev/null | sort -u || true)
|
||||
else
|
||||
# Fallback: grep for 'policies:' lines and extract values
|
||||
referenced_policies=$(grep -E '^\s*policies:' "$ROLES_YAML" 2>/dev/null | \
|
||||
sed -E 's/.*policies:\s*\[(.*)\].*/\1/' | \
|
||||
tr ',' '\n' | \
|
||||
sed 's/^[[:space:]]*"//;s/"[[:space:]]*$//' | \
|
||||
sort -u || true)
|
||||
fi
|
||||
|
||||
if [ -z "$referenced_policies" ]; then
|
||||
echo "vault/policies/validate.sh: no policies referenced in roles.yaml" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
for policy in $referenced_policies; do
|
||||
if ! echo "$policy_names" | grep -q "^${policy}$"; then
|
||||
echo "vault/policies/validate.sh: ERROR: policy '$policy' referenced in roles.yaml but not found" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
|
||||
return $failed
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Main
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
check_refs=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--check-exists)
|
||||
check_refs=1
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1" >&2
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo "vault/policies/validate.sh — validating policy HCL files"
|
||||
echo " VAULT_ADDR: $VAULT_ADDR"
|
||||
echo " roles.yaml: $ROLES_YAML (exists: $([ -f "$ROLES_YAML" ] && echo yes || echo no))"
|
||||
echo ""
|
||||
|
||||
# Run fmt check
|
||||
fmt_check || exit 1
|
||||
|
||||
# Run syntax validation
|
||||
validate_syntax || exit 1
|
||||
|
||||
# Run reference check if requested
|
||||
if [ "$check_refs" -eq 1 ]; then
|
||||
check_policy_references || exit 2
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "vault/policies/validate.sh: all checks passed"
|
||||
exit 0
|
||||
220
vault/validate-roles.sh
Executable file
220
vault/validate-roles.sh
Executable file
|
|
@ -0,0 +1,220 @@
|
|||
#!/usr/bin/env bash
|
||||
# vault/validate-roles.sh — Validate roles.yaml for Vault workload identity
|
||||
#
|
||||
# Usage: vault/validate-roles.sh
|
||||
#
|
||||
# This script validates the roles.yaml file for Nomad workload identity:
|
||||
# 1. yamllint check — ensures YAML syntax is valid
|
||||
# 2. Policy reference check — each role references a policy that exists
|
||||
# 3. Required fields check — each role has required fields (name, policies, auth)
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — all checks pass
|
||||
# 1 — YAML syntax or validation error
|
||||
#
|
||||
# Environment:
|
||||
# VAULT_POLICY_DIR — Directory containing policy HCL files (default: vault/policies/)
|
||||
#
|
||||
# CI usage:
|
||||
# vault/validate-roles.sh
|
||||
#
|
||||
# Notes:
|
||||
# - Requires yamllint to be installed
|
||||
# - Policy existence check requires Vault policy files to exist
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROLES_YAML="${SCRIPT_DIR}/roles.yaml"
|
||||
VAULT_POLICY_DIR="${VAULT_POLICY_DIR:-${SCRIPT_DIR}/policies}"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "$0")
|
||||
|
||||
Validate vault/roles.yaml for Vault workload identity configuration.
|
||||
|
||||
Options:
|
||||
--help Show this help message
|
||||
|
||||
Environment:
|
||||
VAULT_POLICY_DIR — Directory containing policy HCL files (default: vault/policies/)
|
||||
|
||||
Exit codes:
|
||||
0 — all checks pass
|
||||
1 — YAML syntax or validation error
|
||||
|
||||
Requires:
|
||||
- yamllint (for YAML syntax check)
|
||||
- yq (optional, for YAML parsing; falls back to grep/sed)
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Check if required tools are available
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
check_dependencies() {
|
||||
if ! command -v yamllint > /dev/null 2>&1; then
|
||||
echo "validate-roles.sh: yamllint not found in PATH" >&2
|
||||
echo " Install with: pip install yamllint" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 1: yamllint check
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
yamllint_check() {
|
||||
echo "yamllint: $ROLES_YAML"
|
||||
if ! yamllint -q "$ROLES_YAML" 2>&1; then
|
||||
echo " ERROR: yamllint found issues" >&2
|
||||
return 1
|
||||
fi
|
||||
echo " OK"
|
||||
return 0
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 2: Extract policy names from roles.yaml
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
get_referenced_policies() {
|
||||
if command -v yq > /dev/null 2>&1; then
|
||||
# yq is available, use it
|
||||
yq -r '.roles[].policies[]?' "$ROLES_YAML" 2>/dev/null | sort -u || true
|
||||
else
|
||||
# Fallback: grep for 'policies:' lines and extract values
|
||||
grep -E '^\s*policies:' "$ROLES_YAML" 2>/dev/null | \
|
||||
sed -E 's/.*policies:\s*\[(.*)\].*/\1/' | \
|
||||
tr ',' '\n' | \
|
||||
sed 's/^[[:space:]]*"//;s/"[[:space:]]*$//' | \
|
||||
grep -v '^$' | \
|
||||
sort -u || true
|
||||
fi
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 3: Check that referenced policies exist
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
check_policy_references() {
|
||||
local failed=0
|
||||
|
||||
if [ ! -d "$VAULT_POLICY_DIR" ]; then
|
||||
echo "validate-roles.sh: policy directory not found: $VAULT_POLICY_DIR" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Get list of policy names (basenames without .hcl)
|
||||
local policy_names
|
||||
policy_names=$(find "$VAULT_POLICY_DIR" -maxdepth 1 -name '*.hcl' -type f -exec basename {} .hcl \; | sort)
|
||||
|
||||
if [ -z "$policy_names" ]; then
|
||||
echo "validate-roles.sh: no .hcl files found in $VAULT_POLICY_DIR" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
local referenced_policies
|
||||
referenced_policies=$(get_referenced_policies)
|
||||
|
||||
if [ -z "$referenced_policies" ]; then
|
||||
echo "validate-roles.sh: no policies referenced in roles.yaml" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
for policy in $referenced_policies; do
|
||||
if ! echo "$policy_names" | grep -q "^${policy}$"; then
|
||||
echo "validate-roles.sh: ERROR: policy '$policy' referenced in roles.yaml but not found" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
|
||||
return $failed
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 4: Check required fields in roles
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
check_required_fields() {
|
||||
local failed=0
|
||||
|
||||
if command -v yq > /dev/null 2>&1; then
|
||||
# Check each role has required fields
|
||||
local roles_count
|
||||
roles_count=$(yq '.roles | length' "$ROLES_YAML" 2>/dev/null || echo "0")
|
||||
|
||||
if [ "$roles_count" -eq 0 ]; then
|
||||
echo "validate-roles.sh: WARNING: no roles defined in roles.yaml" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
for ((i=0; i<roles_count; i++)); do
|
||||
local role_name
|
||||
role_name=$(yq -r ".roles[$i].name // \"<unnamed>\"" "$ROLES_YAML" 2>/dev/null || echo "<unnamed>")
|
||||
|
||||
# Check for name field
|
||||
if [ "$role_name" = "<unnamed>" ]; then
|
||||
echo "validate-roles.sh: ERROR: role missing 'name' field" >&2
|
||||
failed=1
|
||||
fi
|
||||
|
||||
# Check for policies field
|
||||
local policies_count
|
||||
policies_count=$(yq ".roles[$i].policies | length" "$ROLES_YAML" 2>/dev/null || echo "0")
|
||||
if [ "$policies_count" -eq 0 ]; then
|
||||
echo "validate-roles.sh: ERROR: role '$role_name' has no policies defined" >&2
|
||||
failed=1
|
||||
fi
|
||||
|
||||
# Check for auth field (JWT auth config)
|
||||
local auth_method
|
||||
auth_method=$(yq -r ".roles[$i].auth // \"<none>\"" "$ROLES_YAML" 2>/dev/null || echo "<none>")
|
||||
if [ "$auth_method" = "<none>" ]; then
|
||||
echo "validate-roles.sh: ERROR: role '$role_name' has no auth configuration" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
return $failed
|
||||
}
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Main
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# Parse arguments
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--help|-h)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1" >&2
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo "vault/validate-roles.sh — validating roles.yaml"
|
||||
echo " roles.yaml: $ROLES_YAML (exists: $([ -f "$ROLES_YAML" ] && echo yes || echo no))"
|
||||
echo " policy dir: $VAULT_POLICY_DIR"
|
||||
echo ""
|
||||
|
||||
# Exit early if roles.yaml doesn't exist (it will be created in a future step)
|
||||
if [ ! -f "$ROLES_YAML" ]; then
|
||||
echo "vault/validate-roles.sh: roles.yaml not found, skipping validation" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check dependencies
|
||||
check_dependencies
|
||||
|
||||
# Run validations
|
||||
yamllint_check || exit 1
|
||||
check_policy_references || exit 1
|
||||
check_required_fields || exit 1
|
||||
|
||||
echo ""
|
||||
echo "vault/validate-roles.sh: all checks passed"
|
||||
exit 0
|
||||
Loading…
Add table
Add a link
Reference in a new issue