Merge pull request 'fix: feat: consolidate secret stores — single granular secrets/*.enc, deprecate .env.vault.enc (#777)' (#806) from fix/issue-777 into main
All checks were successful
ci/woodpecker/push/ci Pipeline was successful

This commit is contained in:
dev-bot 2026-04-15 18:46:12 +00:00
commit defec3b255
14 changed files with 254 additions and 130 deletions

View file

@ -1,8 +1,7 @@
# Secrets — prevent .env files from being baked into the image # Secrets — prevent .env files and encrypted secrets from being baked into the image
.env .env
.env.enc .env.enc
.env.vault secrets/
.env.vault.enc
# Version control — .git is huge and not needed in image # Version control — .git is huge and not needed in image
.git .git

View file

@ -83,16 +83,17 @@ FORWARD_AUTH_SECRET= # [SECRET] Shared secret for Caddy ↔
# ── Vault-only secrets (DO NOT put these in .env) ──────────────────────── # ── Vault-only secrets (DO NOT put these in .env) ────────────────────────
# These tokens grant access to external systems (GitHub, ClawHub, deploy targets). # These tokens grant access to external systems (GitHub, ClawHub, deploy targets).
# They live ONLY in .env.vault.enc and are injected into the ephemeral runner # They live ONLY in secrets/<NAME>.enc (age-encrypted, one file per key) and are
# container at fire time (#745). lib/env.sh explicitly unsets them so agents # decrypted into the ephemeral runner container at fire time (#745, #777).
# can never hold them directly — all external actions go through vault dispatch. # lib/env.sh explicitly unsets them so agents can never hold them directly —
# all external actions go through vault dispatch.
# #
# GITHUB_TOKEN — GitHub API access (publish, deploy, post) # GITHUB_TOKEN — GitHub API access (publish, deploy, post)
# CLAWHUB_TOKEN — ClawHub registry credentials (publish) # CLAWHUB_TOKEN — ClawHub registry credentials (publish)
# CADDY_SSH_KEY — SSH key for Caddy log collection
# (deploy keys) — SSH keys for deployment targets # (deploy keys) — SSH keys for deployment targets
# #
# To manage vault secrets: disinto secrets edit-vault # To manage secrets: disinto secrets add/show/remove/list
# (vault redesign in progress: PR-based approval, see #73-#77)
# ── Project-specific secrets ────────────────────────────────────────────── # ── Project-specific secrets ──────────────────────────────────────────────
# Store all project secrets here so formulas reference env vars, never hardcode. # Store all project secrets here so formulas reference env vars, never hardcode.

1
.gitignore vendored
View file

@ -3,7 +3,6 @@
# Encrypted secrets — safe to commit (SOPS-encrypted with age) # Encrypted secrets — safe to commit (SOPS-encrypted with age)
!.env.enc !.env.enc
!.env.vault.enc
!.sops.yaml !.sops.yaml
# Per-box project config (generated by disinto init) # Per-box project config (generated by disinto init)

View file

@ -86,7 +86,7 @@ Each agent has a `.profile` repository on Forgejo storing `knowledge/lessons-lea
- All scripts start with `#!/usr/bin/env bash` and `set -euo pipefail` - All scripts start with `#!/usr/bin/env bash` and `set -euo pipefail`
- Source shared environment: `source "$(dirname "$0")/../lib/env.sh"` - Source shared environment: `source "$(dirname "$0")/../lib/env.sh"`
- Log to `$LOGFILE` using the `log()` function from env.sh or defined locally - Log to `$LOGFILE` using the `log()` function from env.sh or defined locally
- Never hardcode secrets — agent secrets come from `.env.enc`, vault secrets from `.env.vault.enc` (or `.env`/`.env.vault` fallback) - Never hardcode secrets — agent secrets come from `.env.enc`, vault secrets from `secrets/<NAME>.enc` (age-encrypted, one file per key)
- Never embed secrets in issue bodies, PR descriptions, or comments — use env var references (e.g. `$BASE_RPC_URL`) - Never embed secrets in issue bodies, PR descriptions, or comments — use env var references (e.g. `$BASE_RPC_URL`)
- ShellCheck must pass (CI runs `shellcheck` on all `.sh` files) - ShellCheck must pass (CI runs `shellcheck` on all `.sh` files)
- Avoid duplicate code — shared helpers go in `lib/` - Avoid duplicate code — shared helpers go in `lib/`
@ -179,8 +179,8 @@ Humans write these. Agents read and enforce them.
| AD-002 | **Concurrency is bounded per LLM backend, not per project.** One concurrent Claude session per OAuth credential pool; one concurrent session per llama-server instance. Containers with disjoint backends may run in parallel. | The single-thread invariant is about *backends*, not pipelines. **(a) Anthropic OAuth credentials race on token refresh** — each container uses a per-session `CLAUDE_CONFIG_DIR`, so Claude Code's native lockfile-based OAuth refresh handles contention automatically without external serialization. (Legacy: set `CLAUDE_EXTERNAL_LOCK=1` to re-enable the old `flock session.lock` wrapper for rollback.) **(b) llama-server has finite VRAM and one KV cache** — parallel inference thrashes the cache and risks OOM. All llama-backed agents serialize on the same lock. **(c) Disjoint backends are free to parallelize.** Today `disinto-agents` (Anthropic OAuth, runs `review,gardener`) runs concurrently with `disinto-agents-llama` (llama, runs `dev`) on the same project — they share neither OAuth state nor llama VRAM. **(d) Per-project work-conflict safety** (no duplicate dev work, no merge conflicts on the same branch) is enforced by `issue_claim` (assignee + `in-progress` label) and per-issue worktrees — that's a separate guard that does NOT depend on this AD. | | AD-002 | **Concurrency is bounded per LLM backend, not per project.** One concurrent Claude session per OAuth credential pool; one concurrent session per llama-server instance. Containers with disjoint backends may run in parallel. | The single-thread invariant is about *backends*, not pipelines. **(a) Anthropic OAuth credentials race on token refresh** — each container uses a per-session `CLAUDE_CONFIG_DIR`, so Claude Code's native lockfile-based OAuth refresh handles contention automatically without external serialization. (Legacy: set `CLAUDE_EXTERNAL_LOCK=1` to re-enable the old `flock session.lock` wrapper for rollback.) **(b) llama-server has finite VRAM and one KV cache** — parallel inference thrashes the cache and risks OOM. All llama-backed agents serialize on the same lock. **(c) Disjoint backends are free to parallelize.** Today `disinto-agents` (Anthropic OAuth, runs `review,gardener`) runs concurrently with `disinto-agents-llama` (llama, runs `dev`) on the same project — they share neither OAuth state nor llama VRAM. **(d) Per-project work-conflict safety** (no duplicate dev work, no merge conflicts on the same branch) is enforced by `issue_claim` (assignee + `in-progress` label) and per-issue worktrees — that's a separate guard that does NOT depend on this AD. |
| AD-003 | The runtime creates and destroys, the formula preserves. | Runtime manages worktrees/sessions/temp. Formulas commit knowledge to git before signaling done. | | AD-003 | The runtime creates and destroys, the formula preserves. | Runtime manages worktrees/sessions/temp. Formulas commit knowledge to git before signaling done. |
| AD-004 | Event-driven > polling > fixed delays. | Never `waitForTimeout` or hardcoded sleep. Use phase files, webhooks, or poll loops with backoff. | | AD-004 | Event-driven > polling > fixed delays. | Never `waitForTimeout` or hardcoded sleep. Use phase files, webhooks, or poll loops with backoff. |
| AD-005 | Secrets via env var indirection, never in issue bodies. | Issue bodies become code. Agent secrets go in `.env.enc`, vault secrets in `.env.vault.enc` (SOPS-encrypted when available; plaintext `.env`/`.env.vault` fallback supported). Referenced as `$VAR_NAME`. Runner gets only vault secrets; agents get only agent secrets. | | AD-005 | Secrets via env var indirection, never in issue bodies. | Issue bodies become code. Agent secrets go in `.env.enc` (SOPS-encrypted), vault secrets in `secrets/<NAME>.enc` (age-encrypted, one file per key). Referenced as `$VAR_NAME`. Runner gets only vault secrets; agents get only agent secrets. |
| AD-006 | External actions go through vault dispatch, never direct. | Agents build addressables; only the vault exercises them (publishes, deploys, posts). Tokens for external systems (`GITHUB_TOKEN`, `CLAWHUB_TOKEN`, deploy keys) live only in `.env.vault.enc` and are injected into the ephemeral runner container. `lib/env.sh` unsets them so agents never hold them. PRs with direct external actions without vault dispatch get REQUEST_CHANGES. (Vault redesign in progress: PR-based approval on ops repo, see #73-#77) | | AD-006 | External actions go through vault dispatch, never direct. | Agents build addressables; only the vault exercises them (publishes, deploys, posts). Tokens for external systems (`GITHUB_TOKEN`, `CLAWHUB_TOKEN`, deploy keys) live only in `secrets/<NAME>.enc` and are decrypted into the ephemeral runner container. `lib/env.sh` unsets them so agents never hold them. PRs with direct external actions without vault dispatch get REQUEST_CHANGES. (Vault redesign in progress: PR-based approval on ops repo, see #73-#77) |
**Who enforces what:** **Who enforces what:**
- **Gardener** checks open backlog issues against ADs during grooming; closes violations with a comment referencing the AD number. - **Gardener** checks open backlog issues against ADs during grooming; closes violations with a comment referencing the AD number.

View file

@ -50,7 +50,7 @@ blast_radius = "low" # optional: overrides policy.toml tier ("low"|"medium
## Secret Names ## Secret Names
Secret names must be defined in `.env.vault.enc` on the ops repo. The vault validates that requested secrets exist in the allowlist before execution. Secret names must have a corresponding `secrets/<NAME>.enc` file (age-encrypted). The vault validates that requested secrets exist in the allowlist before execution.
Common secret names: Common secret names:
- `CLAWHUB_TOKEN` - Token for ClawHub skill publishing - `CLAWHUB_TOKEN` - Token for ClawHub skill publishing

View file

@ -28,7 +28,7 @@ fi
# VAULT ACTION VALIDATION # VAULT ACTION VALIDATION
# ============================================================================= # =============================================================================
# Allowed secret names - must match keys in .env.vault.enc # Allowed secret names - must match files in secrets/<NAME>.enc
VAULT_ALLOWED_SECRETS="CLAWHUB_TOKEN GITHUB_TOKEN CODEBERG_TOKEN DEPLOY_KEY NPM_TOKEN DOCKER_HUB_TOKEN" VAULT_ALLOWED_SECRETS="CLAWHUB_TOKEN GITHUB_TOKEN CODEBERG_TOKEN DEPLOY_KEY NPM_TOKEN DOCKER_HUB_TOKEN"
# Allowed mount aliases — well-known file-based credential directories # Allowed mount aliases — well-known file-based credential directories

View file

@ -1133,8 +1133,6 @@ disinto_secrets() {
local subcmd="${1:-}" local subcmd="${1:-}"
local enc_file="${FACTORY_ROOT}/.env.enc" local enc_file="${FACTORY_ROOT}/.env.enc"
local env_file="${FACTORY_ROOT}/.env" local env_file="${FACTORY_ROOT}/.env"
local vault_enc_file="${FACTORY_ROOT}/.env.vault.enc"
local vault_env_file="${FACTORY_ROOT}/.env.vault"
# Shared helper: ensure sops+age and .sops.yaml exist # Shared helper: ensure sops+age and .sops.yaml exist
_secrets_ensure_sops() { _secrets_ensure_sops() {
@ -1257,6 +1255,37 @@ disinto_secrets() {
sops -d "$enc_file" sops -d "$enc_file"
fi fi
;; ;;
remove)
local name="${2:-}"
if [ -z "$name" ]; then
echo "Usage: disinto secrets remove <NAME>" >&2
exit 1
fi
local enc_path="${secrets_dir}/${name}.enc"
if [ ! -f "$enc_path" ]; then
echo "Error: ${enc_path} not found" >&2
exit 1
fi
rm -f "$enc_path"
echo "Removed: ${enc_path}"
;;
list)
if [ ! -d "$secrets_dir" ]; then
echo "No secrets directory found." >&2
exit 0
fi
local found=false
for enc_file_path in "${secrets_dir}"/*.enc; do
[ -f "$enc_file_path" ] || continue
found=true
local secret_name
secret_name=$(basename "$enc_file_path" .enc)
echo "$secret_name"
done
if [ "$found" = false ]; then
echo "No secrets stored." >&2
fi
;;
edit) edit)
if [ ! -f "$enc_file" ]; then if [ ! -f "$enc_file" ]; then
echo "Error: ${enc_file} not found. Run 'disinto secrets migrate' first." >&2 echo "Error: ${enc_file} not found. Run 'disinto secrets migrate' first." >&2
@ -1280,54 +1309,100 @@ disinto_secrets() {
rm -f "$env_file" rm -f "$env_file"
echo "Migrated: .env -> .env.enc (plaintext removed)" echo "Migrated: .env -> .env.enc (plaintext removed)"
;; ;;
edit-vault) migrate-from-vault)
if [ ! -f "$vault_enc_file" ]; then # One-shot migration: split .env.vault.enc into secrets/<KEY>.enc files (#777)
echo "Error: ${vault_enc_file} not found. Run 'disinto secrets migrate-vault' first." >&2 local vault_enc_file="${FACTORY_ROOT}/.env.vault.enc"
local vault_env_file="${FACTORY_ROOT}/.env.vault"
local source_file=""
if [ -f "$vault_enc_file" ] && command -v sops &>/dev/null; then
source_file="$vault_enc_file"
elif [ -f "$vault_env_file" ]; then
source_file="$vault_env_file"
else
echo "Error: neither .env.vault.enc nor .env.vault found — nothing to migrate." >&2
exit 1 exit 1
fi fi
sops "$vault_enc_file"
;; _secrets_ensure_age_key
show-vault) mkdir -p "$secrets_dir"
if [ ! -f "$vault_enc_file" ]; then
echo "Error: ${vault_enc_file} not found." >&2 # Decrypt vault to temp dotenv
local tmp_dotenv
tmp_dotenv=$(mktemp /tmp/disinto-vault-migrate-XXXXXX)
trap 'rm -f "$tmp_dotenv"' RETURN
if [ "$source_file" = "$vault_enc_file" ]; then
if ! sops -d --output-type dotenv "$vault_enc_file" > "$tmp_dotenv" 2>/dev/null; then
rm -f "$tmp_dotenv"
echo "Error: failed to decrypt .env.vault.enc" >&2
exit 1
fi
else
cp "$vault_env_file" "$tmp_dotenv"
fi
# Parse each KEY=VALUE and encrypt into secrets/<KEY>.enc
local count=0
local failed=0
while IFS='=' read -r key value; do
# Skip empty lines and comments
[[ -z "$key" || "$key" =~ ^[[:space:]]*# ]] && continue
# Trim whitespace from key
key=$(echo "$key" | xargs)
[ -z "$key" ] && continue
local enc_path="${secrets_dir}/${key}.enc"
if printf '%s' "$value" | age -r "$AGE_PUBLIC_KEY" -o "$enc_path" 2>/dev/null; then
# Verify round-trip
local check
check=$(age -d -i "$age_key_file" "$enc_path" 2>/dev/null) || { failed=$((failed + 1)); echo " FAIL (verify): ${key}" >&2; continue; }
if [ "$check" = "$value" ]; then
echo " OK: ${key} -> secrets/${key}.enc"
count=$((count + 1))
else
echo " FAIL (mismatch): ${key}" >&2
failed=$((failed + 1))
fi
else
echo " FAIL (encrypt): ${key}" >&2
failed=$((failed + 1))
fi
done < "$tmp_dotenv"
rm -f "$tmp_dotenv"
if [ "$failed" -gt 0 ]; then
echo "Error: ${failed} secret(s) failed migration. Vault files NOT removed." >&2
exit 1 exit 1
fi fi
sops -d "$vault_enc_file"
;; if [ "$count" -eq 0 ]; then
migrate-vault) echo "Warning: no secrets found in vault file." >&2
if [ ! -f "$vault_env_file" ]; then else
echo "Error: ${vault_env_file} not found — nothing to migrate." >&2 echo "Migrated ${count} secret(s) to secrets/*.enc"
echo " Create .env.vault with vault secrets (GITHUB_TOKEN, deploy keys, etc.)" >&2 # Remove old vault files on success
exit 1 rm -f "$vault_enc_file" "$vault_env_file"
echo "Removed: .env.vault.enc / .env.vault"
fi fi
_secrets_ensure_sops
encrypt_env_file "$vault_env_file" "$vault_enc_file"
# Verify decryption works before removing plaintext
if ! sops -d "$vault_enc_file" >/dev/null 2>&1; then
echo "Error: failed to verify .env.vault.enc decryption" >&2
rm -f "$vault_enc_file"
exit 1
fi
rm -f "$vault_env_file"
echo "Migrated: .env.vault -> .env.vault.enc (plaintext removed)"
;; ;;
*) *)
cat <<EOF >&2 cat <<EOF >&2
Usage: disinto secrets <subcommand> Usage: disinto secrets <subcommand>
Individual secrets (secrets/<NAME>.enc): Secrets (secrets/<NAME>.enc — age-encrypted, one file per key):
add <NAME> Prompt for value, encrypt, store in secrets/<NAME>.enc add <NAME> Prompt for value, encrypt, store in secrets/<NAME>.enc
show <NAME> Decrypt and print an individual secret show <NAME> Decrypt and print a secret
remove <NAME> Remove a secret
list List all stored secrets
Agent secrets (.env.enc): Agent secrets (.env.enc — sops-encrypted dotenv):
edit Edit agent secrets (FORGE_TOKEN, CLAUDE_API_KEY, etc.) edit Edit agent secrets (FORGE_TOKEN, CLAUDE_API_KEY, etc.)
show Show decrypted agent secrets (no argument) show Show decrypted agent secrets (no argument)
migrate Encrypt .env -> .env.enc migrate Encrypt .env -> .env.enc
Vault secrets (.env.vault.enc): Migration:
edit-vault Edit vault secrets (GITHUB_TOKEN, deploy keys, etc.) migrate-from-vault Split .env.vault.enc into secrets/<KEY>.enc (one-shot)
show-vault Show decrypted vault secrets
migrate-vault Encrypt .env.vault -> .env.vault.enc
EOF EOF
exit 1 exit 1
;; ;;
@ -1339,7 +1414,8 @@ EOF
disinto_run() { disinto_run() {
local action_id="${1:?Usage: disinto run <action-id>}" local action_id="${1:?Usage: disinto run <action-id>}"
local compose_file="${FACTORY_ROOT}/docker-compose.yml" local compose_file="${FACTORY_ROOT}/docker-compose.yml"
local vault_enc="${FACTORY_ROOT}/.env.vault.enc" local secrets_dir="${FACTORY_ROOT}/secrets"
local age_key_file="${HOME}/.config/sops/age/keys.txt"
if [ ! -f "$compose_file" ]; then if [ ! -f "$compose_file" ]; then
echo "Error: docker-compose.yml not found" >&2 echo "Error: docker-compose.yml not found" >&2
@ -1347,29 +1423,42 @@ disinto_run() {
exit 1 exit 1
fi fi
if [ ! -f "$vault_enc" ]; then if [ ! -d "$secrets_dir" ]; then
echo "Error: .env.vault.enc not found — create vault secrets first" >&2 echo "Error: secrets/ directory not found — create secrets first" >&2
echo " Run 'disinto secrets migrate-vault' after creating .env.vault" >&2 echo " Run 'disinto secrets add <NAME>' to add secrets" >&2
exit 1 exit 1
fi fi
if ! command -v sops &>/dev/null; then if ! command -v age &>/dev/null; then
echo "Error: sops not found — required to decrypt vault secrets" >&2 echo "Error: age not found — required to decrypt secrets" >&2
exit 1 exit 1
fi fi
# Decrypt vault secrets to temp file if [ ! -f "$age_key_file" ]; then
echo "Error: age key not found at ${age_key_file}" >&2
exit 1
fi
# Decrypt all secrets/*.enc into a temp env file for the runner
local tmp_env local tmp_env
tmp_env=$(mktemp /tmp/disinto-vault-XXXXXX) tmp_env=$(mktemp /tmp/disinto-secrets-XXXXXX)
trap 'rm -f "$tmp_env"' EXIT trap 'rm -f "$tmp_env"' EXIT
if ! sops -d --output-type dotenv "$vault_enc" > "$tmp_env" 2>/dev/null; then local count=0
rm -f "$tmp_env" for enc_path in "${secrets_dir}"/*.enc; do
echo "Error: failed to decrypt .env.vault.enc" >&2 [ -f "$enc_path" ] || continue
exit 1 local key
fi key=$(basename "$enc_path" .enc)
local val
val=$(age -d -i "$age_key_file" "$enc_path" 2>/dev/null) || {
echo "Warning: failed to decrypt ${enc_path}" >&2
continue
}
printf '%s=%s\n' "$key" "$val" >> "$tmp_env"
count=$((count + 1))
done
echo "Vault secrets decrypted to tmpfile" echo "Decrypted ${count} secret(s) to tmpfile"
# Run action in ephemeral runner container # Run action in ephemeral runner container
local rc=0 local rc=0

View file

@ -8,7 +8,7 @@
# 2. Scan vault/actions/ for TOML files without .result.json # 2. Scan vault/actions/ for TOML files without .result.json
# 3. Verify TOML arrived via merged PR with admin merger (Forgejo API) # 3. Verify TOML arrived via merged PR with admin merger (Forgejo API)
# 4. Validate TOML using vault-env.sh validator # 4. Validate TOML using vault-env.sh validator
# 5. Decrypt .env.vault.enc and extract only declared secrets # 5. Decrypt declared secrets from secrets/<NAME>.enc (age-encrypted)
# 6. Launch: docker run --rm disinto/agents:latest <action-id> # 6. Launch: docker run --rm disinto/agents:latest <action-id>
# 7. Write <action-id>.result.json with exit code, timestamp, logs summary # 7. Write <action-id>.result.json with exit code, timestamp, logs summary
# #
@ -27,19 +27,34 @@ source "${SCRIPT_ROOT}/../lib/env.sh"
# the shallow clone only has .toml.example files. # the shallow clone only has .toml.example files.
PROJECTS_DIR="${PROJECTS_DIR:-${FACTORY_ROOT:-/opt/disinto}-projects}" PROJECTS_DIR="${PROJECTS_DIR:-${FACTORY_ROOT:-/opt/disinto}-projects}"
# Load vault secrets after env.sh (env.sh unsets them for agent security) # Load granular secrets from secrets/*.enc (age-encrypted, one file per key).
# Vault secrets must be available to the dispatcher # These are decrypted on demand and exported so the dispatcher can pass them
if [ -f "$FACTORY_ROOT/.env.vault.enc" ] && command -v sops &>/dev/null; then # to runner containers. Replaces the old monolithic .env.vault.enc store (#777).
set -a _AGE_KEY_FILE="${HOME}/.config/sops/age/keys.txt"
eval "$(sops -d --output-type dotenv "$FACTORY_ROOT/.env.vault.enc" 2>/dev/null)" \ _SECRETS_DIR="${FACTORY_ROOT}/secrets"
|| echo "Warning: failed to decrypt .env.vault.enc — vault secrets not loaded" >&2
set +a # decrypt_secret <NAME> — decrypt secrets/<NAME>.enc and print the plaintext value
elif [ -f "$FACTORY_ROOT/.env.vault" ]; then decrypt_secret() {
set -a local name="$1"
# shellcheck source=/dev/null local enc_path="${_SECRETS_DIR}/${name}.enc"
source "$FACTORY_ROOT/.env.vault" if [ ! -f "$enc_path" ]; then
set +a return 1
fi fi
age -d -i "$_AGE_KEY_FILE" "$enc_path" 2>/dev/null
}
# load_secrets <NAME ...> — decrypt each secret and export it
load_secrets() {
if [ ! -f "$_AGE_KEY_FILE" ]; then
echo "Warning: age key not found at ${_AGE_KEY_FILE} — secrets not loaded" >&2
return 1
fi
for name in "$@"; do
local val
val=$(decrypt_secret "$name") || continue
export "$name=$val"
done
}
# Ops repo location (vault/actions directory) # Ops repo location (vault/actions directory)
OPS_REPO_ROOT="${OPS_REPO_ROOT:-/home/debian/disinto-ops}" OPS_REPO_ROOT="${OPS_REPO_ROOT:-/home/debian/disinto-ops}"
@ -452,17 +467,18 @@ launch_runner() {
fi fi
# Add environment variables for secrets (if any declared) # Add environment variables for secrets (if any declared)
# Secrets are decrypted per-key from secrets/<NAME>.enc (#777)
if [ -n "$secrets_array" ]; then if [ -n "$secrets_array" ]; then
for secret in $secrets_array; do for secret in $secrets_array; do
secret=$(echo "$secret" | xargs) secret=$(echo "$secret" | xargs)
if [ -n "$secret" ]; then if [ -n "$secret" ]; then
# Verify secret exists in vault local secret_val
if [ -z "${!secret:-}" ]; then secret_val=$(decrypt_secret "$secret") || {
log "ERROR: Secret '${secret}' not found in vault for action ${action_id}" log "ERROR: Secret '${secret}' not found in secrets/*.enc for action ${action_id}"
write_result "$action_id" 1 "Secret not found in vault: ${secret}" write_result "$action_id" 1 "Secret not found: ${secret} (expected secrets/${secret}.enc)"
return 1 return 1
fi }
cmd+=(-e "${secret}=${!secret}") cmd+=(-e "${secret}=${secret_val}")
fi fi
done done
else else

View file

@ -173,9 +173,40 @@ PROJECT_TOML="${PROJECT_TOML:-projects/disinto.toml}"
sleep 1200 # 20 minutes sleep 1200 # 20 minutes
done) & done) &
# ── Load required secrets from secrets/*.enc (#777) ────────────────────
# Edge container declares its required secrets; missing ones cause a hard fail.
_AGE_KEY_FILE="${HOME}/.config/sops/age/keys.txt"
_SECRETS_DIR="/opt/disinto/secrets"
EDGE_REQUIRED_SECRETS="CADDY_SSH_KEY CADDY_SSH_HOST CADDY_SSH_USER CADDY_ACCESS_LOG"
_edge_decrypt_secret() {
local enc_path="${_SECRETS_DIR}/${1}.enc"
[ -f "$enc_path" ] || return 1
age -d -i "$_AGE_KEY_FILE" "$enc_path" 2>/dev/null
}
if [ -f "$_AGE_KEY_FILE" ] && [ -d "$_SECRETS_DIR" ]; then
_missing=""
for _secret_name in $EDGE_REQUIRED_SECRETS; do
_val=$(_edge_decrypt_secret "$_secret_name") || { _missing="${_missing} ${_secret_name}"; continue; }
export "$_secret_name=$_val"
done
if [ -n "$_missing" ]; then
echo "FATAL: required secrets missing from secrets/*.enc:${_missing}" >&2
echo " Run 'disinto secrets add <NAME>' for each missing secret." >&2
echo " If migrating from .env.vault.enc, run 'disinto secrets migrate-from-vault' first." >&2
exit 1
fi
echo "edge: loaded required secrets: ${EDGE_REQUIRED_SECRETS}" >&2
else
echo "FATAL: age key (${_AGE_KEY_FILE}) or secrets dir (${_SECRETS_DIR}) not found — cannot load required secrets" >&2
echo " Ensure age is installed and secrets/*.enc files are present." >&2
exit 1
fi
# Start daily engagement collection cron loop in background (#745) # Start daily engagement collection cron loop in background (#745)
# Runs collect-engagement.sh daily at ~23:50 UTC via a sleep loop that # Runs collect-engagement.sh daily at ~23:50 UTC via a sleep loop that
# calculates seconds until the next 23:50 window. SSH key from .env.vault.enc. # calculates seconds until the next 23:50 window. SSH key from secrets/*.enc (#777).
(while true; do (while true; do
# Calculate seconds until next 23:50 UTC # Calculate seconds until next 23:50 UTC
_now=$(date -u +%s) _now=$(date -u +%s)
@ -186,26 +217,21 @@ done) &
_sleep_secs=$(( _target - _now )) _sleep_secs=$(( _target - _now ))
echo "edge: collect-engagement scheduled in ${_sleep_secs}s (next 23:50 UTC)" >&2 echo "edge: collect-engagement scheduled in ${_sleep_secs}s (next 23:50 UTC)" >&2
sleep "$_sleep_secs" sleep "$_sleep_secs"
# Set CADDY_ACCESS_LOG so the script reads from the fetched local copy
_fetch_log="/tmp/caddy-access-log-fetch.log" _fetch_log="/tmp/caddy-access-log-fetch.log"
if [ -n "${CADDY_SSH_KEY:-}" ]; then _ssh_key_file=$(mktemp)
_ssh_key_file=$(mktemp) printf '%s\n' "$CADDY_SSH_KEY" > "$_ssh_key_file"
printf '%s\n' "$CADDY_SSH_KEY" > "$_ssh_key_file" chmod 0600 "$_ssh_key_file"
chmod 0600 "$_ssh_key_file" scp -i "$_ssh_key_file" -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 -o BatchMode=yes \
scp -i "$_ssh_key_file" -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 -o BatchMode=yes \ "${CADDY_SSH_USER}@${CADDY_SSH_HOST}:${CADDY_ACCESS_LOG}" \
"${CADDY_SSH_USER:-debian}@${CADDY_SSH_HOST:-disinto.ai}:${CADDY_ACCESS_LOG:-/var/log/caddy/access.log}" \ "$_fetch_log" 2>&1 | tee -a /opt/disinto-logs/collect-engagement.log || true
"$_fetch_log" 2>&1 | tee -a /opt/disinto-logs/collect-engagement.log || true rm -f "$_ssh_key_file"
rm -f "$_ssh_key_file" if [ -s "$_fetch_log" ]; then
if [ -s "$_fetch_log" ]; then CADDY_ACCESS_LOG="$_fetch_log" bash /opt/disinto/site/collect-engagement.sh 2>&1 \
CADDY_ACCESS_LOG="$_fetch_log" bash /opt/disinto/site/collect-engagement.sh 2>&1 \ | tee -a /opt/disinto-logs/collect-engagement.log || true
| tee -a /opt/disinto-logs/collect-engagement.log || true
else
echo "edge: collect-engagement: fetched log is empty, skipping parse" >&2
fi
rm -f "$_fetch_log"
else else
echo "edge: collect-engagement: CADDY_SSH_KEY not set, skipping" >&2 echo "edge: collect-engagement: fetched log is empty, skipping parse" >&2
fi fi
rm -f "$_fetch_log"
done) & done) &
# Caddy as main process — run in foreground via wait so background jobs survive # Caddy as main process — run in foreground via wait so background jobs survive

View file

@ -50,7 +50,7 @@ description = """
Fetch today's Caddy access log segment from the remote host using SCP. Fetch today's Caddy access log segment from the remote host using SCP.
The SSH key is read from the environment (CADDY_SSH_KEY), which is The SSH key is read from the environment (CADDY_SSH_KEY), which is
decrypted from .env.vault.enc by the dispatcher. It is NEVER hardcoded. decrypted from secrets/CADDY_SSH_KEY.enc by the edge entrypoint. It is NEVER hardcoded.
1. Write the SSH key to a temporary file with restricted permissions: 1. Write the SSH key to a temporary file with restricted permissions:
_ssh_key_file=$(mktemp) _ssh_key_file=$(mktemp)

View file

@ -79,28 +79,23 @@ AND set CADDY_ACCESS_LOG in the factory environment to match.
[[steps]] [[steps]]
id = "store-private-key" id = "store-private-key"
title = "Add the private key to .env.vault.enc as CADDY_SSH_KEY" title = "Add the private key as CADDY_SSH_KEY secret"
needs = ["generate-keypair"] needs = ["generate-keypair"]
description = """ description = """
Store the private key in the factory's encrypted vault secrets. Store the private key in the factory's encrypted secrets store.
1. Read the private key: 1. Add the private key using `disinto secrets add`:
cat caddy-collect
2. Add it to .env.vault.enc (or .env.vault for plaintext fallback) as cat caddy-collect | disinto secrets add CADDY_SSH_KEY
CADDY_SSH_KEY. The key is multi-line, so use the base64-encoded form:
echo "CADDY_SSH_KEY=$(base64 -w0 caddy-collect)" >> .env.vault.enc This encrypts the key with age and stores it as secrets/CADDY_SSH_KEY.enc.
Or, if using SOPS-encrypted vault, decrypt first, add the variable, 2. IMPORTANT: After storing, securely delete the local private key file:
then re-encrypt.
3. IMPORTANT: After storing, securely delete the local private key file:
shred -u caddy-collect 2>/dev/null || rm -f caddy-collect shred -u caddy-collect 2>/dev/null || rm -f caddy-collect
rm -f caddy-collect.pub rm -f caddy-collect.pub
The public key is already installed on the Caddy host; the private key The public key is already installed on the Caddy host; the private key
now lives only in the vault. now lives only in secrets/CADDY_SSH_KEY.enc.
Never commit the private key to any git repository. Never commit the private key to any git repository.
""" """
@ -109,20 +104,19 @@ Never commit the private key to any git repository.
[[steps]] [[steps]]
id = "store-caddy-host" id = "store-caddy-host"
title = "Add the Caddy host address to .env.vault.enc as CADDY_HOST" title = "Add the Caddy host details as secrets"
needs = ["install-public-key"] needs = ["install-public-key"]
description = """ description = """
Store the Caddy host connection string so collect-engagement.sh knows Store the Caddy connection details so collect-engagement.sh knows
where to SSH. where to SSH.
1. Add to .env.vault.enc (or .env.vault for plaintext fallback): 1. Add each value using `disinto secrets add`:
echo "CADDY_HOST=user@caddy-host-ip-or-domain" >> .env.vault.enc echo 'disinto.ai' | disinto secrets add CADDY_SSH_HOST
echo 'debian' | disinto secrets add CADDY_SSH_USER
echo '/var/log/caddy/access.log' | disinto secrets add CADDY_ACCESS_LOG
Replace user@caddy-host-ip-or-domain with the actual SSH user and host Replace values with the actual SSH host, user, and log path for your setup.
(e.g. debian@203.0.113.42 or deploy@caddy.disinto.ai).
2. If using SOPS, decrypt/add/re-encrypt as above.
""" """
# ── Step 5: Test the connection ────────────────────────────────────────────── # ── Step 5: Test the connection ──────────────────────────────────────────────

View file

@ -213,7 +213,7 @@ should file a vault item instead of executing directly.
**Exceptions** (do NOT flag these): **Exceptions** (do NOT flag these):
- Code inside `vault/` the vault system itself is allowed to handle secrets - Code inside `vault/` the vault system itself is allowed to handle secrets
- References in comments or documentation explaining the architecture - References in comments or documentation explaining the architecture
- `bin/disinto` setup commands that manage `.env.vault.enc` and the `run` subcommand - `bin/disinto` setup commands that manage `secrets/*.enc` and the `run` subcommand
- Local operations (git push to forge, forge API calls with `FORGE_TOKEN`) - Local operations (git push to forge, forge API calls with `FORGE_TOKEN`)
## 6. Re-review (if previous review is provided) ## 6. Re-review (if previous review is provided)

View file

@ -158,8 +158,8 @@ export WOODPECKER_SERVER="${WOODPECKER_SERVER:-http://localhost:8000}"
export CLAUDE_TIMEOUT="${CLAUDE_TIMEOUT:-7200}" export CLAUDE_TIMEOUT="${CLAUDE_TIMEOUT:-7200}"
# Vault-only token guard (#745): external-action tokens (GITHUB_TOKEN, CLAWHUB_TOKEN) # Vault-only token guard (#745): external-action tokens (GITHUB_TOKEN, CLAWHUB_TOKEN)
# must NEVER be available to agents. They live in .env.vault.enc and are injected # must NEVER be available to agents. They live in secrets/*.enc and are decrypted
# only into the ephemeral runner container at fire time. Unset them here so # only into the ephemeral runner container at fire time (#777). Unset them here so
# even an accidental .env inclusion cannot leak them into agent sessions. # even an accidental .env inclusion cannot leak them into agent sessions.
unset GITHUB_TOKEN 2>/dev/null || true unset GITHUB_TOKEN 2>/dev/null || true
unset CLAWHUB_TOKEN 2>/dev/null || true unset CLAWHUB_TOKEN 2>/dev/null || true

View file

@ -372,8 +372,8 @@ services:
PLANNER_INTERVAL: ${PLANNER_INTERVAL:-43200} PLANNER_INTERVAL: ${PLANNER_INTERVAL:-43200}
# IMPORTANT: agents get explicit environment variables (forge tokens, CI tokens, config). # IMPORTANT: agents get explicit environment variables (forge tokens, CI tokens, config).
# Vault-only secrets (GITHUB_TOKEN, CLAWHUB_TOKEN, deploy keys) live in # Vault-only secrets (GITHUB_TOKEN, CLAWHUB_TOKEN, deploy keys) live in
# .env.vault.enc and are NEVER injected here — only the runner # secrets/*.enc and are NEVER injected here — only the runner
# container receives them at fire time (AD-006, #745). # container receives them at fire time (AD-006, #745, #777).
depends_on: depends_on:
forgejo: forgejo:
condition: service_healthy condition: service_healthy