Merge pull request 'fix: feat: consolidate secret stores — single granular secrets/*.enc, deprecate .env.vault.enc (#777)' (#806) from fix/issue-777 into main
All checks were successful
ci/woodpecker/push/ci Pipeline was successful
All checks were successful
ci/woodpecker/push/ci Pipeline was successful
This commit is contained in:
commit
defec3b255
14 changed files with 254 additions and 130 deletions
|
|
@ -1,8 +1,7 @@
|
|||
# Secrets — prevent .env files from being baked into the image
|
||||
# Secrets — prevent .env files and encrypted secrets from being baked into the image
|
||||
.env
|
||||
.env.enc
|
||||
.env.vault
|
||||
.env.vault.enc
|
||||
secrets/
|
||||
|
||||
# Version control — .git is huge and not needed in image
|
||||
.git
|
||||
|
|
|
|||
11
.env.example
11
.env.example
|
|
@ -83,16 +83,17 @@ FORWARD_AUTH_SECRET= # [SECRET] Shared secret for Caddy ↔
|
|||
|
||||
# ── Vault-only secrets (DO NOT put these in .env) ────────────────────────
|
||||
# These tokens grant access to external systems (GitHub, ClawHub, deploy targets).
|
||||
# They live ONLY in .env.vault.enc and are injected into the ephemeral runner
|
||||
# container at fire time (#745). lib/env.sh explicitly unsets them so agents
|
||||
# can never hold them directly — all external actions go through vault dispatch.
|
||||
# They live ONLY in secrets/<NAME>.enc (age-encrypted, one file per key) and are
|
||||
# decrypted into the ephemeral runner container at fire time (#745, #777).
|
||||
# lib/env.sh explicitly unsets them so agents can never hold them directly —
|
||||
# all external actions go through vault dispatch.
|
||||
#
|
||||
# GITHUB_TOKEN — GitHub API access (publish, deploy, post)
|
||||
# CLAWHUB_TOKEN — ClawHub registry credentials (publish)
|
||||
# CADDY_SSH_KEY — SSH key for Caddy log collection
|
||||
# (deploy keys) — SSH keys for deployment targets
|
||||
#
|
||||
# To manage vault secrets: disinto secrets edit-vault
|
||||
# (vault redesign in progress: PR-based approval, see #73-#77)
|
||||
# To manage secrets: disinto secrets add/show/remove/list
|
||||
|
||||
# ── Project-specific secrets ──────────────────────────────────────────────
|
||||
# Store all project secrets here so formulas reference env vars, never hardcode.
|
||||
|
|
|
|||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -3,7 +3,6 @@
|
|||
|
||||
# Encrypted secrets — safe to commit (SOPS-encrypted with age)
|
||||
!.env.enc
|
||||
!.env.vault.enc
|
||||
!.sops.yaml
|
||||
|
||||
# Per-box project config (generated by disinto init)
|
||||
|
|
|
|||
|
|
@ -86,7 +86,7 @@ Each agent has a `.profile` repository on Forgejo storing `knowledge/lessons-lea
|
|||
- All scripts start with `#!/usr/bin/env bash` and `set -euo pipefail`
|
||||
- Source shared environment: `source "$(dirname "$0")/../lib/env.sh"`
|
||||
- Log to `$LOGFILE` using the `log()` function from env.sh or defined locally
|
||||
- Never hardcode secrets — agent secrets come from `.env.enc`, vault secrets from `.env.vault.enc` (or `.env`/`.env.vault` fallback)
|
||||
- Never hardcode secrets — agent secrets come from `.env.enc`, vault secrets from `secrets/<NAME>.enc` (age-encrypted, one file per key)
|
||||
- Never embed secrets in issue bodies, PR descriptions, or comments — use env var references (e.g. `$BASE_RPC_URL`)
|
||||
- ShellCheck must pass (CI runs `shellcheck` on all `.sh` files)
|
||||
- Avoid duplicate code — shared helpers go in `lib/`
|
||||
|
|
@ -179,8 +179,8 @@ Humans write these. Agents read and enforce them.
|
|||
| AD-002 | **Concurrency is bounded per LLM backend, not per project.** One concurrent Claude session per OAuth credential pool; one concurrent session per llama-server instance. Containers with disjoint backends may run in parallel. | The single-thread invariant is about *backends*, not pipelines. **(a) Anthropic OAuth credentials race on token refresh** — each container uses a per-session `CLAUDE_CONFIG_DIR`, so Claude Code's native lockfile-based OAuth refresh handles contention automatically without external serialization. (Legacy: set `CLAUDE_EXTERNAL_LOCK=1` to re-enable the old `flock session.lock` wrapper for rollback.) **(b) llama-server has finite VRAM and one KV cache** — parallel inference thrashes the cache and risks OOM. All llama-backed agents serialize on the same lock. **(c) Disjoint backends are free to parallelize.** Today `disinto-agents` (Anthropic OAuth, runs `review,gardener`) runs concurrently with `disinto-agents-llama` (llama, runs `dev`) on the same project — they share neither OAuth state nor llama VRAM. **(d) Per-project work-conflict safety** (no duplicate dev work, no merge conflicts on the same branch) is enforced by `issue_claim` (assignee + `in-progress` label) and per-issue worktrees — that's a separate guard that does NOT depend on this AD. |
|
||||
| AD-003 | The runtime creates and destroys, the formula preserves. | Runtime manages worktrees/sessions/temp. Formulas commit knowledge to git before signaling done. |
|
||||
| AD-004 | Event-driven > polling > fixed delays. | Never `waitForTimeout` or hardcoded sleep. Use phase files, webhooks, or poll loops with backoff. |
|
||||
| AD-005 | Secrets via env var indirection, never in issue bodies. | Issue bodies become code. Agent secrets go in `.env.enc`, vault secrets in `.env.vault.enc` (SOPS-encrypted when available; plaintext `.env`/`.env.vault` fallback supported). Referenced as `$VAR_NAME`. Runner gets only vault secrets; agents get only agent secrets. |
|
||||
| AD-006 | External actions go through vault dispatch, never direct. | Agents build addressables; only the vault exercises them (publishes, deploys, posts). Tokens for external systems (`GITHUB_TOKEN`, `CLAWHUB_TOKEN`, deploy keys) live only in `.env.vault.enc` and are injected into the ephemeral runner container. `lib/env.sh` unsets them so agents never hold them. PRs with direct external actions without vault dispatch get REQUEST_CHANGES. (Vault redesign in progress: PR-based approval on ops repo, see #73-#77) |
|
||||
| AD-005 | Secrets via env var indirection, never in issue bodies. | Issue bodies become code. Agent secrets go in `.env.enc` (SOPS-encrypted), vault secrets in `secrets/<NAME>.enc` (age-encrypted, one file per key). Referenced as `$VAR_NAME`. Runner gets only vault secrets; agents get only agent secrets. |
|
||||
| AD-006 | External actions go through vault dispatch, never direct. | Agents build addressables; only the vault exercises them (publishes, deploys, posts). Tokens for external systems (`GITHUB_TOKEN`, `CLAWHUB_TOKEN`, deploy keys) live only in `secrets/<NAME>.enc` and are decrypted into the ephemeral runner container. `lib/env.sh` unsets them so agents never hold them. PRs with direct external actions without vault dispatch get REQUEST_CHANGES. (Vault redesign in progress: PR-based approval on ops repo, see #73-#77) |
|
||||
|
||||
**Who enforces what:**
|
||||
- **Gardener** checks open backlog issues against ADs during grooming; closes violations with a comment referencing the AD number.
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ blast_radius = "low" # optional: overrides policy.toml tier ("low"|"medium
|
|||
|
||||
## Secret Names
|
||||
|
||||
Secret names must be defined in `.env.vault.enc` on the ops repo. The vault validates that requested secrets exist in the allowlist before execution.
|
||||
Secret names must have a corresponding `secrets/<NAME>.enc` file (age-encrypted). The vault validates that requested secrets exist in the allowlist before execution.
|
||||
|
||||
Common secret names:
|
||||
- `CLAWHUB_TOKEN` - Token for ClawHub skill publishing
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ fi
|
|||
# VAULT ACTION VALIDATION
|
||||
# =============================================================================
|
||||
|
||||
# Allowed secret names - must match keys in .env.vault.enc
|
||||
# Allowed secret names - must match files in secrets/<NAME>.enc
|
||||
VAULT_ALLOWED_SECRETS="CLAWHUB_TOKEN GITHUB_TOKEN CODEBERG_TOKEN DEPLOY_KEY NPM_TOKEN DOCKER_HUB_TOKEN"
|
||||
|
||||
# Allowed mount aliases — well-known file-based credential directories
|
||||
|
|
|
|||
193
bin/disinto
193
bin/disinto
|
|
@ -1133,8 +1133,6 @@ disinto_secrets() {
|
|||
local subcmd="${1:-}"
|
||||
local enc_file="${FACTORY_ROOT}/.env.enc"
|
||||
local env_file="${FACTORY_ROOT}/.env"
|
||||
local vault_enc_file="${FACTORY_ROOT}/.env.vault.enc"
|
||||
local vault_env_file="${FACTORY_ROOT}/.env.vault"
|
||||
|
||||
# Shared helper: ensure sops+age and .sops.yaml exist
|
||||
_secrets_ensure_sops() {
|
||||
|
|
@ -1257,6 +1255,37 @@ disinto_secrets() {
|
|||
sops -d "$enc_file"
|
||||
fi
|
||||
;;
|
||||
remove)
|
||||
local name="${2:-}"
|
||||
if [ -z "$name" ]; then
|
||||
echo "Usage: disinto secrets remove <NAME>" >&2
|
||||
exit 1
|
||||
fi
|
||||
local enc_path="${secrets_dir}/${name}.enc"
|
||||
if [ ! -f "$enc_path" ]; then
|
||||
echo "Error: ${enc_path} not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
rm -f "$enc_path"
|
||||
echo "Removed: ${enc_path}"
|
||||
;;
|
||||
list)
|
||||
if [ ! -d "$secrets_dir" ]; then
|
||||
echo "No secrets directory found." >&2
|
||||
exit 0
|
||||
fi
|
||||
local found=false
|
||||
for enc_file_path in "${secrets_dir}"/*.enc; do
|
||||
[ -f "$enc_file_path" ] || continue
|
||||
found=true
|
||||
local secret_name
|
||||
secret_name=$(basename "$enc_file_path" .enc)
|
||||
echo "$secret_name"
|
||||
done
|
||||
if [ "$found" = false ]; then
|
||||
echo "No secrets stored." >&2
|
||||
fi
|
||||
;;
|
||||
edit)
|
||||
if [ ! -f "$enc_file" ]; then
|
||||
echo "Error: ${enc_file} not found. Run 'disinto secrets migrate' first." >&2
|
||||
|
|
@ -1280,54 +1309,100 @@ disinto_secrets() {
|
|||
rm -f "$env_file"
|
||||
echo "Migrated: .env -> .env.enc (plaintext removed)"
|
||||
;;
|
||||
edit-vault)
|
||||
if [ ! -f "$vault_enc_file" ]; then
|
||||
echo "Error: ${vault_enc_file} not found. Run 'disinto secrets migrate-vault' first." >&2
|
||||
migrate-from-vault)
|
||||
# One-shot migration: split .env.vault.enc into secrets/<KEY>.enc files (#777)
|
||||
local vault_enc_file="${FACTORY_ROOT}/.env.vault.enc"
|
||||
local vault_env_file="${FACTORY_ROOT}/.env.vault"
|
||||
local source_file=""
|
||||
|
||||
if [ -f "$vault_enc_file" ] && command -v sops &>/dev/null; then
|
||||
source_file="$vault_enc_file"
|
||||
elif [ -f "$vault_env_file" ]; then
|
||||
source_file="$vault_env_file"
|
||||
else
|
||||
echo "Error: neither .env.vault.enc nor .env.vault found — nothing to migrate." >&2
|
||||
exit 1
|
||||
fi
|
||||
sops "$vault_enc_file"
|
||||
;;
|
||||
show-vault)
|
||||
if [ ! -f "$vault_enc_file" ]; then
|
||||
echo "Error: ${vault_enc_file} not found." >&2
|
||||
|
||||
_secrets_ensure_age_key
|
||||
mkdir -p "$secrets_dir"
|
||||
|
||||
# Decrypt vault to temp dotenv
|
||||
local tmp_dotenv
|
||||
tmp_dotenv=$(mktemp /tmp/disinto-vault-migrate-XXXXXX)
|
||||
trap 'rm -f "$tmp_dotenv"' RETURN
|
||||
|
||||
if [ "$source_file" = "$vault_enc_file" ]; then
|
||||
if ! sops -d --output-type dotenv "$vault_enc_file" > "$tmp_dotenv" 2>/dev/null; then
|
||||
rm -f "$tmp_dotenv"
|
||||
echo "Error: failed to decrypt .env.vault.enc" >&2
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
cp "$vault_env_file" "$tmp_dotenv"
|
||||
fi
|
||||
|
||||
# Parse each KEY=VALUE and encrypt into secrets/<KEY>.enc
|
||||
local count=0
|
||||
local failed=0
|
||||
while IFS='=' read -r key value; do
|
||||
# Skip empty lines and comments
|
||||
[[ -z "$key" || "$key" =~ ^[[:space:]]*# ]] && continue
|
||||
# Trim whitespace from key
|
||||
key=$(echo "$key" | xargs)
|
||||
[ -z "$key" ] && continue
|
||||
|
||||
local enc_path="${secrets_dir}/${key}.enc"
|
||||
if printf '%s' "$value" | age -r "$AGE_PUBLIC_KEY" -o "$enc_path" 2>/dev/null; then
|
||||
# Verify round-trip
|
||||
local check
|
||||
check=$(age -d -i "$age_key_file" "$enc_path" 2>/dev/null) || { failed=$((failed + 1)); echo " FAIL (verify): ${key}" >&2; continue; }
|
||||
if [ "$check" = "$value" ]; then
|
||||
echo " OK: ${key} -> secrets/${key}.enc"
|
||||
count=$((count + 1))
|
||||
else
|
||||
echo " FAIL (mismatch): ${key}" >&2
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
else
|
||||
echo " FAIL (encrypt): ${key}" >&2
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
done < "$tmp_dotenv"
|
||||
|
||||
rm -f "$tmp_dotenv"
|
||||
|
||||
if [ "$failed" -gt 0 ]; then
|
||||
echo "Error: ${failed} secret(s) failed migration. Vault files NOT removed." >&2
|
||||
exit 1
|
||||
fi
|
||||
sops -d "$vault_enc_file"
|
||||
;;
|
||||
migrate-vault)
|
||||
if [ ! -f "$vault_env_file" ]; then
|
||||
echo "Error: ${vault_env_file} not found — nothing to migrate." >&2
|
||||
echo " Create .env.vault with vault secrets (GITHUB_TOKEN, deploy keys, etc.)" >&2
|
||||
exit 1
|
||||
|
||||
if [ "$count" -eq 0 ]; then
|
||||
echo "Warning: no secrets found in vault file." >&2
|
||||
else
|
||||
echo "Migrated ${count} secret(s) to secrets/*.enc"
|
||||
# Remove old vault files on success
|
||||
rm -f "$vault_enc_file" "$vault_env_file"
|
||||
echo "Removed: .env.vault.enc / .env.vault"
|
||||
fi
|
||||
_secrets_ensure_sops
|
||||
encrypt_env_file "$vault_env_file" "$vault_enc_file"
|
||||
# Verify decryption works before removing plaintext
|
||||
if ! sops -d "$vault_enc_file" >/dev/null 2>&1; then
|
||||
echo "Error: failed to verify .env.vault.enc decryption" >&2
|
||||
rm -f "$vault_enc_file"
|
||||
exit 1
|
||||
fi
|
||||
rm -f "$vault_env_file"
|
||||
echo "Migrated: .env.vault -> .env.vault.enc (plaintext removed)"
|
||||
;;
|
||||
*)
|
||||
cat <<EOF >&2
|
||||
Usage: disinto secrets <subcommand>
|
||||
|
||||
Individual secrets (secrets/<NAME>.enc):
|
||||
add <NAME> Prompt for value, encrypt, store in secrets/<NAME>.enc
|
||||
show <NAME> Decrypt and print an individual secret
|
||||
Secrets (secrets/<NAME>.enc — age-encrypted, one file per key):
|
||||
add <NAME> Prompt for value, encrypt, store in secrets/<NAME>.enc
|
||||
show <NAME> Decrypt and print a secret
|
||||
remove <NAME> Remove a secret
|
||||
list List all stored secrets
|
||||
|
||||
Agent secrets (.env.enc):
|
||||
edit Edit agent secrets (FORGE_TOKEN, CLAUDE_API_KEY, etc.)
|
||||
show Show decrypted agent secrets (no argument)
|
||||
migrate Encrypt .env -> .env.enc
|
||||
Agent secrets (.env.enc — sops-encrypted dotenv):
|
||||
edit Edit agent secrets (FORGE_TOKEN, CLAUDE_API_KEY, etc.)
|
||||
show Show decrypted agent secrets (no argument)
|
||||
migrate Encrypt .env -> .env.enc
|
||||
|
||||
Vault secrets (.env.vault.enc):
|
||||
edit-vault Edit vault secrets (GITHUB_TOKEN, deploy keys, etc.)
|
||||
show-vault Show decrypted vault secrets
|
||||
migrate-vault Encrypt .env.vault -> .env.vault.enc
|
||||
Migration:
|
||||
migrate-from-vault Split .env.vault.enc into secrets/<KEY>.enc (one-shot)
|
||||
EOF
|
||||
exit 1
|
||||
;;
|
||||
|
|
@ -1339,7 +1414,8 @@ EOF
|
|||
disinto_run() {
|
||||
local action_id="${1:?Usage: disinto run <action-id>}"
|
||||
local compose_file="${FACTORY_ROOT}/docker-compose.yml"
|
||||
local vault_enc="${FACTORY_ROOT}/.env.vault.enc"
|
||||
local secrets_dir="${FACTORY_ROOT}/secrets"
|
||||
local age_key_file="${HOME}/.config/sops/age/keys.txt"
|
||||
|
||||
if [ ! -f "$compose_file" ]; then
|
||||
echo "Error: docker-compose.yml not found" >&2
|
||||
|
|
@ -1347,29 +1423,42 @@ disinto_run() {
|
|||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$vault_enc" ]; then
|
||||
echo "Error: .env.vault.enc not found — create vault secrets first" >&2
|
||||
echo " Run 'disinto secrets migrate-vault' after creating .env.vault" >&2
|
||||
if [ ! -d "$secrets_dir" ]; then
|
||||
echo "Error: secrets/ directory not found — create secrets first" >&2
|
||||
echo " Run 'disinto secrets add <NAME>' to add secrets" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v sops &>/dev/null; then
|
||||
echo "Error: sops not found — required to decrypt vault secrets" >&2
|
||||
if ! command -v age &>/dev/null; then
|
||||
echo "Error: age not found — required to decrypt secrets" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Decrypt vault secrets to temp file
|
||||
if [ ! -f "$age_key_file" ]; then
|
||||
echo "Error: age key not found at ${age_key_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Decrypt all secrets/*.enc into a temp env file for the runner
|
||||
local tmp_env
|
||||
tmp_env=$(mktemp /tmp/disinto-vault-XXXXXX)
|
||||
tmp_env=$(mktemp /tmp/disinto-secrets-XXXXXX)
|
||||
trap 'rm -f "$tmp_env"' EXIT
|
||||
|
||||
if ! sops -d --output-type dotenv "$vault_enc" > "$tmp_env" 2>/dev/null; then
|
||||
rm -f "$tmp_env"
|
||||
echo "Error: failed to decrypt .env.vault.enc" >&2
|
||||
exit 1
|
||||
fi
|
||||
local count=0
|
||||
for enc_path in "${secrets_dir}"/*.enc; do
|
||||
[ -f "$enc_path" ] || continue
|
||||
local key
|
||||
key=$(basename "$enc_path" .enc)
|
||||
local val
|
||||
val=$(age -d -i "$age_key_file" "$enc_path" 2>/dev/null) || {
|
||||
echo "Warning: failed to decrypt ${enc_path}" >&2
|
||||
continue
|
||||
}
|
||||
printf '%s=%s\n' "$key" "$val" >> "$tmp_env"
|
||||
count=$((count + 1))
|
||||
done
|
||||
|
||||
echo "Vault secrets decrypted to tmpfile"
|
||||
echo "Decrypted ${count} secret(s) to tmpfile"
|
||||
|
||||
# Run action in ephemeral runner container
|
||||
local rc=0
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
# 2. Scan vault/actions/ for TOML files without .result.json
|
||||
# 3. Verify TOML arrived via merged PR with admin merger (Forgejo API)
|
||||
# 4. Validate TOML using vault-env.sh validator
|
||||
# 5. Decrypt .env.vault.enc and extract only declared secrets
|
||||
# 5. Decrypt declared secrets from secrets/<NAME>.enc (age-encrypted)
|
||||
# 6. Launch: docker run --rm disinto/agents:latest <action-id>
|
||||
# 7. Write <action-id>.result.json with exit code, timestamp, logs summary
|
||||
#
|
||||
|
|
@ -27,19 +27,34 @@ source "${SCRIPT_ROOT}/../lib/env.sh"
|
|||
# the shallow clone only has .toml.example files.
|
||||
PROJECTS_DIR="${PROJECTS_DIR:-${FACTORY_ROOT:-/opt/disinto}-projects}"
|
||||
|
||||
# Load vault secrets after env.sh (env.sh unsets them for agent security)
|
||||
# Vault secrets must be available to the dispatcher
|
||||
if [ -f "$FACTORY_ROOT/.env.vault.enc" ] && command -v sops &>/dev/null; then
|
||||
set -a
|
||||
eval "$(sops -d --output-type dotenv "$FACTORY_ROOT/.env.vault.enc" 2>/dev/null)" \
|
||||
|| echo "Warning: failed to decrypt .env.vault.enc — vault secrets not loaded" >&2
|
||||
set +a
|
||||
elif [ -f "$FACTORY_ROOT/.env.vault" ]; then
|
||||
set -a
|
||||
# shellcheck source=/dev/null
|
||||
source "$FACTORY_ROOT/.env.vault"
|
||||
set +a
|
||||
fi
|
||||
# Load granular secrets from secrets/*.enc (age-encrypted, one file per key).
|
||||
# These are decrypted on demand and exported so the dispatcher can pass them
|
||||
# to runner containers. Replaces the old monolithic .env.vault.enc store (#777).
|
||||
_AGE_KEY_FILE="${HOME}/.config/sops/age/keys.txt"
|
||||
_SECRETS_DIR="${FACTORY_ROOT}/secrets"
|
||||
|
||||
# decrypt_secret <NAME> — decrypt secrets/<NAME>.enc and print the plaintext value
|
||||
decrypt_secret() {
|
||||
local name="$1"
|
||||
local enc_path="${_SECRETS_DIR}/${name}.enc"
|
||||
if [ ! -f "$enc_path" ]; then
|
||||
return 1
|
||||
fi
|
||||
age -d -i "$_AGE_KEY_FILE" "$enc_path" 2>/dev/null
|
||||
}
|
||||
|
||||
# load_secrets <NAME ...> — decrypt each secret and export it
|
||||
load_secrets() {
|
||||
if [ ! -f "$_AGE_KEY_FILE" ]; then
|
||||
echo "Warning: age key not found at ${_AGE_KEY_FILE} — secrets not loaded" >&2
|
||||
return 1
|
||||
fi
|
||||
for name in "$@"; do
|
||||
local val
|
||||
val=$(decrypt_secret "$name") || continue
|
||||
export "$name=$val"
|
||||
done
|
||||
}
|
||||
|
||||
# Ops repo location (vault/actions directory)
|
||||
OPS_REPO_ROOT="${OPS_REPO_ROOT:-/home/debian/disinto-ops}"
|
||||
|
|
@ -452,17 +467,18 @@ launch_runner() {
|
|||
fi
|
||||
|
||||
# Add environment variables for secrets (if any declared)
|
||||
# Secrets are decrypted per-key from secrets/<NAME>.enc (#777)
|
||||
if [ -n "$secrets_array" ]; then
|
||||
for secret in $secrets_array; do
|
||||
secret=$(echo "$secret" | xargs)
|
||||
if [ -n "$secret" ]; then
|
||||
# Verify secret exists in vault
|
||||
if [ -z "${!secret:-}" ]; then
|
||||
log "ERROR: Secret '${secret}' not found in vault for action ${action_id}"
|
||||
write_result "$action_id" 1 "Secret not found in vault: ${secret}"
|
||||
local secret_val
|
||||
secret_val=$(decrypt_secret "$secret") || {
|
||||
log "ERROR: Secret '${secret}' not found in secrets/*.enc for action ${action_id}"
|
||||
write_result "$action_id" 1 "Secret not found: ${secret} (expected secrets/${secret}.enc)"
|
||||
return 1
|
||||
fi
|
||||
cmd+=(-e "${secret}=${!secret}")
|
||||
}
|
||||
cmd+=(-e "${secret}=${secret_val}")
|
||||
fi
|
||||
done
|
||||
else
|
||||
|
|
|
|||
|
|
@ -173,9 +173,40 @@ PROJECT_TOML="${PROJECT_TOML:-projects/disinto.toml}"
|
|||
sleep 1200 # 20 minutes
|
||||
done) &
|
||||
|
||||
# ── Load required secrets from secrets/*.enc (#777) ────────────────────
|
||||
# Edge container declares its required secrets; missing ones cause a hard fail.
|
||||
_AGE_KEY_FILE="${HOME}/.config/sops/age/keys.txt"
|
||||
_SECRETS_DIR="/opt/disinto/secrets"
|
||||
EDGE_REQUIRED_SECRETS="CADDY_SSH_KEY CADDY_SSH_HOST CADDY_SSH_USER CADDY_ACCESS_LOG"
|
||||
|
||||
_edge_decrypt_secret() {
|
||||
local enc_path="${_SECRETS_DIR}/${1}.enc"
|
||||
[ -f "$enc_path" ] || return 1
|
||||
age -d -i "$_AGE_KEY_FILE" "$enc_path" 2>/dev/null
|
||||
}
|
||||
|
||||
if [ -f "$_AGE_KEY_FILE" ] && [ -d "$_SECRETS_DIR" ]; then
|
||||
_missing=""
|
||||
for _secret_name in $EDGE_REQUIRED_SECRETS; do
|
||||
_val=$(_edge_decrypt_secret "$_secret_name") || { _missing="${_missing} ${_secret_name}"; continue; }
|
||||
export "$_secret_name=$_val"
|
||||
done
|
||||
if [ -n "$_missing" ]; then
|
||||
echo "FATAL: required secrets missing from secrets/*.enc:${_missing}" >&2
|
||||
echo " Run 'disinto secrets add <NAME>' for each missing secret." >&2
|
||||
echo " If migrating from .env.vault.enc, run 'disinto secrets migrate-from-vault' first." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "edge: loaded required secrets: ${EDGE_REQUIRED_SECRETS}" >&2
|
||||
else
|
||||
echo "FATAL: age key (${_AGE_KEY_FILE}) or secrets dir (${_SECRETS_DIR}) not found — cannot load required secrets" >&2
|
||||
echo " Ensure age is installed and secrets/*.enc files are present." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Start daily engagement collection cron loop in background (#745)
|
||||
# Runs collect-engagement.sh daily at ~23:50 UTC via a sleep loop that
|
||||
# calculates seconds until the next 23:50 window. SSH key from .env.vault.enc.
|
||||
# calculates seconds until the next 23:50 window. SSH key from secrets/*.enc (#777).
|
||||
(while true; do
|
||||
# Calculate seconds until next 23:50 UTC
|
||||
_now=$(date -u +%s)
|
||||
|
|
@ -186,26 +217,21 @@ done) &
|
|||
_sleep_secs=$(( _target - _now ))
|
||||
echo "edge: collect-engagement scheduled in ${_sleep_secs}s (next 23:50 UTC)" >&2
|
||||
sleep "$_sleep_secs"
|
||||
# Set CADDY_ACCESS_LOG so the script reads from the fetched local copy
|
||||
_fetch_log="/tmp/caddy-access-log-fetch.log"
|
||||
if [ -n "${CADDY_SSH_KEY:-}" ]; then
|
||||
_ssh_key_file=$(mktemp)
|
||||
printf '%s\n' "$CADDY_SSH_KEY" > "$_ssh_key_file"
|
||||
chmod 0600 "$_ssh_key_file"
|
||||
scp -i "$_ssh_key_file" -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 -o BatchMode=yes \
|
||||
"${CADDY_SSH_USER:-debian}@${CADDY_SSH_HOST:-disinto.ai}:${CADDY_ACCESS_LOG:-/var/log/caddy/access.log}" \
|
||||
"$_fetch_log" 2>&1 | tee -a /opt/disinto-logs/collect-engagement.log || true
|
||||
rm -f "$_ssh_key_file"
|
||||
if [ -s "$_fetch_log" ]; then
|
||||
CADDY_ACCESS_LOG="$_fetch_log" bash /opt/disinto/site/collect-engagement.sh 2>&1 \
|
||||
| tee -a /opt/disinto-logs/collect-engagement.log || true
|
||||
else
|
||||
echo "edge: collect-engagement: fetched log is empty, skipping parse" >&2
|
||||
fi
|
||||
rm -f "$_fetch_log"
|
||||
_ssh_key_file=$(mktemp)
|
||||
printf '%s\n' "$CADDY_SSH_KEY" > "$_ssh_key_file"
|
||||
chmod 0600 "$_ssh_key_file"
|
||||
scp -i "$_ssh_key_file" -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 -o BatchMode=yes \
|
||||
"${CADDY_SSH_USER}@${CADDY_SSH_HOST}:${CADDY_ACCESS_LOG}" \
|
||||
"$_fetch_log" 2>&1 | tee -a /opt/disinto-logs/collect-engagement.log || true
|
||||
rm -f "$_ssh_key_file"
|
||||
if [ -s "$_fetch_log" ]; then
|
||||
CADDY_ACCESS_LOG="$_fetch_log" bash /opt/disinto/site/collect-engagement.sh 2>&1 \
|
||||
| tee -a /opt/disinto-logs/collect-engagement.log || true
|
||||
else
|
||||
echo "edge: collect-engagement: CADDY_SSH_KEY not set, skipping" >&2
|
||||
echo "edge: collect-engagement: fetched log is empty, skipping parse" >&2
|
||||
fi
|
||||
rm -f "$_fetch_log"
|
||||
done) &
|
||||
|
||||
# Caddy as main process — run in foreground via wait so background jobs survive
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ description = """
|
|||
Fetch today's Caddy access log segment from the remote host using SCP.
|
||||
|
||||
The SSH key is read from the environment (CADDY_SSH_KEY), which is
|
||||
decrypted from .env.vault.enc by the dispatcher. It is NEVER hardcoded.
|
||||
decrypted from secrets/CADDY_SSH_KEY.enc by the edge entrypoint. It is NEVER hardcoded.
|
||||
|
||||
1. Write the SSH key to a temporary file with restricted permissions:
|
||||
_ssh_key_file=$(mktemp)
|
||||
|
|
|
|||
|
|
@ -79,28 +79,23 @@ AND set CADDY_ACCESS_LOG in the factory environment to match.
|
|||
|
||||
[[steps]]
|
||||
id = "store-private-key"
|
||||
title = "Add the private key to .env.vault.enc as CADDY_SSH_KEY"
|
||||
title = "Add the private key as CADDY_SSH_KEY secret"
|
||||
needs = ["generate-keypair"]
|
||||
description = """
|
||||
Store the private key in the factory's encrypted vault secrets.
|
||||
Store the private key in the factory's encrypted secrets store.
|
||||
|
||||
1. Read the private key:
|
||||
cat caddy-collect
|
||||
1. Add the private key using `disinto secrets add`:
|
||||
|
||||
2. Add it to .env.vault.enc (or .env.vault for plaintext fallback) as
|
||||
CADDY_SSH_KEY. The key is multi-line, so use the base64-encoded form:
|
||||
cat caddy-collect | disinto secrets add CADDY_SSH_KEY
|
||||
|
||||
echo "CADDY_SSH_KEY=$(base64 -w0 caddy-collect)" >> .env.vault.enc
|
||||
This encrypts the key with age and stores it as secrets/CADDY_SSH_KEY.enc.
|
||||
|
||||
Or, if using SOPS-encrypted vault, decrypt first, add the variable,
|
||||
then re-encrypt.
|
||||
|
||||
3. IMPORTANT: After storing, securely delete the local private key file:
|
||||
2. IMPORTANT: After storing, securely delete the local private key file:
|
||||
shred -u caddy-collect 2>/dev/null || rm -f caddy-collect
|
||||
rm -f caddy-collect.pub
|
||||
|
||||
The public key is already installed on the Caddy host; the private key
|
||||
now lives only in the vault.
|
||||
now lives only in secrets/CADDY_SSH_KEY.enc.
|
||||
|
||||
Never commit the private key to any git repository.
|
||||
"""
|
||||
|
|
@ -109,20 +104,19 @@ Never commit the private key to any git repository.
|
|||
|
||||
[[steps]]
|
||||
id = "store-caddy-host"
|
||||
title = "Add the Caddy host address to .env.vault.enc as CADDY_HOST"
|
||||
title = "Add the Caddy host details as secrets"
|
||||
needs = ["install-public-key"]
|
||||
description = """
|
||||
Store the Caddy host connection string so collect-engagement.sh knows
|
||||
Store the Caddy connection details so collect-engagement.sh knows
|
||||
where to SSH.
|
||||
|
||||
1. Add to .env.vault.enc (or .env.vault for plaintext fallback):
|
||||
1. Add each value using `disinto secrets add`:
|
||||
|
||||
echo "CADDY_HOST=user@caddy-host-ip-or-domain" >> .env.vault.enc
|
||||
echo 'disinto.ai' | disinto secrets add CADDY_SSH_HOST
|
||||
echo 'debian' | disinto secrets add CADDY_SSH_USER
|
||||
echo '/var/log/caddy/access.log' | disinto secrets add CADDY_ACCESS_LOG
|
||||
|
||||
Replace user@caddy-host-ip-or-domain with the actual SSH user and host
|
||||
(e.g. debian@203.0.113.42 or deploy@caddy.disinto.ai).
|
||||
|
||||
2. If using SOPS, decrypt/add/re-encrypt as above.
|
||||
Replace values with the actual SSH host, user, and log path for your setup.
|
||||
"""
|
||||
|
||||
# ── Step 5: Test the connection ──────────────────────────────────────────────
|
||||
|
|
|
|||
|
|
@ -213,7 +213,7 @@ should file a vault item instead of executing directly.
|
|||
**Exceptions** (do NOT flag these):
|
||||
- Code inside `vault/` — the vault system itself is allowed to handle secrets
|
||||
- References in comments or documentation explaining the architecture
|
||||
- `bin/disinto` setup commands that manage `.env.vault.enc` and the `run` subcommand
|
||||
- `bin/disinto` setup commands that manage `secrets/*.enc` and the `run` subcommand
|
||||
- Local operations (git push to forge, forge API calls with `FORGE_TOKEN`)
|
||||
|
||||
## 6. Re-review (if previous review is provided)
|
||||
|
|
|
|||
|
|
@ -158,8 +158,8 @@ export WOODPECKER_SERVER="${WOODPECKER_SERVER:-http://localhost:8000}"
|
|||
export CLAUDE_TIMEOUT="${CLAUDE_TIMEOUT:-7200}"
|
||||
|
||||
# Vault-only token guard (#745): external-action tokens (GITHUB_TOKEN, CLAWHUB_TOKEN)
|
||||
# must NEVER be available to agents. They live in .env.vault.enc and are injected
|
||||
# only into the ephemeral runner container at fire time. Unset them here so
|
||||
# must NEVER be available to agents. They live in secrets/*.enc and are decrypted
|
||||
# only into the ephemeral runner container at fire time (#777). Unset them here so
|
||||
# even an accidental .env inclusion cannot leak them into agent sessions.
|
||||
unset GITHUB_TOKEN 2>/dev/null || true
|
||||
unset CLAWHUB_TOKEN 2>/dev/null || true
|
||||
|
|
|
|||
|
|
@ -372,8 +372,8 @@ services:
|
|||
PLANNER_INTERVAL: ${PLANNER_INTERVAL:-43200}
|
||||
# IMPORTANT: agents get explicit environment variables (forge tokens, CI tokens, config).
|
||||
# Vault-only secrets (GITHUB_TOKEN, CLAWHUB_TOKEN, deploy keys) live in
|
||||
# .env.vault.enc and are NEVER injected here — only the runner
|
||||
# container receives them at fire time (AD-006, #745).
|
||||
# secrets/*.enc and are NEVER injected here — only the runner
|
||||
# container receives them at fire time (AD-006, #745, #777).
|
||||
depends_on:
|
||||
forgejo:
|
||||
condition: service_healthy
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue