From 2465841b84eb2368894c1133b8f0ad2cc9c198d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 15 Apr 2026 20:22:11 +0000 Subject: [PATCH 1/2] =?UTF-8?q?fix:=20[nomad-prep]=20P8=20=E2=80=94=20spot?= =?UTF-8?q?-check=20lib/mirrors.sh=20against=20empty=20Forgejo=20target=20?= =?UTF-8?q?(#796)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/mirror-bootstrap.md | 59 ++++++++++++++++++++++++++++++++ lib/AGENTS.md | 2 +- lib/mirrors.sh | 72 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 131 insertions(+), 2 deletions(-) create mode 100644 docs/mirror-bootstrap.md diff --git a/docs/mirror-bootstrap.md b/docs/mirror-bootstrap.md new file mode 100644 index 0000000..686e51e --- /dev/null +++ b/docs/mirror-bootstrap.md @@ -0,0 +1,59 @@ +# Mirror Bootstrap — Pull-Mirror Cutover Path + +How to populate an empty Forgejo repo from an external source using +`lib/mirrors.sh`'s `mirror_pull_register()`. + +## Prerequisites + +| Variable | Example | Purpose | +|---|---|---| +| `FORGE_URL` | `http://forgejo:3000` | Forgejo instance base URL | +| `FORGE_API` | `${FORGE_URL}/api/v1` | API base (set by `lib/env.sh`) | +| `FORGE_TOKEN` | (admin or org-owner token) | Must have `repo:create` scope | + +The target org/user must already exist on the Forgejo instance. + +## Command + +```bash +source lib/env.sh +source lib/mirrors.sh + +# Register a pull mirror — creates the repo and starts the first sync. +mirror_pull_register \ + "https://codeberg.org/johba/disinto.git" \ # source URL + "disinto-admin" \ # target owner + "disinto" \ # target repo name + "8h0m0s" # sync interval (optional, default 8h) +``` + +The function calls `POST /api/v1/repos/migrate` with `mirror: true`. +Forgejo creates the repo and immediately queues the first sync. + +## Verifying the sync + +```bash +# Check mirror status via API +forge_api GET "/repos/disinto-admin/disinto" | jq '.mirror, .mirror_interval' + +# Confirm content arrived — should list branches +forge_api GET "/repos/disinto-admin/disinto/branches" | jq '.[].name' +``` + +The first sync typically completes within a few seconds for small-to-medium +repos. For large repos, poll the branches endpoint until content appears. + +## Cutover scenario (Nomad migration) + +At cutover to the Nomad box: + +1. Stand up fresh Forgejo on the Nomad cluster (empty instance). +2. Create the `disinto-admin` org via `disinto init` or API. +3. Run `mirror_pull_register` pointing at the Codeberg source. +4. Wait for sync to complete (check branches endpoint). +5. Once content is confirmed, proceed with `disinto init` against the + now-populated repo — all subsequent `mirror_push` calls will push + to any additional mirrors configured in `projects/*.toml`. + +No manual `git clone` + `git push` step is needed. The Forgejo pull-mirror +handles the entire transfer. diff --git a/lib/AGENTS.md b/lib/AGENTS.md index f746217..4564cfa 100644 --- a/lib/AGENTS.md +++ b/lib/AGENTS.md @@ -14,7 +14,7 @@ sourced as needed. | `lib/parse-deps.sh` | Extracts dependency issue numbers from an issue body (stdin → stdout, one number per line). Matches `## Dependencies` / `## Depends on` / `## Blocked by` sections and inline `depends on #N` / `blocked by #N` patterns. Inline scan skips fenced code blocks to prevent false positives from code examples in issue bodies. Not sourced — executed via `bash lib/parse-deps.sh`. | dev-poll | | `lib/formula-session.sh` | `acquire_run_lock()`, `load_formula()`, `load_formula_or_profile()`, `build_context_block()`, `ensure_ops_repo()`, `ops_commit_and_push()`, `build_prompt_footer()`, `build_sdk_prompt_footer()`, `formula_worktree_setup()`, `formula_prepare_profile_context()`, `formula_lessons_block()`, `profile_write_journal()`, `profile_load_lessons()`, `ensure_profile_repo()`, `_profile_has_repo()`, `_count_undigested_journals()`, `_profile_digest_journals()`, `_profile_restore_lessons()`, `_profile_commit_and_push()`, `resolve_agent_identity()`, `build_graph_section()`, `build_scratch_instruction()`, `read_scratch_context()`, `cleanup_stale_crashed_worktrees()` — shared helpers for formula-driven polling-loop agents (lock, .profile repo management, prompt assembly, worktree setup). Memory guard is provided by `memory_guard()` in `lib/env.sh` (not duplicated here). `resolve_agent_identity()` — sets `FORGE_TOKEN`, `AGENT_IDENTITY`, `FORGE_REMOTE` from per-agent token env vars and FORGE_URL remote detection. `build_graph_section()` generates the structural-analysis section (runs `lib/build-graph.py`, formats JSON output) — previously duplicated in planner-run.sh and predictor-run.sh, now shared here. `cleanup_stale_crashed_worktrees()` — thin wrapper around `worktree_cleanup_stale()` from `lib/worktree.sh` (kept for backwards compatibility). **Journal digestion guards (#702)**: `_profile_digest_journals()` respects `PROFILE_DIGEST_TIMEOUT` (default 300s) and `PROFILE_DIGEST_MAX_BATCH` (default 5 journals per run); `_profile_restore_lessons()` restores the previous lessons-learned.md on digest failure. | planner-run.sh, predictor-run.sh, gardener-run.sh, supervisor-run.sh, dev-agent.sh | | `lib/guard.sh` | `check_active(agent_name)` — reads `$FACTORY_ROOT/state/.{agent_name}-active`; exits 0 (skip) if the file is absent. Factory is off by default — state files must be created to enable each agent. **Logs a message to stderr** when skipping (`[check_active] SKIP: state file not found`), so agent dropout is visible in loop logs. Sourced by dev-poll.sh, review-poll.sh, predictor-run.sh, supervisor-run.sh. | polling-loop entry points | -| `lib/mirrors.sh` | `mirror_push()` — pushes `$PRIMARY_BRANCH` + tags to all configured mirror remotes (fire-and-forget background pushes). Reads `MIRROR_NAMES` and `MIRROR_*` vars exported by `load-project.sh` from the `[mirrors]` TOML section. Failures are logged but never block the pipeline. Sourced by dev-poll.sh — called after every successful merge. | dev-poll.sh | +| `lib/mirrors.sh` | `mirror_push()` — pushes `$PRIMARY_BRANCH` + tags to all configured mirror remotes (fire-and-forget background pushes). Reads `MIRROR_NAMES` and `MIRROR_*` vars exported by `load-project.sh` from the `[mirrors]` TOML section. Failures are logged but never block the pipeline. `mirror_pull_register(clone_url, owner, repo_name, [interval])` — registers a Forgejo pull mirror via `POST /repos/migrate` with `mirror: true`. Creates the target repo and queues the first sync automatically. Works against empty Forgejo instances — no pre-existing content required. Used for Nomad migration cutover: point at Codeberg source, wait for sync, then proceed with `disinto init`. See [docs/mirror-bootstrap.md](../docs/mirror-bootstrap.md) for the full cutover path. Sourced by dev-poll.sh — called after every successful merge. | dev-poll.sh | | `lib/build-graph.py` | Python tool: parses VISION.md, prerequisites.md (from ops repo), AGENTS.md, formulas/*.toml, evidence/ (from ops repo), and forge issues/labels into a NetworkX DiGraph. Runs structural analyses (orphaned objectives, stale prerequisites, thin evidence, circular deps) and outputs a JSON report. Used by `review-pr.sh` (per-PR changed-file analysis) and `predictor-run.sh` (full-project analysis) to provide structural context to Claude. | review-pr.sh, predictor-run.sh | | `lib/secret-scan.sh` | `scan_for_secrets()` — detects potential secrets (API keys, bearer tokens, private keys, URLs with embedded credentials) in text; returns 1 if secrets found. `redact_secrets()` — replaces detected secret patterns with `[REDACTED]`. | issue-lifecycle.sh | | `lib/stack-lock.sh` | File-based lock protocol for singleton project stack access. `stack_lock_acquire(holder, project)` — polls until free, breaks stale heartbeats (>10 min old), claims lock. `stack_lock_release(project)` — deletes lock file. `stack_lock_check(project)` — inspect current lock state. `stack_lock_heartbeat(project)` — update heartbeat timestamp (callers must call every 2 min while holding). Lock files at `~/data/locks/-stack.lock`. | docker/edge/dispatcher.sh, reproduce formula | diff --git a/lib/mirrors.sh b/lib/mirrors.sh index 3ba561d..7bcd41d 100644 --- a/lib/mirrors.sh +++ b/lib/mirrors.sh @@ -1,8 +1,10 @@ #!/usr/bin/env bash -# mirrors.sh — Push primary branch + tags to configured mirror remotes. +# mirrors.sh — Mirror helpers: push to remotes + register pull mirrors via API. # # Usage: source lib/mirrors.sh; mirror_push +# source lib/mirrors.sh; mirror_pull_register [interval] # Requires: PROJECT_REPO_ROOT, PRIMARY_BRANCH, MIRROR_* vars from load-project.sh +# FORGE_API, FORGE_TOKEN for pull-mirror registration # shellcheck disable=SC2154 # globals set by load-project.sh / calling script @@ -37,3 +39,71 @@ mirror_push() { log "mirror: pushed to ${name} (pid $!)" done } + +# --------------------------------------------------------------------------- +# mirror_pull_register — register a Forgejo pull mirror via the /repos/migrate API. +# +# Creates a new repo as a pull mirror of an external source. Works against +# empty target repos (the repo is created by the API call itself). +# +# Usage: +# mirror_pull_register [interval] +# +# Args: +# clone_url — HTTPS URL of the source repo (e.g. https://codeberg.org/johba/disinto.git) +# owner — Forgejo org or user that will own the mirror repo +# repo_name — name of the new mirror repo on Forgejo +# interval — sync interval (default: "8h0m0s"; Forgejo duration format) +# +# Requires: +# FORGE_API, FORGE_TOKEN (from env.sh) +# +# Returns 0 on success, 1 on failure. Prints the new repo JSON to stdout. +# --------------------------------------------------------------------------- +mirror_pull_register() { + local clone_url="$1" + local owner="$2" + local repo_name="$3" + local interval="${4:-8h0m0s}" + + if [ -z "${FORGE_API:-}" ] || [ -z "${FORGE_TOKEN:-}" ]; then + echo "ERROR: FORGE_API and FORGE_TOKEN must be set" >&2 + return 1 + fi + + if [ -z "$clone_url" ] || [ -z "$owner" ] || [ -z "$repo_name" ]; then + echo "Usage: mirror_pull_register [interval]" >&2 + return 1 + fi + + local payload + payload=$(cat <&2 + return 1 + fi +} From b6f2d83a2887407629de7fcc41ffac48fd2f6413 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 15 Apr 2026 20:29:27 +0000 Subject: [PATCH 2/2] fix: use FORGE_API_BASE for /repos/migrate endpoint, build payload with jq - FORGE_API is repo-scoped; /repos/migrate needs the global FORGE_API_BASE - Use jq -n --arg for safe JSON construction (no shell interpolation) - Update docs to reference FORGE_API_BASE Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/mirror-bootstrap.md | 2 +- lib/mirrors.sh | 34 ++++++++++++++++++---------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/docs/mirror-bootstrap.md b/docs/mirror-bootstrap.md index 686e51e..ca91d32 100644 --- a/docs/mirror-bootstrap.md +++ b/docs/mirror-bootstrap.md @@ -8,7 +8,7 @@ How to populate an empty Forgejo repo from an external source using | Variable | Example | Purpose | |---|---|---| | `FORGE_URL` | `http://forgejo:3000` | Forgejo instance base URL | -| `FORGE_API` | `${FORGE_URL}/api/v1` | API base (set by `lib/env.sh`) | +| `FORGE_API_BASE` | `${FORGE_URL}/api/v1` | Global API base (set by `lib/env.sh`) | | `FORGE_TOKEN` | (admin or org-owner token) | Must have `repo:create` scope | The target org/user must already exist on the Forgejo instance. diff --git a/lib/mirrors.sh b/lib/mirrors.sh index 7bcd41d..9b135c4 100644 --- a/lib/mirrors.sh +++ b/lib/mirrors.sh @@ -4,7 +4,7 @@ # Usage: source lib/mirrors.sh; mirror_push # source lib/mirrors.sh; mirror_pull_register [interval] # Requires: PROJECT_REPO_ROOT, PRIMARY_BRANCH, MIRROR_* vars from load-project.sh -# FORGE_API, FORGE_TOKEN for pull-mirror registration +# FORGE_API_BASE, FORGE_TOKEN for pull-mirror registration # shellcheck disable=SC2154 # globals set by load-project.sh / calling script @@ -56,7 +56,7 @@ mirror_push() { # interval — sync interval (default: "8h0m0s"; Forgejo duration format) # # Requires: -# FORGE_API, FORGE_TOKEN (from env.sh) +# FORGE_API_BASE, FORGE_TOKEN (from env.sh) # # Returns 0 on success, 1 on failure. Prints the new repo JSON to stdout. # --------------------------------------------------------------------------- @@ -66,8 +66,8 @@ mirror_pull_register() { local repo_name="$3" local interval="${4:-8h0m0s}" - if [ -z "${FORGE_API:-}" ] || [ -z "${FORGE_TOKEN:-}" ]; then - echo "ERROR: FORGE_API and FORGE_TOKEN must be set" >&2 + if [ -z "${FORGE_API_BASE:-}" ] || [ -z "${FORGE_TOKEN:-}" ]; then + echo "ERROR: FORGE_API_BASE and FORGE_TOKEN must be set" >&2 return 1 fi @@ -77,23 +77,25 @@ mirror_pull_register() { fi local payload - payload=$(cat <