From 02dd03eaaf9deb3f508d9742fa6f6ad370532fc0 Mon Sep 17 00:00:00 2001 From: johba Date: Sat, 28 Mar 2026 11:14:42 +0000 Subject: [PATCH] chore: remove BOOTSTRAP.md, slim CLAUDE.md BOOTSTRAP.md is superseded by the disinto-factory skill (SKILL.md). CLAUDE.md now just points to AGENTS.md and the skill. Updated AGENTS.md reference accordingly. Co-Authored-By: Claude Opus 4.6 (1M context) --- AGENTS.md | 2 +- BOOTSTRAP.md | 460 --------------------------------------- CLAUDE.md | 18 +- disinto-factory/SKILL.md | 1 - 4 files changed, 3 insertions(+), 478 deletions(-) delete mode 100644 BOOTSTRAP.md diff --git a/AGENTS.md b/AGENTS.md index 4d5a91f..ffc5561 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ gardener, supervisor, planner, predictor, action, vault) that pick up issues fro implement them, review PRs, plan from the vision, gate dangerous actions, and keep the system healthy — all via cron and `claude -p`. -See `README.md` for the full architecture and `BOOTSTRAP.md` for setup. +See `README.md` for the full architecture and `disinto-factory/SKILL.md` for setup. ## Directory layout diff --git a/BOOTSTRAP.md b/BOOTSTRAP.md deleted file mode 100644 index 80e7408..0000000 --- a/BOOTSTRAP.md +++ /dev/null @@ -1,460 +0,0 @@ -# Bootstrapping a New Project - -How to point disinto at a new target project and get all agents running. - -## Prerequisites - -Before starting, ensure you have: - -- [ ] A **git repo** (GitHub, Codeberg, or any URL) with at least one issue labeled `backlog` -- [ ] A **Woodpecker CI** pipeline (`.woodpecker/` dir with at least one `.yml`) -- [ ] **Docker** installed (for local Forgejo provisioning) — or a running Forgejo instance -- [ ] A **local clone** of the target repo on the same machine as disinto -- [ ] `claude` CLI installed and authenticated (`claude --version`) -- [ ] `tmux` installed (`tmux -V`) — required for persistent dev sessions (issue #80+) - -## Quick Start - -The fastest path is `disinto init`, which provisions a local Forgejo instance, creates bot users and tokens, clones the repo, and sets up cron — all in one command: - -```bash -disinto init https://github.com/org/repo -``` - -This will: -1. Start a local Forgejo instance via Docker (at `http://localhost:3000`) -2. Create admin + bot users (dev-bot, review-bot) with API tokens -3. Create the repo on Forgejo and push your code -4. Generate a `projects/.toml` config -5. Create standard labels (backlog, in-progress, blocked, etc.) -6. Install cron entries for the agents - -No external accounts or tokens needed. - -## 1. Secret Management (SOPS + age) - -Disinto encrypts secrets at rest using [SOPS](https://github.com/getsops/sops) with [age](https://age-encryption.org/) encryption. When `sops` and `age` are installed, `disinto init` automatically: - -1. Generates an age key at `~/.config/sops/age/keys.txt` (if none exists) -2. Creates `.sops.yaml` pinning the age public key -3. Encrypts all secrets into `.env.enc` (safe to commit) -4. Removes the plaintext `.env` - -**Install the tools:** - -```bash -# age (key generation) -apt install age # Debian/Ubuntu -brew install age # macOS - -# sops (encryption/decryption) -# Download from https://github.com/getsops/sops/releases -``` - -**The age private key** at `~/.config/sops/age/keys.txt` is the single file that must be protected. Back it up securely — without it, `.env.enc` cannot be decrypted. LUKS disk encryption on the VPS protects this key at rest. - -**Managing secrets after setup:** - -```bash -disinto secrets edit # Opens .env.enc in $EDITOR, re-encrypts on save -disinto secrets show # Prints decrypted secrets (for debugging) -disinto secrets migrate # Converts existing plaintext .env -> .env.enc -``` - -**Fallback:** If `sops`/`age` are not installed, `disinto init` writes secrets to a plaintext `.env` file with a warning. All agents load secrets transparently — `lib/env.sh` checks for `.env.enc` first, then falls back to `.env`. - -## 2. Configure `.env` - -```bash -cp .env.example .env -``` - -Fill in: - -```bash -# ── Forge (auto-populated by disinto init) ───────────────── -FORGE_URL=http://localhost:3000 # local Forgejo instance -FORGE_TOKEN= # dev-bot token (auto-generated) -FORGE_REVIEW_TOKEN= # review-bot token (auto-generated) - -# ── Woodpecker CI ─────────────────────────────────────────── -WOODPECKER_TOKEN=tok_xxxxxxxx -WOODPECKER_SERVER=http://localhost:8000 -# WOODPECKER_REPO_ID — now per-project, set in projects/*.toml [ci] section - -# Woodpecker Postgres (for direct pipeline queries) -WOODPECKER_DB_PASSWORD=secret -WOODPECKER_DB_USER=woodpecker -WOODPECKER_DB_HOST=127.0.0.1 -WOODPECKER_DB_NAME=woodpecker - -# ── Tuning ────────────────────────────────────────────────── -CLAUDE_TIMEOUT=7200 # seconds per Claude invocation -``` - -### Backwards compatibility - -If you have an existing deployment using `CODEBERG_TOKEN` / `REVIEW_BOT_TOKEN` in `.env`, those still work — `env.sh` falls back to the old names automatically. No migration needed. - -## 3. Configure Project TOML - -Each project needs a `projects/.toml` file with box-specific settings -(absolute paths, Woodpecker CI IDs, forge URL). These files are -**gitignored** — they are local installation config, not shared code. - -To create one: - -```bash -# Automatic — generates TOML, clones repo, sets up cron: -disinto init https://github.com/org/repo - -# Manual — copy a template and fill in your values: -cp projects/myproject.toml.example projects/myproject.toml -vim projects/myproject.toml -``` - -The `forge_url` field in the TOML tells all agents where to find the forge API: - -```toml -name = "myproject" -repo = "org/myproject" -forge_url = "http://localhost:3000" -``` - -The repo ships `projects/*.toml.example` templates showing the expected -structure. See any `.toml.example` file for the full field reference. - -## 4. Claude Code Global Settings - -Configure `~/.claude/settings.json` with **only** permissions and `skipDangerousModePermissionPrompt`. Do not add hooks to the global settings — `agent-session.sh` injects per-worktree hooks automatically. - -Match the configuration from harb-staging exactly. The file should contain only permission grants and the dangerous-mode flag: - -```json -{ - "permissions": { - "allow": [ - "..." - ] - }, - "skipDangerousModePermissionPrompt": true -} -``` - -### Seed `~/.claude.json` - -Run `claude --dangerously-skip-permissions` once interactively to create `~/.claude.json`. This file must exist before cron-driven agents can run. - -```bash -claude --dangerously-skip-permissions -# Exit after it initializes successfully -``` - -## 5. File Ownership - -Everything under `/home/debian` must be owned by `debian:debian`. Root-owned files cause permission errors when agents run as the `debian` user. - -```bash -chown -R debian:debian /home/debian/harb /home/debian/dark-factory -``` - -Verify no root-owned files exist in agent temp directories: - -```bash -# These should return nothing -find /tmp/dev-* /tmp/harb-* /tmp/review-* -not -user debian 2>/dev/null -``` - -## 5b. Woodpecker CI + Forgejo Integration - -`disinto init` automatically configures Woodpecker to use the local Forgejo instance as its forge backend if `WOODPECKER_SERVER` is set in `.env`. This includes: - -1. Creating an OAuth2 application on Forgejo for Woodpecker -2. Writing `WOODPECKER_FORGEJO_*` env vars to `.env` -3. Activating the repo in Woodpecker - -### Manual setup (if Woodpecker runs outside of `disinto init`) - -If you manage Woodpecker separately, configure these env vars in its server config: - -```bash -WOODPECKER_FORGEJO=true -WOODPECKER_FORGEJO_URL=http://localhost:3000 -WOODPECKER_FORGEJO_CLIENT= -WOODPECKER_FORGEJO_SECRET= -``` - -To create the OAuth2 app on Forgejo: - -```bash -# Create OAuth2 application (redirect URI = Woodpecker authorize endpoint) -curl -X POST \ - -H "Authorization: token ${FORGE_TOKEN}" \ - -H "Content-Type: application/json" \ - "http://localhost:3000/api/v1/user/applications/oauth2" \ - -d '{"name":"woodpecker-ci","redirect_uris":["http://localhost:8000/authorize"],"confidential_client":true}' -``` - -The response contains `client_id` and `client_secret` for `WOODPECKER_FORGEJO_CLIENT` / `WOODPECKER_FORGEJO_SECRET`. - -To activate the repo in Woodpecker: - -```bash -woodpecker-cli repo add / -# Or via API: -curl -X POST \ - -H "Authorization: Bearer ${WOODPECKER_TOKEN}" \ - "http://localhost:8000/api/repos" \ - -d '{"forge_remote_id":"/"}' -``` - -Woodpecker will now trigger pipelines on pushes to Forgejo and push commit status back. Disinto queries Woodpecker directly for CI status (with a forge API fallback), so pipeline results are visible even if Woodpecker's status push to Forgejo is delayed. - -## 6. Prepare the Target Repo - -### Required: CI pipeline - -The repo needs at least one Woodpecker pipeline. Disinto monitors CI status to decide when a PR is ready for review and when it can merge. - -### Required: `CLAUDE.md` - -Create a `CLAUDE.md` in the repo root. This is the context document that dev-agent and review-agent read before working. It should cover: - -- **What the project is** (one paragraph) -- **Tech stack** (languages, frameworks, DB) -- **How to build/run/test** (`npm install`, `npm test`, etc.) -- **Coding conventions** (import style, naming, linting rules) -- **Project structure** (key directories and what lives where) - -The dev-agent reads this file via `claude -p` before implementing any issue. The better this file, the better the output. - -### Required: Issue labels - -`disinto init` creates these automatically. If setting up manually, create these labels on the forge repo: - -| Label | Purpose | -|-------|---------| -| `backlog` | Issues ready to be picked up by dev-agent | -| `in-progress` | Managed by dev-agent (auto-applied, auto-removed) | - -Optional but recommended: - -| Label | Purpose | -|-------|---------| -| `tech-debt` | Gardener can promote these to `backlog` | -| `blocked` | Dev-agent marks issues with unmet dependencies | -| `formula` | **Not yet functional.** Formula dispatch lives on the unmerged `feat/formula` branch. Dev-agent will skip any issue with this label until that branch is merged. Template files exist in `formulas/` for future use. | - -### Required: Branch protection - -On Forgejo, set up branch protection for your primary branch: - -- **Require pull request reviews**: enabled -- **Required approvals**: 1 (from the review bot account) -- **Restrict push**: only allow merges via PR - -This ensures dev-agent can't merge its own PRs — it must wait for review-agent (running as the bot account) to approve. - -> **Common pitfall:** Approvals alone are not enough. You must also: -> 1. Add `review-bot` as a **write** collaborator on the repo (Settings → Collaborators) -> 2. Set both `approvals_whitelist_username` **and** `merge_whitelist_usernames` to include `review-bot` in the branch protection rule -> -> Without write access, the bot's approval is counted but the merge API returns HTTP 405. - -### Required: Seed the `AGENTS.md` tree - -The planner maintains an `AGENTS.md` tree — architecture docs with -per-file `` watermarks. You must seed this before -the first planner run, otherwise the planner sees no watermarks and treats the -entire repo as "new", generating a noisy first-run diff. - -1. **Create `AGENTS.md` in the repo root** with a one-page overview of the - project: what it is, tech stack, directory layout, key conventions. Link - to sub-directory AGENTS.md files. - -2. **Create sub-directory `AGENTS.md` files** for each major directory - (e.g. `frontend/AGENTS.md`, `backend/AGENTS.md`). Keep each under ~200 - lines — architecture and conventions, not implementation details. - -3. **Set the watermark** on line 1 of every AGENTS.md file to the current HEAD: - ```bash - SHA=$(git rev-parse --short HEAD) - for f in $(find . -name "AGENTS.md" -not -path "./.git/*"); do - sed -i "1s/^/\n/" "$f" - done - ``` - -4. **Symlink `CLAUDE.md`** so Claude Code picks up the same file: - ```bash - ln -sf AGENTS.md CLAUDE.md - ``` - -5. Commit and push. The planner will now see 0 changes on its first run and - only update files when real commits land. - -See `formulas/run-planner.toml` (agents-update step) for the full AGENTS.md conventions. - -## 7. Write Good Issues - -Dev-agent works best with issues that have: - -- **Clear title** describing the change (e.g., "Add email validation to customer form") -- **Acceptance criteria** — what "done" looks like -- **Dependencies** — reference blocking issues with `#NNN` in the body or a `## Dependencies` section: - ``` - ## Dependencies - - #4 - - #7 - ``` - -Dev-agent checks that all referenced issues are closed (= merged) before starting work. If any are open, the issue is skipped and checked again next cycle. - -## 8. Install Cron - -```bash -crontab -e -``` - -### Single project - -Add (adjust paths): - -```cron -FACTORY_ROOT=/home/you/disinto - -# Supervisor — health checks, auto-healing (every 10 min) -0,10,20,30,40,50 * * * * $FACTORY_ROOT/supervisor/supervisor-poll.sh - -# Review agent — find unreviewed PRs (every 10 min, offset +3) -3,13,23,33,43,53 * * * * $FACTORY_ROOT/review/review-poll.sh $FACTORY_ROOT/projects/myproject.toml - -# Dev agent — find ready issues, implement (every 10 min, offset +6) -6,16,26,36,46,56 * * * * $FACTORY_ROOT/dev/dev-poll.sh $FACTORY_ROOT/projects/myproject.toml - -# Gardener — backlog grooming (daily) -15 8 * * * $FACTORY_ROOT/gardener/gardener-poll.sh - -# Planner — AGENTS.md maintenance + gap analysis (weekly) -0 9 * * 1 $FACTORY_ROOT/planner/planner-poll.sh -``` - -`review-poll.sh`, `dev-poll.sh`, and `gardener-poll.sh` all take a project TOML file as their first argument. - -### Multiple projects - -Stagger each project's polls so they don't overlap. With the example below, cross-project gaps are 2 minutes: - -```cron -FACTORY_ROOT=/home/you/disinto - -# Supervisor (shared) -0,10,20,30,40,50 * * * * $FACTORY_ROOT/supervisor/supervisor-poll.sh - -# Project A — review +3, dev +6 -3,13,23,33,43,53 * * * * $FACTORY_ROOT/review/review-poll.sh $FACTORY_ROOT/projects/project-a.toml -6,16,26,36,46,56 * * * * $FACTORY_ROOT/dev/dev-poll.sh $FACTORY_ROOT/projects/project-a.toml - -# Project B — review +8, dev +1 (2-min gap from project A) -8,18,28,38,48,58 * * * * $FACTORY_ROOT/review/review-poll.sh $FACTORY_ROOT/projects/project-b.toml -1,11,21,31,41,51 * * * * $FACTORY_ROOT/dev/dev-poll.sh $FACTORY_ROOT/projects/project-b.toml - -# Gardener — per-project backlog grooming (daily) -15 8 * * * $FACTORY_ROOT/gardener/gardener-poll.sh $FACTORY_ROOT/projects/project-a.toml -45 8 * * * $FACTORY_ROOT/gardener/gardener-poll.sh $FACTORY_ROOT/projects/project-b.toml - -# Planner — AGENTS.md maintenance + gap analysis (weekly) -0 9 * * 1 $FACTORY_ROOT/planner/planner-poll.sh -``` - -The staggered offsets prevent agents from competing for resources. Each project gets its own lock file (`/tmp/dev-agent-{name}.lock`) derived from the `name` field in its TOML, so concurrent runs across projects are safe. - -## 9. Verify - -```bash -# Should complete with "all clear" (no problems to fix) -bash supervisor/supervisor-poll.sh - -# Should list backlog issues (or "no backlog issues") -bash dev/dev-poll.sh - -# Should find no unreviewed PRs (or review one if exists) -bash review/review-poll.sh -``` - -Check logs after a few cycles: - -```bash -tail -30 supervisor/supervisor.log -tail -30 dev/dev-agent.log -tail -30 review/review.log -``` - -## Lifecycle - -Once running, the system operates autonomously: - -``` -You write issues (with backlog label) - → dev-poll finds ready issues - → dev-agent implements in a worktree, opens PR - → CI runs (Woodpecker) - → review-agent reviews, approves or requests changes - → dev-agent addresses feedback (if any) - → merge, close issue, clean up - -Meanwhile: - supervisor-poll monitors health, kills stale processes, manages resources - gardener grooms backlog: closes duplicates, promotes tech-debt, escalates ambiguity - planner rebuilds AGENTS.md from git history, gap-analyses against VISION.md -``` - -## Troubleshooting - -| Symptom | Check | -|---------|-------| -| Dev-agent not picking up issues | `cat /tmp/dev-agent.lock` — is another instance running? Issues labeled `backlog`? Dependencies met? | -| PR not getting reviewed | `tail review/review.log` — CI must pass first. Review bot token valid? | -| CI stuck | `bash lib/ci-debug.sh` — check Woodpecker. Rate-limited? (exit 128 = wait 15 min) | -| Claude not found | `which claude` — must be in PATH. Check `lib/env.sh` adds `~/.local/bin`. | -| Merge fails | Branch protection misconfigured? Review bot needs write access to the repo. | -| Memory issues | Supervisor auto-heals at <500 MB free. Check `supervisor/supervisor.log` for P0 alerts. | -| Works on one box but not another | Diff configs first (`~/.claude/settings.json`, `.env`, crontab, branch protection). Write code never — config mismatches are the #1 cause of cross-box failures. | - -### Multi-project common blockers - -| Symptom | Cause | Fix | -|---------|-------|-----| -| Dev-agent for project B never starts | Shared lock file path | Each TOML `name` field must be unique — lock is `/tmp/dev-agent-{name}.lock` | -| Review-poll skips all PRs | CI gate with no CI configured | Set `woodpecker_repo_id = 0` in the TOML `[ci]` section to bypass the CI check | -| Approved PRs never merge (HTTP 405) | `review-bot` not in merge/approvals whitelist | Add as write collaborator; set both `approvals_whitelist_username` and `merge_whitelist_usernames` in branch protection | -| Dev-agent churns through issues without waiting for open PRs to land | No single-threaded enforcement | `WAITING_PRS` check in dev-poll holds new work — verify TOML `name` is consistent across invocations | -| Label ping-pong (issue reopened then immediately re-closed) | `already_done` handler doesn't close issue | Review dev-agent log; `already_done` status should auto-close the issue | - -## Security: Docker Socket Sharing in CI - -The `woodpecker-agent` service mounts `/var/run/docker.sock` to execute `type: docker` CI pipelines. This grants root-equivalent access to the Docker host — any CI pipeline step can run privileged containers, mount arbitrary host paths, or access other containers' data. - -**Mitigations:** - -- **Run disinto in an LXD/VM container, not on bare metal.** When the Docker daemon runs inside an LXD container, LXD's user namespace mapping and resource limits contain the blast radius. A compromised CI step cannot reach the real host. -- **`WOODPECKER_MAX_WORKFLOWS: 1`** limits concurrent CI resource usage, preventing a runaway pipeline from exhausting host resources. -- **`WOODPECKER_AGENT_SECRET`** authenticates the agent↔server gRPC connection. `disinto init` auto-generates this secret and stores it in `.env` (or `.env.enc` when SOPS is available). -- Consider setting `WOODPECKER_BACKEND_DOCKER_VOLUMES` on the agent to restrict which host volumes CI pipelines can mount. - -**Threat model:** PRs are created by the dev-agent (Claude) and auto-reviewed by the review-bot. A crafted backlog issue could theoretically produce a PR whose CI step exploits the Docker socket. The LXD containment boundary is the primary defense — treat the LXD container as the trust boundary, not the Docker daemon inside it. - -## Action Runner — disinto (harb-staging) - -Added 2026-03-19. Polls disinto repo for `action`-labeled issues. - -``` -*/5 * * * * cd /home/debian/dark-factory && bash action/action-poll.sh projects/disinto.toml >> /tmp/action-disinto-cron.log 2>&1 -``` - -Runs locally on harb-staging — same box where Caddy/site live. For formulas that need local resources (publish-site, etc). - -### Fix applied: action-agent.sh needs +x -The script wasn't executable after git clone. Run: -```bash -chmod +x action/action-agent.sh action/action-poll.sh -``` diff --git a/CLAUDE.md b/CLAUDE.md index 63927a1..9671180 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,19 +2,5 @@ This repo is **disinto** — an autonomous code factory. -For setup and operations, load the `disinto-factory` skill from `disinto-factory/SKILL.md`. - -Quick references: -- `AGENTS.md` — per-agent architecture and file-level docs -- `VISION.md` — project philosophy -- `BOOTSTRAP.md` — detailed init walkthrough -- `disinto-factory/references/troubleshooting.md` — common issues and fixes -- `disinto-factory/scripts/factory-status.sh` — quick status check - -## Code conventions - -- Bash for checks, AI for judgment -- Zero LLM tokens when idle (cron polls are pure bash) -- Fire-and-forget mirror pushes (never block the pipeline) -- Issues are the unit of work; PRs are the delivery mechanism -- See `AGENTS.md` for per-file watermarks and coding conventions +Read `AGENTS.md` for architecture, coding conventions, and per-file documentation. +For setup and operations, load the `disinto-factory` skill (`disinto-factory/SKILL.md`). diff --git a/disinto-factory/SKILL.md b/disinto-factory/SKILL.md index 45186fc..8e17508 100644 --- a/disinto-factory/SKILL.md +++ b/disinto-factory/SKILL.md @@ -203,7 +203,6 @@ docker exec disinto-forgejo-1 su -c "forgejo admin user change-password --userna - Read `AGENTS.md` for per-agent architecture and file-level docs - Read `VISION.md` for project philosophy -- Read `BOOTSTRAP.md` for detailed init walkthrough - The factory uses a single internal Forgejo as its forge, regardless of where mirrors go - Dev-agent uses `claude -p --resume` for session continuity across CI/review cycles - Mirror pushes happen automatically after every merge (fire-and-forget)