183 lines
4.2 KiB
Markdown
183 lines
4.2 KiB
Markdown
# Nomad Cutover Runbook
|
|
|
|
End-to-end procedure to cut over the disinto factory from docker-compose on
|
|
disinto-dev-box to Nomad on disinto-nomad-box.
|
|
|
|
**Target**: disinto-nomad-box (10.10.10.216) becomes production; disinto-dev-box
|
|
stays warm for rollback.
|
|
|
|
**Downtime budget**: <5 min blue-green flip.
|
|
|
|
**Data scope**: Forgejo issues + disinto-ops git bundle only. Everything else is
|
|
regenerated or discarded. OAuth secrets are regenerated on fresh init (all
|
|
sessions invalidated).
|
|
|
|
---
|
|
|
|
## 1. Pre-cutover readiness checklist
|
|
|
|
- [ ] Nomad + Vault stack healthy on a fresh wipe+init (step 5 verified)
|
|
- [ ] Codeberg mirror current — `git log` parity between dev-box Forgejo and
|
|
Codeberg
|
|
- [ ] SSH key pair generated for nomad-box, registered on DO edge (see §4.6)
|
|
- [ ] Companion tools landed:
|
|
- `disinto backup create` (#1057)
|
|
- `disinto backup import` (#1058)
|
|
- [ ] Backup tarball produced and tested against a scratch LXC (see §3)
|
|
|
|
---
|
|
|
|
## 2. Pre-cutover artifact: backup
|
|
|
|
On disinto-dev-box:
|
|
|
|
```bash
|
|
./bin/disinto backup create /tmp/disinto-backup-$(date +%Y%m%d).tar.gz
|
|
```
|
|
|
|
Copy the tarball to nomad-box (and optionally to a local workstation for
|
|
safekeeping):
|
|
|
|
```bash
|
|
scp /tmp/disinto-backup-*.tar.gz nomad-box:/tmp/
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Pre-cutover dry-run
|
|
|
|
On a throwaway LXC:
|
|
|
|
```bash
|
|
lxc launch ubuntu:24.04 cutover-dryrun
|
|
# inside the container:
|
|
disinto init --backend=nomad --import-env .env --with edge
|
|
./bin/disinto backup import /tmp/disinto-backup-*.tar.gz
|
|
```
|
|
|
|
Verify:
|
|
|
|
- Issue count matches source Forgejo
|
|
- disinto-ops repo refs match source bundle
|
|
|
|
Destroy the LXC once satisfied:
|
|
|
|
```bash
|
|
lxc delete cutover-dryrun --force
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Cutover T-0 (operator executes; <5 min target)
|
|
|
|
### 4.1 Stop dev-box services
|
|
|
|
```bash
|
|
# On disinto-dev-box — stop, do NOT remove volumes (rollback needs them)
|
|
docker-compose stop
|
|
```
|
|
|
|
### 4.2 Provision nomad-box (if not already done)
|
|
|
|
```bash
|
|
# On disinto-nomad-box
|
|
disinto init --backend=nomad --import-env .env --with edge
|
|
```
|
|
|
|
### 4.3 Import backup
|
|
|
|
```bash
|
|
# On disinto-nomad-box
|
|
./bin/disinto backup import /tmp/disinto-backup-*.tar.gz
|
|
```
|
|
|
|
### 4.4 Configure Codeberg pull mirror
|
|
|
|
Manual, one-time step in the new Forgejo UI:
|
|
|
|
1. Create a mirror repository pointing at the Codeberg upstream
|
|
2. Confirm initial sync completes
|
|
|
|
### 4.5 Claude login
|
|
|
|
```bash
|
|
# On disinto-nomad-box
|
|
claude login
|
|
```
|
|
|
|
Set up Anthropic OAuth so agents can authenticate.
|
|
|
|
### 4.6 Autossh tunnel swap
|
|
|
|
> **Operator step** — cross-host, no dev-agent involvement. Do NOT automate.
|
|
|
|
1. Stop the tunnel on dev-box:
|
|
```bash
|
|
# On disinto-dev-box
|
|
systemctl stop reverse-tunnel
|
|
```
|
|
|
|
2. Copy or regenerate the tunnel unit on nomad-box:
|
|
```bash
|
|
# Copy from dev-box, or let init regenerate it
|
|
scp dev-box:/etc/systemd/system/reverse-tunnel.service \
|
|
nomad-box:/etc/systemd/system/
|
|
```
|
|
|
|
3. Register nomad-box's public key on DO edge:
|
|
```bash
|
|
# On DO edge box — same restricted-command as the dev-box key
|
|
echo "<nomad-box-pubkey>" >> /home/johba/.ssh/authorized_keys
|
|
```
|
|
|
|
4. Start the tunnel on nomad-box:
|
|
```bash
|
|
# On disinto-nomad-box
|
|
systemctl enable --now reverse-tunnel
|
|
```
|
|
|
|
5. Verify end-to-end:
|
|
```bash
|
|
curl https://self.disinto.ai/api/v1/version
|
|
# Should return the new box's Forgejo version
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Post-cutover smoke
|
|
|
|
- [ ] `curl https://self.disinto.ai` → Forgejo welcome page
|
|
- [ ] Create a test PR → Woodpecker pipeline runs → agents assign and work
|
|
- [ ] Claude chat login via Forgejo OAuth succeeds
|
|
|
|
---
|
|
|
|
## 6. Rollback (if any step 4 gate fails)
|
|
|
|
1. Stop the tunnel on nomad-box:
|
|
```bash
|
|
systemctl stop reverse-tunnel # on nomad-box
|
|
```
|
|
|
|
2. Restore the tunnel on dev-box:
|
|
```bash
|
|
systemctl start reverse-tunnel # on dev-box
|
|
```
|
|
|
|
3. Bring dev-box services back up:
|
|
```bash
|
|
docker-compose up -d # on dev-box
|
|
```
|
|
|
|
4. DO Caddy config is unchanged — traffic restores in <5 min.
|
|
|
|
5. File a post-mortem issue. Keep nomad-box state intact for debugging.
|
|
|
|
---
|
|
|
|
## 7. Post-stable cleanup (T+1 week)
|
|
|
|
- `docker-compose down -v` on dev-box
|
|
- Archive `/var/lib/docker/volumes/disinto_*` to cold storage
|
|
- Delete disinto-dev-box LXC or keep as permanent rollback reserve (operator
|
|
decision)
|