disinto/docs/agents-llama.md

Local-Model Agents

Local-model agents run the same agent code as the Claude-backed agents, but connect to a local llama-server (or compatible OpenAI-API endpoint) instead of the Anthropic API. This document describes the current activation flow using disinto hire-an-agent and [agents.X] TOML configuration.

Overview

Local-model agents are configured via [agents.<name>] sections in projects/<project>.toml. Each agent gets:

• Its own Forgejo bot user with dedicated API token and password
• A dedicated compose service agents-<name>
• Isolated credentials stored as FORGE_TOKEN_<USER_UPPER> and FORGE_PASS_<USER_UPPER> in .env

Prerequisites

• llama-server (or compatible OpenAI-API endpoint) running on the host, reachable from inside Docker at the URL you will configure.
• A disinto factory already initialized (disinto init completed).

Hiring a local-model agent

Use disinto hire-an-agent with --local-model to create a bot user and configure the agent:

# Hire a local-model agent for the dev role
disinto hire-an-agent dev-qwen dev \
  --local-model http://10.10.10.1:8081 \
  --model unsloth/Qwen3.5-35B-A3B

The command performs these steps:

1. Creates a Forgejo user dev-qwen with a random password
2. Generates an API token for the user
3. Writes credentials to .env:
   • FORGE_TOKEN_DEV_QWEN — the API token
   • FORGE_PASS_DEV_QWEN — the password
   • ANTHROPIC_BASE_URL — the llama endpoint (required by the agent)
4. Writes [agents.dev-qwen] to projects/<project>.toml with:
   • base_url, model, api_key
   • roles = ["dev"]
   • forge_user = "dev-qwen"
   • compact_pct = 60
   • poll_interval = 60
5. Regenerates docker-compose.yml to include the agents-dev-qwen service

Anthropic backend agents

For agents that use Anthropic API instead of a local model, omit --local-model:

# Anthropic backend agent (requires ANTHROPIC_API_KEY in environment)
export ANTHROPIC_API_KEY="sk-..."
disinto hire-an-agent dev-claude dev

This writes ANTHROPIC_API_KEY to .env instead of ANTHROPIC_BASE_URL.

Activation and running

Default activation (single agent)

Once hired, the agent service is added to docker-compose.yml but not started by default. To start a single agent:

# Start just the dev-qwen agent
COMPOSE_PROFILES=agents-dev-qwen docker compose up -d

Important: Local-model agent services are profile-gated. Running docker compose up -d without COMPOSE_PROFILES will not start them, and --remove-orphans may remove them as unmanaged containers.

Starting multiple agents

# Start multiple agents
COMPOSE_PROFILES=agents-dev-qwen COMPOSE_PROFILES=agents-planner docker compose up -d

Stopping agents

# Stop specific agents
COMPOSE_PROFILES=agents-dev-qwen docker compose down

# Stop all agents
docker compose down

Credential rotation

Re-running disinto hire-an-agent <same-name> with the same parameters rotates credentials idempotently:

# Re-hire the same agent to rotate token and password
disinto hire-an-agent dev-qwen dev \
  --local-model http://10.10.10.1:8081 \
  --model unsloth/Qwen3.5-35B-A3B

# The command will:
# 1. Detect the user already exists
# 2. Reset the password to a new random value
# 3. Create a new API token
# 4. Update .env with the new credentials

This is the recommended way to rotate agent credentials. The .env file is updated in place, so no manual editing is required.

If you need to manually rotate credentials:

1. Generate a new token in Forgejo admin UI
2. Edit .env and replace FORGE_TOKEN_<USER_UPPER> and FORGE_PASS_<USER_UPPER>
3. Restart the agent service: docker compose restart agents-<name>

Configuration reference

Environment variables (.env)

Variable	Description	Example
FORGE_TOKEN_<USER_UPPER>	Forgejo API token for the bot user	FORGE_TOKEN_DEV_QWEN
FORGE_PASS_<USER_UPPER>	Forgejo password for the bot user	FORGE_PASS_DEV_QWEN
ANTHROPIC_BASE_URL	Local llama endpoint (local model agents)	http://host.docker.internal:8081
ANTHROPIC_API_KEY	Anthropic API key (Anthropic backend agents)	sk-...

Project TOML ([agents.<name>] section)

[agents.dev-qwen]
base_url = "http://10.10.10.1:8081"
model = "unsloth/Qwen3.5-35B-A3B"
api_key = "sk-no-key-required"
roles = ["dev"]
forge_user = "dev-qwen"
compact_pct = 60
poll_interval = 60

Field	Description
base_url	llama-server endpoint
model	Model name (for logging/identification)
api_key	Required by API; set to placeholder for llama
roles	Agent roles this instance handles
forge_user	Forgejo bot username
compact_pct	Context compaction threshold (lower = more aggressive)
poll_interval	Seconds between polling cycles

Behaviour

• Each agent runs with AGENT_ROLES set to its configured roles
• CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=60 — more aggressive compaction for smaller context windows
• Agents serialize on the llama-server's single KV cache (AD-002)

Troubleshooting

Agent service not starting

Check that you're using COMPOSE_PROFILES:

# Wrong: this won't start profile-gated agent services
docker compose up -d

# Correct: explicitly specify the profile
COMPOSE_PROFILES=agents-dev-qwen docker compose up -d

Model endpoint unreachable

Verify llama-server is accessible from inside Docker:

docker compose -f docker-compose.yml exec agents curl -sf http://host.docker.internal:8081/health

If using a custom host IP, update ANTHROPIC_BASE_URL in .env:

# Update the base URL
sed -i 's|^ANTHROPIC_BASE_URL=.*|ANTHROPIC_BASE_URL=http://192.168.1.100:8081|' .env

# Restart the agent
COMPOSE_PROFILES=agents-dev-qwen docker compose restart agents-dev-qwen

Invalid agent name

Agent names must match ^[a-z]([a-z0-9]|-[a-z0-9])*$ (lowercase letters, digits, hyphens; starts with letter, ends with alphanumeric). Invalid names like dev-qwen2 (trailing digit is OK) or dev--qwen (consecutive hyphens) will be rejected.