Upgrading Hivemind

Hivemind ships signed releases via the managed-update path (the hivemind-updater sidecar). The normal upgrade is a one-click apply from the admin Updates page; a manual install.sh re-run is the fallback. For v1, rollback is manual — back up before you click Apply.

The managed-update path (normal)

The hivemind-updater sidecar polls updates.seglamater.app on a schedule for new signed releases on your channel (your bundle pins the channel; the default is canary for new installs). When a new release is available, the admin Updates page in the web UI surfaces it with the version, the cosign signature status, and the release notes. Hit Apply to update.

The sidecar then:

1. Pulls the new hivemind-server image at its pinned digest from registry.seglamater.app.
2. Verifies the image signature against the pinned cosign public key (/srv/hivemind/secrets/cosign_pub — pinned at install time).
3. Re-renders the channel pointer in your bundle (the pinned image digest advances; channel stays the same).
4. Sends a signed POST /v1/apply to the in-process orchestrator over the internal updater_internal Docker network (HMAC-signed with updater_token). The orchestrator:
5. Drains the current hivemind-server (graceful shutdown).
6. Recreates the container at the new image digest.
7. Waits for the healthcheck (/api/v1/health) to return ok.
8. Reports success / failure back to the Updates page.

If the healthcheck doesn't pass within the grace window, the orchestrator rolls back automatically to the prior image digest. You'll see a Rollback status on the Updates page and a WARN in docker compose logs hivemind-updater.

Expected runtime: 30 s - 2 min. There is a brief window (a few seconds) where the API is unreachable as the container is replaced. The Postgres volume is untouched — schema migrations run on first start of the new version.

Manual upgrade (fallback)

If you want to skip the in-app path — e.g. you've held back the channel on the sidecar and you have a newer bundle in hand — you can re-run the install script with the new bundle:

cd /srv/hivemind                                 # or your install dir
./scripts/install.sh --bundle /path/to/new-bundle.json

install.sh is idempotent: it reuses your existing credential files, re-renders the environment file only for keys it doesn't already see, pulls the new image digest from the bundle, and runs docker compose up -d to recreate the affected containers. The same migrations + healthcheck gates apply. Time is similar (30 s - 2 min on a warm cache).

If you do not have a new bundle file, the simplest path is to ask your Seglamater contact to mint a fresh invite — that bootstrap fetches the latest bundle for your customer + re-applies cleanly.

Pre-upgrade checklist (recommended)

For any non-canary instance:

cd /srv/hivemind

# 1. Note the running version (so you can compare after).
curl -fsS http://localhost:8585/api/v1/health
# {"status":"ok","version":"0.1.9"}

# 2. Back up the postgres volume. The simplest snapshot is pg_dump:
docker compose exec -T postgres pg_dump -U hivemind hivemind \
  > "/var/backups/hivemind-pre-upgrade-$(date +%Y%m%d-%H%M%S).sql"

# 3. If you're running Borg / Restic against /var/lib/docker/volumes/,
#    run a manual snapshot now so you have a known-good restore point.

# 4. Skim the boot log so any pre-existing WARNs aren't blamed on the upgrade.
docker compose logs --tail=50 hivemind-server | grep -E "WARN|ERROR"

The pg_dump file is your point-in-time restore target if the upgrade introduces a schema problem. Hivemind migrations are forward-only — the v1 rollback strategy is restore-from-dump, not down-migration. See the "Rollback" section below.

Verify post-upgrade

# 1. Server is healthy at the new version.
curl -fsS http://localhost:8585/api/v1/health
# {"status":"ok","version":"<new-version>"}

# 2. Migrations applied without error.
docker compose logs --tail=30 hivemind-server | grep -E "migration|database"

# 3. Provider init still green (no new LLM warnings).
docker compose logs --tail=30 hivemind-server | grep -E "LLM|provider|integration credential"

# 4. Background workers came up.
docker compose logs --tail=30 hivemind-server | grep -E "watchdog|prekey-replenish|bot-ws supervisor"

# 5. Sanity-call a real endpoint with your admin API key.
curl -fsS -H "X-API-Key: $ADMIN_API_KEY" \
  http://localhost:8585/api/v1/agents/me | jq .

If any of those don't return the expected shape, see troubleshooting.md — most upgrade-time failures are the same boot-time failures as fresh installs (missing key, port conflict, DB not ready, etc.).

Rollback (v1: manual)

Hivemind v1 has no automated down-migration. Migrations are forward-only and Postgres state is the source of truth.

To roll back to the prior version:

cd /srv/hivemind

# 1. Stop the new stack.
docker compose down

# 2. Restore the postgres dump you took pre-upgrade.
#    (If you didn't take one, restore from your borg / restic snapshot of
#    /var/lib/docker/volumes/hivemind_pgdata.)
docker compose up -d postgres
sleep 5
cat /var/backups/hivemind-pre-upgrade-<timestamp>.sql | \
  docker compose exec -T postgres psql -U hivemind hivemind

# 3. Re-pin the OLD image digest. Easiest: ask Seglamater for the prior
#    bundle (they archive every cut), copy it into ./scripts/bundle.json,
#    then re-run install.sh:
./scripts/install.sh --bundle ./scripts/bundle.json

# 4. Bring the stack back up.
docker compose up -d

Confirm /api/v1/health returns the prior version + the integration credential key load line is present. Open the Updates page in the admin UI and pin the prior channel pointer (or temporarily switch the sidecar to a paused channel) so the next sidecar tick doesn't immediately re-apply the version you just rolled back from.

Where the updater logs live

docker compose logs --tail=80 hivemind-updater

A successful apply sequence reads roughly:

INFO  hivemind-updater: poll: new release 0.1.10 on channel canary (digest sha256:…)
INFO  hivemind-updater: cosign verify OK (pubkey sha256: efca…)
INFO  hivemind-updater: pulling hivemind-server@sha256:…
INFO  hivemind-updater: stop_old: graceful shutdown of hivemind-server
INFO  hivemind-updater: start_new: starting hivemind-server at new digest
INFO  hivemind-updater: healthcheck passed; apply succeeded

A failed apply (the rollback path) reads:

WARN  hivemind-updater: healthcheck did not pass within grace window
WARN  hivemind-updater: rolling back to prior digest sha256:…
INFO  hivemind-updater: rollback applied; hivemind-server back at prior version

If you see the WARN healthcheck did not pass, the new container booted but /api/v1/health didn't return ok in time. Pull the server logs for that container:

docker compose logs --tail=200 hivemind-server | tail -200

and check troubleshooting.md. The most common causes: missing HIVEMIND_INTEGRATION_ENCRYPTION_KEY_FILE mount (compose mis-edit), DB schema mismatch (a migration partially applied), port conflict on the host (some other process took 8585 between stop_old and start_new).

Pausing or changing your channel

Channels are pinned in ./scripts/bundle.json. To pause auto-applies, ask Seglamater to issue a bundle pinned to a closed channel. To switch between canary, beta, and stable, the same — a new bundle from your Seglamater contact.

Hivemind does not currently expose a one-click channel switcher in the admin UI; that is on the roadmap. Until then, the channel is part of your bundle and managed out-of-band.