Commander link¶

The Commander link lets someone using the dashboard assistant put a question to the Commander running on a workstation, and get the answer back in the same conversation. It arrived in 0.1.94 and needs Spawn 0.1.6 or later on the workstation.

This page covers setting it up, putting it behind SSO forward auth, and what to check when asks never arrive. The wire contract is in the design doc, docs/design/commander-link.md, in the Hivemind repository.

How it works¶

An administrator asks Facet, the dashboard assistant, in plain words, to ask the Commander something. Facet proposes it as a card, and the administrator confirms the card once, in a signed-in browser. The ask is now queued on your instance. Only an administrator can ask the Commander; Facet does not raise the card for anyone else. Facet itself must be on first: see Turn on Facet.
On the workstation, spawn hive-ask tick runs once a minute. It dials out to your instance, lists confirmed asks, and delivers one into the Commander session.
When the Commander answers, a later tick posts the answer back. Hivemind writes it into the conversation the ask came from.

The instance never connects to the workstation, holds no workstation secret, and never types into a terminal. An ask carries the administrator's words, never an approval. Confirming the card executes nothing, and it never counts as an operator OK for anything the Commander would otherwise need a human to approve.

The workstation reaches these endpoints, all under one prefix:

Request	Purpose
`GET /api/v1/commander-link/asks`	List confirmed asks that are unanswered and not yet delivered
`GET /api/v1/commander-link/v2/asks`	The same list, with the asks your agents raised as well as the chat's, each labeled with where it came from. Spawn uses this form when it labels asks by origin
`POST /api/v1/commander-link/asks/{id}/delivered`	Report that an ask reached the Commander
`POST /api/v1/commander-link/asks/{id}/reply`	Post the Commander's answer (recorded once)
`POST /api/v1/commander-link/asks/{id}/question`	The Commander asks the person back, with choices; the question appears as a card in the conversation
`GET /api/v1/commander-link/topology-changes`	List the hierarchy change requests that have no result yet
`POST /api/v1/commander-link/topology-changes/{id}/result`	Report the result of one hierarchy change (recorded once)

Every request is signed by the workstation, and Hivemind verifies the signature itself. That is why this prefix must reach Hivemind without passing through SSO. See Behind SSO forward auth.

Setup¶

1. Workstation: generate a key¶

spawn hive-ask keygen

This writes an Ed25519 private key to a mode-600 file, then prints where it wrote it, the public key, and the key's fingerprint. It also prints the credential_path line to use in step 3. Use --out PATH to choose where the key goes, and --force to replace an existing key. The private key is never printed and never sent. Each request carries only a signature.

2. Instance: trust the key¶

As an admin, open Settings, then the Commander link: trusted workstation panel. Paste the public key and choose Trust. The panel shows the key's fingerprint. Compare it with the fingerprint keygen printed. They must match.

Trusting the key is an authority grant: whoever holds the matching private key can read queued asks and answer them. So the panel only works from an interactive admin session (a local login or SSO). An API key is refused, whatever its role. Every set, clear and read is written to the audit log with the key's fingerprint.

There is nothing to restart. The key is stored in the database and read on every request, so it needs no compose change and no mounted file.

Rotate: run spawn hive-ask keygen --force on the workstation, then paste the new public key in the same panel. The instance refuses the new key's signatures until you do.
Revoke: choose Revoke this key. The link answers 503 until a key is set again.

Only one workstation key is trusted at a time.

The HMAC key file

A shared HMAC key, mounted as a file on the instance (HIVEMIND_COMMANDER_LINK_KEY_FILE), is also accepted. It needs a compose override, which an upgrade never writes, so it is for operators who manage the compose file by hand. The trusted workstation key above is the normal path. Revoking the workstation key does not remove an HMAC key file. The panel shows whether one is configured.

3. Workstation: point Spawn at the instance¶

Create hive-ask.toml in Spawn's config directory. You can also point $SPAWN_HIVE_ASK at a file elsewhere.

url = "https://hivemind.example.com"
credential_path = "/path/printed/by/keygen"

url is your instance's base URL. It must be https. The only exception is http to a loopback host, for a development instance on the same machine.
credential_path is the key file keygen wrote. The tick refuses a key file that the group or others can read, and names the path and mode.
If your instance uses a certificate from a private CA, add ca_bundle = "/path/to/ca.pem". It adds to the public roots and never turns verification off.

If the file is absent, a tick prints "not configured" and exits 0.

Try one tick by hand before you enable the timer:

spawn hive-ask tick

4. Workstation: run it every minute¶

Spawn's source tree ships two systemd user units, spawn-hive-ask.service and spawn-hive-ask.timer, in deploy/systemd/. The service runs %h/.local/bin/spawn hive-ask tick. The timer fires one minute after boot and then once a minute. Ticks never overlap: the service is a oneshot, and each tick also holds a lock.

From a Spawn source checkout:

mkdir -p ~/.config/systemd/user
cp deploy/systemd/spawn-hive-ask.service deploy/systemd/spawn-hive-ask.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now spawn-hive-ask.timer

Tick output goes to the journal:

journalctl --user -u spawn-hive-ask.service -n 50

User timers stop when the account's last session ends. To keep the link running while nobody is logged in, enable lingering for that account (loginctl enable-linger <account>).

5. Check it is alive¶

While an ask is waiting, its card in the dashboard shows the link's state, such as Commander link: online — last contact 12s ago or Commander link: OFFLINE — last contact 10 min ago. An admin can read the same state from GET /api/v1/assistant/commander-link. Only a request whose signature Hivemind accepted counts as contact. A refused request records nothing, so traffic from anyone else can never make a dead link look alive.

Behind SSO forward auth¶

If your instance sits behind a reverse proxy doing forward auth (Authentik, Authelia, oauth2-proxy, and similar), the proxy intercepts every request that arrives without an SSO session. That includes the workstation's requests. The proxy answers them with a 302 to your login page. The tick refuses redirects by design, because following one would send a signed request somewhere other than your instance. So no ask ever reaches the Commander.

The fix: route /api/v1/commander-link/* straight to Hivemind, outside forward auth, and strip the forward-auth identity headers on that route. This is the same kind of bypass the SSO setup guide describes for /api/v1/health and for API-key requests.

Why this is safe¶

Hivemind authenticates every one of these requests itself. Each request carries a timestamp and an Ed25519 signature over the method, the path, the timestamp and a SHA-256 of the body. Hivemind checks it against the trusted workstation key before doing anything.
Unsigned requests are refused with 401, and so are requests with a malformed timestamp or a wrong signature. A request signed for a different path or body fails the signature check.
Stale requests are refused. A timestamp more than 300 seconds from the server's clock gets 401, so a captured request stops working within five minutes.
A replay inside that window changes nothing. An answer is recorded once, and delivery is recorded once; a second copy gets 409. A replayed list request returns only what the key holder could already read.
A delivered ask is not offered again. Once delivery is recorded, the ask stops appearing in the listing, so a workstation that holds no local record of it — a replacement, or one that lost its state — does not deliver it to the Commander a second time. It remains answerable: the workstation that delivered it can still post the reply, because delivery is not the answer.
The key opens nothing else. It authenticates only these endpoints. It is not an API key, maps to no user, and cannot propose, confirm or decide anything.
Forward-auth identity plays no part here. These routes do not read forward-auth identity headers, and HIVEMIND_FORWARD_AUTH_TRUSTED_NETS does not apply to them. Strip the identity headers on this route anyway. Then no identity header a client supplies ever reaches Hivemind through an unauthenticated route.

Two things the proxy must get right:

Pass the path through unchanged. The signature covers the exact path /api/v1/commander-link/.... Rewriting or stripping it, for example with Caddy's handle_path, makes every signature fail. Serve the instance at the root of its host.
Rate-limit in front, if the site is public. As with the API-key bypass, Hivemind does not rate-limit failed signatures. A rate limiter in front of the proxy keeps a flood of bogus requests from costing you anything.

Caddy (`forward_auth`)¶

Put the link route in its own handle block, and put forward_auth inside a catch-all handle block, not at the top level of the site block. Caddy's directive order runs a top-level forward_auth before any handle block, so a bypass written next to a top-level forward_auth never actually bypasses it.

hivemind.example.com {
    # Strip client-supplied identity headers on every request. Caddy runs a
    # site-level request_header before any handle block.
    request_header -X-Authentik-Username
    request_header -X-Authentik-Email
    request_header -X-Authentik-Groups
    request_header -X-Authentik-Uid
    request_header -X-Authentik-Name

    # The Commander link: signed by the workstation, verified by Hivemind.
    # Outside forward_auth, with the identity headers removed.
    handle /api/v1/commander-link/* {
        reverse_proxy 127.0.0.1:8585 {
            header_up -X-Authentik-Username
            header_up -X-Authentik-Email
            header_up -X-Authentik-Groups
            header_up -X-Authentik-Uid
            header_up -X-Authentik-Name
        }
    }

    # Everything else goes through SSO.
    handle {
        forward_auth http://127.0.0.1:9000 {
            uri /outpost.goauthentik.io/auth/caddy
            copy_headers X-Authentik-Username X-Authentik-Email X-Authentik-Groups X-Authentik-Uid X-Authentik-Name
        }
        reverse_proxy 127.0.0.1:8585
    }
}

The site-level request_header lines and the header_up lines on the link route both remove the identity headers. The second is a local guarantee that survives someone later deleting the first. SSO setup explains why the strip lines must not go inside forward_auth.

Replace 127.0.0.1:8585 with the address your other reverse_proxy lines use for Hivemind, and 127.0.0.1:9000 with your Authentik outpost. Keep any other bypass routes you already have, such as health and API-key requests, as their own handle blocks beside the link route. If your IdP uses different header names, remove the ones your HIVEMIND_FORWARD_AUTH_USER_HEADER, HIVEMIND_FORWARD_AUTH_EMAIL_HEADER and HIVEMIND_FORWARD_AUTH_GROUPS_HEADER name.

nginx (`auth_request`)¶

# The Commander link: signed by the workstation, verified by Hivemind.
location /api/v1/commander-link/ {
    auth_request off;

    # Clear the forward-auth identity headers on this route.
    proxy_set_header X-Authentik-Username "";
    proxy_set_header X-Authentik-Email    "";
    proxy_set_header X-Authentik-Groups   "";
    proxy_set_header X-Authentik-Uid      "";
    proxy_set_header X-Authentik-Name     "";

    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:8585;
}

location / {
    auth_request /outpost.goauthentik.io/auth/nginx;
    # ... your existing auth_request configuration ...
    proxy_pass http://127.0.0.1:8585;
}

Notes:

Write proxy_pass without a trailing path (as shown), so nginx forwards the original path unchanged.
nginx drops a request header whose value is set to the empty string, which is how the identity headers are removed.
Any proxy_set_header in a location replaces all of the ones it would otherwise inherit from the server block. So repeat Host and any other headers you normally forward.

Authelia, oauth2-proxy and others¶

The rule is the same for any forward-auth gateway. Requests under /api/v1/commander-link/ must reach Hivemind without an SSO check, with their path unchanged, and with no forward-auth identity headers attached.

Simplest: exclude the path in the reverse proxy itself, as in the Caddy and nginx examples. The gateway never sees these requests.
If your gateway decides access per path: add a rule that lets this path through without login. In Authelia that is an access-control rule with policy: bypass for resources matching ^/api/v1/commander-link/. In oauth2-proxy it is --skip-auth-route. Check your gateway's own documentation for the exact syntax. Then make sure the proxy still removes that gateway's identity headers on the route.

Never exclude more than this prefix. A broader rule, such as all of /api/v1/*, would open routes that rely on the proxy for authentication.

Troubleshooting¶

The workstation log says "answered the ask list with 302 — refused, not followed"¶

Your SSO proxy is intercepting the link. The 302 is a redirect to your login page, not an answer from Hivemind. Add the bypass route from Behind SSO forward auth.

In the dashboard, a waiting ask's card shows the link as OFFLINE, or says the workstation has never checked in. Nothing counts as contact until a signed request reaches Hivemind and is accepted.

Check the route from outside¶

Send an unsigned request to the link path, from any machine:

curl -s -i https://hivemind.example.com/api/v1/commander-link/asks | head -n 1

You get	Meaning
`401` with `{"error":"the request carries no link signature"}`	Correct. Hivemind answered, and refused the unsigned request.
`503` with `{"error":"the commander link is not configured on this instance"}`	The route is correct, but no workstation key is trusted yet. Do step 2.
`302` to your login page	The proxy is still intercepting. The bypass is missing, or it comes after forward auth.

Then check that SSO still protects everything else:

curl -s -i https://hivemind.example.com/ | head -n 1

This must still be a 302 to your login page. If it is not, your change opened more than the link route.

The tick gets `401` from Hivemind¶

Hivemind reached the request and refused its signature. The response body says why:

Body	Cause
`the link signature does not match`	The trusted key is not the one in `credential_path` (compare fingerprints), or the proxy changed the path.
`the link timestamp is outside the allowed window`	The workstation's clock and the server's clock differ by more than 300 seconds. Fix time sync on both.
`the request carries no link signature`	Something between the workstation and Hivemind removed the `X-Hivemind-Link-*` headers.

The tick gets `503`¶

No workstation key is trusted on the instance (and no HMAC key file is configured), or the instance could not read its keys. Trust the key in Settings; see step 2.