Credential governance runbook

This runbook is the operator procedure for Enterprise credential governance. That page explains what each part does. This page says what to run, in what order, and what to check. The commands assume a Kubernetes install from the elitea chart in namespace <ns>, and an administrator’s personal API token in $TOKEN. Every admin route below is mounted only while the flag is on. With the flag off, each one answers 404.

Prerequisites

• A release that contains migration 0193. The migration Job applies it on upgrade.
• An administrator who holds models.agent_identity.bindings.manage, and a second administrator who holds models.agent_identity.bindings.approve. Activation is a two-person step. Both people are read from the caller’s token, never from the request body.
• A role that holds the models.credentials.governed.* permissions in each project that will run governed agents. No role holds them by default (see Credential use grants).
• A policy revision stored with elitea-main credential-policy put and named in main.env.ELITEA_CREDENTIAL_GOVERNANCE_POLICY_REVISION.
• To keep static secrets in Vault/OpenBao: the ELITEA_SECRETSTORE_VAULT_* variables on elitea-main, a KV v2 mount with cas_required on, and two roles, <prefix>-credential-broker (read) and <prefix>-credential-migration (create and read). Both roles need the paths elitea/projects/*/credentials/*.
• The nats CLI, in the natsio/nats-box image that the chart’s bootstrap Job uses.

Check the governed durables

Governed commands travel on elitea.rt.v1.<route>.g.*. Only three durables read them, one per stream:

Stream	Durable
ELITEA_RT_V1_VALIDATE	elitea-configuration-worker-governed-v1
ELITEA_RT_V1_AGENT	elitea-agent-worker-governed-v1
ELITEA_RT_V1_INDEX	elitea-index-worker-governed-v1

NATS accepts mTLS clients only, and its NetworkPolicy admits port 4222 only from the platform’s own pods. So run the CLI in a pod labelled like the bootstrap Job, as the runtime account’s bootstrap identity:

kubectl -n <ns> run nats-check --rm -it --restart=Never \
  --image=natsio/nats-box:0.19.7 \
  --labels=app.kubernetes.io/name=nats-bootstrap \
  --overrides='{"spec":{"volumes":[{"name":"tls","secret":{"secretName":"elitea-nats-bootstrap-runtime-nats-client-tls"}}],"containers":[{"name":"nats-check","image":"natsio/nats-box:0.19.7","stdin":true,"tty":true,"command":["sh"],"volumeMounts":[{"name":"tls","mountPath":"/tls","readOnly":true}]}]}}'

Inside the pod:

NATS="nats --server tls://elitea-nats:4222 --tlsca /tls/ca.crt --tlscert /tls/tls.crt --tlskey /tls/tls.key --inbox-prefix _INBOX_elitea-nats-bootstrap-runtime"
$NATS consumer info ELITEA_RT_V1_VALIDATE elitea-configuration-worker-governed-v1
$NATS consumer info ELITEA_RT_V1_AGENT elitea-agent-worker-governed-v1
$NATS consumer info ELITEA_RT_V1_INDEX elitea-index-worker-governed-v1

Each durable must exist and filter on its .g.* subject. “Waiting Pulls” above zero means a governed worker is bound to it, and “Unprocessed Messages” is the governed backlog. A missing durable means the bootstrap Job has not run on this release. Sync the release again, or run helm upgrade again, and wait for elitea-nats-bootstrap to succeed.

Enable

Turn the flag on only after governed workers run. Otherwise governed commands wait in their streams and fail at their deadline.

1. Run governed workers first

   Set worker.governedContext: "true" and roll the worker out, with the flag still off. The worker then binds the governed durables next to the legacy ones. Check that “Waiting Pulls” is above zero on all three durables.

2. Stage the policy revision

   kubectl -n <ns> exec deploy/elitea-main -- /elitea-main credential-policy show --id <revision> must print the revision you will boot with.

   Check asserted_user_identity_modes in it. With ["elitea_native"] (as in the shipped deploy/governed/credential-policy.json), an approved delegated_background mandate can stand in for a user’s delegation. Without the field, every user-attended governed run needs a verified delegation (an Entra sign-in) and is otherwise denied with agent_session.asserted_user_not_accepted. See Asserted users.

3. Turn the flag on

   Set main.env.ELITEA_ENTERPRISE_CREDENTIAL_GOVERNANCE_ENABLED: "true" and main.env.ELITEA_CREDENTIAL_GOVERNANCE_POLICY_REVISION: "<revision>". Leave worker.governedContext set: when it is empty, the worker follows the flag. elitea-main refuses to boot when a command stream lacks its governed durable, and when the revision is missing or does not match its digest.

4. Verify

   GET /api/v2/admin/credential_references?project_id=<id> answers 200 rather than 404. Every new run is pinned governed:

   SELECT profile, count(*) FROM elitea_runtime.execution_admission_profiles
   WHERE pinned_at > now() - interval '15 minutes' GROUP BY profile;

Stage credential references

A governed run resolves a toolkit credential only by its exact credential_reference_id, which only the governed admin API writes. Existing toolkits hold configuration references by title. migrate-references proposes and stages one credential reference, with its target binding, for each configuration that a governed agent needs.

kubectl -n <ns> exec deploy/elitea-main -- /elitea-main credential-governance migrate-references --project 42
kubectl -n <ns> exec deploy/elitea-main -- /elitea-main credential-governance migrate-references --project 42 --apply --operator 17

Flag	Meaning
--dry-run	The default. It lists the proposals and writes nothing: it reads the toolkits and existing references and stops before every write. It works through PgBouncer in transaction pooling mode.
--apply	Stage the references. --apply writes the state staged only, never active.
--operator <user id>	Required with --apply. The numeric id of an existing user, recorded as staged_by. That user can never activate the reference. The id is compared as a number, so 007 is user 7. An id that is not a number, or names no user, is a usage error (exit 2) and nothing is written.
--project <id>	One tenant project. If you leave it out, every project is read.
--copy-to vault	With --apply, copy each static secret from the built-in vault to Vault/OpenBao (see below).
--policy-revision <id>	With --copy-to vault, or an --apply that writes a basic-auth key map, the revision that the secret store calls name. It defaults to ELITEA_CREDENTIAL_GOVERNANCE_POLICY_REVISION.
--json	One JSON object per toolkit per line, instead of the table

The exit code is 0 when the run finished, 1 when it stopped on an error, and 2 for a usage error. The command is idempotent. Each reference is keyed by its source (configuration:<uuid>:<backend>, or mcp:<toolkit id>), so a second --apply stages nothing new and reports exists. Toolkits that share a configuration share one reference. A retired reference no longer holds its source: the next --apply stages a new reference for it and reports staged.

What it covers. Each line has an action: propose (in a dry run), staged, exists or skipped with a reason.

Toolkit type	Target	Class
jira, confluence	base_url	static_secret when token or api_key holds a {{secret.NAME}} reference (token wins, as in a legacy run). An api_key with a username is basic auth: see below. Otherwise oauth_delegated.
qtest	base_url	static_secret from qtest_api_token, or oauth_delegated
aha	base_url	static_secret from api_key, or oauth_delegated
sharepoint	site_url	oauth_delegated
mcp	the toolkit’s url	oauth_delegated

A static secret stays where it is, in the built-in vault (backend pgvault, generation 1), unless you pass --copy-to vault or it is basic auth.

Basic auth (Jira Cloud, Confluence Cloud). A configuration with no token, a sealed api_key and a non-empty username (plain or sealed) is an account email and API token. Its reference points at one secret that holds both, as the key map {"token": "<api key>", "username": "<email>"}, and a redemption releases both keys. --apply writes that secret through the secret write ledger: in the built-in vault under a new name, governed_<configuration uuid without dashes>, or with --copy-to vault at the usual Vault path. The reference is staged at generation 0 and takes the written version, as for a copy, and each write appends a credential_reference.copy_to_pgvault (or copy_to_vault) audit decision. The configuration’s own secrets are not changed. The write reads the built-in vault, so it needs SECRETS_MASTER_KEY and a policy revision (--policy-revision or ELITEA_CREDENTIAL_GOVERNANCE_POLICY_REVISION). The dry run shows the secret as pgvault:governed_…@<new>, and the material field of each line reads token+username or token. A token-only reference (a PAT, or one staged before this release) keeps working.

Reason	Meaning and what to do
provider_not_governed	No governed adapter exists for the type. Its toolkits stay legacy.
private_credential_not_governed	The toolkit names a private (personal) configuration. Move the credential into the project.
configuration_not_in_project	The title does not resolve in the toolkit’s own project. A governed run never reads the personal or public project.
secret_not_sealed	The secret field holds plain text. Save the configuration again so the secret is sealed, then rerun.
credential_reference_ambiguous	The toolkit names more than one configuration
configuration_type_mismatch, configuration_unreadable	The configuration is not the toolkit’s type, or its data does not parse
target_not_canonical	The target URL is not a plain https origin and path
class_not_supported	The adapter does not accept the class
toolkit_not_governable	An MCP toolkit with static headers, or without a usable URL
settings_unreadable	The toolkit’s stored settings are not a JSON object, so the rule cannot read them. Save the toolkit again, then rerun.

Copy to Vault/OpenBao. --copy-to vault reads each static secret from the built-in vault (it needs SECRETS_MASTER_KEY). It stages the reference first, at generation 0, and then writes the secret to <kv mount>/elitea/projects/<project>/credentials/<configuration uuid> with check-and-set 0, through the secret write ledger. The reference then takes the version that was written as its generation. A reference at generation 0 cannot be activated. Each copy appends a credential_reference.copy_to_vault audit decision, whose id authorises the read, and the decision and the ledger row both name the staged reference. If a run stops between the staging and the write, the next run finishes the copy for the same reference. If the path already holds a secret, the run reads the path’s current version. It uses that version when the value is the same, and stops when it differs. Nothing is ever overwritten.

Warning

A basic-auth configuration staged before this release has a token-only reference, so its governed calls get no username and fail. Retire that reference, then run migrate-references --apply again: it stages a replacement that points at the key map. Activate, attach and grant the replacement as usual.

Stage a reference from scratch

When no toolkit configuration holds the credential yet, stage the reference through the admin API instead. POST /api/v2/admin/credential_references (bindings.manage) uses the same staging step as migrate-references --apply: it records the target binding, writes the reference as staged with the caller as staged_by, and appends a security audit decision in the same transaction. Activation stays a second person’s step.

curl -fsS -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"project_id": 42, "provider_type": "jira", "target_identity": "https://jira.example.com",
       "class": "static_secret", "backend": "vault",
       "locator": "elitea/projects/42/credentials/jira-prod"}' \
  "$DEPLOYMENT_URL/api/v2/admin/credential_references"

Field	Rule
provider_type, target_identity	A governed provider, and a target in the adapter’s canonical form
class	static_secret or oauth_delegated, and accepted by the adapter
backend, locator	static_secret: pgvault (the secret’s name) or vault (a path under elitea/projects/<project_id>/credentials/). oauth_delegated: pgvault and oauth_grant.
generation	Optional. For static_secret it defaults to the secret’s current version; a version that does not exist is refused (409). oauth_delegated takes 0.

For a static secret, Main checks that the locator exists and is readable through the secret store’s metadata call, under the credential-migration workload scope (on Vault/OpenBao, the <role prefix>-credential-migration role needs metadata read on the path). It never reads, returns or logs the value. A locator that does not exist answers 409, and a secret store that cannot be reached answers 503.

Activate, attach and grant

Do these through the admin API. Each write appends a security audit decision in the same transaction.

Route	Permission	Body
GET /api/v2/admin/credential_references?project_id=<id>	bindings.manage or .approve	Query: state, limit, offset
GET /api/v2/admin/credential_references/{id}	bindings.manage or .approve
POST /api/v2/admin/credential_references	bindings.manage	See Stage a reference from scratch
POST /api/v2/admin/credential_references/{id}:activate	bindings.approve	{}
POST /api/v2/admin/credential_references/{id}:retire	bindings.manage	{}
POST /api/v2/admin/credential_references/{id}:attach	bindings.manage	{"project_id": 42, "toolkit_id": 7}
POST /api/v2/admin/credential_references/{id}:detach	bindings.manage	{"project_id": 42, "toolkit_id": 7}
PUT /api/v2/admin/agent_identities/{id}/toolkits/{toolkit_id}	bindings.manage	{"project_id": 42}
DELETE /api/v2/admin/agent_identities/{id}/toolkits/{toolkit_id}	bindings.manage	{"project_id": 42}, or ?project_id=42
POST /api/v2/admin/agent_identities/{id}:repin	bindings.manage	none
POST /api/v2/admin/agent_identities/{id}:approve	bindings.approve	none

The permissions are the models.agent_identity.* permissions.

1. Activate

   A second administrator activates the reference. Only a staged reference with a recorded stager can be activated, and the stager’s own activation is refused with 403. A reference created before migration 0193 has no stager, so it can never be activated: stage a new one.

   curl -fsS -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{}' \
     "$DEPLOYMENT_URL/api/v2/admin/credential_references/<reference id>:activate"

2. Attach it to each toolkit

   :attach writes credential_reference_id into the toolkit’s stored configuration reference. The reference’s provider must be the toolkit’s type, and the toolkit must name exactly one project configuration. A toolkit that is attached to another reference must be detached first (409). An MCP toolkit has no configuration reference, so there is nothing to attach. It is governed by its digest and the one usable grant.

   Warning

   Attaching re-pins the toolkit. It edits the stored settings, so the toolkit’s digest changes (the answer carries the new toolkit_digest). Every agent binding that was pinned to the old digest now fails admission with toolkit_digest_mismatch, until you pin it again in the next step. :detach restores the old settings and the old digest.

3. Pin the toolkit to the agent binding

   PUT /api/v2/admin/agent_identities/{binding id}/toolkits/{toolkit id} records the toolkit’s current digest for the binding. Pin after every attach, detach or toolkit edit. The binding must be in the toolkit’s project, and not revoked.

4. Re-pin the agent binding

   The agent’s definition embeds its toolkits’ stored settings, so the attach also changed the agent’s definition digest. Until the binding pins the new digest, runs of the agent get no agent identity, and admission records definition_digest_mismatch. Compare definition_digest with computed_definition_digest in the binding’s view. If they differ, one administrator calls :repin, and a second one calls :approve. The re-pinner cannot approve its own re-pin. Re-pin after every attach, detach, toolkit edit or agent edit.

5. Propose and approve a grant

   Use the per-project grant routes in Credential use grants. A grant is two-person too.

Drain

Drain before you turn the flag off, and before an upgrade whose notes say that in-flight governed runs fail. Stop starting new governed work first: pause the governed schedules and triggers, or detach their mandates. Then wait until both queries return 0:

SELECT count(*) FROM elitea_runtime.execution_jobs j
JOIN elitea_runtime.execution_admission_profiles p USING (execution_id)
WHERE p.profile = 'elitea.credential-profile.governed.v1'
  AND j.state IN ('PENDING', 'DISPATCHED', 'CLAIMED', 'RUNNING', 'SETTLING');

SELECT count(*) FROM elitea_runtime.execution_interrupts i
JOIN elitea_runtime.execution_admission_profiles p USING (execution_id)
WHERE p.profile = 'elitea.credential-profile.governed.v1'
  AND i.state IN ('PENDING', 'DECIDED');

The second query counts governed runs that wait for a human answer. Their continuation stays governed. Answer or cancel them. “Unprocessed Messages” on the three governed durables must be 0 too.

Rollback

Turn the flag off. This is the normal rollback. Set main.env.ELITEA_ENTERPRISE_CREDENTIAL_GOVERNANCE_ENABLED: "false", and keep worker.governedContext: "true" until the drain queries return 0. Then remove worker.governedContext. Check afterwards:

• GET /api/v2/admin/credential_references?project_id=<id> answers 404, and so do the other governed admin routes.
• New pins are legacy: the query in Enable shows only elitea.credential-profile.legacy.v1 rows for new runs.
• Legacy runs resolve toolkit credentials by title again. Every reader except the governed resolver ignores credential_reference_id, so attached toolkits need no change.
• The data stays: references, grants, pins and the audit chain are kept for the next enable.

Roll the release back. The migration runner refuses a database that has a migration its binary does not know (“database version 0193 is not present in the binary manifest”). So you cannot roll back below the release that added 0193 without a database restore. Turn the flag off instead.

Incident response

Revoke at the narrowest record that covers the incident. Each transition bumps the record’s epoch, or for a credential reference, moves it out of active. The next RedeemCredential re-reads the epochs and is denied authority_revoked. A running run is refused at its next claim, and cancelled at its next lease renewal. Every replica also drops its cached material through elitea.rt.v1.security.invalidation. No body is needed: each call is POST with an empty body or {}.

Compromised	Call	Effect
One credential (leaked secret, wrong target)	POST /api/v2/admin/credential_references/{id}:retire	The reference leaves active, so every grant and run that uses it is denied. Retired is final.
One agent’s use of one credential	POST /api/v2/projects/{project_id}/credential_grants/{id}:revoke (or :suspend)	The grant’s epoch increases by 1
One agent version	POST /api/v2/admin/agent_identities/{id}:revoke (or :suspend)	The binding’s revocation_epoch increases by 1. Its mandates stop too.
One unattended authority	POST /api/v2/admin/mandates/{id}:revoke (or :suspend)	The mandate’s epoch increases by 1. Its schedules and triggers are refused as not usable at their next fire.
The agent’s external identity	POST /api/v2/admin/external_agent_identities/{id}:suspend	The binding counts as suspended

curl -fsS -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{}' \
  "$DEPLOYMENT_URL/api/v2/admin/credential_references/<reference id>:retire"

Then confirm:

• The revocation took effect: elitea_credential_revocation_denials_total and elitea_runtime_governed_authority_revoked_total{point} increase. elitea_credential_revocation_deny_latency_seconds shows how long the first denial took.
• The audit chain is intact and holds the transition: kubectl -n <ns> exec deploy/elitea-main -- /elitea-main security-audit verify --tenant <project id>.
• For a leaked static secret, rotate it at the provider as well. Retiring stops Elitea from releasing it, but does not change it. Then stage, activate, attach and pin a new reference. Save the new secret in the configuration, then run migrate-references --apply again, which stages the replacement. With --copy-to vault the Vault path is the same for each configuration, and its current version still holds the old value, so the run stops. Write the new value to that path in Vault/OpenBao first. The rerun then pins that version.