Enterprise credential governance
Enterprise credential governance is an operator switch for deployments that must prove which agent used which credential, on whose authority. With it on, every new run is pinned to a governed profile and an immutable policy revision. Later releases enforce that pin at each credential use. The design is ADR-0039 in the architecture records.
Off by default
Section titled “Off by default”main.env.ELITEA_ENTERPRISE_CREDENTIAL_GOVERNANCE_ENABLED is "false" by default. With it off:
- Admissions, routes and wire bytes are unchanged. The governed admin routes return 404.
- No new dependency is built or dialled: no Vault/OpenBao client, no audit relay.
- The new tables and columns (migrations 0180–0193) are created on upgrade and stay empty.
The value must be exactly "true" or "false". Anything else, such as "1" or "TRUE", refuses boot rather than picking a profile.
Turn it on
Section titled “Turn it on”-
Write a policy document. It holds parameters only, never a secret:
{"schema": "elitea.credential-policy.v1","allowed_identity_modes": ["dedicated_agent"],"vault_required_credential_classes": [],"max_mandate_ttl_seconds": 86400,"audit_backlog_alert_threshold": 1000}allowed_identity_modestakesdedicated_agent,runtime_domain_bindingandelitea_native. A revision that allows none is refused, because then no agent could ever use a governed credential. Unknown fields are refused too. -
Store it as a revision. Revisions are immutable: a change is a new id.
Terminal window elitea-main credential-policy put --id 2026-10-r1 --file policy.json --operator "$USER"elitea-main credential-policy show --id 2026-10-r1putruns the same checks boot runs (see below), so a revision that could not boot is refused when you publish it. -
Point the deployment at it and set the flag:
main:env:ELITEA_ENTERPRISE_CREDENTIAL_GOVERNANCE_ENABLED: "true"ELITEA_CREDENTIAL_GOVERNANCE_POLICY_REVISION: "2026-10-r1"
Boot refuses to start when the revision is missing, its stored bytes no longer match its digest, it allows no identity mode, the security audit outbox is missing, or it lists a vault_required_credential_classes entry while no Vault/OpenBao store is configured.
What a run records
Section titled “What a run records”Every new run is pinned to elitea.credential-profile.governed.v1 and the active revision, in the same transaction that admits it. A pin never changes. A resumed run (a human-in-the-loop answer, a continued agent turn) keeps the pin it started under, even if the flag or the revision changed since. A run with no pin is legacy.
A resume is admitted again, so it checks the grant, the reference and the binding as they are now. If one was revoked, suspended or retired while the run waited for an answer, the resume is refused with 403 governed_credential_refused, and the credential is never used. The refusal also settles the paused turn, as a stop does: the approval card is removed, the answer ends as an error with error_code governed_credential_refused and a reason that names no credential (for example, that the agent’s access to a toolkit was revoked while it waited), and the paused execution is cancelled. The turn’s question and partial answer are kept (#1485).
A toolkit’s resolved settings never leave the worker in a run’s events: the final message of a turn names its tools, never their settings, so no credential reaches the browser or the replay log, governed or not.
The same transaction stores the run’s execution security context: the policy revision and digest, the tenant project, the subject and actor chain, the calling application, and the agent version’s active identity binding. It holds references, epochs and digests only, never a credential. It is immutable. Every worker command of the run carries the context’s digest inside the signed command, so a worker can prove it holds the context Main admitted.
A binding is recorded only when exactly one binding of the agent version is active. With none, or more than one, the context names no binding, and every governed credential use of the run is denied.
Governed workers
Section titled “Governed workers”A governed run’s commands go out on their own NATS subjects, elitea.rt.v1.<route>.g.*, and only the governed durables elitea-configuration-worker-governed-v1, elitea-agent-worker-governed-v1 and elitea-index-worker-governed-v1 read them. The NATS bootstrap Job creates them beside the legacy durables. A worker that does not verify the context never binds them, so a governed command never reaches it.
The chart opts the Rust worker in with the same flag: it sets ELITEA_WORKER_GOVERNED_CONTEXT from main.env.ELITEA_ENTERPRISE_CREDENTIAL_GOVERNANCE_ENABLED, so Main and the worker cannot disagree. Outside the chart, set it yourself (true/false; any other value refuses startup). An opted-in worker:
- binds the governed durables next to the legacy ones;
- tells Main on each claim that it verifies the context;
- before it reads any input, fetches the context under its claim and checks it. It must be this run’s, at most 16 KiB, match the signed digest, decode with no unknown field, and name the governed profile. A failed check fails the run as an authorization failure, which is never retried.
A governed run’s worker does not hold the user’s personal access token. Its client token is a run token (elmrt_…). The token reaches two things: the model gateway (/llm) and the run project’s artifacts, and nothing else. On artifacts it works only on the five routes the worker uses: create a bucket, list objects, upload, download and stat an object. The token works only for the run’s own project and execution, and only while the run’s claim holds its lease. A request that names another project gets 403. Lease renewal keeps the token valid through a long run. When the run ends or loses its lease, the token stops working within two seconds (Main reuses a live lease check for that long, so a turn’s many model calls cost one database read). Every other route answers 403 for it. Model usage and artifact access are attributed to the same project and user as before. Toolkit credentials come only from the credential broker. The token is signed with a key derived from the personal access token signing key, so there is nothing new to configure.
Main also checks every claim:
| Refusal | When |
|---|---|
GOVERNED_WORKER_REQUIRED | A governed command is claimed by a worker that did not declare it verifies the context |
GOVERNED_CONTEXT_MISSING | A command without a context is claimed for a run pinned governed |
Neither is retried. A GOVERNED_WORKER_REQUIRED command stays in its stream for a governed worker. A GOVERNED_CONTEXT_MISSING run can never be claimed, so Main also cancels it, and the turn ends at once without waiting for the deadline. A worker with node recovery on still claims governed commands as fresh claims, because the recovery claims do not fetch the context. With the flag on, Main refuses to boot when a command stream lacks its governed subject or durable, so re-run the NATS bootstrap Job before turning the flag on.
Agent identity bindings and mandates
Section titled “Agent identity bindings and mandates”With the flag on, two admin API groups are mounted:
| Route | Purpose |
|---|---|
/api/v2/admin/agent_identities | List and propose bindings of an agent version to an identity |
/api/v2/admin/agent_identities/{id}:approve, :suspend, :revoke | Binding lifecycle |
/api/v2/admin/agent_identities/{id}:repin | Pin the binding to its version’s computed definition digest, as a new proposal |
/api/v2/admin/mandates | List and issue mandates (who authorised an unattended run, for which agent and operations, until when) |
/api/v2/admin/mandates/{id}:approve, :suspend, :revoke | Mandate lifecycle |
Four permissions gate them, held by super_admin and admin only:
models.agent_identity.bindings.manage(list, propose, suspend, revoke) andmodels.agent_identity.bindings.approve(list, approve)models.agent_identity.mandates.manage(list, issue, suspend, revoke) andmodels.agent_identity.mandates.approve(list, approve)
Approval is a two-person step: the approver can never be the proposer. A mandate cannot outlive the revision’s max_mandate_ttl_seconds, and suspending or revoking a binding also stops its mandates. Every change writes a security audit event.
The definition digest
Section titled “The definition digest”A binding pins a definition_digest: the SHA-256 of the agent version’s definition. At admission, Main computes the canonical digest of the version’s stored definition and compares it with the digest of the version’s single active binding. The canonical digest is the same value a root run records as source_definition_sha256: the SHA-256 of elitea.runtime.application-definition.v1, a NUL byte, the project, application and version ids, and the stored definition (the agent’s settings with its toolkits and their stored settings).
If the digests differ, the run gets no agent identity. It runs like a run of an unbound version, so the broker denies every governed use, and a run that needs the binding’s authority (a mandate-bound schedule or trigger) is refused. Admission writes a security audit deny decision with the reason definition_digest_mismatch and the binding’s id. An agent from another project’s catalogue is not in the run’s project, so its digest cannot be computed and it binds no identity.
Every binding view (list, propose and each transition) carries computed_definition_digest: the digest Main computes now, or null when the version cannot be read. When it differs from definition_digest, runs of that version are unbound.
Editing the agent changes the digest, and so does editing a toolkit that the agent embeds, including a credential reference :attach or :detach. After such a change, re-pin the binding:
- A manager calls
POST /api/v2/admin/agent_identities/{id}:repin(bindings.manage, no body). Main computes the digest under the binding’s row lock and writes it. The binding becomesproposed, the caller becomes its proposer, the approval is cleared andrevocation_epochincreases by 1. The binding keeps its id, so its toolkit pins, grants and mandates still name it. A binding that already pins the computed digest, a suspended or revoked binding, and a version that cannot be read answer 409. A suspension is lifted only by:approve: approve a suspended binding first, then re-pin it. - A different administrator calls
:approve(bindings.approve). The binding is active again with the new digest.
Between the two steps the binding is not usable, so its runs have no agent identity. A re-pin is a new proposal, so it follows the same two-person rule as the first approval.
External agent identities
Section titled “External agent identities”A binding in dedicated_agent mode names an identity the agent holds at an external provider, such as an Entra agent identity. Those identities have their own admin routes:
| Route | Purpose |
|---|---|
/api/v2/admin/external_agent_identities | List and create external identities |
/api/v2/admin/external_agent_identities/{id}:provision, :activate, :suspend, :resume, :retire | Identity lifecycle |
models.agent_identity.external.manage gates every route; it is held by super_admin and admin. Activation also needs models.agent_identity.bindings.approve, and needs an approved binding: provisioning never activates an identity. All versions of an agent share one identity. A fork gets a new one. While its external identity is not active, a binding is treated as suspended. Retiring takes two steps: the identity is retiring until the provider confirms, then retired.
Drift reconciliation
Section titled “Drift reconciliation”An administrator can change an external identity at the provider without Elitea. No provider sends a notice when that happens. So, with the flag on, a reconcile loop compares each record with the provider’s live object, for each configured provider. One replica runs each pass; the others wait for the next interval.
| Live state at the provider | What Elitea does |
|---|---|
Disabled while the record is active | Record goes to suspended, epoch +1, audit external_identity.drift.disabled |
Not found while the record is active | Record goes to suspended, epoch +1, audit external_identity.drift.missing |
| Recreated, or the immutable id or generation changed | Record goes to replaced (terminal), epoch +1, audit external_identity.drift.replaced |
| A credential or the owner changed (when the provider reports it) | Record goes to suspended, epoch +1, audit external_identity.drift.changed |
| An Elitea-tagged object with no record | Audit external_identity.orphan only; nothing changes |
A suspension stops new sessions and every Credential call of the agent at once: each check reads the record’s state and epoch. A replaced record can never be resumed. To recover, retire what is left and create a new external identity for the agent. Elitea never re-provisions or deletes a provider object itself.
Two optional revision fields set the loop. Leave them out to keep the defaults:
{ "reconcile_interval_seconds": 600, "reconcile_outage_threshold": 3}reconcile_interval_seconds takes 60 to 86400. reconcile_outage_threshold takes 1 to 100. A provider outage changes no record. Each pass that the provider fails increases agentidentity_reconcile_errors_total{provider}. A pass cut short by a shutdown, or one that fails on the database, does not count as an outage. After reconcile_outage_threshold failed passes in a row, new agent sessions for that provider are denied (provider_unreconciled) on every replica, until a pass succeeds.
GET /api/v2/admin/external_agent_identities/drift lists the open findings, newest first, with each provider’s reconcile health (admission_denied, consecutive_failures, last_success_at). It takes provider, limit and offset, and needs models.agent_identity.external.manage. A later pass closes a finding it no longer sees: the record was resumed or retired, the orphan is gone or has a record, or a replaced agent has a new live identity.
Elitea-native agent assertions
Section titled “Elitea-native agent assertions”Many clouds and identity providers have no agent object, but accept a federated JWT from an issuer they trust: AWS AssumeRoleWithWebIdentity, GCP Workload Identity Federation, Keycloak federated client authentication, Vault JWT auth and MCP servers. With the elitea_native identity mode, Elitea is that issuer. At each credential use of a governed run, Main signs one short-lived assertion for exactly one relying party. No cloud secret is stored.
It is active only when all three hold: the flag is on, the active revision lists elitea_native in allowed_identity_modes, and main.env.ELITEA_AGENT_ASSERTION_CONFIG names at least one relying party. When ELITEA_AGENT_ASSERTION_CONFIG is set, the key ring is also loaded without relying parties, to sign the delegation assertions of calls to remote MCP servers (see Remote agents); in a governed deployment whose revision does not list elitea_native, it is not. Without the variable, no key is loaded and the routes below return 404. The external identity uses provider elitea; provisioning makes no external call.
Configuration
Section titled “Configuration”ELITEA_AGENT_ASSERTION_CONFIG is one JSON document. Unknown fields refuse boot.
{ "origin": "https://elitea.example.com", "rotation_overlap": "24h", "keys": [ {"kid": "agent-es-2026-10", "alg": "ES256", "file": "/etc/elitea/agent-assertion/es256.pem"}, {"kid": "agent-rs-2026-10", "alg": "RS256", "vault": {"mount": "kv", "path": "elitea/agent-assertion/rs256", "version": 3}} ], "relying_parties": [ {"id": "aws-prod", "audience": "sts.amazonaws.com", "claims_profile": "aws"}, {"id": "vault", "audience": "https://vault.example.com", "algorithm": "RS256", "max_ttl": "5m", "claims_profile": "vault_jwt"} ]}| Field | Value |
|---|---|
origin | The stable https origin of the issuer. Defaults to DEPLOYMENT_URL. Relying parties pin it, so do not change it. |
tenant | The elitea_tenant claim. Defaults to the origin’s host. |
rotation_overlap | How long a new key is published before it signs. Default 24h. It must be more than 5m, the JWKS cache lifetime, so every relying party has fetched the new key before it signs; at most 720h. |
relying_parties[].id | A name for logs and adapters |
relying_parties[].audience | The exact aud. Each audience names one relying party. |
relying_parties[].algorithm | ES256 (default) or RS256. EdDSA is not offered: AWS and GCP federation do not accept it. |
relying_parties[].max_ttl | Default 10m, from 30s to 15m |
relying_parties[].claims_profile | aws, gcp_wif, keycloak, vault_jwt or mcp |
A credential request for an audience that is not configured is denied before anything is signed.
Key ring
Section titled “Key ring”The keys are dedicated to agent assertions and sign nothing else. Boot refuses a key whose id or key material is also the command signing key (which also signs code-sandbox owner grants), the code owner TLS key or the egress token key.
Each key is a PKCS #8 PEM private key: P-256 for ES256, RSA of at least 2048 bits for RS256. Its source is one of:
file: an absolute path to a key in a mounted Secret. In the chart, setmain.agentAssertion.keysSecretto a Secret that holds only these keys: it is mounted whole atmain.agentAssertion.mountPath(default/etc/elitea/agent-assertion, mode0444), so each data keyes256.pemis read as/etc/elitea/agent-assertion/es256.pem. Main follows the Secret volume’s links but refuses a link that leaves the mount, and a file or directory writable by group or other.vault: a Vault/OpenBao KV v2 secret, read once at boot through the secret store above, with an explicitversion. The read uses workload scopeagent-assertion, so the role is<prefix>-agent-assertion, and needsELITEA_SECRETSTORE_VAULT_*configured.
The private key is kept only in the signer and never leaves it. Only the agent session broker reaches the signer; no route signs on request.
Rotation
Section titled “Rotation”To rotate, add the new key with a not_before time (RFC 3339) and restart. From not_before the key is in the JWKS. From not_before + rotation_overlap it signs every new assertion. The old key stays in the JWKS until the longest max_ttl plus one minute has passed after that, so every assertion it signed can still be verified. Then remove it from the configuration. At most one key per algorithm may leave out not_before.
JWKS and issuer metadata
Section titled “JWKS and issuer metadata”| Route | Content |
|---|---|
GET /.well-known/elitea-agent-jwks.json | The public keys (kid, alg, use: sig) |
GET /agent-issuer/.well-known/openid-configuration | issuer and jwks_uri |
Both are public and cacheable for five minutes, and carry no tenant data. The issuer is <origin>/agent-issuer, so a relying party that discovers keys through OIDC metadata (AWS IAM, GCP WIF, Vault oidc_discovery_url) finds them under the issuer. Others can use the JWKS URL directly.
Claims
Section titled “Claims”The JWS header has typ: elitea-agent+jwt and the signing key’s kid. The same keys also sign the delegation assertions of remote MCP calls, but those have typ: elitea-delegation+jwt and are never signed for the audience of a relying party. A relying party that checks typ therefore cannot be given a delegation assertion in place of a federation assertion.
| Claim | Value |
|---|---|
iss | <origin>/agent-issuer |
sub | agent:<external agent identity id> |
aud | The one relying party’s audience |
iat, nbf, exp, jti | exp is at most the relying party’s max_ttl and never past the session’s expiry. jti is unique. |
sub_profile | ai_agent |
agent_instance_id | <execution id>:<generation> |
elitea_tenant, elitea_project, elitea_binding | The tenant, the project id and the binding id |
elitea_version_digest | The binding’s agent definition digest (SHA-256, hex) |
elitea_policy_revision | The run’s pinned policy revision |
elitea_epoch | The external identity’s revocation epoch |
elitea_user | {"sub": "<user ref>", "evidence": "asserted"}, only when the run has user attribution |
No email, display name or token appears in an assertion.
Suspending or retiring the external identity, or any epoch change, stops signing at once: each assertion re-reads the record’s state and epoch first. Assertions already issued expire within max_ttl.
Credential use grants
Section titled “Credential use grants”A grant lets one agent binding use one credential, through one toolkit, against one target. Grants live per project:
| Route | Permission |
|---|---|
GET /api/v2/projects/{project_id}/credential_grants | models.credentials.governed.manage or .approve |
POST /api/v2/projects/{project_id}/credential_grants (propose) | models.credentials.governed.manage |
POST …/credential_grants/{id}:approve | models.credentials.governed.approve |
POST …/credential_grants/{id}:suspend, :revoke | models.credentials.governed.manage |
The five models.credentials.governed.* permissions (use, reveal, manage, probe, approve) are granted to no role by default. Grant them to a role in Settings → Roles. Approval is two-person, as for bindings, and the database enforces it. A use grant never permits reveal, manage or probe, and a grant for one target never permits another.
Each credential use is decided by eight gates that must all permit: authentication source, RBAC, entity access, binding, toolkit, grant, operation and delegation or mandate. Any gate’s error is a deny. The credential broker calls the decision at each redemption (see below).
Governed credential resolution
Section titled “Governed credential resolution”In a governed run, a toolkit’s credential is chosen by its exact credential reference id, never by its configuration title. Resolution never leaves the run’s project: there is no lookup in the user’s personal project or in the public project.
A stored configuration reference names its governed credential in an optional field:
{ "elitea_title": "jira-prod", "private": false, "credential_reference_id": "<credential reference uuid>" }The toolkit create and update routes refuse this field. Only the governed admin API writes it. Every other reader ignores it, so a legacy run resolves the reference by its title exactly as before.
At admission. For a run of an agent version with a usable binding, Main checks every top-level toolkit in the frozen version. For each toolkit whose stored settings hold a configuration reference, it checks these rules in order, and the first one that fails refuses the run:
| Rule | Code |
|---|---|
| The toolkit names one reference, not more. | credential_reference_ambiguous |
The reference is not private. An isolated personal credential is not governed (customer gate D1). | private_credential_not_governed |
The reference names a credential_reference_id. A reference with only a title is refused. | credential_reference_required |
| The id is a UUID. | credential_reference_invalid |
| The reference exists in the run’s project. | credential_reference_not_found |
The reference is active. | credential_reference_not_active |
| The binding has an agent toolkit binding for the toolkit. | toolkit_not_bound |
| The toolkit binding’s digest is the stored toolkit’s digest. | toolkit_digest_mismatch |
An active, unexpired grant for the binding, toolkit, reference and target allows use. | credential_grant_missing |
A refused start answers 403 with {"error": "governed_credential_refused", "code": "<code>"}. The code names the rule, never the credential.
A toolkit with no configuration reference that the binding binds is frozen at its digest alone. A toolkit that the binding does not bind is not governed.
Main writes each resolved id into its toolkit’s frozen settings as credential_reference_id, where the worker reads it. It also stores the list in the run’s security context: per toolkit, the reference, its target, its generation and the toolkit digest, with the run’s application and version. If the binding changes between resolution and the admission transaction, the admission fails, and a retry resolves again.
A run of a version with no usable binding, and an ad-hoc run, freeze nothing. The broker denies every governed use of such a run.
The toolkit digest. An agent toolkit binding’s toolkit_digest is the SHA-256 of elitea.credential-governance.toolkit-digest.v1, a NUL byte, the toolkit type, a NUL byte, and the toolkit’s stored settings as compact JSON with object keys sorted. The stored settings are the saved row with its configuration references unexpanded. Editing the toolkit changes the digest. Editing a configuration that it names does not.
At redemption. The broker accepts only the reference that admission froze for the toolkit. If the worker sends no reference, the broker uses the frozen one. For a toolkit frozen at its digest alone, the broker uses the one usable grant of the binding for the toolkit and operation. These redemptions are denied with the reason credential_reference_not_frozen, before any decision or secret read:
- the reference is not the frozen one;
- the toolkit has no frozen entry;
- the worker names a reference for a toolkit frozen at its digest alone.
The binding gate then compares the agent binding with the application version that admission ran, and the toolkit binding with the frozen digest. A binding moved to another version, or a toolkit bound again at another digest after admission, is denied. A governed run admitted before this release has no frozen list, so its redemptions are denied, and a new run is needed.
Inside a governed resolution, the legacy title expansion refuses to run (ErrLegacyResolverInGovernedRun), as defence in depth.
Credential references and toolkit pins
Section titled “Credential references and toolkit pins”Credential references and agent toolkit bindings are managed through admin routes. They are gated on the models.agent_identity.bindings.* permissions, and are mounted only while the flag is on:
| Route | Purpose |
|---|---|
GET /api/v2/admin/credential_references?project_id=<id>, GET …/{id} | List and read references |
POST /api/v2/admin/credential_references | Stage a new reference from scratch (bindings.manage) |
POST …/credential_references/{id}:activate | staged to active, by an approver who is not the stager (bindings.approve) |
POST …/credential_references/{id}:retire | Any state to retired, which is final. The next redemption is denied authority_revoked. |
POST …/credential_references/{id}:attach, :detach | Write or remove credential_reference_id in a toolkit’s configuration reference |
PUT, DELETE /api/v2/admin/agent_identities/{id}/toolkits/{toolkit_id} | Pin a toolkit to an agent binding at its current digest, or remove the pin |
Migration 0193 records who staged, activated and last changed each reference, and the reference’s migration source. The database refuses an activation by the stager. A reference with no recorded stager cannot be activated. The stager and approver always come from the caller’s token, never from the body. Every transition writes a security audit decision in the same transaction.
Attaching re-pins the toolkit. :attach changes the stored settings, so the toolkit’s digest changes, and every pin at the old digest fails admission with toolkit_digest_mismatch until it is pinned again. The agent’s definition embeds the toolkit’s settings, so its definition digest changes too: re-pin the agent binding as well. An MCP toolkit has no configuration reference to attach to: it is governed by its pin and the one usable grant.
elitea-main credential-governance migrate-references stages references for existing toolkits. Its --apply mode never activates anything. The credential governance runbook has the flags, the order of the steps and the incident procedure.
Provider adapters and probe
Section titled “Provider adapters and probe”Each governed provider has an adapter descriptor in the platform. It lists the credential classes the provider accepts, the target the credential is bound to, the canonical operations (provider.resource.verb, for example jira.issue.read), the operation that probes a credential, and the envelope keys the broker may release for each class. A grant’s operation_scopes must name operations from these descriptors. An unknown operation is a deny. Targets are canonical URLs: https only, lower-case host, no user name, query or fragment, no trailing slash.
| Provider | Target | static_secret | oauth_delegated | Probe operation |
|---|---|---|---|---|
| Jira | Site base URL | fixture | fixture | jira.project.read |
| Confluence | Site base URL | fixture | fixture | confluence.space.read |
| Aha! | Account base URL | fixture | fixture | aha.product.read |
| SharePoint | Tenant site URL | not accepted | fixture | sharepoint.site.read |
| qTest | Site base URL | fixture | not accepted | qtest.project.read |
| Remote MCP server | MCP resource URI (see Governed MCP) | not accepted | fixture | none |
fixture means a class has been tested against test doubles only, not against a real tenant. This table is not a support claim. No provider and class counts as qualified until a qualification run against a customer environment links its evidence. Provider notes:
- Jira and Confluence Cloud apply point-based rate limits. A search costs more than a single read.
- Aha! has no service accounts. A static key always acts as a person.
- SharePoint accepts only a Microsoft Graph token from Entra on-behalf-of, and prefers the
Sites.Selectedscope. - qTest tokens are long lived. Set a rotation policy before you rely on them.
To check that a credential works, call POST /api/v2/projects/{project_id}/credential_references/{id}:probe. The caller needs models.credentials.governed.probe in the project, and the project needs an active grant for that credential reference that lists the probe capability. A use grant is not enough (403). The platform runs the provider’s probe operation itself. The answer gives the outcome only (succeeded, rejected, unreachable or unavailable) and never contains the credential. Every probe, allowed or denied, writes a security audit event. In this release no provider probe runs yet, so every allowed probe answers unavailable (“probe not available for provider”). Like the other governance routes, this one is mounted only while the flag is on.
Entra delegation
Section titled “Entra delegation”With the flag on, an OIDC identity provider marked as Entra keeps the signed-in user’s delegated middle-tier token, encrypted, so a governed run can act on the user’s behalf through on-behalf-of exchange. Set it on the provider’s stored settings:
| Setting | Value |
|---|---|
entra | true |
entra_tenants | The tenant ids (lower-case GUIDs) allowed to sign in. Required. |
entra_api_scope | The middle-tier scope requested at sign-in. Default api://<client_id>/.default. |
entra_federated_token_file | Path of a projected workload identity token for on-behalf-of exchange. Without it, classic OBO is off; client secrets are refused. |
entra_agent_blueprint_id | The agent identity blueprint application id, for Agent ID OBO |
Sign-in then accepts only those tenants’ v2 issuers and requires the tid and oid claims. The middle-tier API’s app registration must issue v2 access tokens: set "requestedAccessTokenVersion": 2 (accessTokenAcceptedVersion in older manifests) in its manifest. With the default v1 tokens, no delegated token is ever kept and every governed run pauses for authorization. Accounts still link by sub, so turning this on does not orphan existing users. Signing out revokes what that session created, in one transaction that wipes the stored ciphertext. A change to these settings takes effect at the next restart. The admin UI does not show these fields yet: set them through the identity provider API, and note that saving the provider from the UI may drop them.
When Entra asks for interaction or a claims challenge, a run can pause with credential authorization required and resume as the same run once the user consents again. The credential broker reports that pause as AUTHORIZATION_REQUIRED with an opaque reference (see below).
Entra Agent ID provider
Section titled “Entra Agent ID provider”The Entra Agent ID provider gives each Elitea agent its own Microsoft Entra agent identity under one agent identity blueprint. All versions of an agent share that identity, as for any external identity. A governed run then gets tokens issued to the agent itself:
- App-only runs get a resource token of the agent identity (
idtyp=app). - Delegated runs get an Agent ID on-behalf-of token for the signed-in user (
idtyp=user). This needs a verified delegation whose audience is the blueprint, not the middle tier. A user that Elitea only asserts on the basis of a mandate is always refused.
Elitea checks every token before it uses it. The token must name the tenant, the agent as its client and the blueprint as its parent (xms_par_app_azp). The blueprint’s own token for an agent (T1) impersonates that agent, so it never leaves elitea-main. Workers and toolkits receive only the short-lived resource token of one invocation. Logs carry the token’s uti and the Entra request and correlation ids, never token text. Each token request sends a client-request-id derived from the run’s execution and generation.
The provider is built only while the flag is on and the active revision allows the dedicated_agent identity mode. Set the block as environment variables, or as one JSON file named by ELITEA_ENTRA_AGENT_ID_CONFIG_FILE in the form {"entra": {"agentid": {...}}}. Use one or the other, not both:
| Variable | File key | Value |
|---|---|---|
ELITEA_ENTRA_AGENT_ID_TENANT_ID | tenant_id | The blueprint’s tenant id, a lower-case GUID. common, organizations and consumers refuse boot. |
ELITEA_ENTRA_AGENT_ID_BLUEPRINT_APP_ID | blueprint_app_id | The agent identity blueprint’s application id |
ELITEA_ENTRA_AGENT_ID_FEDERATED_TOKEN_FILE | federated_token_file | A projected service account token with audience api://AzureADTokenExchange. Elitea re-reads it for every token request. |
ELITEA_ENTRA_AGENT_ID_ALLOWED_RESOURCE_SCOPES | allowed_resource_scopes | The only scopes a run may ask for, comma separated. App-only runs use the <resource>/.default entries, one per token, for example https://graph.microsoft.com/.default. Delegated runs may use any entry, for example https://graph.microsoft.com/Sites.Selected. |
ELITEA_ENTRA_AGENT_ID_SPONSOR_GROUPS | sponsor_groups | The Entra group that sponsors each project’s agent identities: default=<group id>,<project id>=<group id> |
ELITEA_ENTRA_AGENT_ID_IDENTITY_LIMIT | identity_limit | Agent identities per blueprint. Default 250. |
ELITEA_ENTRA_AGENT_ID_AUTHORITY | authority | Token endpoint host. Default https://login.microsoftonline.com. |
ELITEA_ENTRA_AGENT_ID_GRAPH_BASE_URL | graph_base_url | Microsoft Graph host. Default https://graph.microsoft.com. |
The blueprint authenticates with the federated token only. A client secret (ELITEA_ENTRA_AGENT_ID_CLIENT_SECRET or client_secret) or a certificate (ELITEA_ENTRA_AGENT_ID_CERTIFICATE_FILE or certificate_file) refuses boot. So does an invalid block. To pass the provider to the Agent ID on-behalf-of exchange, also set entra_agent_blueprint_id on the Entra identity provider (see Entra delegation) to the same blueprint.
Operator notes:
- The blueprint needs the Graph permission to create and manage its agent identities. Grant every permission the agents use ahead of time, as an administrator. An agent identity cannot consent interactively: Entra answers
AADSTS82014, and Elitea reports a configuration error, not a pause for the user. - Provision creates the agent identity disabled (
accountEnabled=false). It is namedElitea agent <application id>and taggedelitea:managedandelitea:record:<id>. A repeat call returns the same object. If two copies of one record ever exist (a retry while Graph’s search index lags), the next Provision keeps the one with the lowest object id and soft-deletes the other. Provision accepts thededicated_agentmode only. Activate and Resume enable it, and Suspend disables it in Entra, so Entra issues the agent no token. Retire soft-deletes it. Purging a deleted object stays an operator action. - Disabling the agent identity in Entra is your own kill switch. The drift reconciliation (above) sees the change and suspends the record. An object deleted and created again under the same name or tag is reported as
replaced. - Entra allows at most 250 agent identities per blueprint when an application creates them. At
identity_limit, Provision is refused, and existing identities keep working. To go further, shard: give each trust boundary its own blueprint and Elitea deployment. Or use delegated creation: an administrator creates the agent identity under the blueprint with the record’selitea:record:<id>tag, and Provision then adopts that object instead of creating one. Adoption disables the object until Activate and adds theelitea:managedtag, keeping its other tags. An object under another blueprint is never adopted. Entra Conditional Access for agents needs its own licence.
Provider OAuth consent
Section titled “Provider OAuth consent”Jira, Confluence and Aha! accept a delegated OAuth credential that the user grants to the provider directly: Atlassian OAuth 2.0 (3LO) or Aha! OAuth2. This grant is its own authority. It does not use Entra. SharePoint still uses the Entra on-behalf-of exchange. Each provider adapter states which of the two it uses.
The platform keeps one grant per tenant project, provider, target binding, user and agent binding. The grant holds the refresh token and the current access token. Both are encrypted with AES-256-GCM under a key that is derived from the vault master key (SECRETS_MASTER_KEY, the same key as the Entra delegation store). Migration 0190 creates the tables.
Configure the OAuth applications
Section titled “Configure the OAuth applications”Register one OAuth application for each provider that you use. Set its callback (redirect) URL to exactly:
https://<your Elitea host>/api/v2/credential_consent/<provider>:callback<provider> is jira, confluence or aha. For Atlassian, create an OAuth 2.0 (3LO) app in the Atlassian developer console. Add the Jira or Confluence API scopes and offline_access, because without offline_access Atlassian gives no refresh token. For Aha!, register an OAuth2 application in your Aha! account settings.
Then set these values on elitea-main. <P> is JIRA, CONFLUENCE or AHA:
| Variable | Value |
|---|---|
ELITEA_PROVIDER_OAUTH_<P>_CLIENT_ID | The application client id. Empty means the provider is not configured. |
ELITEA_PROVIDER_OAUTH_<P>_CLIENT_SECRET | The client secret. Keep it in a Kubernetes Secret. |
ELITEA_PROVIDER_OAUTH_<P>_REDIRECT_URI | The exact callback URL above. A different path, a query or a fragment refuses boot. |
ELITEA_PROVIDER_OAUTH_<P>_AUTHORIZE_URL | Atlassian default https://auth.atlassian.com/authorize. Aha!: required, for example https://<account>.aha.io/oauth/authorize. |
ELITEA_PROVIDER_OAUTH_<P>_TOKEN_URL | Atlassian default https://auth.atlassian.com/oauth/token. Aha!: required, for example https://<account>.aha.io/oauth/token. |
ELITEA_PROVIDER_OAUTH_<P>_SCOPES | Space-separated scopes. Atlassian has least-privilege defaults that include offline_access. Aha! uses none. |
ELITEA_PROVIDER_OAUTH_<P>_TIMEOUT | The bound of one token request, default 10s, at most 25s (it must finish inside the 30s refresh lease). |
These values are read only with the governance flag on. Endpoints must be https; plain http is accepted only for a loopback host, for tests. If a provider is configured and SECRETS_MASTER_KEY is missing, elitea-main refuses to start. If no provider is configured, the consent routes are not mounted and a provider-OAuth use is denied with BACKEND_UNAVAILABLE.
The consent flow
Section titled “The consent flow”- The client calls
POST /api/v2/credential_consent/{provider}:startwithproject_id,agent_binding_idandtarget_binding_id. The caller needsmodels.credentials.governed.usein the project. The target binding must be for that provider in the project, and the agent binding must be active there. - The platform stores a random state value as a digest, bound to the user, the project, the agent binding, the target binding and the provider. It also stores an encrypted PKCE verifier. The state is valid for 10 minutes and can be used one time. The answer is the provider authorization URL, with an S256 PKCE challenge and the exact redirect URI.
- Open that URL in the user’s browser. After the user consents, the provider sends the browser to
GET /api/v2/credential_consent/{provider}:callback. - The callback uses up the state. An unknown, expired, used or foreign state, or one for another user, is refused (400). The platform then checks again that the user holds
models.credentials.governed.usein the state’s project, that the target binding still exists (404 if not) and that the agent binding is still active (409 if not). It exchanges the code with the PKCE verifier and the same redirect URI, stores the encrypted tokens and writes a security audit event. If the user declines, the answer is 403. If another consent for the same grant finished first, the answer is 409: start consent again. Every refused callback writes a denied audit event.
A consent again for an existing grant replaces its tokens. If the provider’s answer has no refresh token, the grant keeps the one it already holds.
A second consent for the same key replaces the stored tokens. No answer, log or error contains a token, a code or a verifier.
How a governed run uses the grant
Section titled “How a governed run uses the grant”For an oauth_delegated Jira, Confluence or Aha! credential, the credential broker reads the grant of the run’s own user for the run’s project, agent binding and target binding. A run with no user, for example under an autonomous mandate, has no such grant and is denied. With no grant, the use is denied with the reason provider_consent_required: send the user through the consent flow. The envelope is OAUTH_ACCESS with the key access_token.
Single-flight refresh
Section titled “Single-flight refresh”An access token is refreshed when it is less than 2 minutes from its expiry. Atlassian refresh tokens rotate: each one works one time. If two replicas refresh at the same time, one of them gets invalid_grant and the grant is lost. To prevent this, a refresh runs one time across all elitea-main replicas:
- In each replica, concurrent callers for one grant share one refresh.
- Across replicas, the refresh needs a lease row in
elitea_runtime.credential_refresh_leases. The lease has a fence number. The new access token and the rotated refresh token are written in one transaction, and only while the fence is still current. A holder whose lease expired cannot write. - The other replicas wait. They read the grant again with a bounded backoff with jitter until the new token is there.
If the provider refuses the refresh token, the grant becomes invalid, its tokens are deleted and an audit event is written. The user must consent again.
Governed MCP
Section titled “Governed MCP”Governed MCP has two halves, and each is off until you configure it. Both are read only with the governance flag on. With the flag off, the MCP endpoint, its 401 answer and the existing MCP OAuth settings of toolkits are unchanged.
Inbound makes the project MCP endpoint (https://<your Elitea host>/app/<project id>/mcp) an OAuth protected resource:
| Variable | Value |
|---|---|
ELITEA_GOVERNED_MCP_INBOUND_ISSUER | The issuer of the access tokens that MCP clients send. Empty leaves inbound off. |
ELITEA_GOVERNED_MCP_INBOUND_JWKS_URL | That issuer’s JWKS URL. Required when the issuer is set. |
ELITEA_GOVERNED_MCP_INBOUND_SCOPES | Space-separated scopes that every token must hold. Empty requires none. |
ELITEA_GOVERNED_MCP_INBOUND_USER_CLAIM | The claim that names the Elitea user, matched to the user’s email. Default email. |
ELITEA_GOVERNED_MCP_INBOUND_OAUTH_ONLY | true refuses platform credentials (the browser session, personal access tokens and API keys) on the MCP endpoint. Default false. |
With inbound on, DEPLOYMENT_URL must be https, because the resource URI cannot follow the request’s Host header. The platform then:
- Serves RFC 9728 protected resource metadata, without authentication, at
/.well-known/oauth-protected-resource/app/<project id>/mcp. The document names the resource, the issuer as its authorization server, the scopes andheaderas the only bearer method. Without inbound this path is404. - Adds
resource_metadata="<metadata URL>"to theWWW-Authenticateheader of every401from the MCP endpoint, so an MCP client can discover where to sign in. - Accepts an access token from the issuer only when its single
audis exactly the MCP endpoint URL of that project. A token for another audience, an expired token or a token without the required scopes is401. A token from an issuer that is not configured is not read as OAuth. - Removes the inbound token from the request after it is checked. It is never put in the run’s input bundle, never sent to a worker or a tool and never logged. The run records the MCP client as its calling application.
Outbound holds the OAuth of remote MCP servers that a governed agent calls:
| Variable | Value |
|---|---|
ELITEA_GOVERNED_MCP_OAUTH_ENABLED | true turns it on. It needs an https DEPLOYMENT_URL and SECRETS_MASTER_KEY. |
ELITEA_GOVERNED_MCP_ALLOW_DYNAMIC_REGISTRATION | true allows RFC 7591 dynamic client registration when the server’s authorization server does not support client ID metadata documents. Default false. |
Each remote server is a target binding with the provider type mcp. Its target identity is the canonical server URL: https, lowercase host, no default port, no query and no fragment. The platform finds the server’s authorization server from the server’s RFC 9728 metadata and RFC 8414 (or OpenID) metadata, and needs S256 PKCE there. Elitea identifies itself with the client ID metadata document at /.well-known/elitea-mcp-client.json. Every authorization request, code exchange and refresh carries the RFC 8707 resource parameter with the server URL, so a token is minted for that one server.
The MCP server names its own authorization server, so every request the platform makes for it (metadata, registration, code exchange and refresh) goes through the same egress guard as the MCP OAuth proxies. A private, loopback or link-local address is refused before any connection, unless you name it in ELITEA_MCP_OAUTH_EGRESS_ALLOWLIST (for a private identity provider). Link-local and multicast addresses stay refused whatever the allowlist says. Redirects are not followed.
Consent uses the provider OAuth consent flow above, with mcp as the provider: POST /api/v2/credential_consent/mcp:start and the callback https://<your Elitea host>/api/v2/credential_consent/mcp:callback. You do not register a callback with the MCP server: it comes from the client ID metadata document or from the registration. The grant is stored and refreshed like every other provider OAuth grant.
On a governed agent run the worker gets every MCP token from the credential broker, which records a decision for each use. It uses two operations of the mcp provider:
mcp.tools.list(read, empty resource) is redeemed once when the agent is built. The worker lists the server’s tools with that token, keeps only their names and schemas, and drops the session and the token. Up to four servers are listed at the same time. If a server refuses or does not answer the listing, for example with a401because the token’s scope is too narrow, only that connection is left out and named in the run notice; the run goes on with the other tools.mcp.tool.call(write, resourcemcp:tool/<tool name>) is redeemed for each tool call. The worker opens a new session to that one server with onlyAuthorization: Bearerset, lists the tools on that session, calls the tool the same way as any other MCP tool, and drops the session and the token. A new session for each call costs one extra MCPinitializeandtools/listper call; this is the current design.
The token is released for one target binding and is sent only to the server whose canonical URL is that binding’s identity: a token for server A is refused for server B before the worker connects. The tool-list cache is not used, and the token is never logged or put in a tool result. Only a saved mcp toolkit can be governed. A prebuilt or internal MCP server, a toolkit that has its own credential in its headers, or a URL that has no canonical form (for example one with %) is left out of the run as toolkit_not_governable. Nested agents, saved pipelines and direct toolkit calls do not govern MCP yet and refuse a governed run that has an MCP connection.
If the user has not consented yet and the run is a chat turn of that user, the use is AUTHORIZATION_REQUIRED with an opaque authorization_reference that starts with mar_. It is bound to the user and the project and is valid for one hour. The worker raises it through the run’s usual authorization prompt, for the listing or for the tool call. The client starts consent with that reference in place of the two binding ids, {"project_id": <project id>, "authorization_reference": "mar_..."} to POST /api/v2/credential_consent/mcp:start, sends the user to the answered URL, and retries the turn after the callback. The reference opens only for the user and the project it was issued to: another user’s start with it, a start that also names binding ids, or a start for another provider is 400 and starts nothing. Any other run, for example an autonomous one, is denied with provider_consent_required.
Credential broker
Section titled “Credential broker”A governed run gets toolkit credentials only from RedeemCredential, a gRPC call on the runtime content service under the worker’s mTLS claim. With the flag off, the call answers UNIMPLEMENTED and runs no statement. Each call names one bound tool invocation: an invocation id, the tool call, the toolkit, the operation, the resource, a digest of the arguments and the credential reference. Main then, in this order:
-
Binds the call to the worker’s live claim: the workload, claim id, fence and session, on a
RUNNINGagent job. A node-recovery inspection claim is refused. A failure is a gRPC error, and no secret is read. -
Checks that the run is pinned governed and that the request’s context digest equals the stored context.
-
Checks the credential reference against the one admission froze for the toolkit, and fills it in if the worker sent none (see Governed credential resolution). For a toolkit frozen at its digest alone, it uses the single active, unexpired grant of the run’s agent binding for that toolkit that allows
useof the operation. If no grant matches, or more than one does, the use is denied. -
Decides the use through the eight gates.
-
Records the invocation and writes the decision to the security audit chain in one transaction (migration 0188). The same id sent again with the same fields is decided again; with any field changed it is refused. If the audit write fails, the use is denied.
-
Opens the run’s agent session at its first redemption, from the stored context, and reuses it after that.
-
Gets the material:
- Static secret: read from the secret store at the reference’s generation.
oauth_delegatedcredential, Entra (SharePoint): exchanged on behalf of the user, through the run’s delegation or the one itsdelegated_backgroundmandate carries. The exchange asks for the adapter’s least-privilege scopes (SharePoint:https://graph.microsoft.com/Sites.Selected). If the adapter names none, it asks for the target’s/.default.oauth_delegatedcredential, provider OAuth (Jira, Confluence, Aha!): the access token of the user’s provider grant, refreshed one time across replicas when it is near expiry (see Provider OAuth consent).
Only the keys that the operation catalogue names for the provider are returned. The first key is required and the others are optional, so a Jira or Confluence PAT stored as a bare token works without a username. Basic auth (Jira Cloud, Confluence Cloud) is one secret holding the key map
{"token", "username"}, the same in the built-in vault and in Vault/OpenBao, and both keys are returned;migrate-referenceswrites it (#1481). A stored JSON object without the required key is denied; it is never sent whole. The redemption is recorded and audited. No material is stored or logged.
The answer is an envelope (STATIC, OAUTH_ACCESS or DYNAMIC_LEASE) that expires after 5 minutes or earlier, or a denial:
| Code | When |
|---|---|
NOT_GOVERNED | The run is not pinned governed |
CONTEXT_MISMATCH | The digest is not the stored context’s, or the context does not match the pin |
INVOCATION_MISMATCH | The invocation id was used before with other fields |
DENIED | A gate refused, the audit write failed, the session could not open, or the material has the wrong shape. The denial carries the decision id. |
AUTHORIZATION_REQUIRED | Entra asks the user to consent again. The denial carries an opaque authorization reference. A run that is not a chat turn gets DENIED instead. |
CLASS_NOT_SUPPORTED | The credential class has no adapter or the catalogue names no keys for it |
BACKEND_UNAVAILABLE | The secret store, the token endpoint or the delegated token store is not available |
The broker reads Vault/OpenBao as the workload scope credential-broker, so create the role <prefix>-credential-broker when a revision requires Vault.
The client token of a governed run
Section titled “The client token of a governed run”GetEliteaClientToken serves a governed run a run token (elmrt_…), because the worker uses it for model calls, embeddings and the run project’s artifacts. The response carries no project secrets header value. Governed toolkit credentials come only from RedeemCredential. The run token reaches the model gateway and the run project’s artifacts, nothing else, and lasts as long as the run’s lease. A legacy run still gets its user’s personal access token.
Which authority a run uses
Section titled “Which authority a run uses”Admission records exactly one authority source in the run’s security context. Admission picks it from these rules, checked in order:
- A run started by a signed-in user, with a delegation. The run uses that user’s newest active, unexpired Entra delegation for the bound agent. A delegation belongs to the user through the sign-in that created it.
- A run started by a user, with no delegation. The run uses the one active
delegated_backgroundmandate of the binding that names that user. - A run with no user. The run uses the one active
autonomousmandate of the binding.
If no rule matches, the run records no authority. If two mandates match, it also records none. In either case, every credential use of the run is denied. An autonomous mandate never serves a run that has a user. The choice is fixed at admission: a delegation or mandate created later does not change a run that is already running.
Whose permissions an autonomous run uses
Section titled “Whose permissions an autonomous run uses”A run under an autonomous mandate has no signed-in user. Its RBAC and entity access are those of the user who approved the mandate, read at each decision, so a revoked role takes effect at the next use. The audit chain records that subject as attributed, not verified. A mandate with no approver has no subject, and every use is denied. A delegated_background mandate keeps the run’s own subject, as an interactive run does.
Metrics
Section titled “Metrics”elitea_credential_redeem_total{result,kind,reason} counts redemptions (result is permit, deny or error), and elitea_credential_redeem_duration_seconds{result} times them. elitea_security_audit_head_lock_wait_seconds{partition,result} times each audit append’s wait for its team’s chain head (see Limits). They carry no tenant or user label.
Revocation and caching
Section titled “Revocation and caching”Every governed authority record has an epoch that each change bumps by one: the agent identity binding, the mandate, the delegation, the credential use grant, the credential reference (its generation) and the external agent identity. PostgreSQL is the authority for all of them. Main checks the epochs a run depends on at three points, for governed runs only:
| Point | What is checked | When an epoch is stale |
|---|---|---|
ClaimCommand | The binding, mandate and delegation the run’s security context recorded at admission | The claim is refused with AUTHORIZATION_FAILED (“The governed execution’s authority was revoked.”), not retryable, and the run is cancelled |
RenewLease | The same, after the fence renewed | The answer is desired state CANCELLED (reason AUTHORITY_REVOKED in the log and metric), and the run is cancelled durably |
RedeemCredential | The same, plus the decision’s grant and credential reference and the binding’s external identity | DENIED (reason authority_revoked), before any cache lookup or secret read |
A record that is missing, revoked, suspended, expired or not active counts as stale at any epoch. All the epochs of one check are read in one statement. A legacy run is never checked: its claim is unchanged, and its lease renewal reads only whether the run has a security context (once per replica). With the flag off, none of these checks exists.
Cache. Each Main replica keeps a bounded in-memory cache of released material, keyed by the full security key: tenant project, agent binding, subject, authority, credential reference, target, scopes and operation class, policy revision and credential generation. An entry stores the epochs read when it was filled. It is served only if the epochs read for this redemption are the same and the entry is not within the skew of its expiry; anything else evicts the entry and zeroes its value. Entra on-behalf-of access tokens are cached until their expiry. Static secrets are not cached unless the run’s policy revision allows it. The cache parameters are optional revision fields; the defaults are test values, not a commitment:
"cache": { "max_entries": 1024, "max_bytes": 4194304, "skew_seconds": 30, "cache_static_secrets": false}The replica’s bounds come from the revision Main booted with. Whether a run may reuse a static secret comes from the revision the run is pinned to. A document without cache keeps its digest.
Invalidation. After a binding, mandate, delegation, grant or external identity change commits, Main publishes the record’s kind, id, new epoch and change time on the core NATS subject elitea.rt.v1.security.invalidation, and every replica drops the entries filled at an older epoch. The message carries no tenant secret and no value. The chart grants the subject to the elitea-main-runtime identity only; workers are denied it. Delivery is advisory: a lost message costs latency only, because every redemption re-reads the epochs.
Outages. If PostgreSQL cannot answer the epoch read, new redemptions are denied BACKEND_UNAVAILABLE, even when a cached entry exists. If Entra’s token endpoint is unavailable, oauth_delegated redemptions that miss the cache are denied the same way. Each dependency has a circuit breaker: after 5 consecutive failures it opens for 10 seconds and denies without trying. The Entra breaker is kept per tenant, so one tenant’s failing identity provider does not deny another’s. A request that the caller cancelled or that ran out of time does not count as a failure. No entry is ever served past its expiry.
Metrics. None carries a tenant or user label.
| Metric | Meaning |
|---|---|
elitea_credential_cache_lookups_total{result} | Cache lookups: hit, miss, stale (epochs changed) or expired |
elitea_credential_invalidation_received_total{source} | Invalidations this replica received, local or remote |
elitea_credential_revocation_denials_total{change_time} | First redemption denied after an epoch change; known when this replica heard of the change |
elitea_credential_revocation_deny_latency_seconds | From the epoch change to that first denial |
elitea_runtime_governed_authority_revoked_total{point} | Governed runs refused at claim or cancelled at renew |
Mandate-bound schedules and triggers
Section titled “Mandate-bound schedules and triggers”With the flag on, a pipeline schedule, an inbound pipeline trigger and an index schedule run only under a mandate that an administrator attached to them. The schedule’s author, the trigger’s creator and the project system token are no longer an authority for the run, and nothing falls back to them. With the flag off, all three behave as before: saving a schedule still makes the caller its author.
Attach or detach a mandate with these routes. They are gated on models.agent_identity.mandates.manage and mounted only with the flag on:
| Route | Target |
|---|---|
PUT, DELETE /api/v2/admin/mandate_attachments/pipeline_schedules/{project_id}/{version_id} | The pipeline version’s schedule |
PUT, DELETE /api/v2/admin/mandate_attachments/pipeline_triggers/{project_id}/{version_id} | The pipeline version’s live inbound trigger |
PUT, DELETE /api/v2/admin/mandate_attachments/index_schedules/{project_id}/{toolkit_id}/{index_name} | The schedule of one index |
PUT takes {"mandate_id": "<uuid>"}. The attach is refused with 409 and a code unless all of these hold:
- the mandate is in the target’s project (
MANDATE_TENANT_MISMATCH); - its kind is
autonomousordelegated_backgroundand the active revision allows it (MANDATE_KIND_NOT_ALLOWED). Aninteractive_delegationis never accepted; - its binding is bound to the version the target starts (
MANDATE_BINDING_MISMATCH); - it is not revoked or expired (
MANDATE_NOT_USABLE). A proposed or suspended mandate can be attached, but it runs nothing until it is active; - its resource scopes cover the target (
MANDATE_SCOPE_MISMATCH).
A revision allows unattended kinds with an optional field. A revision without the field allows none, so every governed schedule and trigger is refused until you list the kinds:
"unattended_mandate_kinds": ["autonomous", "delegated_background"]Asserted users. A user-attended governed run needs authority over its user. A verified delegation (an Entra sign-in) gives it. Otherwise it comes from an approved delegated_background mandate that names the user, and then the agent session carries a user Elitea asserts on the basis of that mandate, not one a provider verified. A revision accepts asserted users only for the identity modes it lists, and only elitea_native can take one. Without the field, no session carries an asserted user, so every user-attended governed run without a delegation is denied with agent_session.asserted_user_not_accepted:
"asserted_user_identity_modes": ["elitea_native"]Each listed mode must also be in allowed_identity_modes, or the revision is refused.
Asserted evidence never satisfies a policy that requires verified evidence, and listing a mode does not change the security profile a run was admitted under. The governed policy shipped in deploy/governed/credential-policy.json lists elitea_native, so a revision started from it accepts asserted users. To require a verified delegation for every user-attended governed run, remove the field (or leave the list empty) in a new revision and boot with that revision. The decision is recorded in ADR-0039, amendment A1.5.
Scopes. A mandate can restrict a schedule with two resource scope prefixes. schedule.cron:<fields> names a cron expression with its fields joined by _, for example schedule.cron:0_3_*_*_*. schedule.input:sha256:<hex> names the SHA-256 of the schedule’s input. If a mandate has neither prefix, it covers any cron and input. A scoped mandate covers nothing that lacks the scoped value: a mandate with a schedule.cron: scope never covers an inbound trigger, and one with a schedule.input:sha256: scope never covers a trigger or an index schedule. An index schedule’s mandate must cover the cron of every schedule the index has. Saving a governed schedule keeps its mandate. A save whose new cron or input the mandate does not cover answers 409 MANDATE_SCOPE_MISMATCH, and nothing changes.
At every fire the mandate is checked again. It must be active and unexpired, the epochs of the mandate, its binding and its delegation must be current, and its scopes must still cover the target. The admission then records that mandate in the run’s security context, so the lease renewal check cancels the run when the mandate is revoked or expires (see Revocation and caching). A delegated_background mandate runs as its subject, whose run permission is still checked at each fire. An autonomous mandate has no subject. For a trigger, the webhook’s credential still has to be valid, but it is recorded as the caller and is not the authority.
A refused fire starts nothing, writes no admission and appends a security audit decision (unattended_run.admit, effect deny):
| Entry point | No mandate | Mandate not usable | Out of scope |
|---|---|---|---|
| Pipeline schedule | last_result skipped_mandate_required | skipped_mandate_not_usable | skipped_mandate_scope |
| Inbound trigger | 403, code MANDATE_REQUIRED | 403, MANDATE_NOT_USABLE | n/a |
| Index schedule | Initialization failure whose reason starts with MANDATE_REQUIRED | MANDATE_NOT_USABLE | MANDATE_SCOPE_MISMATCH |
After a rollback. A mandate’s binding pins one agent version and its definition digest. When the agent is rolled back to a release of another version, a fire whose mandate is bound to the current version is refused with MANDATE_RELEASE_ROLLED_BACK (an inbound trigger answers 403 with that code): running the older definition under the newer version’s mandate would let a rollback widen what the mandate approved. Roll forward again, or attach a mandate bound to the release’s version.
last_run is not stamped on a skip. Each attach and detach also appends a security audit decision.
A governed index schedule never gets the project system token: neither the occurrence preflight nor the worker’s client token request issues it. The broker does not yet release the index.ingest capability, so a governed index run that needs the platform’s client token fails closed for now.
Vault / OpenBao secret store
Section titled “Vault / OpenBao secret store”Governed credentials live in the built-in encrypted PostgreSQL vault unless the revision lists their class under vault_required_credential_classes. Those classes need an external HashiCorp Vault or OpenBao KV v2 store, configured only by these variables (read only with the flag on):
| Variable | Value |
|---|---|
ELITEA_SECRETSTORE_VAULT_ADDR | https://host[:port]. Plain http is refused. |
ELITEA_SECRETSTORE_VAULT_CA_FILE | CA bundle that signs the server certificate |
ELITEA_SECRETSTORE_VAULT_AUTH_METHOD | jwt or kubernetes |
ELITEA_SECRETSTORE_VAULT_AUTH_MOUNT | The auth method’s mount path |
ELITEA_SECRETSTORE_VAULT_ROLE_PREFIX | Roles are <prefix>-<workload scope>, one per scope |
ELITEA_SECRETSTORE_VAULT_KV_MOUNT | The KV v2 mount. It must have cas_required on. |
ELITEA_SECRETSTORE_VAULT_TOKEN_FILE | The projected workload token, re-read at every login |
ELITEA_SECRETSTORE_VAULT_NAMESPACE | Vault Enterprise namespace. Leave empty for OpenBao and Vault CE. |
ELITEA_SECRETSTORE_VAULT_READINESS_SCOPE | The workload scope the boot check logs in as |
All empty means “not configured”. Setting any one of them makes all of them required: a half configuration refuses boot. There are no defaults for anything that selects authority.
Security audit chain
Section titled “Security audit chain”Governed events (revisions, bindings, mandates, and later every credential decision) are written to a per-tenant hash chain in the same transaction as the change, then relayed to durable storage. A relay or exporter failure leaves rows pending; none is dropped. A backlog above the revision’s audit_backlog_alert_threshold raises an alert.
Check a chain at any time:
elitea-main security-audit verify --tenant 42 # one tenant projectelitea-main security-audit verify --tenant 0 --source outbox # the platform partition, not yet relayed rowsThe scheduler can delete relayed outbox rows older than a window. SECURITY_AUDIT_OUTBOX_RETENTION_DAYS is "0" by default, which deletes nothing. SECURITY_AUDIT_OUTBOX_RETENTION_INTERVAL (default 1h), ..._BATCH_SIZE and ..._MAX_BATCHES bound each sweep.
Scale (synthetic)
Section titled “Scale (synthetic)”These numbers come from one synthetic run. They hold for the dimensions and the machine below, and for nothing beyond them. They are not a throughput claim for a production deployment.
Dimensions. 300 teams (projects) × 60 calling applications, 1 agent per application, 2 grants per agent: 18,000 agents, bindings, external identities, delegations and mandates, and 36,000 credential references, target bindings, toolkit bindings and grants, under one policy revision. The seed writes bindings, identities, delegations, mandates and grants through the production stores, so the seed adds 126,000 audit rows.
Machine. Apple M4 Pro (14 cores, 48 GB). The test process ran on the host. PostgreSQL 18.6 ran in a podman VM (7 vCPUs, 2 GiB) with stock settings (synchronous_commit=on, 100 connections). NATS 2.12 ran in the same VM. Go 1.26.9.
What is real and what is not. The broker, the AND-composed decider, the admission transaction, the epoch reads, the refresh coordinator and its lease, the provider OAuth token source and the audit relay are the production code, against a real PostgreSQL. These parts are stubs: the RBAC and entity-access gates (always allow), the secret backend (in memory), the identity provider (the fake provider), and the Jira token endpoint (an in-process rotating fake that waits 40 ms per refresh). There is no network hop between the replicas and PostgreSQL other than the VM’s. Each of the 3 “replicas” is its own pool of 16 connections and its own broker in one process.
| Measurement | Load | p50 | p95 | p99 | max |
|---|---|---|---|---|---|
| Admission, legacy (#999 method) | 5 bursts of 100 starts, 40 concurrent, 3 replicas | 29.5 ms | 37.4 ms | 41.2 ms | 47.2 ms |
| Admission, governed (resolve + freeze + pin + mandate bind + security context) | same | 41.6 ms | 61.1 ms | 66.0 ms | 67.6 ms |
| Redeem, spread across 276 teams | 6,000 redemptions, 32 workers, 3 replicas, 1,500 runs (972/s) | 30.9 ms | 43.6 ms | 47.0 ms | 70.2 ms |
| Redeem, first of a run (opens the agent session) | 1,500 of the above | 41.1 ms | 46.6 ms | 60.3 ms | 70.2 ms |
| Redeem, repeat | 4,500 of the above | 29.9 ms | 34.0 ms | 36.0 ms | 43.6 ms |
| Redeem, one team | 3,000 redemptions, 32 workers, 60 runs of one team (407/s) | 63.8 ms | 191.6 ms | 277.5 ms | 381.8 ms |
| Refresh contention | 5 rounds × 60 concurrent redemptions of one expiring grant, 3 replicas | 124.1 ms | 189.5 ms | 207.4 ms | 209.5 ms |
| Revocation to deny, all kinds | 50 trials (10 each: grant, mandate, binding, delegation, credential reference) | 4.2 ms | 5.2 ms | 5.8 ms | 5.8 ms |
| Audit outbox lag (append to durable store) | the redeem loads above, relay at its defaults (1 s, batch 500) | 471 ms | 879 ms | 951 ms | 986 ms |
| NATS invalidation fan-out | 500 epoch changes, 2 hubs over core NATS | 0.21 ms | 0.30 ms | 0.37 ms | 0.51 ms |
What the run also checked:
- Refresh. Every round made exactly one provider refresh call (5 in total) and no
invalid_grant, across 3 replicas with their own coordinators. All 300 contended redemptions were permitted. - Revocation. No redemption that started after the revocation committed was permitted, on a replica other than the writer. The broker re-reads the epochs on every redemption, so the deny does not depend on the invalidation fan-out. Revocation to deny is about one redemption’s duration.
- Audit. The relay drained the seed’s 126,000 rows at about 29,000 rows/s before the load. During the load the backlog peaked at 2,158 rows. The lag is set by the relay interval. Every outbox row reached the durable store with its hash.
- One team. A team’s audit events form one hash chain. Every append takes that chain’s head row until its transaction commits, so a single team’s redemptions serialise on commit time. On this machine that capped one team at about 400 redemptions/s, and the p99 rose to 277 ms. Load spread across teams did not show it.
Reproduce, against a throwaway PostgreSQL 18 and NATS (the CI workflow ci-governed-scale.yml runs the same nightly and on dispatch, and uploads the report):
cd services/elitea-mainELITEA_GOVERNED_SCALE=1 \ELITEA_TEST_DATABASE_URL='postgres://postgres:pw@localhost:55434/postgres?sslmode=disable' \ELITEA_TEST_NATS_URL=nats://127.0.0.1:54222 \go test -v -count=1 -run '^TestGovernedScale$' -timeout 30m ./internal/credentialgovernance/scale/ELITEA_GOVERNED_SCALE_TEAMS, _APPS, _AGENTS and _GRANTS change the dimensions, _REDEEMS the spread redeem count, and ELITEA_GOVERNED_SCALE_REPORT writes the report to a file.
Limits
Section titled “Limits”One team: about 400 governed redemptions per second. Each team (project) has one security audit chain, and every governed decision and redemption locks that chain’s head until its transaction commits. One team’s governed credential use is therefore serial on PostgreSQL commit time. In the scale run above (PostgreSQL 18, stock settings) one team reached about 400 redemptions/s with a p99 of 277 ms, against about 970/s with a p99 of 47 ms when the same load was spread across teams. More Main replicas do not raise the ceiling; faster commits do. Teams do not slow each other down. This is a documented limit, not a defect (#1480).
Watch elitea_security_audit_head_lock_wait_seconds{partition,result}: how long each audit append waited for its chain head. partition is tenant or platform, and there is no tenant label. When its p99 climbs towards the time of a whole redemption, a team is at the ceiling, and elitea_credential_redeem_duration_seconds rises with it.
Turning it off
Section titled “Turning it off”Turning the flag off stops new governed admissions. Runs already pinned stay governed, and their continuations still go out on the governed subjects. The chart turns the worker’s governed lane off with the flag, so set worker.governedContext: "true" until those runs have drained, then remove it. Such a run fails closed at its next credential use (UNIMPLEMENTED) instead of falling back to legacy behaviour. To drain without failures, wait until no governed run is active before turning the flag off. Rolling back to an earlier release is refused by its migration runner when the database holds a migration the release does not know. Turn the flag off instead. The runbook lists the checks.
Conformance
Section titled “Conformance”A conformance suite checks the governed profile end to end on the standalone stack:
deploy/scripts/governed-conformance.sh run # stack up, all phases, canary scan, teardowndeploy/scripts/governed-conformance.sh run --keep # leave the stack up afterwardsIt uses the test-only overlay deploy/docker-compose.governed.yml. The overlay adds OpenBao (TLS, KV v2 with cas_required, JWT auth) and test doubles for Jira, SharePoint/Graph, Entra and the inbound MCP token issuer. No chart references it.
The script takes one stack through five phases in this order:
- Flag off.
- Flag on.
- Flag on, with a worker that has no governed context.
- The governed worker restored.
- Flag off again (rollback).
It runs cases GOV-001 to GOV-010 and the rollback test, and checks every skip against scripts/go/declared-skips.txt.
Every credential the suite creates is a unique canary value. At the end the script scans for those values in:
- The NATS traffic.
- A dump of every database.
- Every container log, including the OpenTelemetry export.
- The audit outbox.
Any match fails the run. The scanner first tests itself on a planted leak.
conformance/governed/EVIDENCE.md maps each acceptance line to a case or unit test, or to the gate that defers it. CI runs the suite nightly, on demand, and on pull requests that change the suite or its stack (ci-governed-conformance.yml).