Day-2 operations

Running the platform after install: upgrading it safely, changing its master key, scaling its four independently-scaled components, and where to look when something breaks.

Upgrades

1. Bump the chart version

   Point your ArgoCD Application (or helm upgrade) at the new published chart version — see GitOps with ArgoCD. The published chart’s image.tag already tracks that version.

2. Let the migration hook run first

   elitea-migrate is a pre-install,pre-upgrade Helm hook. Helm applies and awaits every such hook — including the migration Job — before it touches any of the release’s other resources, so new pods never roll against an un-migrated schema. A failed migration aborts the release, and the previously running pods keep serving; nothing is torn down until the new version proves it can start.

3. helm upgrade

   deploy/helm/elitea/build-deps.sh   # from a clone: the chart's dependencies
   helm upgrade elitea deploy/helm/elitea \
     -f deploy/helm/elitea/values-standalone.yaml \
     --reuse-values \
     --set image.tag=<new-tag>

   Or, under ArgoCD, let selfHeal: true reconcile the bumped targetRevision/parameter automatically.

4. Roll back on failure

   helm rollback elitea <revision> (or the ArgoCD equivalent, argocd app rollback elitea <history-id>) returns to the previous release’s values and image tags. The migration hook runs again on the way there — a rollback that also needs a schema rollback is a database operation this mechanism does not perform for you.

Changing SECRETS_MASTER_KEY

The key has three states, and moving between them is not symmetric:

State	Effect
Set, valid	Every project’s vault key is wrapped with it; rows are Fernet tokens.
Not set	The service still starts, with a log warning — every project key is stored in the clear. Anyone who can read the database can open every project secret.
Set, malformed	The service refuses to start, naming the variable.

Rows written under one key (or under no key) do not become readable when the key changes — there is no in-place re-encryption. To rotate the key: run scripts/rewrap-centry-vault.py against a copy of the database first. It rewraps the project key and never rewrites the secret values themselves, so a rotation that fails partway leaves the plaintext secrets untouched.

Warning

Both elitea-main and elitea-llm-gateway read the same centry.secrets_key table. Give the same value to every service that reads it — two services with two answers would put two row formats in one table, and neither could read what the other wrote.

Scaling

Three components carry a native HorizontalPodAutoscaler; the fourth uses a different mechanism on purpose:

elitea-main

CPU/memory HPA, minReplicas: 2 / maxReplicas: 10 by default. Scaling this past what postgresql.maxConnections was sized for is refused at render time — see Production deployment overview.

elitea-web

CPU/memory HPA. Stateless; a rollout is safe at any time.

llm-gateway

Scales on concurrent /llm streams (gateway_llm_sse_active_connections) through KEDA’s Prometheus trigger or a custom metrics adapter, and falls back to CPU when neither is there. Off by default (llmGateway.autoscaling.enabled). The chart picks the source it can see and refuses one you name that is missing; see Scaling.

agent worker

No native HPA. An agent execution spends most of its life waiting on a model, so CPU/memory never reflects backlog. Queue-depth autoscaling through KEDA is available (worker.autoscaling, off by default) — it scales on the worker durable’s waiting plus held commands on the NATS command bus (num_pending + num_ack_pending, read from the NATS monitoring port). The chart refuses to install a KEDA ScaledObject unless KEDA is actually present, rather than creating an object nothing reads.

Warning

KEDA’s scaler reads /jsz on the bundled NATS’s headless service, port 8222 (plain HTTP, no credential); the NATS NetworkPolicy admits the KEDA operator there (nats.networkPolicy.monitoringFrom). A scaler that cannot connect reports no lag, which scales the fleet down while the backlog grows; the chart cannot detect this for you.

Note

The agent worker’s own replica count stays correct at >1 — the database enforces one active claim per execution regardless (execution_claims_active_idx is unique on (execution_id, generation) while a claim is live) — but scaling out is not free capacity by default. Measure the command bus durable’s delivery rate before raising it or turning KEDA on; a long cooldownSeconds (300s default) is deliberate, because an in-flight agent execution can run for minutes and a short cooldown turns every quiet moment into a round of terminations.

Agent runs dashboard

The chart ships a Grafana dashboard of agent runs, errors, tool calls and spend. Turn it on with main.grafanaDashboard.enabled: true; it is rendered as a ConfigMap labelled grafana_dashboard: "1" for the Grafana sidecar (main.grafanaDashboard.labels and .annotations adjust the labels and folder).

• Prometheus panels (datasource variable): audit export delivery, backlog, lag and dead letters by stream, concurrent LLM streams, gateway refusals, and budget enforcement health. They are platform-wide: no Elitea metric carries an agent label, because an agent id is unbounded cardinality.
• Per-agent panels (pgdatasource variable): runs, model calls, tool calls, retrievals and errors per agent, tool calls per agent and tool, and billed spend per agent. They read the run step records and the billing ledger through a PostgreSQL data source. Give that data source a read-only role with SELECT on elitea_runtime.run_step_heads, elitea_runtime.run_steps and gateway.llm_usage_events only.

Spend per agent covers model calls made by agent runs that produced step records; spend outside agent runs is not attributed to an agent.

Runtime safety alerts

elitea-main serves the runtime safety metrics on its metrics port (ELITEA_AUDIT_EXPORT_METRICS_ADDR, default :9464, port name audit-metrics), also when the audit export is off: elitea_safety_suspensions_active{scope,source}, elitea_breaker_trips_total{rule}, elitea_breaker_storms_total and elitea_breaker_pass_failures_total. Wherever the Prometheus Operator API is available, the chart renders a PodMonitor (only when the audit export’s PodMonitor does not already scrape the port) and a PrometheusRule with these alerts:

Alert	Severity
EliteaPlatformExecutionSuspended	critical
EliteaAgentBreakerStorm	critical
EliteaAgentBreakerTripped	warning
EliteaExecutionSuspensionsActive	warning
EliteaAgentBreakerPassFailing	warning

Turn them off with main.safetyAlerts.podMonitor.enabled and main.safetyAlerts.prometheusRule.enabled; add the labels your Prometheus selects with .labels. main.safetyAlerts.prometheusRule.activeFor (default 30m) sets how long a non-platform suspension stays active before it alerts. Without the Prometheus Operator, scrape the port with your own monitoring and copy the rules from deploy/helm/elitea/templates/main/safety-monitoring.yaml. Route the alerts to your on-call: Elitea does not page a person. See Agent circuit breaker.

Backups

PostgreSQL

Not provisioned or backed up by any chart here — bring your own PostgreSQL operator/managed service and its backup mechanism. The connection budget reserves headroom (reservedConnections: 25) explicitly for backup tooling and interactive psql alongside the application’s own pools, so a backup job does not compete with the application for its last connection slot.

Object storage

Also brought by you — the chart only creates the bucket (optionally, via objectInit) and sets its retention lifecycle. Use your object store’s own versioning/replication for backup.

NATS JetStream

NATS runs inside the elitea release (the bundled upstream NATS chart, its JetStream volume elitea-nats-js-elitea-nats-0), but its state is rebuildable; back up the JetStream KV buckets and stream through NATS’s own tooling if their loss matters to you. Losing the JetStream store resets budget-enforcement counters to zero, but does not lose spend history: the Usage page’s accumulators are written independently by elitea-scheduler’s write-back consumer from the gateway’s billing deltas, not read out of the counter itself.

Restoring from a backup that predates a worker's session row

runtime-session-init (or the Helm workload-session Job) is what authorizes the agent worker’s certificate identity in elitea_runtime.workload_sessions; a database restored from a backup that predates it has no such row. Re-run standalone-stack.sh seed-runtime (compose) or re-trigger the workload-session hook Job (Helm) to recover, rather than assuming a restore alone brings chat back.

CI/CD: what publishes, and how it’s checked

.github/workflows/publish.yml never runs on a merge. It has three channels:

Channel	Started by	Version	Moves latest
release	dispatch from main	semantic-release from the conventional commits since the last stable tag	yes, and <major>.<minor>
nightly	the 02:07 UTC schedule on main, skipped when nothing landed since the last one	<next>-nightly.<yyyymmdd>.<run> (a GitHub pre-release)	no
dev	dispatch from any branch	<next>-dev.<run> (a GitHub pre-release)	no

<next> is the version a release would cut from that ref now, so a pre-release sorts before the release it previews. The newest 3 pre-releases of each channel are kept, with their images and charts, and older ones are deleted; a stable release deletes every pre-release cut before it. Cut a release with gh workflow run publish.yml --ref main -f channel=release. A stable release also deploys the docs site. Every channel also attaches the signed and notarised macOS desktop app (Apple silicon, Elitea_<version>_aarch64.dmg) to its GitHub release.

Release retention

Stable releases are never deleted automatically. A repository admin cuts them with the Release Retention workflow (.github/workflows/release-retention.yml, dispatched from main). It keeps the newest 10 stable releases and the newest 3 pre-releases of each channel cut after the newest stable one, and deletes every other GitHub release, its tag, the desktop app attached to it, and its images and charts with their signatures. It also deletes the untagged image and chart versions older than the oldest kept release. It also deletes the whole container packages you name in retired_packages (space- or comma-separated), which no workflow publishes any more. Each must belong to this repository and must not be a release image or chart, a ci-* image or buildkit; a name that fails this check refuses the run before anything is deleted.

The workflow starts as a dry run that only lists what it would delete. Read that log, then run it again with dry_run off. Anyone who is not an admin is refused, including someone who re-runs an admin’s run. A run that reaches the API rate limit waits for it to reset, and stops before the job times out; run it again and it carries on. Deleted versions cannot be pulled again, so pin every deployment to a kept release first.

gh workflow run release-retention.yml --ref main -f retired_packages='elitea-ui elitea-worker-python'                   # dry run
gh workflow run release-retention.yml --ref main -f retired_packages='elitea-ui elitea-worker-python' -f dry_run=false

For every channel, every first-party image is built, scanned twice (once against the artefact this release actually pushes, in addition to a same-day CI scan of a rebuild), and cosign-signed keylessly before the charts (elitea, which carries NATS and the sandbox-runtime installer, and elitea-sandbox-runtime, still published on its own) are packaged, stamped with that version, pushed to oci://ghcr.io/elitea-ng/charts/<chart>, and signed the same way. The chart job runs strictly after the images it references are pushed, so a chart never reaches the registry pointing at a tag that does not exist yet. If any stage fails, a rollback job deletes everything that stage’s run had already pushed — charts and images alike.

cosign verify ghcr.io/elitea-ng/elitea-main:<tag> \
  --certificate-identity-regexp '^https://github.com/elitea-ng/elitea-platform/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Each image and chart also gets a signed SLSA build provenance attestation, and each platform image a CycloneDX SBOM attestation. Both are pushed to GHCR next to the artifact. The macOS desktop app gets the same for its .dmg and .app.zip. See Verify release artifacts.

Separately, .github/workflows/helm-install-smoke.yml installs the chart into a real kind cluster (not just helm template) on every PR touching deploy/helm/**, and asserts every pod reaches Ready, the DeepWiki/Inventory provider pod’s two containers run the runners the chart set, the facade’s provider registration is healthy over its mTLS hop, and a fixture generation completes end to end — it installs the chart’s shipped defaults (the runtime plane, the bundled Qdrant and elitea-vector, DeepWiki and Inventory), with the worker, the LLM gateway, the web UI and the scheduler off for weight, so it proves the chart installs cleanly, not the full product.

Security posture

Image scanning

Every published image is scanned by Trivy against the exact digest this release pushes (not a same-day rebuild), gated by scripts/ci/image-scan-gate.sh; an exempted finding must be recorded in .github/image-scan-exempt.txt rather than silently passing.

mTLS runtime material

The agent-execution plane’s three listeners (control, output, content) all require and verify a client certificate; there is no cleartext mode. The runtime and auth material both arrive as Kubernetes Secrets, copied by an init container into an owner-only emptyDir because a plain Secret volume cannot satisfy the permission profile the code enforces.

Identity headers stop at elitea-main

elitea-main deletes caller-supplied X-Auth-* and X-Elitea-* identity headers itself, before any route runs, and reads no X-Auth-* header as an identity, so a plain Ingress that forwards them is safe. The chart’s HTTPRoute removes the same names again (main.ingress.stripIdentityHeaders) as defence in depth.

GATEWAY_IDENTITY_SECRET

Required, not optional, on both elitea-main and the gateway, and the two must agree — see Install with Helm.

Troubleshooting

Operator-facing infrastructure checks (/healthz, /readyz, the application-level Health tab, common agent/pipeline/toolkit errors) live on one page: see Troubleshooting. This page covers the deployment mechanics around those checks; that one covers what each check actually reports.