Scaling
Every Elitea workload either has an autoscaler or is a singleton for a stated reason. All the autoscalers are on by default. The rule behind them: an autoscaler scales on its best signal when that signal’s controller and source are present, and on CPU when they are not. An autoscaler that reads a metric nothing publishes holds the fleet at its floor while the load grows, and reports nothing. So each source below is detected from the cluster, installed by the release, or stated by you. A source you name but do not have is refused at render time, with the reason.
What scales on what
Section titled “What scales on what”| Component | Autoscaler | Scales on | Needs | Default range |
|---|---|---|---|---|
| elitea-main | HPA | CPU (70 %) and memory (80 %) | metrics-server | 2 to 8 |
| elitea-web | HPA | CPU (70 %) | metrics-server | 2 to 6 |
| elitea-vector | HPA | CPU (70 %) | metrics-server | 2 to 6 |
| LLM gateway | KEDA prometheus + CPU | sum(gateway_llm_sse_active_connections) in the namespace, 40 streams per replica | KEDA, Prometheus scraping :8083/metrics | 2 to 4; the ceiling divides the degraded-mode budget cap (see The gateway’s expected replicas) |
| HPA, Pods metric | gateway_llm_sse_active_connections per pod, through custom.metrics.k8s.io | a custom metrics adapter | ||
| HPA | CPU (70 %) | metrics-server | ||
| Agent worker | KEDA nats-jetstream + CPU | consumer lag on the agent command stream; one replica per delivery_max_concurrency (32) commands | KEDA | 1 to 10 |
| HPA | CPU (70 %) | metrics-server | ||
| DeepWiki | KEDA prometheus + CPU | running invocations plus the generations refused fleet-full in the last 5 minutes, one replica per ELITEA_DEEPWIKI_MAX_PARALLEL_WORKERS | KEDA, Prometheus scraping the host’s admin port | 1 to 3 |
KEDA postgresql + CPU | the same, counted in the invocations table | KEDA, the host’s database | ||
| HPA | CPU (70 %) | metrics-server | ||
| Inventory | as DeepWiki | running ingestions, counted in inventory_graph.invocations | as DeepWiki; the host’s database is required for more than one replica | 1 to 3 |
| OTel collector | HPA | CPU (70 %) | metrics-server | 1 to 3 |
| Egress proxy (off by default) | HPA | CPU (70 %) | metrics-server | 2 to 6 |
| Scheduler | none: an active/active pair | scheduler.replicaCount 1 or 2 | 2 | |
| NATS, Qdrant | none: stateful | fixed size; see High availability | 1 | |
| PgBouncer (off by default) | none: a singleton | 1 | ||
| Sandbox supervisor (off by default) | none: a singleton | 1 |
source: auto (the default for the gateway, the worker, DeepWiki and Inventory) picks the first source that is available, in the order of the table. Name a source in <component>.autoscaling.source to require it: a named source that is not available is refused, with the reason.
Every KEDA ScaledObject also has a CPU trigger (targetCPUUtilizationPercentage, 70 % by default). The HPA that KEDA makes takes the larger of the two, so the component still scales when Prometheus or the database cannot be read.
A CPU HPA without metrics-server reports FailedGetResourceMetric and holds the Deployment at minReplicas. That is the old fixed count, not an outage. Turn an autoscaler off with <component>.autoscaling.enabled: false; the Deployment then runs replicaCount.
While an autoscaler owns a Deployment, the chart renders no replicas: field on it. A value there would be written back by every helm upgrade and every Argo CD sync, snap the fleet to replicaCount, and leave the autoscaler to scale it out again.
DeepWiki and Inventory
Section titled “DeepWiki and Inventory”An invocation runs on the replica that accepted it, from start to end. Its row is in a shared invocation store in the engine’s database (DeepWiki: invocations; Inventory: inventory_graph.invocations), so any replica answers a poll, a cancel or a deletion:
- A cancel goes to the owner replica over NATS. The owner also reads the stop requests on its own rows at each claim check (10 s), so a cancel arrives without NATS too.
- A deletion stops the matching runs on every replica, and waits until they end. While it runs, a stop fence (
host_stop_fences) refuses a new run that matches it on every replica, so a generation that starts during the deletion cannot publish the index again. - When a replica stops, another replica marks the runs it left behind as terminal.
- Generation slots are fleet-wide. A replica writes its slot count (
ELITEA_DEEPWIKI_MAX_PARALLEL_WORKERS) inhost_replicas, and a heartbeat apart from the claim check keeps the row live. A generation is admitted while the running generations are fewer than the slots of all live replicas that admit work. A replica whose own slots are full still admits while the fleet has a free slot (it runs above its own count; the fleet total is the bound), so a call thatround_robinsends to a busy replica is not refused while a sibling is idle. A replica that drains, or that is not live, leaves the count with its slots and its running generations together. A call is never queued: when no slot is free, the call is accepted and ends at once withservice_busy(“All N generation slots are busy”), and the UI shows that message. The slots page reports the same fleet numbers. - The setting is per replica, so the fleet’s capacity grows with the fleet: fleet slots =
ELITEA_DEEPWIKI_MAX_PARALLEL_WORKERS× live replicas that admit work. The default is 1 slot per replica and 1 to 3 replicas, so 1 to 3 generations at once. One slot per replica is deliberate: a generation holds a whole engine (memory, CPU and 4 database connections), and a second generation on the same pod slows both. To run more generations at once, raisedeepwiki.autoscaling.maxReplicas(capacity added on demand), raiseminReplicas(capacity always there), or raise the slots per replica together with the engine’s memory and the connection budget. - With
source: cpu(no KEDA), a refused generation does not make the fleet grow: only CPU does. If your users start generations at the same time, setminReplicasto the number of generations you expect at once, or install KEDA so the refused demand counts.
The generations refused because the whole fleet was full are part of the scaling signal: elitea_subapp_slot_refusals_total{reason="fleet_full"} on the admin port, or the rows whose result has busy_reason fleet_full. Without them the running count could never be above the slots, and the fleet could not grow. The other reasons (draining, and local_full on a one-replica install without a database) are not demand and are not counted.
More than one replica needs the host’s database (postgresql.existingSecret, or deepwiki.secrets.ELITEA_DEEPWIKI_DATABASE_URL and inventory.secrets.ELITEA_INVENTORY_DATABASE_URL). For Inventory, the host also needs the migrate Job that creates its tables (the host runner sidecar and inventory.migrate.enabled: true); without the Job the host gets no database URL. Without a store, the chart refuses maxReplicas above 1: each replica would keep its invocations in memory. With a fixture runner and no database, set deepwiki.autoscaling.enabled: false or inventory.autoscaling.enabled: false.
Singletons and their high-availability story
Section titled “Singletons and their high-availability story”| Workload | Why one | Availability |
|---|---|---|
| Scheduler | Not CPU-bound. Every job is safe on two replicas: singleton work takes an advisory lock, the retention sweep uses SKIP LOCKED, the write-back consumer deduplicates. A third replica adds only connections, so the chart refuses more than 2. | Two replicas by default, a PDB and spread: one keeps working while the other moves. |
| NATS | Stateful (JetStream). | values-nats-ha.yaml: three nodes, streams replicated three times. |
| Qdrant | Stateful (the vectors). | values-qdrant-ha.yaml: a three-node cluster. |
| PgBouncer | Its pool is counted once in the connection budget; a second replica would double it. | A restart takes seconds; elitea-main retries its connections. |
| Sandbox supervisor | It writes a fixed owner identity in the sandbox ledger and keeps its live runs in memory. Two would fight over the same runs. Recreate strategy. | A restart reconciles the ledger; runs in flight end as failed and are started again. |
| Sandbox image warmup | A DaemonSet: one pod per node by design. |
How gRPC load spreads
Section titled “How gRPC load spreads”A gRPC client keeps one HTTP/2 connection and sends every call on it. Behind a ClusterIP Service that is one replica for the life of the connection, whatever the autoscaler adds. So the hops that carry the load use client-side round_robin over a headless Service (one connection to each replica), and each server closes connections after a maximum age, so the clients reconnect and see new replicas:
| Hop | Service | Server connection age |
|---|---|---|
| elitea-main → LLM gateway | headless elitea-llm-gateway-grpc, round_robin | 5 min |
| elitea-main → DeepWiki | headless elitea-deepwiki-grpc, round_robin | 1 min |
| elitea-main → Inventory | headless elitea-inventory-grpc, round_robin | 1 min |
| elitea-main → elitea-vector | headless elitea-vector-grpc, round_robin | ELITEA_VECTOR_MAX_CONNECTION_AGE_SECONDS, 300 s |
| the worker, the engines → elitea-vector | ClusterIP elitea-vector | ELITEA_VECTOR_MAX_CONNECTION_AGE_SECONDS, 300 s |
At its maximum age a connection closes gracefully: the calls on it finish, and the next call opens a new connection. The worker and the engines are Rust clients that dial the ClusterIP Service of elitea-vector: each new connection goes to a replica that kube-proxy selects, so their load spreads over the replicas within a few connection ages. The certificates do not change: elitea-main still checks the ClusterIP Service name (ELITEA_DEEPWIKI_SERVER_NAME, ELITEA_INVENTORY_SERVER_NAME, ELITEA_VECTOR_GRPC_SERVER_NAME).
Preconditions: KEDA and a custom metrics adapter
Section titled “Preconditions: KEDA and a custom metrics adapter”KEDA and a custom metrics adapter are cluster-wide: CRDs, plus an APIService each (external.metrics.k8s.io for KEDA, custom.metrics.k8s.io for the adapter). A cluster runs one of each. Two of them fight over the same cluster-scoped objects, and the scalers of whichever loses read nothing.
| You have | Set |
|---|---|
| KEDA already in the cluster | nothing on helm install (the chart sees keda.sh/v1alpha1); scaling.keda.clusterInstalled: true for helm template and for Argo CD |
| No KEDA, and you want the release to bring it | keda.enabled: true (the pinned kedacore chart, 2.21.0) |
| A custom metrics adapter already | scaling.customMetrics.clusterInstalled: true where the chart cannot see it, and a rule for gateway_llm_sse_active_connections in that adapter |
| No adapter, and the gateway should use one | prometheusAdapter.enabled: true and prometheusAdapter.prometheus.url (the pinned prometheus-community chart, 5.3.0; its rules expose exactly the gateway metric and nothing else) |
Prometheus for the KEDA prometheus triggers | scaling.prometheus.serverAddress, plus a scrape: a PodMonitor (rendered by default wherever the Prometheus Operator’s API is served), <component>.metrics.scrapeAnnotations: true, or <component>.metrics.scrapedExternally: true for a scrape config of your own |
helm install and helm upgrade refuse keda.enabled or prometheusAdapter.enabled when they can see one that this release does not own. helm template and Argo CD cannot look into the cluster, so the install notes warn instead. Enable the bundled ones only on a cluster that has none.
Without KEDA. The defaults need no KEDA. With worker.autoscaling.enabled and source: auto, the worker falls back to a CPU HorizontalPodAutoscaler when no KEDA is served, bundled or declared, and the install notes (helm get notes) say so. CPU rises later than the backlog, because a turn waits on a model. A component with its autoscaler off runs its fixed replicaCount. Set scaling.keda.clusterInstalled: true only where KEDA runs: it makes the chart render ScaledObjects in place of the CPU fallback.
The Prometheus labels
Section titled “The Prometheus labels”The generated prometheus queries select the release’s namespace by a label. Its name is scaling.prometheus.namespaceLabel:
| Your scrape | Set |
|---|---|
| A Prometheus Operator PodMonitor (the default) | nothing: the label is namespace |
An annotation-based scrape config that writes kubernetes_namespace | scaling.prometheus.namespaceLabel: kubernetes_namespace |
| One Prometheus for one release, no namespace label | scaling.prometheus.namespaceLabel: "" (the queries do not filter on the namespace) |
| One Prometheus for more than one cluster | scaling.prometheus.extraMatchers: ['cluster="prod-eu"']: each matcher is added to every generated query |
A value that is not a Prometheus label name is refused. <component>.autoscaling.prometheus.query replaces a generated query completely. The bundled Prometheus Adapter’s rule maps the namespace and pod labels (prometheusAdapter.rules.custom[0].resources.overrides); change it to the same names.
The first install with the bundled KEDA
Section titled “The first install with the bundled KEDA”KEDA’s chart ships its CRDs as templates, so on the first helm install a ScaledObject in the same release has no API to map to yet, and Helm refuses the whole install. The chart avoids that. While the cluster does not serve keda.sh/v1alpha1 and this release installs KEDA, each ScaledObject and TriggerAuthentication renders as a post-install hook that carries the release’s ownership metadata. The next helm upgrade renders them as ordinary resources and Helm adopts the existing objects. --wait is not needed: the hook only creates the objects, and KEDA marks them Ready once its operator is up. The kind smoke in CI proves this path on every chart change: a ScaledObject created on the first install must turn Ready=True.
For Argo CD, every ScaledObject and TriggerAuthentication carries argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true. On the first sync the dry run skips them, and Argo CD applies the CRDs before them.
Uninstalling a release that bundles KEDA
Section titled “Uninstalling a release that bundles KEDA”helm uninstall removes KEDA’s operator together with its CRDs. A ScaledObject whose finalizer never ran then keeps the CRD, and the namespace, in Terminating. So with keda.enabled, the chart has a pre-delete hook (elitea-keda-cleanup, a Job in the elitea-main image). It runs before Helm deletes anything:
- It deletes the release’s ScaledObjects and TriggerAuthentications (
app.kubernetes.io/instance=<release>). - It waits up to
kedaCleanup.timeoutSeconds(120) for KEDA’s operator, which is still running, to remove their finalizers. - If the operator does not, it removes the finalizers itself.
The hook has a Role for exactly that: list, deletecollection and patch on those two resources in the release’s namespace.
With a KEDA of the cluster’s own (scaling.keda.clusterInstalled), no hook renders: that operator outlives the release and runs the finalizers.
The PostgreSQL connection budget
Section titled “The PostgreSQL connection budget”Every pool size is explicit, and the chart adds them up at each component’s replica ceiling: autoscaling.maxReplicas while an autoscaler owns the Deployment, otherwise replicaCount. The question is what the release can open without anybody doing anything. The render is refused when
Σ (replica ceiling × connections per replica) ≤ postgresql.maxConnections − postgresql.reservedConnections
does not hold. The refusal lists every component’s share. Exhaustion does not show up on the component that caused it: the next one to connect, usually a migration Job or the scheduler, fails with sorry, too many clients already.
| Component | Connections per replica | Shipped default |
|---|---|---|
| elitea-main | ELITEA_DATABASE_MAX_CONNS + the six runtime-plane pools (36) + elitea-vector’s introspection pool (2) + 4 per distinct AgentState receipt DSN | 6 + 36 + 2 = 44 |
| elitea-main behind PgBouncer | pgbouncer.poolSize + reservePoolSize once, plus the AgentState receipt pools per replica | |
| LLM gateway | GATEWAY_DATABASE_MAX_CONNS | 8 |
| Scheduler | ELITEA_DATABASE_MAX_CONNS | 4 |
| DeepWiki host | ELITEA_DEEPWIKI_DATABASE_MAX_CONNS + 1 (the owner claim) | 4 + 1 |
| DeepWiki engine | 2 + 2 + ELITEA_DEEPWIKI_QUERY_POOL_SIZE + 4 × ELITEA_DEEPWIKI_MAX_PARALLEL_WORKERS | 2 + 2 + 8 + 4 = 16 |
| Inventory host | ELITEA_INVENTORY_DATABASE_MAX_CONNS + 1 (the owner claim) | 2 + 1 |
| Inventory engine | ELITEA_INVENTORY_POOL_SIZE | 8 |
| Worker | limits.agentstate_max_connections AgentState connections, when the agent-state database is on this server (postgresql.agentStateSharesServer) | 8 |
The worker’s pool is not delivery_max_concurrency. A turn holds an AgentState connection only for a short transaction, never across a model or tool call, so 32 turns in flight share 8 connections. A turn that finds none free waits up to limits.agentstate_acquire_timeout_millis (30 000). agentstate_max_connections above delivery_max_concurrency is refused: the extra connections could never be used.
reservedConnections (25) covers what the chart does not model: hook Jobs, an interactive psql, backups. A pool left unset is refused rather than guessed: each service then falls back to pgx’s max(4, numCPU), which depends on the node the pod lands on.
With the shipped defaults, every autoscaler at its ceiling: main 8 × 44 = 352, gateway 4 × 8 = 32, scheduler 2 × 4 = 8, DeepWiki host 3 × 5 = 15, DeepWiki engine 3 × 16 = 48, Inventory host 3 × 3 = 9, Inventory engine 3 × 8 = 24, worker 10 × 8 = 80. That totals 568. postgresql.maxConnections defaults to 600, so 575 are available. To raise a ceiling, size the server with it, put elitea-main behind PgBouncer, or move the agent-state database to another server (postgresql.agentStateSharesServer: false).
values-standalone.yaml states a server of 420 (the same components with elitea-main at 4: 392 + 25 reserved).
values-gke.yaml sizes for a Cloud SQL instance with 4 vCPU and 16 GB. Cloud SQL sets max_connections from memory, and from 15 GB up to 30 GB the value is 500. The arithmetic, also in the file: main 3 × 44 = 132, gateway 3 × 8 = 24, scheduler 2 × 4 = 8, DeepWiki host 3 × 5 = 15, DeepWiki engine 3 × 16 = 48, Inventory host 3 × 3 = 9, Inventory engine 3 × 8 = 24, worker 4 × 8 = 32. That totals 292 of the 475 available. If you set the flag yourself, state that number in postgresql.maxConnections.
Per-replica capacity
Section titled “Per-replica capacity”| Component | Cap per replica | Setting |
|---|---|---|
| elitea-main | 256 concurrent SSE streams, 16 per principal, 128 per project | main.runtime.sse.* |
| LLM gateway | 50 concurrent provider calls; the autoscaler targets 40 streams | GATEWAY_PROVIDER_CONCURRENCY, llmGateway.autoscaling.targetSSEConnectionsPerPod |
| Agent worker | 32 commands in flight | worker.runtime.limits.delivery_max_concurrency |
| DeepWiki | 1 generation at a time; the fleet admits the sum over its replicas | ELITEA_DEEPWIKI_MAX_PARALLEL_WORKERS |
| Inventory | no slot limit; ingestions of one graph run one at a time (an engine lock) | ELITEA_INVENTORY_POOL_SIZE (its database pool) |
The fleet’s capacity is the cap times the replica ceiling. 1000 concurrent agent flows need about 1000 live streams: four elitea-main replicas at 256 each.
High availability
Section titled “High availability”The defaults run NATS and Qdrant as single nodes. Two overlays make them highly available:
values-nats-ha.yaml: three NATS nodes, the streams and key-value buckets replicated three times (natsBootstrap.replicas: 3), and routes over mutual TLS.values-qdrant-ha.yaml: a three-node Qdrant cluster with peer TLS. Raisevector.collection.replicationFactoralong with it.
helm upgrade --install elitea oci://ghcr.io/elitea-ng/charts/elitea -n elitea \ -f your-values.yaml -f values-nats-ha.yaml -f values-qdrant-ha.yamlDisruption budgets and spread
Section titled “Disruption budgets and spread”Each workload that can run more than one replica gets two soft topology spread constraints, over zones and then over nodes (whenUnsatisfiable: ScheduleAnyway). Soft on purpose: a single-zone or single-node cluster still schedules every replica. <component>.topologySpreadConstraints replaces them, and scaling.topologySpread.enabled: false turns them off everywhere.
A PodDisruptionBudget (minAvailable: 1) is rendered only where the component’s floor is above one replica: autoscaling.minReplicas, else replicaCount. Over a single replica it would block every voluntary eviction for good: kubectl drain, a node-pool upgrade, the cluster autoscaler. With the defaults, elitea-main, elitea-web, elitea-vector, the LLM gateway and the scheduler have one. DeepWiki, Inventory, the worker and the OTel collector get one when you raise their floor to two.
Drain and grace periods
Section titled “Drain and grace periods”Kubernetes starts terminationGracePeriodSeconds when it runs the preStop hook, so the hook, the drain and what follows all spend the same clock. Past it, the kubelet sends SIGKILL in the middle of the drain. The chart refuses a grace period that is too short:
| Component | preStop | Drain | Grace (default) | Rule |
|---|---|---|---|---|
| elitea-main | sleep 10 | ELITEA_SHUTDOWN_TIMEOUT 30s | 50 | ≥ 10 + drain + 10 |
| LLM gateway | sleep 10 | GATEWAY_SHUTDOWN_TIMEOUT 150s | 190 | ≥ 10 + drain + 30 |
| DeepWiki, Inventory | sleep 5 (the engine is a native sidecar and stops after the host) | *_DRAIN_TIMEOUT_SECONDS 3600 | 3630 | ≥ preStop + 10 + drain + 10 |
| elitea-vector | sleep 5 | the calls it holds, up to 20 s | 30 | ≥ preStop + 20 |
| OTel collector | sleep 5 | the batch flush, up to 10 s | 30 | ≥ preStop + 10 |
| Egress proxy | drainDelaySeconds 5 + shutdownTimeoutSeconds 60 | 80 | ≥ delay + timeout + 5 | |
| Agent worker | limits.shutdown_timeout_millis 30 000 | 45 | ≥ drain + 15 |
The preStop hook is the native sleep action, which needs Kubernetes 1.30 or newer; the images are distroless and have no sleep binary. It gives the endpoint controllers, the DNS of the headless Services and the load balancer time to take the pod out of rotation before the process gets SIGTERM. At its deadline elitea-main closes the SSE streams that are still open, and the browser reconnects and resumes on another replica.
Why DeepWiki and Inventory drain for an hour
Section titled “Why DeepWiki and Inventory drain for an hour”A wiki generation or an ingestion runs for minutes, and sometimes for most of an hour. When the autoscaler removes a replica, or a rollout replaces it, the replica finishes its work:
- On SIGTERM the host stops admitting. Its slots leave the fleet’s total at once, so a new generation goes to another replica or is refused busy.
- The host waits up to the drain timeout for its running invocations. Polls and cancels for them reach any replica through the shared store.
- The engine is a native sidecar (an init container with
restartPolicy: Always). The kubelet sends it SIGTERM only after the host container has exited, so every engine call of a draining invocation finds the engine serving. Then the engine closes its own open streams and exits. In compose, the hostdepends_onthe engine, sodocker compose stopalso stops the host first. - At the deadline, what is left is cancelled and recorded as terminal.
The other option, to make a build resumable on another replica, needs the request and its git credentials kept in the store, and the work done so far is not reusable: a generation is staged and published in one step, so a cut-off build leaves no partial wiki. A drain long enough to finish is simpler, and loses nothing.
The gateway’s expected replicas
Section titled “The gateway’s expected replicas”While NATS is down, the gateway counts spend in each replica and cannot see the other replicas. The degraded-mode budget cap is the bound for the whole fleet, so each replica gets one share of it:
cap = LLM_BUDGET_NATS_DEGRADED_CAP_USD, or 10% of the budget's hard limit when that is 0per-replica share = floor(cap / LLM_BUDGET_EXPECTED_REPLICAS)reservation = LLM_BUDGET_NATS_DEGRADED_RESERVATION_USD (default 0.10 USD)admit a request only if billed + reserved + reservation ≤ share (one atomic step)fleet total ≤ cap + the cost above its reservation of each request in flight when the shares fillA request’s cost is known only after it runs. So each request admitted while NATS is down reserves a fixed amount against its replica’s share, and gives it back after its real cost is billed. The reservations on one replica never total more than its share, so concurrent requests cannot all pass against the same spend. Set the reservation at or above the cost of your largest request, and the fleet total is at most the cap. A request that costs more than its reservation adds the difference. A reservation goes back only after the request’s cost is billed: a client that disconnects does not free it early.
A share smaller than the reservation admits nothing, and that replica refuses every request for the budget until NATS returns. With the default cap (10% of the hard limit) and the default reservation, a replica admits while the hard limit is at least N USD, where N is LLM_BUDGET_EXPECTED_REPLICAS (4 USD with the chart default of 4). Keep the reservation at or below cap ÷ N.
The share rounds down to the nano-USD, so the shares together never go above the cap. A replica exactly at its share refuses. A cap smaller than the replica count, in nano-USD, gives a share of 0: that replica then refuses every request for the budget until NATS returns. Below the soft-alert threshold this cap applies; above it, each replica gets the same share of the budget that remains. More replicas give a smaller share each. To keep a per-replica share as you add replicas, raise llmGateway.env.LLM_BUDGET_NATS_DEGRADED_CAP_USD with the replica count.
A replica that starts during an outage gets no share: it cannot reach NATS, so it refuses every spend request with 503 until NATS returns. So the replicas that were running when the outage began are the only ones that spend. A live realtime session keeps its reservation until it ends. When other spend fills the share, the session’s next budget check refuses, and the session closes, as it does for any exhausted budget.
The gateway refuses to start when LLM_BUDGET_EXPECTED_REPLICAS is set to anything but a whole number of 1 or more, written with digits only (+3, 007, 0, an empty value, spaces and text are refused; unset it for 1). If that number is below the real fleet, every replica spends a share meant for fewer, and the fleet overspends while it is degraded. The chart derives it from the gateway’s replica ceiling: autoscaling.maxReplicas while autoscaling (4 by default), else replicaCount. Leave it unset. An explicit value below the ceiling is refused, and one above it is accepted. While fewer replicas run than the ceiling, the fleet spends less than the cap in degraded mode, never more.