> For the complete documentation index, see [llms.txt](https://docs.labs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.labs.ai/deployment-and-infrastructure/kubernetes.md).

# Kubernetes

EDDI runs natively on Kubernetes. This guide covers deployment options from a simple quickstart to production-grade configurations.

## Prerequisites

* **Kubernetes cluster** (1.26+) — minikube, kind, GKE, EKS, AKS, or any conformant cluster
* **kubectl** configured to access your cluster
* **Helm 3** (optional, for Helm chart deployment)

## Quick Start (5 minutes)

### Option A: Single-file manifest

Deploy EDDI + MongoDB with one command:

```bash
kubectl apply -f https://raw.githubusercontent.com/labsai/EDDI/main/k8s/quickstart.yaml
```

The EDDI pod now waits in `ContainerCreating` — it mounts the `eddi-secrets` Secret, which no manifest creates. That is deliberate: a Secret shipped in the manifest would be re-applied on every `kubectl apply` and would overwrite your vault master key, making everything already encrypted with it undecryptable. Generate and store the key, and the pod starts by itself:

```bash
set -euo pipefail

# EDDI reads its secrets from a mounted properties FILE rather than environment
# variables, so the Secret holds exactly one key:
# "application-secrets.properties".
#
# mktemp gives an unpredictable name created 0600, so it cannot be pre-created
# or symlinked by another local user.
secrets_file=$(mktemp "${TMPDIR:-/tmp}/eddi-secrets.XXXXXX")
trap 'shred -u "$secrets_file" 2>/dev/null || rm -f "$secrets_file"' EXIT

# Fail closed. This block mints a NEW key every time it runs, so re-running it
# over a live eddi-secrets — after a failed later step, or in a second session
# — would leave everything already encrypted under the old key permanently
# undecryptable. Checked before the key is generated, and `kubectl create`
# below is deliberately NOT piped through `--dry-run=client | kubectl apply`,
# which would replace the Secret silently and exit 0.
if kubectl get secret eddi-secrets --namespace=eddi >/dev/null 2>&1; then
  echo "eddi-secrets already exists — this would install a NEW master key and" >&2
  echo "make everything encrypted under the current one unrecoverable." >&2
  echo "To rotate deliberately: bash k8s/create-secrets.sh --force" >&2
  exit 1
fi

key=$(openssl rand -base64 24)
[ -n "$key" ] || { echo "vault key generation failed" >&2; exit 1; }
printf 'eddi.vault.master-key=%s\n' "$key" > "$secrets_file"

# The key name must be application-secrets.properties — that is the filename the
# Deployment mounts. Passing the temp path bare would name the key after it.
kubectl create secret generic eddi-secrets \
  --namespace=eddi \
  --from-file=application-secrets.properties="$secrets_file"

# Only needed if EDDI was already running with a different key
# kubectl rollout restart deployment/eddi -n eddi

# Access EDDI
kubectl port-forward svc/eddi 7070:7070 -n eddi
```

Open <http://localhost:7070>.

### Option B: Using the helper script

The Secret must exist **before** the first apply — `kubectl apply -k` never creates or reconciles it, so the pod cannot mount it otherwise:

```bash
# Clone the repo
git clone https://github.com/labsai/EDDI.git && cd EDDI

# Generate vault key + create K8s secret
bash k8s/create-secrets.sh

# MongoDB runs with authentication — create its Secret too: the root user for
# mongod, EDDI's own least-privilege user (readWrite + dbAdmin on `eddi`, which
# the ensure-mongodb-user init container creates) and the connection string EDDI
# mounts as a file — for that user, never root. The Secret IS the stored copy of
# the root password: run this once, never on an upgrade.
# A fresh password over an initialised database locks EDDI out — mongod keeps
# the user it created at first start. The guard refuses when one exists.
if kubectl get secret mongodb-secrets -n eddi >/dev/null 2>&1; then
  echo "mongodb-secrets already exists — keeping it" >&2
else
pw=$(openssl rand -hex 24)
apppw=$(openssl rand -hex 24)
kubectl create secret generic mongodb-secrets -n eddi \
  --from-literal=MONGO_INITDB_ROOT_USERNAME=eddi \
  --from-literal=MONGO_INITDB_ROOT_PASSWORD="$pw" \
  --from-literal=MONGO_APP_USERNAME=eddi_app \
  --from-literal=MONGO_APP_PASSWORD="$apppw" \
  --from-literal=mongodb-secrets.properties="mongodb.connectionString=mongodb://eddi_app:${apppw}@mongodb:27017/eddi?authSource=eddi&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=30000"
fi

# Deploy with MongoDB
kubectl apply -k k8s/overlays/mongodb/
```

PowerShell — **PowerShell 7 (`pwsh`)**, not the Windows PowerShell 5.1 that `powershell.exe` starts. The script declares `#Requires -Version 7` and refuses to run under 5.1, whose .NET Framework has no `RandomNumberGenerator.Fill` to generate a key with. [Install it](https://learn.microsoft.com/powershell/scripting/install/installing-powershell-on-windows) or use `winget install Microsoft.PowerShell`.

```powershell
pwsh -File .\k8s\create-secrets.ps1
# Once, never on an upgrade — see the bash note above. kubectl create refuses
# with AlreadyExists when the Secret is there.
$pw = [Security.Cryptography.RandomNumberGenerator]::GetHexString(48, $true)
$apppw = [Security.Cryptography.RandomNumberGenerator]::GetHexString(48, $true)
kubectl create secret generic mongodb-secrets -n eddi `
  --from-literal=MONGO_INITDB_ROOT_USERNAME=eddi `
  --from-literal=MONGO_INITDB_ROOT_PASSWORD="$pw" `
  --from-literal=MONGO_APP_USERNAME=eddi_app `
  --from-literal=MONGO_APP_PASSWORD="$apppw" `
  --from-literal=mongodb-secrets.properties="mongodb.connectionString=mongodb://eddi_app:${apppw}@mongodb:27017/eddi?authSource=eddi&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=30000"
kubectl apply -k k8s\overlays\mongodb\
```

The script generates a **new** master key, so it stops if `eddi-secrets` already exists rather than replacing what is there — re-running it would leave every secret encrypted under the old key unrecoverable. Pass `--force` (`-Force` in PowerShell) only when you mean to rotate.

### Option C: Helm

```bash
helm install eddi ./helm/eddi \
  --set eddi.vaultMasterKey="$(openssl rand -base64 24)" \
  --set mongodb.rootPassword="$(openssl rand -base64 24)" \
  --set eddi.security.allowUnauthenticatedMcp=true \
  --set eddi.security.allowUnauthenticatedSecretStore=true \
  --namespace eddi --create-namespace
```

This is the local, port-forward-only shape: no OIDC, so the chart refuses to render until the MCP server and the secrets vault are opted in to unauthenticated access explicitly — without both, EDDI's `HighValueSurfaceGuard` would refuse to boot. For anything others can reach, enable OIDC instead (`eddi.oidc.enabled=true` with `keycloak.enabled=true` or `eddi.oidc.authServerUrl`) and drop the two opt-ins. Keep the generated `mongodb.rootPassword`: MongoDB only reads it when its volume is first initialised.

#### Upgrading to an authenticated MongoDB

Releases from before MongoDB authentication have a data volume that was initialised without a user. The mongo image creates `mongodb.rootUsername` only on an **empty** volume, but turns `--auth` on regardless — so a plain upgrade leaves MongoDB demanding a user that does not exist, and EDDI loses its database. A live `helm upgrade` detects this and refuses to render. Create the user first, in the still-unauthenticated database, entering the password you will pass as `mongodb.rootPassword` at the prompt:

```bash
kubectl exec -it -n eddi eddi-mongodb-0 -- mongosh admin --quiet \
  --eval 'db.createUser({user: "eddi", pwd: passwordPrompt(), roles: ["root"]})'

# Release WITH OIDC (eddi.oidc.enabled=true):
helm upgrade eddi ./helm/eddi --namespace eddi --reuse-values \
  --set mongodb.rootPassword='<the same password>' \
  --set mongodb.authMigrated=true

# Release WITHOUT OIDC: --reuse-values cannot carry values the old chart never
# had, so the two high-value opt-ins must be added here too, or the chart
# refuses to render (see Option C).
helm upgrade eddi ./helm/eddi --namespace eddi --reuse-values \
  --set mongodb.rootPassword='<the same password>' \
  --set mongodb.authMigrated=true \
  --set eddi.security.allowUnauthenticatedMcp=true \
  --set eddi.security.allowUnauthenticatedSecretStore=true
```

(`eddi-mongodb-0` and `user: "eddi"` assume release `eddi` and the default `mongodb.rootUsername`; the render error prints the exact names for yours.) Existing data is untouched. Leave the opt-ins off an OIDC release: OIDC already satisfies the guard there, and the opt-ins are escape hatches, not settings to carry by default.

## Deployment Options

EDDI provides modular overlays (Kustomize) and Helm values for different deployment profiles:

### Database Backend

| Backend               | Kustomize                                                                                                        | Helm                                                                                        |
| --------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| **MongoDB** (default) | `kubectl apply -k k8s/overlays/mongodb/` (create `mongodb-secrets` first)                                        | `--set mongodb.enabled=true`                                                                |
| **PostgreSQL**        | `kubectl apply -k k8s/overlays/postgres/` (create `postgres-secrets` first — see `postgres-secret.yaml.example`) | `--set postgres.enabled=true --set mongodb.enabled=false --set eddi.datastoreType=postgres` |

### Optional Components

Everything under `k8s/overlays/` except `mongodb/` and `postgres/` is a kustomize **Component**. Components have no resource set of their own — `kubectl apply -k` on one does not work by design — and are composed with a database overlay.

| Component          | Description                                                                                                                        | Helm Values                                                                                                                                                     |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Keycloak Auth**  | OIDC authentication — ⚠️ Kustomize needs the `keycloak-admin` Secret created first, see [Authentication](#authentication-keycloak) | `--set keycloak.enabled=true --set eddi.oidc.enabled=true --set eddi.oidc.publicUrl=http://localhost:8080 --set keycloak.adminPassword=…`                       |
| **NATS JetStream** | Cluster mode: 3-node JetStream cluster + 3 EDDI replicas — see [Scaling](#scaling)                                                 | `--set eddi.messagingType=nats --set eddi.replicas=3 --set nats.enabled=true --set nats.auth.password=… --set nats.cluster.routePassword=…`                     |
| **Monitoring**     | Prometheus (with the shipped alert rules) + Grafana (with the three dashboards)                                                    | Prometheus Operator only: `--set serviceMonitor.enabled=true --set prometheusRule.enabled=true`. The stack itself is Kustomize only: `k8s/overlays/monitoring/` |
| **Ingress**        | External HTTPS access                                                                                                              | `--set ingress.enabled=true --set ingress.hosts[0].host=eddi.example.com`                                                                                       |
| **Production**     | PDB, NetworkPolicy                                                                                                                 | `--set podDisruptionBudget.enabled=true --set networkPolicy.enabled=true`                                                                                       |

> **Manager UI**: there is nothing to enable. EDDI serves it from its own Service at `/manage`.

### Composing Kustomize Overlays

Kustomize takes **one directory** as input. Combine the pieces with a `kustomization.yaml` that lists the standalone overlay under `resources:` and the components under `components:`:

```yaml
# my-deployment/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: eddi
resources:
  - ../k8s/overlays/mongodb        # standalone: base + MongoDB
components:
  - ../k8s/overlays/auth           # Keycloak, realm import, OIDC ConfigMap keys
  - ../k8s/overlays/monitoring     # Prometheus + Grafana
```

Two rules decide whether this works:

* `resources:` may name a **directory** outside your root — kustomize builds it as its own kustomization root — but never a loose **file** outside your root. That is a hard `file ... is not in or below ...` error.
* A component's patches are applied into **your** resource set, so they reach `eddi-config` and the `eddi` Deployment. The same patches inside a `kind: Kustomization` referenced under `resources:` are built in isolation first, match nothing, and are dropped — with no error and exit code 0.

Ready-made examples are provided in `k8s/examples/`:

```bash
# Create the vault Secret once, before the first apply
bash k8s/create-secrets.sh

# This example includes the auth component, which has no default admin
# password — create its Secret too, or Keycloak never starts
kubectl create secret generic keycloak-admin -n eddi \
  --from-literal=password="$(openssl rand -base64 24)"

# ...and the monitoring component, which has no default Grafana password
kubectl create secret generic grafana-admin -n eddi \
  --from-literal=password="$(openssl rand -base64 24)"

# MongoDB runs with authentication — create its Secret too: the root user for
# mongod, EDDI's own least-privilege user (readWrite + dbAdmin on `eddi`, which
# the ensure-mongodb-user init container creates) and the connection string EDDI
# mounts as a file — for that user, never root. The Secret IS the stored copy of
# the root password: run this once, never on an upgrade.
# A fresh password over an initialised database locks EDDI out — mongod keeps
# the user it created at first start. The guard refuses when one exists.
if kubectl get secret mongodb-secrets -n eddi >/dev/null 2>&1; then
  echo "mongodb-secrets already exists — keeping it" >&2
else
pw=$(openssl rand -hex 24)
apppw=$(openssl rand -hex 24)
kubectl create secret generic mongodb-secrets -n eddi \
  --from-literal=MONGO_INITDB_ROOT_USERNAME=eddi \
  --from-literal=MONGO_INITDB_ROOT_PASSWORD="$pw" \
  --from-literal=MONGO_APP_USERNAME=eddi_app \
  --from-literal=MONGO_APP_PASSWORD="$apppw" \
  --from-literal=mongodb-secrets.properties="mongodb.connectionString=mongodb://eddi_app:${apppw}@mongodb:27017/eddi?authSource=eddi&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=30000"
fi

# MongoDB + Keycloak auth + Monitoring
kubectl apply -k k8s/examples/mongodb-full/

# PostgreSQL + Production hardening (PDB, NetworkPolicy, resource limits).
# No database password is shipped: create postgres-secrets with the command in
# k8s/overlays/postgres/postgres-secret.yaml.example first.
kubectl apply -k k8s/examples/postgres-ha/
```

## Architecture on Kubernetes

```
┌──────────────────────────────────────────────┐
│                  Ingress                      │
│            (nginx / traefik)                  │
└──────────────┬───────────────────────────────┘
               │
     ┌─────────▼──────────┐
     │    EDDI Service     │
     │   (ClusterIP:7070)  │
     └─────────┬──────────┘
               │
    ┌──────────▼──────────┐    ┌─────────────┐
    │  EDDI Deployment     │───▶│  MongoDB    │
    │  (labsai/eddi:6.6.0) │    │ StatefulSet │
    │                      │    └─────────────┘
    │  replicas: 1         │    ┌─────────────┐
    │  (N with NATS)       │───▶│ PostgreSQL  │
    └──────────────────────┘    │ StatefulSet │
               │                └─────────────┘
    ┌──────────▼──────────┐
    │   NATS JetStream     │  (cluster mode: leases, shared
    │   StatefulSet (×3)   │   state, events, dead letters)
    └──────────────────────┘
```

## Security

### Vault Master Key

The vault master key encrypts all stored API keys and secrets. **If you lose this key, encrypted secrets are unrecoverable.**

Three ways to manage it:

1. **Helper script** (recommended for initial setup):

   ```bash
   bash k8s/create-secrets.sh
   ```
2. **Manual kubectl** — the Secret holds one key, `application-secrets.properties`, which the Deployment mounts as a file and loads via `QUARKUS_CONFIG_LOCATIONS`:

   ```bash
   set -euo pipefail

   # mktemp gives an unpredictable name created 0600, so another local user
   # cannot pre-create the path or point it at a symlink.
   secrets_file=$(mktemp "${TMPDIR:-/tmp}/eddi-secrets.XXXXXX")
   trap 'shred -u "$secrets_file" 2>/dev/null || rm -f "$secrets_file"' EXIT

   # Fail closed on a key that already exists. This mints a NEW one, so
   # replacing a live eddi-secrets makes everything encrypted under the current
   # key permanently undecryptable. `kubectl create` is deliberately not piped
   # through `--dry-run=client | kubectl apply`: that would replace it silently
   # and exit 0. Rotate with `bash k8s/create-secrets.sh --force` instead.
   if kubectl get secret eddi-secrets --namespace=eddi >/dev/null 2>&1; then
     echo "eddi-secrets already exists — refusing to install a new master key." >&2
     exit 1
   fi

   # Fail closed: an empty key would create a Secret that silently leaves the
   # vault inert and secrets in plaintext.
   key=$(openssl rand -base64 24)
   [ -n "$key" ] || { echo "vault key generation failed" >&2; exit 1; }
   printf 'eddi.vault.master-key=%s\n' "$key" > "$secrets_file"

   # The Secret key must be named application-secrets.properties — that is the
   # filename the Deployment mounts. A bare temp path would name it otherwise.
   kubectl create secret generic eddi-secrets \
     --namespace=eddi \
     --from-file=application-secrets.properties="$secrets_file"
   ```
3. **External secrets** (production): Use [External Secrets Operator](https://external-secrets.io/) to sync from AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, etc. With Helm, point the chart at the synced Secret — see [Bringing your own Secrets](#bringing-your-own-secrets-helm) — and no credential ever passes through a values file.

### Bringing your own Secrets (Helm)

Every credential the chart renders can instead come from a Secret you manage (External Secrets Operator, Sealed Secrets, `kubectl create secret`). Name it in the matching `existingSecret` value and the chart renders **no** Secret of its own for that component; the corresponding password value is then ignored and need not be set:

| Value                     | Replaces                  | Keys the Secret must carry                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eddi.existingSecret`     | `<release>-eddi-secrets`  | `application-secrets.properties` — a Quarkus properties file with `eddi.vault.master-key=…` and the datastore line (`mongodb.connectionString=…` for MongoDB, percent-encoding the password; `quarkus.datasource.username=` / `.password=` for an external PostgreSQL), plus `eddi.nats.username=` / `eddi.nats.password=` for an in-chart NATS with auth or an external NATS that needs credentials |
| `mongodb.existingSecret`  | `<release>-eddi-mongodb`  | `MONGO_INITDB_ROOT_USERNAME`, `MONGO_INITDB_ROOT_PASSWORD`. Requires `eddi.existingSecret` too: the chart can no longer build EDDI's connection string without the password, so it refuses to render otherwise                                                                                                                                                                                       |
| `postgres.existingSecret` | `<release>-eddi-postgres` | `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` and `postgres-secrets.properties` (`quarkus.datasource.username=` / `quarkus.datasource.password=`)                                                                                                                                                                                                                                              |
| `keycloak.existingSecret` | `<release>-eddi-keycloak` | `password` — the bootstrap admin password, the same shape as the Kustomize overlay's `keycloak-admin` Secret                                                                                                                                                                                                                                                                                         |

The chart's own Keycloak Secret is where `keycloak.adminPassword` now lands: the StatefulSet reads `KC_BOOTSTRAP_ADMIN_PASSWORD` through a `secretKeyRef`, never as a plain `value:` in the pod spec (which showed the master-realm superuser password in every `kubectl get sts -o yaml` and in the release manifest). The reference is not `optional`, so a missing Secret leaves the pod in `CreateContainerConfigError` rather than booting with no admin.

> **`start-dev` behind an Ingress.** The in-chart Keycloak runs in development mode (embedded H2 on one PVC, no HA, no production hardening). The chart therefore refuses `keycloak.enabled=true` together with `ingress.enabled=true` unless you set `keycloak.devModeAcknowledged=true` — a demo or staging cluster you understand. For production, run Keycloak with `start` against an external database and point `eddi.oidc.authServerUrl` at it with `keycloak.enabled=false`. Setting `keycloak.publicOrigin` also strips the `http://localhost:*` redirect URIs and the `+` wildcard web origin from the realm's `eddi-frontend` client, so a realm published under a real hostname no longer completes a login for a page on the operator's own machine; the `eddi-mcp` client keeps its loopback redirects, which is how MCP clients sign in.
>
> With an Ingress and OIDC on, the chart also pins the MCP resource identifier EDDI advertises (`QUARKUS_OIDC_RESOURCE_METADATA_RESOURCE`) to `https://<ingress.hosts[0].host>/mcp` (`http://` when neither `ingress.tls` nor `eddi.oidc.resourceMetadata.forceHttpsScheme` is set) instead of deriving it from each request's `Host` header. An explicit `eddi.oidc.resourceMetadata.resource` still wins.

> **No manifest ever writes this Secret.** `k8s/base/eddi-secret.yaml.example` is a commented template that kustomize does not include, and `k8s/quickstart.yaml` ships no Secret at all. A reconciled Secret would replace a live vault master key with a placeholder on the next apply, and every secret encrypted under the old key would be permanently undecryptable.

### Authentication (Keycloak)

Both delivery paths ship the `eddi` realm and import it into Keycloak on **first boot** — `k8s/overlays/auth/eddi-realm.json` for Kustomize, `helm/eddi/files/eddi-realm.json` for Helm. Both are cluster-calibrated copies of `keycloak/eddi-realm.json`, which is calibrated for docker-compose: it allows redirects to `localhost` only and names a login theme that only the compose file mounts.

Realm import is **one-shot**. It seeds an empty database and is skipped once the realm exists, so later edits to the JSON do not reach a running Keycloak — change those in the admin console. Keycloak keeps that database on a PVC, so a restart no longer wipes it.

A realm imported from EDDI 6.4.0 — the first release to ship one for Helm and Kustomize — lacks the client scopes that put the user's identity in a token, so EDDI cannot tell its users apart and a non-admin opening their own conversation gets HTTP 500. Run the one-time repair in [security.md → Identity claims](https://docs.labs.ai/security-and-compliance/security.md#identity-claims-and-realms-imported-from-eddi-610640).

#### No credential in this component has a default

Neither delivery path ships a password for the Keycloak superuser or for the privileged EDDI account. Both used to, and both were reachable: the `keycloak` Service is a `ClusterIP`, so `admin`/`admin` and `eddi`/`eddi` were a guessable master-realm superuser and a guessable full EDDI administrator for anything running in the cluster — "development component" is a property of the manifests, not of the network they are applied to.

**1. Create the Keycloak admin Secret before the first apply (Kustomize only — the pod does not start without it):**

```bash
kubectl create secret generic keycloak-admin -n eddi \
  --from-literal=password="$(openssl rand -base64 24)"
```

`keycloak-statefulset.yaml` reads `KC_BOOTSTRAP_ADMIN_PASSWORD` from it through `secretKeyRef` with no default, so a missing Secret fails closed — the pod stays in `CreateContainerConfigError` with `secret "keycloak-admin" not found` rather than coming up with a known password. Helm asks for the same value as `keycloak.adminPassword`, which `required` refuses to default, renders it into the `<release>-eddi-keycloak` Secret and reads it the same way; or name your own Secret in `keycloak.existingSecret` (see [Bringing your own Secrets](#bringing-your-own-secrets-helm)).

**2. Give the `eddi` account a password after the first boot.** The realm seeds it with the `eddi-admin`, `eddi-editor` and `eddi-viewer` roles and **no credential**, so it cannot be logged into until you set one: admin console → *Users* → `eddi` → *Credentials* → *Set password*. Or grant those realm roles to an account you create yourself and leave `eddi` unused. The realm also seeds `viewer` (`eddi-viewer`) and `user` (`eddi-user`), likewise without credentials.

The unprivileged fixtures `viewer` (`eddi-viewer`) and `user` (`eddi-user`) are opt-in the same way: they ship with their roles and **no password**, so they cannot log in until you set one. They used to ship as `viewer`/`viewer` and `user`/`user`, and with the password grant then enabled on the public `eddi-frontend` client a single `curl` against a reachable Keycloak returned a token good enough to run LLM turns. The grant is now off on `eddi-frontend` too — the Manager uses the authorization-code flow and never needed it.

#### TLS

The realm ships `"sslRequired": "external"`, Keycloak's own default: HTTPS is required for requests from outside the local network, and loopback and private addresses are still served over HTTP. The documented quick start — `kubectl port-forward svc/keycloak 8080:8080`, browser on `http://localhost:8080` — arrives at the pod as `127.0.0.1` and is unaffected, and so is EDDI's in-cluster backchannel to `http://keycloak:8080` from an RFC 1918 pod address. Exposing Keycloak on a public hostname now requires TLS in front of it, with the proxy forwarding the HTTPS scheme (`X-Forwarded-Proto`) so Keycloak sees it.

> On a cluster whose pod CIDR is outside RFC 1918 — `100.64.0.0/10` on some managed offerings — Keycloak does not count the backchannel as local and will demand HTTPS for it. Terminate TLS in front of Keycloak, or set the realm's `sslRequired` back to `none` on that cluster deliberately.

Four settings have to change together, and none of them can be derived — but they carry **two different URLs**. Behind an Ingress those are two different hosts, because the Ingress fronts EDDI and not Keycloak.

**Three name the browser-facing URL of Keycloak** (the IdP, e.g. `https://auth.example.com`):

| Setting                                                                                                                                                 | Kustomize                                                                                     | Helm                                                                                 |
| ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Keycloak's own hostname (what it stamps as the token issuer)                                                                                            | `KC_HOSTNAME` in `keycloak-statefulset.yaml`                                                  | `eddi.oidc.publicUrl`                                                                |
| URL handed to the Manager SPA                                                                                                                           | `EDDI_KEYCLOAK_PUBLIC_URL` patch                                                              | `eddi.oidc.publicUrl`                                                                |
| Issuer EDDI validates tokens against                                                                                                                    | `QUARKUS_OIDC_TOKEN_ISSUER` patch                                                             | derived from `eddi.oidc.publicUrl`                                                   |
| Keycloak trusts `X-Forwarded-*` from a TLS-terminating proxy, so its token/JWKS/userinfo URLs are `https` (MCP clients refuse an `http` token endpoint) | add `KC_PROXY_HEADERS: xforwarded` to `keycloak-statefulset.yaml` when a proxy terminates TLS | `keycloak.proxyHeaders` — automatic `xforwarded` when `eddi.oidc.publicUrl` is https |

**One names the browser-facing origin of EDDI** — where the Manager SPA is served, i.e. `ingress.hosts[0].host` (e.g. `https://eddi.example.com`), *not* the Keycloak host:

| Setting                                                           | Kustomize                                                         | Helm                    |
| ----------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------- |
| Realm `redirectUris` / `webOrigins` of the `eddi-frontend` client | edit `eddi-realm.json` (placeholder `https://eddi.example.com/*`) | `keycloak.publicOrigin` |

The Manager redirects back to where it is itself served, so the realm has to list **that** origin. Putting the Keycloak host into `redirectUris` is the common way to get `Invalid parameter: redirect_uri`; so is omitting your host altogether. The shipped defaults cover `kubectl port-forward svc/keycloak 8080:8080`, where both URLs happen to be `localhost` and the distinction does not bite.

#### Upgrading an existing Keycloak install

> ⚠️ **Kustomize: delete the old Deployment before applying this version.**
>
> ```bash
> kubectl delete deployment keycloak -n eddi
> ```
>
> Keycloak used to be a `Deployment` named `keycloak`; it is now a `StatefulSet` under the **same name and the same pod labels**. `kubectl apply -k` never prunes, so on an existing install the old Deployment's ReplicaSet keeps running and **both** pods match the one `keycloak` Service selector — one with the imported `eddi` realm, one without. Requests round-robin between them, so `/realms/eddi` 404s on roughly half of them and logins fail intermittently, with nothing in `kubectl get` to explain it. Run the delete before the first apply, or immediately after if you have already applied.
>
> Helm needs no manual step: `helm upgrade` removes resources that left the release. The Deployment→StatefulSet change still replaces the pod, and because the old Deployment had no volume there is no Keycloak state to carry over — the new pod imports the realm into an empty database, and anything configured by hand in the old admin console is gone. Export it first if you need it.

> ⚠️ **Kustomize: restart EDDI after applying the auth component.**
>
> ```bash
> kubectl rollout restart deployment/eddi -n eddi
> ```
>
> The component patches `eddi-config` and nothing else, and the EDDI Deployment reads that ConfigMap through `envFrom`. Environment variables are fixed at container start, so `kubectl apply -k` updates the ConfigMap while the running pod keeps `QUARKUS_OIDC_TENANT_ENABLED: "false"` and the three `ALLOW_UNAUTHENTICATED` escape hatches it booted with. Every object reports as applied and the install stays **unauthenticated** — the exact state the previously ineffective overlay left behind — until the pod is replaced.
>
> Helm does this on its own: the pod template carries `checksum/config` and `checksum/secret` annotations, so a ConfigMap or Secret change alters the Deployment spec and `helm upgrade` rolls the pod.

### Update check (Helm)

`eddi.updateCheck.enabled` (default `true`) becomes `EDDI_UPDATE_CHECK_ENABLED`. The check — one anonymous request a day to `api.github.com`, and a notification for the administrators when a newer release exists — is on unless you set it to `false`, which stops it entirely, even the first check after the first start, and locks the switch on the Manager's Updates page. See [Docker → Update check](https://docs.labs.ai/deployment-and-infrastructure/docker.md#update-check).

### White-labeling (Helm)

`eddi.branding.*` in `values.yaml` seeds the deployment's brand: product name, tagline, colours, the Powered-by and update-check toggles, `locked` (the values become the whole brand and the Manager's Branding page turns read-only) and `assets.configMap`, an existing ConfigMap of images mounted read-only as `eddi.branding.assets-dir`. Everything is empty by default and an empty value is not rendered. The chart does not mount a login theme into its in-cluster Keycloak; the theme ZIP the Branding page offers has to reach Keycloak the way you manage it. See [White-Labeling → Helm](https://docs.labs.ai/deployment-and-infrastructure/white-labeling.md#helm).

### Pod Security

EDDI runs as non-root user (UID 185) and is compatible with `restricted` Pod Security Standards:

```yaml
securityContext:
  runAsNonRoot: true
  runAsUser: 185
  runAsGroup: 185
```

No pod mounts a service-account token it does not use (`automountServiceAccountToken: false` on EDDI, MongoDB, PostgreSQL, NATS, Keycloak and Grafana): none of them calls the Kubernetes API, so a process that gets into one finds no API credential to escalate with. Prometheus keeps its token, which pod discovery needs. Helm exposes `serviceAccount.automountToken` for the rare setup that adds something needing one.

### Datastore authentication and isolation

The in-cluster MongoDB used to run **without authentication**, behind a Service every pod in the cluster could reach. Any workload could read and rewrite every agent, conversation and the audit ledger. Now:

* **Helm** runs MongoDB with authentication (`mongodb.rootPassword`, required — see [Option C](#option-c-helm) and [Upgrading to an authenticated MongoDB](#upgrading-to-an-authenticated-mongodb)). With `networkPolicy.enabled=true` the chart also admits only the EDDI pod to MongoDB, PostgreSQL, NATS and Keycloak.
* **EDDI connects as its own MongoDB user, not as root.** Helm and `overlays/mongodb` give EDDI a user with `readWrite` and `dbAdmin` on the `eddi` database and no role anywhere else (Helm: `mongodb.appUsername`, `mongodb.appPassword`, which defaults to a value derived from `mongodb.rootPassword`). An `ensure-mongodb-user` init container in the EDDI pod creates the user, or resets its password and roles, before every start, with the root credentials the `eddi` container itself never receives. It runs every time rather than as a `/docker-entrypoint-initdb.d` script because the mongo image runs those only on an empty volume, so existing releases are migrated by the upgrade itself. A kustomize Secret created before this change has no `MONGO_APP_*` keys; the init container then does nothing and EDDI keeps connecting as root until you add them ([Upgrading from 6.5](https://docs.labs.ai/deployment-and-infrastructure/upgrading-from-6.5.md#mongodb-least-privilege-user)).
* **Kustomize** `overlays/mongodb` runs MongoDB with authentication from the `mongodb-secrets` Secret you create (see Option B), and `overlays/mongodb` and `overlays/postgres` each ship a NetworkPolicy admitting only EDDI. `overlays/postgres` no longer ships `eddi`/`eddi`: `postgres-secrets` is created by hand from `postgres-secret.yaml.example`.
* **quickstart.yaml** stays one file for evaluation: its MongoDB is still unauthenticated, but a NetworkPolicy admits only EDDI.

NetworkPolicies need a CNI that enforces them (Calico, Cilium and most managed offerings do); authentication is what holds without one.

> ⚠️ **Upgrading a Kustomize install whose MongoDB ran without authentication.** The mongo image creates the root user only when it initialises an **empty** data directory, but it switches `--auth` on every time. So create the user in the running database **first**, with the password you are about to configure:
>
> ```bash
> kubectl exec -n eddi statefulset/mongodb -- mongosh admin \
>   --eval 'db.createUser({user: "eddi", pwd: "<password>", roles: ["root"]})'
> ```
>
> then create `mongodb-secrets` with that password and `kubectl apply -k`. Done the other way round, mongod restarts with `--auth` and no user and EDDI cannot log in until you run the same `createUser` (the localhost exception still allows it). For Helm, see [Upgrading to an authenticated MongoDB](#upgrading-to-an-authenticated-mongodb).

### Network Policy

The production overlay includes a `NetworkPolicy` that restricts EDDI to:

* **Ingress**: HTTP port 7070 from within the namespace + the `ingress-nginx` controller (namespace and pods labelled `ingress-nginx`; patch the policy for another controller)
* **Egress**: Database (MongoDB/PG), NATS, Keycloak, DNS, and external HTTPS (port 443 for LLM APIs — to public addresses only: private, loopback, link-local, CGNAT and multicast ranges are excluded)

The datastores carry their own ingress policies — see [Datastore authentication and isolation](#datastore-authentication-and-isolation).

## Scaling

### Single replica (default)

The default, `eddi.messaging.type=in-memory`, runs **exactly one replica**: the turns of a conversation are serialised with a JVM-local lock, and replay nonces, rate limits and caches live in the JVM. The Helm chart refuses `eddi.replicas` above 1, `autoscaling.enabled=true` and `eddi.updateStrategy=RollingUpdate` in this mode, and rolls out with `Recreate` so two JVMs never share the database. Scale **vertically** via `eddi.resources`.

### Sizing a node

EDDI waits on LLMs far more than it computes, so **memory, not CPU**, decides how many conversation turns a node runs at once. Each in-flight turn holds its conversation (about 1 MB for short conversations in load tests, more with long histories and tool payloads).

| Setting        | Default                                          | Why                                                                                                                                                           |
| -------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Memory limit   | 3 GiB (Helm, base), 4 GiB (production overlay)   | The heap is **75 %** of it (`JAVA_MAX_MEM_RATIO` in the image); the rest covers metaspace, thread stacks and direct buffers. Raise it for more turns per node |
| Memory request | equal to the limit                               | Guaranteed QoS: the pod is not the first evicted under node pressure                                                                                          |
| CPU            | request 1 (2 in the production overlay), limit 4 | The JVM sizes its GC, virtual-thread carrier and Jlama threads from the **limit**; without one it sees every core on the node                                 |
| GC             | G1 (`GC_CONTAINER_OPTIONS`)                      | Short pauses with hundreds of turns in flight. For heaps of 8 GB and more, `-XX:+UseZGC`                                                                      |
| Out of memory  | the JVM exits (`-XX:+ExitOnOutOfMemoryError`)    | The orchestrator restarts it instead of a half-working JVM holding its turns                                                                                  |

Guidelines:

* **Keep the database close.** A turn makes about 20 database round trips, so latency to MongoDB or PostgreSQL matters more than CPU speed. Size its pool with the node (MongoDB `maxPoolSize` in the connection string, default 100; PostgreSQL `quarkus.datasource.jdbc.max-size`, Quarkus default 50).
* **File descriptors:** allow at least 65536 (`nofile`). The compose file sets it; on Kubernetes it is a property of the node's container runtime.
* **Scale out** with cluster mode (below) when one node would need a heap above about 16 GB, when CPU stays above 60–70 %, or when you need availability during upgrades.
* **In-process Jlama models** are memory-mapped outside the heap: lower `JAVA_MAX_MEM_RATIO` and add the model size to the memory limit, see [LLM configuration](https://docs.labs.ai/agent-configuration/langchain.md).

### Upgrades replace the pod (no surge)

With in-memory messaging the EDDI Deployment uses the **`Recreate`** strategy (Helm chart: `eddi.updateStrategy`, empty by default, which picks per messaging type; the Kustomize base and `quickstart.yaml` likewise): an upgrade stops the running pod before the new one starts, so each upgrade is unavailable for as long as EDDI takes to boot and become ready. A `RollingUpdate` would avoid that only by running **two EDDI JVMs against the same database** for the length of every rollout — and everything a single node keeps per process (the per-conversation turn lock, replay-protection nonces, rate limits and cost budgets, the agent and workflow caches, HITL crash recovery) diverges between them in that window. The chart therefore refuses `RollingUpdate` with in-memory messaging. For upgrades without downtime run cluster mode, where the chart (and the `nats` Kustomize component) roll with `maxSurge: 1`, `maxUnavailable: 0`: the old and the new pod coordinate through NATS like any two replicas.

### Cluster mode (horizontal scaling)

With `eddi.messaging.type=nats` the **published image** runs as a cluster: any number of replicas behind the plain round-robin Service, sharing the database and a NATS JetStream cluster. Every replica takes a cluster-wide lease per conversation, so the turns of one conversation never overlap whichever pod receives them; a pod that lost its lease cannot overwrite the turn that ran after it (fencing). Shared security and tool state lives in JetStream KV, caches are invalidated on every pod, and cancel, GDPR erasure, undeploy and HITL reach the pod that runs the work. When NATS is unreachable every pod keeps serving in a documented degraded mode — readiness stays UP — except a pod that has never connected since it started, or whose connection fails on credentials or TLS: that one reports not-ready (it cannot take a lease). The guarantees, the failure modes and the runbook are in [Clustering](https://docs.labs.ai/deployment-and-infrastructure/clustering.md).

**Kustomize** — PostgreSQL, production hardening and cluster mode (three EDDI replicas, three NATS nodes):

```bash
kubectl apply -k k8s/examples/postgres-ha/
```

The cluster part is the `k8s/overlays/nats` component; list it **after** `../../overlays/production`, whose replicas patch it overrides. NATS clients and routes authenticate, with credentials from a `nats-auth` Secret you create first — the NATS pods do not start without it — and EDDI's user may touch only EDDI's subjects and never delete a stream:

```bash
NATS_USER_PW=$(openssl rand -hex 24)
kubectl create secret generic nats-auth -n eddi \
  --from-literal=eddi-password="$NATS_USER_PW" \
  --from-literal=route-password="$(openssl rand -hex 24)"
# and in eddi-secrets' application-secrets.properties:
#   eddi.nats.username=eddi
#   eddi.nats.password=<NATS_USER_PW>
```

The component also restricts the NATS ports with NetworkPolicies, pins the NATS image by digest, keeps one NATS pod per node (a cluster with fewer than three nodes needs that patched to `preferred`), leaves 20 % of the NATS volume free (`max_file_store`), sets `EDDI_CLUSTER_DEGRADED_TURNS=reject` and gives EDDI a 5 s `preStop` sleep. TLS on NATS itself is not configured, so by default EDDI sends its NATS password over plaintext `nats://` inside the cluster network — use the Helm chart's `nats.tls`, or patch a `tls {}` block into `nats-config` **and** switch EDDI to it: set `EDDI_NATS_URL` to `tls://` URLs (or `eddi.nats.tls.enabled=true`) and give it the CA with `eddi.nats.tls.truststore-path`, mounted from a Secret. Patching only the server leaves EDDI dialling `nats://`, which a TLS-only server refuses.

**Helm** — three replicas, an in-chart three-node NATS cluster with authentication, and a PodDisruptionBudget:

```bash
helm install eddi ./helm/eddi \
  --set eddi.messagingType=nats \
  --set eddi.replicas=3 \
  --set nats.enabled=true \
  --set nats.auth.password="$(openssl rand -base64 24)" \
  --set nats.cluster.routePassword="$(openssl rand -hex 24)" \
  --set podDisruptionBudget.enabled=true \
  --namespace eddi --create-namespace
```

| Value                                                  | Default                   | In cluster mode                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eddi.replicas`                                        | `1`                       | Any number; omitted while `autoscaling.enabled=true`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `autoscaling.enabled`                                  | `false`                   | Renders an HPA (`minReplicas` 2, `maxReplicas` 6, CPU 70 %). `targetMemoryUtilizationPercentage` is `null` since chart 2.14.0: a JVM keeps the heap it grew to, so a memory target ratcheted the HPA up to `maxReplicas`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `autoscaling.extraMetrics` / `autoscaling.behavior`    | `[]` / `{}`               | Further `autoscaling/v2` metrics and scaling behavior, as given. CPU is a weak signal for a node that mostly waits on LLMs; scale on in-flight turns instead — see [Clustering → Autoscaling](https://docs.labs.ai/deployment-and-infrastructure/clustering.md#autoscaling). A target set to `null` drops that metric                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `eddi.updateStrategy`                                  | `""`                      | Empty picks `RollingUpdate` (maxSurge 1, maxUnavailable 0); in-memory picks `Recreate`. A string, or the Deployment's own strategy object (`{type: RollingUpdate, rollingUpdate: {...}}`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `networkPolicy.natsEgressTo`                           | `[]`                      | With `networkPolicy.enabled` and `nats.externalUrl` the chart opens egress to the URL's port; this narrows the destination (NetworkPolicyPeers)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `eddi.terminationGracePeriodSeconds`                   | `""`                      | 75 s; the chart refuses a grace that does not exceed the `preStop` sleep plus the drain plus 3 s                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `eddi.preStopSleepSeconds`                             | `5`                       | A `preStop` sleep before SIGTERM, in every mode: Kubernetes removes the pod from the Service endpoints while it runs, and requests that still arrive are served. After SIGTERM EDDI still admits turns for its 3 s readiness grace. `0` renders no hook                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `eddi.cluster.degradedTurns`                           | `""`                      | `EDDI_CLUSTER_DEGRADED_TURNS` when set. **`reject` is recommended for production**: while NATS is unreachable a turn is answered 409 + `Retry-After` instead of running unfenced (the application default, `local`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `eddi.shutdownDrainTimeoutSeconds`                     | `""`                      | 65 s: a terminating pod lets its running turns finish                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `podDisruptionBudget.*`                                | off                       | `minAvailable: 1` when the release never runs fewer than two replicas (`eddi.replicas`, or `autoscaling.minReplicas` while autoscaling), so a drain never takes the last pod; `maxUnavailable: 1` otherwise                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `nats.enabled` / `nats.externalUrl`                    | off / `""`                | One of the two is required; both at once is refused. `externalUrl` must not carry credentials (an `@` is refused: it is written to a ConfigMap); put them in `nats.externalUsername` / `nats.externalPassword` (both or neither), which go to the Secret. With those set every server must be `tls://` (or `wss://`): EDDI then connects over TLS, verifying the certificate and that it names the host, and a `nats://` server is refused unless `nats.externalAllowPlaintext=true` (for a transport encrypted underneath EDDI, such as a service mesh). With `eddi.existingSecret` they are refused too: put `eddi.nats.username=` / `eddi.nats.password=` in that Secret, and use `tls://`                                                                             |
| `nats.externalCaSecretName`                            | `""`                      | A Secret with the external NATS' CA as `ca.crt` (PEM), mounted into EDDI as its NATS trust store. Empty: the JVM's public CAs                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `nats.externalReplicas`                                | `3`                       | `EDDI_NATS_REPLICAS` for an external NATS — R3 buckets and streams on a JetStream cluster of three or more servers. Set `1` only for a single-server NATS. Before chart 2.14.0 an external NATS silently got R1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `nats.replicas`, `nats.cluster.enabled`                | `3`, `true`               | Three routed nodes, R3 buckets and streams                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `nats.cluster.routeUsername` / `routePassword`         | `route` / `""`            | The routes between the NATS nodes authenticate with their own credentials, which client permissions do not cover; `routePassword` is required whenever the in-chart NATS runs more than one node                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `nats.auth.*`                                          | on, user `eddi`           | `nats.auth.password` is required; the user may touch only EDDI's subjects, and may not delete streams or touch JetStream placement. `nats.auth.secondaryUsername` / `secondaryPassword` add a second user with the same permissions, for rotating the password without downtime: set the secondary to the new credentials and upgrade, swap it into `username`/`password` and upgrade (EDDI rolls onto it), then clear the secondary and upgrade. With `eddi.existingSecret` the chart neither updates EDDI's NATS credentials nor rolls its pods: before clearing the old user, put the new password in that Secret (`eddi.nats.password=`) and restart the EDDI pods (`kubectl rollout restart deployment/<fullname>`), or every node loses NATS when the old user goes |
| `nats.isolation.*`                                     | off                       | `enabled: true` puts EDDI's user in its own NATS account (`account`, default `EDDI`) with its own reply-subject prefix (`_INBOX_<account>`), named in its permissions — for a NATS that other tenants share. It moves the buckets and streams into the new account: enable it on a fresh install                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `nats.tls.*`                                           | off                       | Client and route TLS from a `kubernetes.io/tls` Secret you provide (cert-manager's carry `tls.crt`, `tls.key` and `ca.crt`). The chart mounts its `ca.crt` — only that key — into EDDI as the trust store and connects over `tls://`; the certificate must name `*.<fullname>-nats`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `nats.image.*`                                         | `2.11.17-alpine` + digest | A full patch version pinned by digest                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `nats.antiAffinity`                                    | `required`                | One NATS pod per node (and spread over zones, best effort); `preferred` for a dev cluster with fewer nodes than `nats.replicas`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `nats.goMemLimit`                                      | `460MiB`                  | `GOMEMLIMIT` for the NATS server, \~90 % of its memory limit                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `nats.storage.maxFileStore`                            | `""`                      | JetStream's `max_file_store`; empty is 80 % of `nats.storage.size`, so JetStream refuses data before the volume is full                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `nats.metrics.*`                                       | off                       | A `prometheus-nats-exporter` sidecar (`-varz -jsz=all`, port 7777), and a ServiceMonitor with `serviceMonitor.enabled`; the `eddi-cluster-nats` alerts read it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `ingress.nginxDefaults` / `ingress.readTimeoutSeconds` | `true` / `300`            | With `ingress.className: nginx`: no response buffering and a 300 s read timeout (SSE), and in cluster mode an `<release>-mcp` Ingress that hashes `/mcp` on the client address. Your own `ingress.annotations` win. Other controllers need the equivalent by hand                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `prometheusRule.enabled`                               | off                       | In cluster mode the PrometheusRule also carries the cluster alerts ([`eddi-cluster-alerts.yml`](https://github.com/labsai/EDDI/tree/main/docs/monitoring/eddi-cluster-alerts.yml))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

The first switch of an existing release from in-memory to `nats` should roll out with `Recreate` — see [Clustering → Moving from one node to a cluster, and back](https://docs.labs.ai/deployment-and-infrastructure/clustering.md#moving-from-one-node-to-a-cluster-and-back).

Each replica's node id is its pod name (`EDDI_CLUSTER_NODE_ID`), and the pods are spread over nodes and zones. Every path balances freely except `/mcp`: an MCP session lives on the replica that opened it, so route `/mcp` with client or cookie affinity on your ingress — the chart's ingress does it for ingress-nginx (see [Clustering → Residual limitations](https://docs.labs.ai/deployment-and-infrastructure/clustering.md#residual-limitations)). `nats.buildProfileImage` from charts before 2.7.0 is no longer read — the published image selects cluster mode at runtime.

## Monitoring

EDDI exposes Prometheus metrics at `/q/metrics`. The EDDI Deployment includes Prometheus scrape annotations by default:

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "7070"
  prometheus.io/path: "/q/metrics"
```

Deploy the monitoring stack with Kustomize. It runs Prometheus with EDDI's alert rules loaded (`docs/monitoring/eddi-alerts.yml`, runbooks in the [monitoring guide](https://docs.labs.ai/deployment-and-infrastructure/monitoring-guide.md#alert-runbooks)) and Grafana with the Prometheus datasource (uid `prometheus`) and all three EDDI dashboards provisioned into an "EDDI" folder. The rules and dashboards come from generated ConfigMaps, each inside the 256 KiB limit of client-side `kubectl apply`. To stay under it, the Full Metrics Reference ships as two linked dashboards, "(1/2)" and "(2/2)"; part 1 keeps its uid `eddi-metrics-all`:

```bash
# Kustomize (with MongoDB + Auth + Monitoring)
kubectl apply -k k8s/examples/mongodb-full/

# Or list `- ../../overlays/monitoring` under `components:` in your own
# kustomization (it is a Component; `kubectl apply -k` on it directly fails)

# Access Grafana — the Prometheus datasource is provisioned for you
kubectl port-forward svc/grafana 3000:3000 -n eddi
# Open http://localhost:3000 — user admin, the password in the grafana-admin
# Secret, which you create before the first apply (Grafana does not start
# without it; it used to run as admin/admin):
#   kubectl create secret generic grafana-admin -n eddi \
#     --from-literal=password="$(openssl rand -base64 24)"
```

Prometheus discovers pods **in its own namespace** only, so it now gets a namespaced `Role`/`RoleBinding` instead of a cluster-wide `ClusterRole`. On an existing install delete the old objects by hand — `kubectl apply -k` does not prune: `kubectl delete clusterrolebinding eddi-prometheus` and `kubectl delete clusterrole eddi-prometheus`.

Prometheus has no login of its own, so the component also ships a `NetworkPolicy` (`prometheus-network-policy`) that admits only the Grafana pod on port 9090; other pods in the cluster cannot query it or list its scrape targets (`kubectl port-forward` still works). To let another client in, such as a federating Prometheus, patch its pod selector into that policy's `from` list. A client in another namespace needs a `namespaceSelector` **in the same `from` entry** as its `podSelector` (a `podSelector` alone selects pods in the policy's own namespace only). It takes effect only where the CNI enforces NetworkPolicy.

With the auth component, `/q/metrics` requires a token and an anonymous scrape answers 401 — see [Scraping with authentication on](https://docs.labs.ai/deployment-and-infrastructure/monitoring-guide.md#scraping-with-authentication-on).

**Helm with the Prometheus Operator.** The chart ships no Prometheus or Grafana of its own (the old `monitoring.*` values rendered nothing and were removed), but it can hand EDDI to a Prometheus Operator such as kube-prometheus-stack. Both switches are off by default because their CRDs exist only where the operator runs:

```bash
helm upgrade --install eddi helm/eddi … \
  --set serviceMonitor.enabled=true \
  --set prometheusRule.enabled=true \
  --set serviceMonitor.labels.release=kube-prometheus-stack \
  --set prometheusRule.labels.release=kube-prometheus-stack
```

The `ServiceMonitor` rewrites the job label to `eddi`, which the rules and the dashboards select on; with authentication on, give it a token through `serviceMonitor.endpointConfig` (`authorization:` or `oauth2:`). If your Prometheus also honours the `prometheus.io/*` pod annotations, clear `eddi.podAnnotations`, or EDDI is scraped twice. With `networkPolicy.enabled=true` the chart's ingress rules admit only the release namespace and `networkPolicy.allowedIngressNamespaces`: a Prometheus in another namespace (kube-prometheus-stack usually runs in `monitoring`) must be listed there, or every scrape times out and `EddiDown` fires. Import the dashboards from `docs/monitoring/` (they pick the datasource through a variable).

**Tracing.** `--set eddi.tracing.otlpEndpoint=http://<collector>:4317` turns the OpenTelemetry SDK on. With `networkPolicy.enabled=true` the chart then also allows egress to `eddi.tracing.otlpPort` (default 4317) for the peers in `networkPolicy.otlpEgressTo` (default: any pod in the cluster) — without it the default-deny policy drops every span bound for an in-cluster collector. A collector **outside** the cluster on 443 is covered by the external-HTTPS rule. On any other port it needs an `ipBlock` for its address in `networkPolicy.otlpEgressTo`, e.g. `--set-json 'networkPolicy.otlpEgressTo=[{"ipBlock":{"cidr":"203.0.113.10/32"}}]'`, because the default peer matches cluster pods only. An empty list (`otlpEgressTo: []`) opens the OTLP port to every destination; use it only where that breadth is acceptable. The Kustomize production NetworkPolicy carries the same rule commented out.

## Health Checks

EDDI provides three probe endpoints:

| Endpoint          | Probe Type         | Purpose                                          |
| ----------------- | ------------------ | ------------------------------------------------ |
| `/q/health/live`  | Liveness + Startup | Process is alive                                 |
| `/q/health/ready` | Readiness          | DB connected, migrations done, ready for traffic |
| `/q/metrics`      | —                  | Prometheus metrics                               |

The startup probe checks **liveness**, not readiness. Readiness stays DOWN for as long as the startup migrations run, which on the first boot of an [EDDI 5 database](https://docs.labs.ai/deployment-and-infrastructure/upgrading-from-5x.md) can take minutes. A startup probe on readiness (as in chart versions before 2.3.0) killed the pod mid-migration, and every restart began the migration again. In the chart its limits are `eddi.startupProbe.path`, `.failureThreshold` and `.periodSeconds`.

Readiness includes the check `Agents are ready health check`. It is DOWN until the startup migrations have run and the first deployment pass has finished. An agent that fails to deploy does **not** make it DOWN: one broken agent would otherwise take every replica out of rotation. Instead, the check counts such deployments in its data, as `agentsInErrorCount`. It gives only the count because `/q/health` is reachable without authentication. Which agents failed is in the log and in the authenticated `GET /administration/{environment}/deploymentstatus`. Watch that count after a deploy or an upgrade. A failed deployment is retried after 10 s, then with a doubling delay capped at 5 minutes, and is logged once when it starts failing and once when it recovers.

## File Structure

```
k8s/
├── base/                    # Core EDDI manifests
├── overlays/
│   ├── mongodb/             # MongoDB backend, authenticated (standalone)
│   ├── postgres/            # PostgreSQL backend (standalone; postgres-secret.yaml.example)
│   ├── nats/                # Cluster mode: 3-node NATS + 3 EDDI replicas (component)
│   ├── auth/                # Keycloak + realm import (component)
│   ├── monitoring/          # Prometheus + Grafana (component)
│   ├── ingress/             # Ingress resource (component)
│   └── production/          # PDB, NetworkPolicy, resource limits (component;
│                            #   eddi-hpa.yaml is an intentionally disabled template)
├── examples/
│   ├── mongodb-full/        # MongoDB + Auth + Monitoring
│   └── postgres-ha/         # PostgreSQL + Production hardening
├── create-secrets.sh        # Vault key generator (bash)
├── create-secrets.ps1       # Vault key generator (PowerShell)
└── quickstart.yaml          # All-in-one manifest

helm/
└── eddi/                    # Helm chart
    ├── Chart.yaml
    ├── values.yaml
    ├── files/               # eddi-realm.json, imported by Keycloak on first boot
    └── templates/
```

> **Note**: Overlays marked **(standalone)** include the base and can be applied directly with `kubectl apply -k`. Overlays marked **(component)** are `kind: Component` — they have no resource set of their own, so `kubectl apply -k` on one fails by design; list them under `components:` in a kustomization that includes a standalone overlay. See [Composing Kustomize Overlays](#composing-kustomize-overlays).

> **Note**: `k8s/base/eddi-secret.yaml.example` is a template, not a resource. The `eddi-secrets` Secret is created out-of-band (`bash k8s/create-secrets.sh`) so that no re-apply can overwrite a live vault master key.

## Troubleshooting

### EDDI pod stuck in ContainerCreating

This is the most likely first-run symptom. `kubectl describe pod` shows:

```
MountVolume.SetUp failed for volume "eddi-secrets" :
  secret "eddi-secrets" not found
```

No shipped manifest creates `eddi-secrets` — a reconciled Secret would overwrite a live vault master key on the next `kubectl apply`. Create it out-of-band and the pod starts on its own, with no restart needed:

```bash
bash k8s/create-secrets.sh            # PowerShell: pwsh -File .\k8s\create-secrets.ps1
kubectl get pods -n eddi -w
```

If `create-secrets` refuses because `eddi-secrets` already exists, that is the guard against replacing a live key — the Secret is there and the mount failure is something else (wrong namespace, or the key inside it is not named `application-secrets.properties`). Check with `kubectl get secret eddi-secrets -n eddi -o jsonpath='{.data}'`.

### EDDI pod stuck in CrashLoopBackOff

Check if the database is reachable:

```bash
kubectl logs -n eddi deployment/eddi
kubectl get pods -n eddi
```

Common causes:

* MongoDB/PostgreSQL not yet ready (wait for StatefulSet pod)
* Incorrect connection string in ConfigMap
* Volume claims pending (check `kubectl get pvc -n eddi`)

### EDDI starts but readiness probe fails

Check the health endpoint:

```bash
kubectl exec -n eddi deployment/eddi -- curl -s localhost:7070/q/health/ready
```

### Vault key issues

On a fresh install a missing key can no longer produce a running pod — the Deployment mounts `eddi-secrets` and the pod stays in `ContainerCreating` until the Secret exists (see above). "vault master key not set" therefore means the Secret exists but its `application-secrets.properties` payload has an empty or absent `eddi.vault.master-key` — for example a Secret created by hand from the old `k8s/base/eddi-secret.yaml` template. Replace it and restart:

```bash
bash k8s/create-secrets.sh --force    # PowerShell: pwsh -File .\k8s\create-secrets.ps1 -Force
kubectl rollout restart deployment/eddi -n eddi
```

⚠️ `--force` installs a **new** key. Anything already encrypted under the old one becomes unrecoverable, so use it only when there is nothing to lose — which is exactly the case when the key was empty.

### PVC stuck in Pending

If PVCs aren't provisioning, check your StorageClass:

```bash
kubectl get sc                    # List available StorageClasses
kubectl get pvc -n eddi           # Check PVC status
kubectl describe pvc -n eddi      # See events / errors
```

If your cluster doesn't have a default StorageClass, uncomment `storageClassName` in the StatefulSet manifests.
