Kubernetes

EDDI runs natively on Kubernetes. This guide covers deployment options from a simple quickstart to production-grade configurations.

Prerequisites

• Kubernetes cluster (1.26+) — minikube, kind, GKE, EKS, AKS, or any conformant cluster

• kubectl configured to access your cluster

• Helm 3 (optional, for Helm chart deployment)

Quick Start (5 minutes)

Option A: Single-file manifest

Deploy EDDI + MongoDB with one command:

kubectl apply -f https://raw.githubusercontent.com/labsai/EDDI/main/k8s/quickstart.yaml

The EDDI pod now waits in ContainerCreating — it mounts the eddi-secrets Secret, which no manifest creates. That is deliberate: a Secret shipped in the manifest would be re-applied on every kubectl apply and would overwrite your vault master key, making everything already encrypted with it undecryptable. Generate and store the key, and the pod starts by itself:

set -euo pipefail

# EDDI reads its secrets from a mounted properties FILE rather than environment
# variables, so the Secret holds exactly one key:
# "application-secrets.properties".
#
# mktemp gives an unpredictable name created 0600, so it cannot be pre-created
# or symlinked by another local user.
secrets_file=$(mktemp "${TMPDIR:-/tmp}/eddi-secrets.XXXXXX")
trap 'shred -u "$secrets_file" 2>/dev/null || rm -f "$secrets_file"' EXIT

# Fail closed. This block mints a NEW key every time it runs, so re-running it
# over a live eddi-secrets — after a failed later step, or in a second session
# — would leave everything already encrypted under the old key permanently
# undecryptable. Checked before the key is generated, and `kubectl create`
# below is deliberately NOT piped through `--dry-run=client | kubectl apply`,
# which would replace the Secret silently and exit 0.
if kubectl get secret eddi-secrets --namespace=eddi >/dev/null 2>&1; then
  echo "eddi-secrets already exists — this would install a NEW master key and" >&2
  echo "make everything encrypted under the current one unrecoverable." >&2
  echo "To rotate deliberately: bash k8s/create-secrets.sh --force" >&2
  exit 1
fi

key=$(openssl rand -base64 24)
[ -n "$key" ] || { echo "vault key generation failed" >&2; exit 1; }
printf 'eddi.vault.master-key=%s\n' "$key" > "$secrets_file"

# The key name must be application-secrets.properties — that is the filename the
# Deployment mounts. Passing the temp path bare would name the key after it.
kubectl create secret generic eddi-secrets \
  --namespace=eddi \
  --from-file=application-secrets.properties="$secrets_file"

# Only needed if EDDI was already running with a different key
# kubectl rollout restart deployment/eddi -n eddi

# Access EDDI
kubectl port-forward svc/eddi 7070:7070 -n eddi

Open http://localhost:7070.

Option B: Using the helper script

The Secret must exist before the first apply — kubectl apply -k never creates or reconciles it, so the pod cannot mount it otherwise:

# Clone the repo
git clone https://github.com/labsai/EDDI.git && cd EDDI

# Generate vault key + create K8s secret
bash k8s/create-secrets.sh

# MongoDB runs with authentication — create its Secret too (one password, the
# root user for mongod and the connection string EDDI mounts as a file). The
# Secret IS the stored copy of the password: run this once, never on an upgrade.
# A fresh password over an initialised database locks EDDI out — mongod keeps
# the user it created at first start. The guard refuses when one exists.
if kubectl get secret mongodb-secrets -n eddi >/dev/null 2>&1; then
  echo "mongodb-secrets already exists — keeping it" >&2
else
pw=$(openssl rand -hex 24)
kubectl create secret generic mongodb-secrets -n eddi \
  --from-literal=MONGO_INITDB_ROOT_USERNAME=eddi \
  --from-literal=MONGO_INITDB_ROOT_PASSWORD="$pw" \
  --from-literal=mongodb-secrets.properties="mongodb.connectionString=mongodb://eddi:${pw}@mongodb:27017/eddi?authSource=admin&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=30000"
fi

# Deploy with MongoDB
kubectl apply -k k8s/overlays/mongodb/

PowerShell — PowerShell 7 (pwsh), not the Windows PowerShell 5.1 that powershell.exe starts. The script declares #Requires -Version 7 and refuses to run under 5.1, whose .NET Framework has no RandomNumberGenerator.Fill to generate a key with. Install it or use winget install Microsoft.PowerShell.

pwsh -File .\k8s\create-secrets.ps1
# Once, never on an upgrade — see the bash note above. kubectl create refuses
# with AlreadyExists when the Secret is there.
$pw = [Security.Cryptography.RandomNumberGenerator]::GetHexString(48, $true)
kubectl create secret generic mongodb-secrets -n eddi `
  --from-literal=MONGO_INITDB_ROOT_USERNAME=eddi `
  --from-literal=MONGO_INITDB_ROOT_PASSWORD="$pw" `
  --from-literal=mongodb-secrets.properties="mongodb.connectionString=mongodb://eddi:${pw}@mongodb:27017/eddi?authSource=admin&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=30000"
kubectl apply -k k8s\overlays\mongodb\

The script generates a new master key, so it stops if eddi-secrets already exists rather than replacing what is there — re-running it would leave every secret encrypted under the old key unrecoverable. Pass --force (-Force in PowerShell) only when you mean to rotate.

Option C: Helm

helm install eddi ./helm/eddi \
  --set eddi.vaultMasterKey="$(openssl rand -base64 24)" \
  --set mongodb.rootPassword="$(openssl rand -base64 24)" \
  --set eddi.security.allowUnauthenticatedMcp=true \
  --set eddi.security.allowUnauthenticatedSecretStore=true \
  --namespace eddi --create-namespace

This is the local, port-forward-only shape: no OIDC, so the chart refuses to render until the MCP server and the secrets vault are opted in to unauthenticated access explicitly — without both, EDDI's HighValueSurfaceGuard would refuse to boot. For anything others can reach, enable OIDC instead (eddi.oidc.enabled=true with keycloak.enabled=true or eddi.oidc.authServerUrl) and drop the two opt-ins. Keep the generated mongodb.rootPassword: MongoDB only reads it when its volume is first initialised.

Upgrading to an authenticated MongoDB

Releases from before MongoDB authentication have a data volume that was initialised without a user. The mongo image creates mongodb.rootUsername only on an empty volume, but turns --auth on regardless — so a plain upgrade leaves MongoDB demanding a user that does not exist, and EDDI loses its database. A live helm upgrade detects this and refuses to render. Create the user first, in the still-unauthenticated database, entering the password you will pass as mongodb.rootPassword at the prompt:

kubectl exec -it -n eddi eddi-mongodb-0 -- mongosh admin --quiet \
  --eval 'db.createUser({user: "eddi", pwd: passwordPrompt(), roles: ["root"]})'

# Release WITH OIDC (eddi.oidc.enabled=true):
helm upgrade eddi ./helm/eddi --namespace eddi --reuse-values \
  --set mongodb.rootPassword='<the same password>' \
  --set mongodb.authMigrated=true

# Release WITHOUT OIDC: --reuse-values cannot carry values the old chart never
# had, so the two high-value opt-ins must be added here too, or the chart
# refuses to render (see Option C).
helm upgrade eddi ./helm/eddi --namespace eddi --reuse-values \
  --set mongodb.rootPassword='<the same password>' \
  --set mongodb.authMigrated=true \
  --set eddi.security.allowUnauthenticatedMcp=true \
  --set eddi.security.allowUnauthenticatedSecretStore=true

(eddi-mongodb-0 and user: "eddi" assume release eddi and the default mongodb.rootUsername; the render error prints the exact names for yours.) Existing data is untouched. Leave the opt-ins off an OIDC release: OIDC already satisfies the guard there, and the opt-ins are escape hatches, not settings to carry by default.

Deployment Options

EDDI provides modular overlays (Kustomize) and Helm values for different deployment profiles:

Database Backend

Backend	Kustomize	Helm
MongoDB (default)	kubectl apply -k k8s/overlays/mongodb/ (create mongodb-secrets first)	--set mongodb.enabled=true
PostgreSQL	kubectl apply -k k8s/overlays/postgres/ (create postgres-secrets first — see postgres-secret.yaml.example)	--set postgres.enabled=true --set mongodb.enabled=false --set eddi.datastoreType=postgres

Optional Components

Everything under k8s/overlays/ except mongodb/ and postgres/ is a kustomize Component. Components have no resource set of their own — kubectl apply -k on one does not work by design — and are composed with a database overlay.

Component	Description	Helm Values
Keycloak Auth	OIDC authentication — ⚠️ Kustomize needs the keycloak-admin Secret created first, see Authentication	--set keycloak.enabled=true --set eddi.oidc.enabled=true --set eddi.oidc.publicUrl=http://localhost:8080 --set keycloak.adminPassword=…
NATS JetStream	Durable, ordered messaging	⚠️ needs an image built with -Dquarkus.profile=nats — see Durable Messaging
Monitoring	Prometheus + Grafana	— (Kustomize only: k8s/overlays/monitoring/)
Ingress	External HTTPS access	--set ingress.enabled=true --set ingress.hosts[0].host=eddi.example.com
Production	PDB, NetworkPolicy	--set podDisruptionBudget.enabled=true --set networkPolicy.enabled=true

Manager UI: there is nothing to enable. EDDI serves it from its own Service at /manage.

Composing Kustomize Overlays

Kustomize takes one directory as input. Combine the pieces with a kustomization.yaml that lists the standalone overlay under resources: and the components under components::

# my-deployment/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: eddi
resources:
  - ../k8s/overlays/mongodb        # standalone: base + MongoDB
components:
  - ../k8s/overlays/auth           # Keycloak, realm import, OIDC ConfigMap keys
  - ../k8s/overlays/monitoring     # Prometheus + Grafana

Two rules decide whether this works:

• resources: may name a directory outside your root — kustomize builds it as its own kustomization root — but never a loose file outside your root. That is a hard file ... is not in or below ... error.

• A component's patches are applied into your resource set, so they reach eddi-config and the eddi Deployment. The same patches inside a kind: Kustomization referenced under resources: are built in isolation first, match nothing, and are dropped — with no error and exit code 0.

Ready-made examples are provided in k8s/examples/:

# Create the vault Secret once, before the first apply
bash k8s/create-secrets.sh

# This example includes the auth component, which has no default admin
# password — create its Secret too, or Keycloak never starts
kubectl create secret generic keycloak-admin -n eddi \
  --from-literal=password="$(openssl rand -base64 24)"

# ...and the monitoring component, which has no default Grafana password
kubectl create secret generic grafana-admin -n eddi \
  --from-literal=password="$(openssl rand -base64 24)"

# MongoDB runs with authentication — create its Secret too (one password, the
# root user for mongod and the connection string EDDI mounts as a file). The
# Secret IS the stored copy of the password: run this once, never on an upgrade.
# A fresh password over an initialised database locks EDDI out — mongod keeps
# the user it created at first start. The guard refuses when one exists.
if kubectl get secret mongodb-secrets -n eddi >/dev/null 2>&1; then
  echo "mongodb-secrets already exists — keeping it" >&2
else
pw=$(openssl rand -hex 24)
kubectl create secret generic mongodb-secrets -n eddi \
  --from-literal=MONGO_INITDB_ROOT_USERNAME=eddi \
  --from-literal=MONGO_INITDB_ROOT_PASSWORD="$pw" \
  --from-literal=mongodb-secrets.properties="mongodb.connectionString=mongodb://eddi:${pw}@mongodb:27017/eddi?authSource=admin&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=30000"
fi

# MongoDB + Keycloak auth + Monitoring
kubectl apply -k k8s/examples/mongodb-full/

# PostgreSQL + Production hardening (PDB, NetworkPolicy, resource limits).
# No database password is shipped: create postgres-secrets with the command in
# k8s/overlays/postgres/postgres-secret.yaml.example first.
kubectl apply -k k8s/examples/postgres-ha/

Architecture on Kubernetes

┌──────────────────────────────────────────────┐
│                  Ingress                      │
│            (nginx / traefik)                  │
└──────────────┬───────────────────────────────┘
               │
     ┌─────────▼──────────┐
     │    EDDI Service     │
     │   (ClusterIP:7070)  │
     └─────────┬──────────┘
               │
    ┌──────────▼──────────┐    ┌─────────────┐
    │  EDDI Deployment     │───▶│  MongoDB    │
    │  (labsai/eddi:6.5.0) │    │ StatefulSet │
    │                      │    └─────────────┘
    │  replicas: 1         │    ┌─────────────┐
    │  (single-writer)     │───▶│ PostgreSQL  │
    └──────────────────────┘    │ StatefulSet │
               │                └─────────────┘
    ┌──────────▼──────────┐
    │   NATS JetStream     │  (optional, durable ordering)
    │   StatefulSet        │
    └──────────────────────┘

Security

Vault Master Key

The vault master key encrypts all stored API keys and secrets. If you lose this key, encrypted secrets are unrecoverable.

Three ways to manage it:

1. Helper script (recommended for initial setup):

   bash k8s/create-secrets.sh

2. Manual kubectl — the Secret holds one key, application-secrets.properties, which the Deployment mounts as a file and loads via QUARKUS_CONFIG_LOCATIONS:

   set -euo pipefail
   
   # mktemp gives an unpredictable name created 0600, so another local user
   # cannot pre-create the path or point it at a symlink.
   secrets_file=$(mktemp "${TMPDIR:-/tmp}/eddi-secrets.XXXXXX")
   trap 'shred -u "$secrets_file" 2>/dev/null || rm -f "$secrets_file"' EXIT
   
   # Fail closed on a key that already exists. This mints a NEW one, so
   # replacing a live eddi-secrets makes everything encrypted under the current
   # key permanently undecryptable. `kubectl create` is deliberately not piped
   # through `--dry-run=client | kubectl apply`: that would replace it silently
   # and exit 0. Rotate with `bash k8s/create-secrets.sh --force` instead.
   if kubectl get secret eddi-secrets --namespace=eddi >/dev/null 2>&1; then
     echo "eddi-secrets already exists — refusing to install a new master key." >&2
     exit 1
   fi
   
   # Fail closed: an empty key would create a Secret that silently leaves the
   # vault inert and secrets in plaintext.
   key=$(openssl rand -base64 24)
   [ -n "$key" ] || { echo "vault key generation failed" >&2; exit 1; }
   printf 'eddi.vault.master-key=%s\n' "$key" > "$secrets_file"
   
   # The Secret key must be named application-secrets.properties — that is the
   # filename the Deployment mounts. A bare temp path would name it otherwise.
   kubectl create secret generic eddi-secrets \
     --namespace=eddi \
     --from-file=application-secrets.properties="$secrets_file"

3. External secrets (production): Use External Secrets Operator to sync from AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, etc.

No manifest ever writes this Secret. k8s/base/eddi-secret.yaml.example is a commented template that kustomize does not include, and k8s/quickstart.yaml ships no Secret at all. A reconciled Secret would replace a live vault master key with a placeholder on the next apply, and every secret encrypted under the old key would be permanently undecryptable.

Authentication (Keycloak)

Both delivery paths ship the eddi realm and import it into Keycloak on first boot — k8s/overlays/auth/eddi-realm.json for Kustomize, helm/eddi/files/eddi-realm.json for Helm. Both are cluster-calibrated copies of keycloak/eddi-realm.json, which is calibrated for docker-compose: it allows redirects to localhost only and names a login theme that only the compose file mounts.

Realm import is one-shot. It seeds an empty database and is skipped once the realm exists, so later edits to the JSON do not reach a running Keycloak — change those in the admin console. Keycloak keeps that database on a PVC, so a restart no longer wipes it.

A realm imported from EDDI 6.4.0 — the first release to ship one for Helm and Kustomize — lacks the client scopes that put the user's identity in a token, so EDDI cannot tell its users apart and a non-admin opening their own conversation gets HTTP 500. Run the one-time repair in security.md → Identity claims.

No credential in this component has a default

Neither delivery path ships a password for the Keycloak superuser or for the privileged EDDI account. Both used to, and both were reachable: the keycloak Service is a ClusterIP, so admin/admin and eddi/eddi were a guessable master-realm superuser and a guessable full EDDI administrator for anything running in the cluster — "development component" is a property of the manifests, not of the network they are applied to.

1. Create the Keycloak admin Secret before the first apply (Kustomize only — the pod does not start without it):

kubectl create secret generic keycloak-admin -n eddi \
  --from-literal=password="$(openssl rand -base64 24)"

keycloak-statefulset.yaml reads KC_BOOTSTRAP_ADMIN_PASSWORD from it through secretKeyRef with no default, so a missing Secret fails closed — the pod stays in CreateContainerConfigError with secret "keycloak-admin" not found rather than coming up with a known password. Helm asks for the same value as keycloak.adminPassword, which required refuses to default.

2. Give the eddi account a password after the first boot. The realm seeds it with the eddi-admin and eddi-editor roles and no credential, so it cannot be logged into until you set one: admin console → Users → eddi → Credentials → Set password. Or grant those two realm roles to an account you create yourself and leave eddi unused.

The unprivileged fixtures viewer (eddi-viewer) and user (eddi-user) are opt-in the same way: they ship with their roles and no password, so they cannot log in until you set one. They used to ship as viewer/viewer and user/user, and with the password grant then enabled on the public eddi-frontend client a single curl against a reachable Keycloak returned a token good enough to run LLM turns. The grant is now off on eddi-frontend too — the Manager uses the authorization-code flow and never needed it.

TLS

The realm ships "sslRequired": "external", Keycloak's own default: HTTPS is required for requests from outside the local network, and loopback and private addresses are still served over HTTP. The documented quick start — kubectl port-forward svc/keycloak 8080:8080, browser on http://localhost:8080 — arrives at the pod as 127.0.0.1 and is unaffected, and so is EDDI's in-cluster backchannel to http://keycloak:8080 from an RFC 1918 pod address. Exposing Keycloak on a public hostname now requires TLS in front of it, with the proxy forwarding the HTTPS scheme (X-Forwarded-Proto) so Keycloak sees it.

On a cluster whose pod CIDR is outside RFC 1918 — 100.64.0.0/10 on some managed offerings — Keycloak does not count the backchannel as local and will demand HTTPS for it. Terminate TLS in front of Keycloak, or set the realm's sslRequired back to none on that cluster deliberately.

Four settings have to change together, and none of them can be derived — but they carry two different URLs. Behind an Ingress those are two different hosts, because the Ingress fronts EDDI and not Keycloak.

Three name the browser-facing URL of Keycloak (the IdP, e.g. https://auth.example.com):

Setting	Kustomize	Helm
Keycloak's own hostname (what it stamps as the token issuer)	KC_HOSTNAME in keycloak-statefulset.yaml	eddi.oidc.publicUrl
URL handed to the Manager SPA	EDDI_KEYCLOAK_PUBLIC_URL patch	eddi.oidc.publicUrl
Issuer EDDI validates tokens against	QUARKUS_OIDC_TOKEN_ISSUER patch	derived from eddi.oidc.publicUrl

One names the browser-facing origin of EDDI — where the Manager SPA is served, i.e. ingress.hosts[0].host (e.g. https://eddi.example.com), not the Keycloak host:

Setting	Kustomize	Helm
Realm redirectUris / webOrigins of the eddi-frontend client	edit eddi-realm.json (placeholder https://eddi.example.com/*)	keycloak.publicOrigin

The Manager redirects back to where it is itself served, so the realm has to list that origin. Putting the Keycloak host into redirectUris is the common way to get Invalid parameter: redirect_uri; so is omitting your host altogether. The shipped defaults cover kubectl port-forward svc/keycloak 8080:8080, where both URLs happen to be localhost and the distinction does not bite.

Upgrading an existing Keycloak install

⚠️ Kustomize: delete the old Deployment before applying this version.

kubectl delete deployment keycloak -n eddi

Keycloak used to be a Deployment named keycloak; it is now a StatefulSet under the same name and the same pod labels. kubectl apply -k never prunes, so on an existing install the old Deployment's ReplicaSet keeps running and both pods match the one keycloak Service selector — one with the imported eddi realm, one without. Requests round-robin between them, so /realms/eddi 404s on roughly half of them and logins fail intermittently, with nothing in kubectl get to explain it. Run the delete before the first apply, or immediately after if you have already applied.

Helm needs no manual step: helm upgrade removes resources that left the release. The Deployment→StatefulSet change still replaces the pod, and because the old Deployment had no volume there is no Keycloak state to carry over — the new pod imports the realm into an empty database, and anything configured by hand in the old admin console is gone. Export it first if you need it.

⚠️ Kustomize: restart EDDI after applying the auth component.

kubectl rollout restart deployment/eddi -n eddi

The component patches eddi-config and nothing else, and the EDDI Deployment reads that ConfigMap through envFrom. Environment variables are fixed at container start, so kubectl apply -k updates the ConfigMap while the running pod keeps QUARKUS_OIDC_TENANT_ENABLED: "false" and the three ALLOW_UNAUTHENTICATED escape hatches it booted with. Every object reports as applied and the install stays unauthenticated — the exact state the previously ineffective overlay left behind — until the pod is replaced.

Helm does this on its own: the pod template carries checksum/config and checksum/secret annotations, so a ConfigMap or Secret change alters the Deployment spec and helm upgrade rolls the pod.

Pod Security

EDDI runs as non-root user (UID 185) and is compatible with restricted Pod Security Standards:

securityContext:
  runAsNonRoot: true
  runAsUser: 185
  runAsGroup: 185

No pod mounts a service-account token it does not use (automountServiceAccountToken: false on EDDI, MongoDB, PostgreSQL, NATS, Keycloak and Grafana): none of them calls the Kubernetes API, so a process that gets into one finds no API credential to escalate with. Prometheus keeps its token, which pod discovery needs. Helm exposes serviceAccount.automountToken for the rare setup that adds something needing one.

Datastore authentication and isolation

The in-cluster MongoDB used to run without authentication, behind a Service every pod in the cluster could reach. Any workload could read and rewrite every agent, conversation and the audit ledger. Now:

• Helm runs MongoDB with authentication (mongodb.rootPassword, required — see Option C and Upgrading to an authenticated MongoDB). With networkPolicy.enabled=true the chart also admits only the EDDI pod to MongoDB, PostgreSQL, NATS and Keycloak.

• Kustomize overlays/mongodb runs MongoDB with authentication from the mongodb-secrets Secret you create (see Option B), and overlays/mongodb and overlays/postgres each ship a NetworkPolicy admitting only EDDI. overlays/postgres no longer ships eddi/eddi: postgres-secrets is created by hand from postgres-secret.yaml.example.

• quickstart.yaml stays one file for evaluation: its MongoDB is still unauthenticated, but a NetworkPolicy admits only EDDI.

NetworkPolicies need a CNI that enforces them (Calico, Cilium and most managed offerings do); authentication is what holds without one.

⚠️ Upgrading a Kustomize install whose MongoDB ran without authentication. The mongo image creates the root user only when it initialises an empty data directory, but it switches --auth on every time. So create the user in the running database first, with the password you are about to configure:

kubectl exec -n eddi statefulset/mongodb -- mongosh admin \
  --eval 'db.createUser({user: "eddi", pwd: "<password>", roles: ["root"]})'

then create mongodb-secrets with that password and kubectl apply -k. Done the other way round, mongod restarts with --auth and no user and EDDI cannot log in until you run the same createUser (the localhost exception still allows it). For Helm, see Upgrading to an authenticated MongoDB.

Network Policy

The production overlay includes a NetworkPolicy that restricts EDDI to:

• Ingress: HTTP port 7070 from within the namespace + Ingress controllers

• Egress: Database (MongoDB/PG), NATS, Keycloak, DNS, and external HTTPS (port 443 for LLM APIs)

The datastores carry their own ingress policies — see Datastore authentication and isolation.

Scaling

Single Replica (default)

Default configuration uses in-memory messaging — suitable for development and low-traffic deployments.

Durable Messaging (production)

EDDI runs at exactly one replica. It serialises the turns of a conversation with a JVM-local lock, so a second replica silently drops turns — the Helm chart refuses to render with eddi.replicas above 1 or autoscaling.enabled=true, and every shipped manifest pins replicas: 1. NATS JetStream is a durable ordering and dead-lettering primitive, not a scale-out enabler: the Callable still executes in the JVM that published it. Scale vertically via eddi.resources.

⚠️ NATS needs a purpose-built image. NatsConversationCoordinator is gated on @IfBuildProfile("nats") — a build-time switch — and the published labsai/eddi image is built without it. No Java code reads eddi.messaging.type at runtime either, so setting it does not swap the coordinator: you get a JetStream StatefulSet with a PVC that EDDI never connects to, and in-memory queues anyway. The Helm chart now refuses to render eddi.messagingType other than in-memory unless you also set nats.buildProfileImage=true to confirm you built the image yourself with -Dquarkus.profile=nats; the Kustomize component carries the same warning.

Kustomize — production hardening with PostgreSQL (still one replica):

kubectl apply -k k8s/examples/postgres-ha/

Add NATS on top only with a -Dquarkus.profile=nats image, by listing - ../../overlays/nats under that example's components:.

Helm — with an image you built with -Dquarkus.profile=nats:

helm install eddi ./helm/eddi \
  --set eddi.image.repository=your-registry/eddi-nats \
  --set nats.enabled=true \
  --set nats.buildProfileImage=true \
  --set eddi.messagingType=nats \
  --namespace eddi --create-namespace

Monitoring

EDDI exposes Prometheus metrics at /q/metrics. The EDDI Deployment includes Prometheus scrape annotations by default:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "7070"
  prometheus.io/path: "/q/metrics"

Deploy the monitoring stack with Kustomize. The Helm chart ships no Prometheus or Grafana templates and no longer offers monitoring.* values — they used to exist and render nothing, which read as success:

# Kustomize (with MongoDB + Auth + Monitoring)
kubectl apply -k k8s/examples/mongodb-full/

# Or list `- ../../overlays/monitoring` under `components:` in your own
# kustomization (it is a Component; `kubectl apply -k` on it directly fails)

# Access Grafana — the Prometheus datasource is provisioned for you
kubectl port-forward svc/grafana 3000:3000 -n eddi
# Open http://localhost:3000 — user admin, the password in the grafana-admin
# Secret, which you create before the first apply (Grafana does not start
# without it; it used to run as admin/admin):
#   kubectl create secret generic grafana-admin -n eddi \
#     --from-literal=password="$(openssl rand -base64 24)"

Prometheus discovers pods in its own namespace only, so it now gets a namespaced Role/RoleBinding instead of a cluster-wide ClusterRole. On an existing install delete the old objects by hand — kubectl apply -k does not prune: kubectl delete clusterrolebinding eddi-prometheus and kubectl delete clusterrole eddi-prometheus.

With the auth component, /q/metrics requires a token and an anonymous scrape answers 401 — see Scraping with authentication on.

Health Checks

EDDI provides three probe endpoints:

Endpoint	Probe Type	Purpose
/q/health/live	Liveness + Startup	Process is alive
/q/health/ready	Readiness	DB connected, migrations done, ready for traffic
/q/metrics	—	Prometheus metrics

The startup probe checks liveness, not readiness. Readiness stays DOWN for as long as the startup migrations run, which on the first boot of an EDDI 5 database can take minutes. A startup probe on readiness (as in chart versions before 2.3.0) killed the pod mid-migration, and every restart began the migration again. In the chart its limits are eddi.startupProbe.path, .failureThreshold and .periodSeconds.

Readiness includes the check Agents are ready health check. It is DOWN until the startup migrations have run and the first deployment pass has finished. An agent that fails to deploy does not make it DOWN: one broken agent would otherwise take every replica out of rotation. Instead, the check counts such deployments in its data, as agentsInErrorCount. It gives only the count because /q/health is reachable without authentication. Which agents failed is in the log and in the authenticated GET /administration/{environment}/deploymentstatus. Watch that count after a deploy or an upgrade. A failed deployment is retried after 10 s, then with a doubling delay capped at 5 minutes, and is logged once when it starts failing and once when it recovers.

File Structure

k8s/
├── base/                    # Core EDDI manifests
├── overlays/
│   ├── mongodb/             # MongoDB backend, authenticated (standalone)
│   ├── postgres/            # PostgreSQL backend (standalone; postgres-secret.yaml.example)
│   ├── nats/                # NATS JetStream (component)
│   ├── auth/                # Keycloak + realm import (component)
│   ├── monitoring/          # Prometheus + Grafana (component)
│   ├── ingress/             # Ingress resource (component)
│   └── production/          # PDB, NetworkPolicy, resource limits (component;
│                            #   eddi-hpa.yaml is an intentionally disabled template)
├── examples/
│   ├── mongodb-full/        # MongoDB + Auth + Monitoring
│   └── postgres-ha/         # PostgreSQL + Production hardening
├── create-secrets.sh        # Vault key generator (bash)
├── create-secrets.ps1       # Vault key generator (PowerShell)
└── quickstart.yaml          # All-in-one manifest

helm/
└── eddi/                    # Helm chart
    ├── Chart.yaml
    ├── values.yaml
    ├── files/               # eddi-realm.json, imported by Keycloak on first boot
    └── templates/

Note: Overlays marked (standalone) include the base and can be applied directly with kubectl apply -k. Overlays marked (component) are kind: Component — they have no resource set of their own, so kubectl apply -k on one fails by design; list them under components: in a kustomization that includes a standalone overlay. See Composing Kustomize Overlays.

Note: k8s/base/eddi-secret.yaml.example is a template, not a resource. The eddi-secrets Secret is created out-of-band (bash k8s/create-secrets.sh) so that no re-apply can overwrite a live vault master key.

Troubleshooting

EDDI pod stuck in ContainerCreating

This is the most likely first-run symptom. kubectl describe pod shows:

MountVolume.SetUp failed for volume "secrets" :
  secret "eddi-secrets" not found

No shipped manifest creates eddi-secrets — a reconciled Secret would overwrite a live vault master key on the next kubectl apply. Create it out-of-band and the pod starts on its own, with no restart needed:

bash k8s/create-secrets.sh            # PowerShell: pwsh -File .\k8s\create-secrets.ps1
kubectl get pods -n eddi -w

If create-secrets refuses because eddi-secrets already exists, that is the guard against replacing a live key — the Secret is there and the mount failure is something else (wrong namespace, or the key inside it is not named application-secrets.properties). Check with kubectl get secret eddi-secrets -n eddi -o jsonpath='{.data}'.

EDDI pod stuck in CrashLoopBackOff

Check if the database is reachable:

kubectl logs -n eddi deployment/eddi
kubectl get pods -n eddi

Common causes:

• MongoDB/PostgreSQL not yet ready (wait for StatefulSet pod)

• Incorrect connection string in ConfigMap

• Volume claims pending (check kubectl get pvc -n eddi)

EDDI starts but readiness probe fails

Check the health endpoint:

kubectl exec -n eddi deployment/eddi -- curl -s localhost:7070/q/health/ready

Vault key issues

On a fresh install a missing key can no longer produce a running pod — the Deployment mounts eddi-secrets and the pod stays in ContainerCreating until the Secret exists (see above). "vault master key not set" therefore means the Secret exists but its application-secrets.properties payload has an empty or absent eddi.vault.master-key — for example a Secret created by hand from the old k8s/base/eddi-secret.yaml template. Replace it and restart:

bash k8s/create-secrets.sh --force    # PowerShell: pwsh -File .\k8s\create-secrets.ps1 -Force
kubectl rollout restart deployment/eddi -n eddi

⚠️ --force installs a new key. Anything already encrypted under the old one becomes unrecoverable, so use it only when there is nothing to lose — which is exactly the case when the key was empty.

PVC stuck in Pending

If PVCs aren't provisioning, check your StorageClass:

kubectl get sc                    # List available StorageClasses
kubectl get pvc -n eddi           # Check PVC status
kubectl describe pvc -n eddi      # See events / errors

If your cluster doesn't have a default StorageClass, uncomment storageClassName in the StatefulSet manifests.