Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Metrics & Monitoring

E.D.D.I exposes comprehensive metrics via Micrometer in Prometheus format, covering conversations, tool execution, caching, rate limiting, cost tracking, multi-agent group discussions, scheduled triggers, tenant quotas, audit integrity, and JVM internals.

Quick Start — Grafana Dashboards

E.D.D.I ships three dashboards, all auto-provisioned into Grafana by docker-compose.monitoring.yml:

Dashboard
UID
File
Shape
Use it for

Operations Command Center

eddi-ops

eddi-operations-dashboard.json

51 panels, KPI strip + 9 rows

The front door. Is the platform healthy, and if not, roughly where.

Full Metrics Reference

eddi-metrics-all

eddi-full-metrics-dashboard.json

138 panels, 19 subsystem rows

Every meter E.D.D.I registers. Go here when the number you need is not on the ops dashboard.

EDDI Observability

eddi-observability

eddi-grafana-dashboard.json

16 panels, 6 rows

The original dashboard: Coordinator Health, Pipeline Tasks, Tool Execution, Vault & Security, NATS, HTTP & JVM.

The Full Metrics Reference covers every eddi.* meter the codebase registers, and that is a checked property rather than an aspiration: MetricsDashboardCoverageTest scans the registration sites in src/main/java and fails the build if any meter has no panel. Add a meter without a panel and ./mvnw test tells you, naming the meter and the file that registers it.

All rows but the first are collapsed; open the subsystem you care about. Two template variables scope everything: the Prometheus data source and the scrape job.

Enable Monitoring

# Docker Compose
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

# Or via the install wizard
./install.sh --with-monitoring     # Linux / macOS
./install.ps1 -WithMonitoring      # Windows
Service
URL
Credentials

Grafana

http://localhost:3000

admin / admin

Prometheus

http://localhost:9090

—

Metrics

http://localhost:7070/q/metrics

—

Log in to Grafana with admin / admin, then open Dashboards → EDDI — the provisioned folder holding all three. Grafana's built-in Home is still the landing page; anonymous access is not enabled.

Dashboard Sections

Row
Title
Key Panels

KPI Strip

(always visible)

Uptime, Agents Deployed, Active Conversations, Messages/sec, Tool Success %, Cache Hit %, Error Rate, Cost/hr

Row 1

Platform Overview & HTTP Traffic

Request rate by status (2xx/4xx/5xx), latency P50/P95/P99, CPU usage, top 10 slowest endpoints

Row 2

Conversations

Start/end/processing rate, processing duration percentiles, active gauge, undo/redo, start vs load latency

Row 3

Tool Execution Engine

Success vs failure rate, per-tool execution duration, cached/rate-limited breakdown, per-tool call counts

Row 4

Tool Cache Performance

Hit rate %, hits vs misses, cache size, get/put duration

Row 5

Rate Limiting & Cost

Allowed vs denied, denied by tool, total cost, budget exceeded events, cost accumulation, cost by tool

Row 6

Multi-Agent Group Discussions

Started vs failed, failure rate gauge, discussion duration

Row 7

Scheduled Triggers

Poll/fire/failed, fire duration, claim conflicts, dead-lettered

Row 8

Tenant Quotas & Audit

Quota allowed vs denied, denied by type, audit entries dropped, tenant usage

Row 9

JVM & Infrastructure

Heap/non-heap memory, threads, GC, MongoDB pool, PostgreSQL Agroal pool, NATS messaging

Database-agnostic: Row 9 includes panels for both MongoDB (mongodb_driver_pool_*) and PostgreSQL (agroal_*). Whichever backend is active shows data; the other gracefully shows "No data".

Full Metrics Reference — rows

Overview (open by default) · Conversations · Coordinator & Lifecycle Pipeline · Tool Execution Engine · Tool Cache · Tool Rate Limiting · LLM — Model Cascade & Streaming · Guardrails, Counterweights & Masking · Human-in-the-Loop · Group Conversations & Standing Teams · Scheduling · Persistent Memory — Dream & Summarization · Integrations — MCP, A2A Identity, OpenAI-compatible API · Capability Registry & Connections · Secrets Vault · Tenancy, Quotas & Audit · Platform Operator · NATS JetStream · Backup — Export, Import & Sync · Runtime context (Quarkus / JVM built-ins)

Naming: what the exposition actually looks like

All metrics are served at /q/metrics. Micrometer mostly uses dot notation in source (eddi.tool.cache.hits) and Prometheus renders it with underscores. The rules below are worth knowing exactly, because guessing wrong yields a panel that is silently empty rather than an error. They were verified against Micrometer 1.17.0 with micrometer-registry-prometheus-simpleclient, the registry Quarkus pulls in.

Meter
Source name
Prometheus series

Counter

eddi.tool.calls

eddi_tool_calls_total

Counter, name already ends in _count

eddi.vault.errors.count

eddi_vault_errors_count_total

Counter, name already ends in _total

eddi_group_cost_ceiling_hit_total

eddi_group_cost_ceiling_hit_total — not doubled

Timer

eddi.pipeline.task.duration

_seconds_count, _seconds_sum, _seconds_max

Distribution summary

eddi.llm.cascade.confidence

_count, _sum, _max

Gauge

eddi_agents_deployed

eddi_agents_deployed

Tag key with a dot

.tag("task.id", …)

label task_id

Tag keys that are already camelCase stay camelCase — connection.resolve.time carries authType, not auth_type. Only dots are rewritten.

Timers do not publish percentiles

Every E.D.D.I timer exposes _seconds_sum, _seconds_count and _seconds_max. Only one — eddi.pipeline.task.duration — calls publishPercentileHistogram(), so it is the only one with a _seconds_bucket series and therefore the only one where histogram_quantile() returns anything.

histogram_quantile over any other EDDI timer returns an empty result, which Grafana renders as "No data" — indistinguishable from an idle system. If you want percentiles on a timer, add .publishPercentileHistogram() at its registration site first; it is not free (one series per bucket per tag combination).

One name, one tag shape

A PrometheusMeterRegistry keeps only the first tag-key shape registered under a given metric name and silently drops every later one — no exception, no log line. Registering foo untagged and then foo{tenant,type} means the tagged series never reaches /q/metrics at all.

So a metric must be registered with the same tag keys at every call site. Where you want both a total and a breakdown, tag everything and aggregate at query time with sum(rate(...)) — do not add a second untagged counter under the same name. TenantQuotaService had exactly this bug; TenantQuotaServiceTest (PrometheusExpositionTests) now pins the exposition against a real scrape, because a SimpleMeterRegistry tolerates the collision and will not catch it.

Metrics Reference

Conversation Metrics

Tool Execution Metrics

All execution metrics support a tool label for per-tool breakdown:

Tool Cache Metrics

eddi_tool_cache_bypassed_total counts tool calls where caching was enabled but no identity could be derived to scope the entry to, so the cache was skipped on both the read and the write side. A sustained non-zero rate means tool calls are running without a user id or a conversation id and are paying full tool cost every time — investigate the caller rather than widening the cache scope.

Rate Limiting Metrics

Per-tool and aggregate queries:

Cost Tracking Metrics

Per-tool breakdown — note the _total, which the exposition adds to every counter. eddi_tool_calls{tool="weather"} without it matches no series and returns an empty result rather than an error:

The total-cost gauge was renamed to eddi_tool_costs_accrued. It used to be registered as eddi.tool.costs.total, which the exposition renders as eddi_tool_costs_total — the same name as the tool-tagged counter above. Prometheus refuses two meters under one name with different tag keys, so the first priced tool call threw, and the tool returned that error instead of its result. eddi_tool_costs_total now always means the per-tool counter; take the all-tools total from the gauge, or as sum(eddi_tool_costs_total). A dashboard or alert written against the old gauge must switch to eddi_tool_costs_accrued.

Group Discussion Metrics

Standing Team (Cadence) Metrics

Scheduled Trigger Metrics

eddi_schedule_fire_skipped_total is neither a success nor a failure. The coordinator dropped the turn without consuming the input — the conversation was already IN_PROGRESS or AWAITING_HUMAN — so the schedule is re-armed at its next cadence with failCount untouched, and it will never dead-letter on skips alone. A skip rate that stays high is therefore the one scheduling problem the failure and dead-letter counters cannot show you: a conversationStrategy=persistent heartbeat aimed at a conversation that is never free (a human is chatting in it, or it is parked on a HITL approval) has its message dropped every single cycle while every other metric stays green.

eddi_schedule_firelog_pruned_total counts rows, not sweeps. Flat while the fire-log table keeps growing means either retention is switched off (eddi.schedule.fire-log-retention=0) or the sweep is throwing — the poller logs that failure at ERROR.

Tenant Quota Metrics

Denials carry tenant and type (conversation / api_call / agent / cost); aggregate at query time rather than expecting an untagged total:

Upgrade note. Before this was fixed, the denial counter was also registered untagged, which — per One name, one tag shape — meant the tagged series never reached /q/metrics and the per-tenant breakdown above did not work at all. After upgrading, Prometheus keeps the old label-less samples for its retention window, so breakdown queries should filter them out with {tenant!=""}. The shipped dashboard already does.

eddi_tenant_quota_allowed_total counts slot acquisitions only. The read-only gates (checkAgentQuota, checkCostBudget) deliberately do not touch it, so allowed / (allowed + denied) is not a true accept rate.

eddi_tenant_quota_unavailable_total is the other reason a request is refused: the quota store could not answer at all — a driver failure in whichever store is configured (a MongoException in MongoTenantQuotaStore, which is the default backend, or a SQLException in PostgresTenantQuotaStore) — so the turn is denied for safety without any limit having been reached. It covers both store calls a gate makes: reading the tenant's configuration and incrementing the counter. The read runs first, so on a full outage it is the one that fails — which is why a failing read used to exit as an opaque 500 on MongoDB and to bypass enforcement silently on PostgreSQL, while only a partial outage (reads up, writes down) ever reached the counter. It used to be counted on eddi_tenant_quota_denied_total and answered 429 with Retry-After: 60, so a database outage looked exactly like a tenant burning through its allowance on the very graph you would use to decide whether to raise a limit. It now answers 503 (quota_accounting_unavailable) and carries the same tenant / type tags, so the two can sit side by side:

Coordinator Metrics

queue_depth rising while total_processed flattens is the signature of a backlog: work is arriving faster than it drains.

Pipeline Metrics

eddi_pipeline_task_duration is the only EDDI meter publishing histogram buckets, so it is the only one where histogram_quantile gives a real percentile. See Timers do not publish percentiles.

Model Cascade Metrics

Full guide: model-cascade.md.

accepted_step is the metric that says whether cascading is working. Mass at step="0" means the cheap model is carrying the load, which is the entire point. Mass at the last step means every turn pays for the cheap attempt and the expensive one.

Streaming Metrics

still_over_budget means eviction ran and the context remains too large — the turn proceeds degraded. A persistent non-zero rate is a configuration problem, not a blip.

Attachment Metrics

Full guide: attachments-guide.md.

HITL Metrics

Full guide: hitl.md.

Pauses without matching resumes are approvals nobody answered. Alert on eddi_hitl_pause_count_total - eddi_hitl_resume_count_total growing without bound, not on either alone.

Platform Operator Metrics

eddi_operator_gate_verified dropping to 0 means the Platform Operator's human-approval gate is no longer proven — treat it as a security alert.

Prompt & Guardrail Metrics

Agent Identity & Signing Metrics

Any sustained verify_fail or replay_rejected rate is an attack signal or a misconfigured peer — never routine.

Capability Registry Metrics

Full guide: capability-match-guide.md.

miss_count tagged by skill names exactly which capability your agents cannot serve — the most directly actionable metric in this list.

Secrets Vault Metrics

Full guide: secrets-vault.md.

MCP & Integration Metrics

eddi_channel_observe_decisions_total is one sample per message an observer saw. reason is the gate that settled it: MATCHED (it replied), NO_TRIGGER (the message was not for it), COOLDOWN / DAILY_RESPONSE_CAP / DAILY_COST_CAP (it wanted to and was stopped), or CONTENTION (it could not book the reply because concurrent events kept winning the compare-and-set -- a load signal, not a configuration one). type is the observer's target type — AGENT today, since observe mode is refused on anything else — not the channel platform. A rising throttle share with a flat MATCHED share is an observer whose triggers are too broad for its budget.

Backup, Export & Sync Metrics

The four resource counters explain a successful no-op. An all-skipped run is answered with 200 OK: source and target already agree, nothing was written and no agent version was burned. (201 Created means something was written, 207 Multi-Status that part of it failed — see IRestImportService.) A created line where updated is expected means the matcher is not joining source and target extensions and every sync is duplicating the configuration tree.

Session & Listing Metrics

Audit Ledger Metrics

eddi_audit_sequence_collisions_total is non-zero only on a multi-replica deployment without conversation affinity, where two nodes allocate the same per-conversation chain positions and /auditstore/verify then grades those conversations BROKEN. See Chain sequences and multi-replica deployments.

Dream (Background Memory Consolidation) Metrics

Read the two failure counters together. A summarization failure the service classifies as transient — a socket timeout, a connection refusal, a rate-limit message — is skipped and the cycle carries on, so eddi_dream_summarization_failed_total rises while eddi_dream_cycles_failed_total stays flat. Every other failure aborts consolidation for that cycle and increments both.

So both counters rising together narrows the cause to a non-transient one; it does not identify it. Missing credentials are the common case — a provider 401 lands here, and stale pruning keeps working, so the cycle looks partly healthy — but so does a bad model name, a rejected request or a provider outage that does not present as a timeout. Confirm from the ERROR line the cycle logs: it names the provider, the model and the configured parameter keys, which is enough to tell a missing apiKey from a wrong llmModel without exposing the value. If credentials are the cause, set userMemoryConfig.dream.parameters — see user-memory.md.

Conversation Summarization Metrics

Connection Resolution Metrics

Deployed Agents

NATS Messaging Metrics

Only active when using the NATS messaging profile. Shows nothing under in-memory messaging.

JVM & HTTP Server (auto-exposed)

Standard Micrometer metrics for Quarkus:

Database Connection Pool (auto-exposed)

MongoDB (when eddi.datastore.type=mongodb):

PostgreSQL / Agroal (when eddi.datastore.type=postgres):

Prometheus Alerts

Sample Alert Rules

REST API Endpoints

EDDI also exposes tool metrics via REST:

All of them live under /llm/tools (RestToolHistory):

The pre-v6 prefix /langchain/tools still answers, because LegacyPathRewriteFilter rewrites it to /llm/tools. That is a compatibility shim for existing clients, not a second spelling to write in new code — it can be withdrawn, and it makes stale documentation look correct while it lives.

Monitoring Best Practices

Key Metrics to Watch

Metric
Target
Why

Cache Hit Rate

> 70%

Below this, tool calls are mostly un-cached → higher latency & cost

Tool Success Rate

> 95%

Dropping below indicates tool integration issues

P95 Latency

< 2s

Conversation responsiveness depends on tool speed

Cost Per Request

< $0.001

Runaway costs indicate misconfigured tools or abuse

Audit Drops

= 0

Any non-zero value is a compliance incident

Error Rate (HTTP 5xx)

< 1%

Proxy for overall platform health

Key PromQL Queries

Cache Hit Rate:

Tool Success Rate:

Mean Conversation Processing Latency:

There is no P95 for this timer. See Timers do not publish percentiles — eddi_conversation_processing_duration exposes no _bucket series, so histogram_quantile over it returns nothing at all. Use the mean above, or max(eddi_conversation_processing_duration_seconds_max) for the peak.

P99 Pipeline Task Duration (the one timer that does have buckets):

Cost Per Hour:

Additional Resources

PreviousRelease Notes — 6.0.2
NextMonitoring & Tracing Guide

Last updated

Was this helpful?