> For the complete documentation index, see [llms.txt](https://docs.labs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.labs.ai/reference/changelog/2026-07.md).

# July 2026

> Archived entries for July 2026, newest first. For recent work see [the live changelog](/reference/changelog.md).

***

## 🔓 fix(ci): secret scanning could never pass on a pull request from a fork (2026-07-31)

**Repo:** EDDI (`ci/gitleaks-cli-for-forks`)

The **Secret Scanning** job failed on every pull request opened from a fork. On #623 it was the *only* failing check: twelve passed and seven were skipped. Nothing downstream was blocked: two jobs list `gitleaks` in `needs:` and neither is stopped by it. `docker` is gated on `github.event_name == 'push'` and skips on pull requests; `notify-slack` runs `if: always()` and reads `needs.gitleaks.result` only to pick a status icon, so every fork pull request posted a Slack card with a red Secret Scan marker. The cost is narrower than a blocked pipeline but not harmless: an outside contributor cannot get a green run, a permanently red check trains reviewers to ignore that check, and it cannot be made a required status without blocking every fork.

**Two causes, stacked.** `gitleaks/gitleaks-action` requires a paid `GITLEAKS_LICENSE` for repositories owned by an organization, and GitHub deliberately withholds repository secrets from `pull_request` runs that originate in a fork. So the license was absent by design, not by misconfiguration, and no amount of secret management on this side would have supplied it. The failure is structural rather than incidental, and the open pull requests are a natural experiment. Of the 21 open, 12 come from forks: all 9 that have a Secret Scanning result failed it, and the other 3 never ran it. Of the 9 from branches in this repository, 7 passed, 1 has no result, and 1 (#430, a Dependabot bump last updated in May) failed — its logs have since expired, so that one is unattributed rather than explained. Dependabot is not the discriminator: five other Dependabot pull requests pass.

**The scanner and its wrapper have different licenses.** The `gitleaks` CLI itself is MIT and needs no key at all — only the Action wrapper is commercially licensed. Running the binary directly restores coverage on forks, which is precisely where an unreviewed secret is most likely to arrive. The two alternatives both lose that: dropping the job to `continue-on-error` keeps it green while scanning nothing, and gating it on `github.event.pull_request.head.repo.fork` skips forks outright.

**Pinned by version and checksum, following the existing convention.** This repository already installs a third-party binary this way: `PREFLIGHT_VERSION` and `PREFLIGHT_SHA256` sit in the workflow-level `env:` block and the install step pipes the recorded hash through `sha256sum -c -`. `GITLEAKS_VERSION` and `GITLEAKS_SHA256` join them there and the install step mirrors that shape, so a retagged or substituted release fails the step rather than executing. This is stronger than the commit-pinned Action it replaces: the Action pin covers the wrapper, not the binary the wrapper downloads at runtime. Output is redacted with `--redact=100`, so a real finding never prints the secret into a public log.

**Design decision — scan the event's commits, never the full history.** A whole-repository scan on a full-depth checkout reports **61 findings across 16 commits, dated 2016 to 2026**. Most are in files deleted years ago: Keycloak property files from 2016-2019, vendored licence HTML from 2022, an old `keycloak-dev.json`. The licence files are plainly false positives, licence prose tripping `generic-api-key`. Several of the old Keycloak entries are UUID-shaped, which is what a real Keycloak client secret looks like, so they are worth a maintainer's eye even though the files are long gone and the history is public either way. No values are reproduced here or in the pull request. Whatever their status, none is something a contributor could fix, so repo-wide scope would fail the build for everyone. Scoping to the event range keeps the same scope `gitleaks-action` used, so this is not a coverage change.

**Correction to an earlier revision of this entry.** It said 82 findings across 9 commits, concentrated in `docs/gdpr-compliance.md` and test fixtures, and called all 82 false positives. That was measured on a **shallow clone**: at the graft boundary a commit appears to add entire files, which both inflated the count and misattributed it. On a full-depth checkout, which is what `fetch-depth: 0` gives CI, the real figure is 61 across 16 commits with a different profile. The blanket "all false positives" reading was also not supported by the measurement. The conclusion is unchanged; the numbers behind it were wrong and are corrected here.

**Design decision — the scanner's own config comes from the base branch on a pull request.** The checkout on a pull request is the contributor's tree, and gitleaks reads two files from it: `.gitleaksignore` (fingerprint allowlist) and `.gitleaks.toml` (auto-detected config). Those two files are the whole surface: `gitleaks.toml` without the dot and `.gitleaks.yaml` are not auto-detected, which I checked rather than assumed. Either can switch a finding off. Verified by building the attack: commit a secret, read back the fingerprint gitleaks reports, add it to `.gitleaksignore` in the same pull request, and the scan passes clean. A permissive `[allowlist] regexes` in an added `.gitleaks.toml` does the same, and the repository has no `.gitleaks.toml` today, so adding one meets nothing. `--gitleaks-ignore-path` does not help: the flag is honoured but the repository-root file is still read alongside it. Restoring both from `base.sha` before scanning does, and it keeps the base's own legitimate entries. This was not a regression — `gitleaks-action` read the checked-out config the same way — but fork pull requests were never scanned before, so there was nothing to bypass; making fork scanning work is what makes the boundary matter.

**Design decision — a range that cannot be resolved is an error, not an empty scan.** On a pull request the step requires both `base.sha` and `head.sha` to exist in the checkout and exits non-zero with a `::error::` annotation if either is missing. The tempting fallback is to narrow the range and carry on, but that converts a broken fetch into a scan that passes without having looked at anything — the failure mode that is worst here, because it is invisible. Only the push path falls back, to the tip commit, and it logs that it did. `fetch-depth: 0` on the checkout is what makes both SHAs reachable and is now load-bearing rather than incidental.

**Verification.** Both steps were extracted from the committed YAML and executed verbatim against this repository with the real 8.30.1 binary. Nine scan cases, three install cases and two configuration-bypass cases:

| Case                                                      | Expected                   | Result                                     |
| --------------------------------------------------------- | -------------------------- | ------------------------------------------ |
| Pull request, valid range, no secrets                     | pass                       | exit 0                                     |
| Pull request, `base.sha` unresolvable                     | fail loudly                | exit 1, `::error::`, scanner never invoked |
| Pull request, `head.sha` unresolvable                     | fail loudly                | exit 1, `::error::`, scanner never invoked |
| Pull request, empty range                                 | pass                       | 0 commits scanned, exit 0                  |
| Pull request planting a Bearer token of 40 hex characters | fail                       | `leaks found: 1`, exit 1                   |
| Push, valid `before`                                      | pass                       | exit 0                                     |
| Push, `before` all zeros (new branch)                     | tip commit, logged         | exit 0                                     |
| Push, `before` absent from the checkout                   | tip commit, logged         | exit 0                                     |
| Event that is neither push nor pull request               | tip commit, logged         | exit 0                                     |
| Install, correct version                                  | pass                       | checksum `OK`, ELF x86-64 extracted        |
| Install, tampered tarball                                 | reject                     | `sha256sum -c` exit 1                      |
| Install, nonexistent version tag                          | fail, leave nothing behind | exit 22, no binary written                 |
| Fork allowlists its own secret via `.gitleaksignore`      | blocked                    | exit 1; base's own entries kept            |
| Fork adds a permissive `.gitleaks.toml`                   | blocked                    | exit 1; file removed, logged               |

The detection test matters because the first attempt at it used the AWS documentation example key, which gitleaks allowlists — it reported clean, which would have made a scanner that detects nothing look correct. The planted-secret case is what proves the job is not vacuous.

**Files:** `.github/workflows/ci.yml` (the `gitleaks` job only; no other job, and no application code, is touched).

***

## 🧨 fix(secrets): export was corrupting the configs it exported (2026-07-30)

**Repo:** EDDI (`fix/code-review-validation`)

The Integration Tests job went from **156 failures + 14 errors down to 18 failures, 0 errors** after the fixture and round-trip fixes. The remaining 18 split two ways, and the second one is the serious find.

### 15 of them: the same outer `type`, hardcoded inline

`ComplexRulesAgentEngineIT`, `HitlToolPauseResumeIT`, `HttpCallsAgentEngineIT`, `LlmAgentEngineIT` and `PropertySetterAgentEngineIT` build their output configs as inline JSON strings rather than loading fixtures, and every one of them wrote `"outputs": [{"type": "text", …}]` — the key `OutputConfiguration.Output` never declared. The file sweep could not see them because they are Java string literals. Removed; the inner `valueAlternatives[].type` (a real field) is untouched.

### The other 3: `SecretScrubber` corrupts exported agents

`ImportMergeIT` imports the ZIP **EDDI itself exported** one test earlier, and it 400'd with:

```
No condition for type ${vault:REDACTED} was created (ai.labs.behavior.conditions.${vault:REDACTED})
```

`SecretScrubber` runs on the export path and replaces suspected secrets with `${vault:REDACTED}`. Its second heuristic — Shannon entropy > 3.5 bits/char on any string ≥ 14 chars — cannot tell a long identifier from a long key. Simulating it against the weather-agent export, it rewrites **four ordinary configuration values**:

| Value                                                | Field                        | Entropy |
| ---------------------------------------------------- | ---------------------------- | ------- |
| `dynamicvaluematcher`                                | a behaviour condition `type` | 3.68    |
| `currentWeatherDescription`                          | a property `name`            | 3.57    |
| `properties.count+1`                                 | a `fromObjectPath`           | 3.61    |
| `memory.current.httpCalls.currentWeatherDescription` | a `fromObjectPath`           | 3.87    |

So **export → import was never lossless**: exported agents came back with a condition that resolves to no class, and property names and memory paths replaced by a vault reference.

**This is pre-existing — `SecretScrubber` is byte-identical to `main`, and `main` is green.** It was invisible because nothing validated a ruleset on write: the corrupt condition stored happily, the rule silently never matched, and the import returned 201. #620's write-time rule validation is what turned a silent corruption into a loud 400. The validation is doing exactly its job.

The fix exempts schema-fixed field names (`type`, `name`, `fromObjectPath`, `toObjectPath`, `expressions`, `actions`, …) from the **entropy** heuristic only. A condition `type` names a Java class and a `fromObjectPath` names a memory path — neither can be a credential, so the exemption costs no secret coverage, and field-name detection still runs first, so a field actually called `apiKey` or `token` is redacted regardless. Mutation-checked: with the exemption removed the new test fails with all four values rewritten to `${vault:REDACTED}`.

***

## 🔀 chore(merge): main (#622, caller-bound MCP) into the wave-4a branch (2026-07-30)

**Repo:** EDDI (`fix/code-review-validation`)

`main` moved by 16 commits while #620 was open (#622, the operator-write foundation, plus caller-bound MCP credentials and HITL endpoint patterns), which put the PR into `CONFLICTING`. Worth recording because a conflicting PR has **no computable merge ref, so no workflow can run at all** — the `Integration Tests` failure GitHub was still showing on #620 was a stale result from before the fixes that address it. Resolving the conflict is what lets CI produce a real answer.

Git reported exactly one textual conflict — `docs/changelog.md`, a pure union of entries — and silently auto-merged three code files: `McpToolProviderManager`, `AgentOrchestrator`, and `McpToolProviderManagerAdditionalTest`. The auto-merges were the part worth checking, and one of them was wrong.

### The silent breakage

\#622 made the MCP **tool** cache credential-scoped, not just the client cache: `toolCache` is now keyed on `cacheKey(config)` (`url|<sha-256 of apiKey>`, or `url|anonymous`) so a tool list discovered with one credential can never be served to another. Our F12 TTL tests seeded that cache under the bare URL. After the merge the seeded entry no longer matched the lookup, and the two tests failed *differently*:

* `freshEntryIsServedFromCache` failed loudly — `expected: <1> but was: <0>`.
* `staleEntryIsNotServed` kept **passing, vacuously**. Nothing was served because nothing matched the key, not because the entry was stale. It would have gone on "passing" while testing nothing.

The second is the reason this is in the changelog: a green suite after a merge is not evidence that the merge is correct.

The helper now derives the key by reflectively calling the production `cacheKey` rather than reconstructing it — the same idiom #622's own tests use — so the next change to the key shape carries these tests along instead of hollowing them out. Production was correct throughout; only the test was stale.

`AgentOrchestrator`'s side of the merge was javadoc-only on our branch and merged cleanly; `McpToolProviderManager`'s validation block interleaved coherently (`validateServerUrl` + our `validateTransport` + #622's `validateCallerBoundKey` in one guard, with our one-time deprecated-transport warning after it). Our URL-keyed `deprecatedTransportWarned` set is unaffected by credential scoping, which is right: the transport is a property of the server, not of the credential.

***

## 🧩 fix(rag): two guards disagreed about chunkStrategy and neither test could tell (2026-07-30)

**Repo:** EDDI (`fix/code-review-validation`)

`chunkStrategy` has no reader — ingestion always builds a `DocumentSplitters.recursive` splitter — so an unsupported value is inert. Rejecting it at save time so the author hears about it is right. It was implemented **twice**, at two layers, with opposite intentions:

| Layer                                          | Behaviour                                                                                                                                                 |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `RestRagStore.prepareForWrite` (create/update) | normalise legacy aliases, else **400** — author-facing, correct                                                                                           |
| `RestRagStore.duplicateRag`                    | normalise only, deliberately **no** rejection — "a copy of an existing document must not be refused just because the rules tightened after it was stored" |
| `RagStore.validate` (store, every write)       | normalise legacy aliases, else **throw**                                                                                                                  |

The store hook runs on `create` and `update`, so it silently overrode the duplicate exemption one layer down. Two paths were broken:

* **`duplicateRag`** — refused to copy a document the same store happily serves through `readRag`.
* **ZIP import** — `RestImportService.createNewRags` writes through `createResourceDirect`, i.e. straight to the store with no REST layer in front, and catches only `ResourceStoreException`. An `IllegalArgumentException` escaped and rolled back the **entire agent import** over a field that changes nothing. (`UpgradeExecutor` replays documents the same way.)

### Why the test suite couldn't see it

This is the interesting part. Both layers were tested, and both tests passed:

* `RestRagStoreWriteValidationTest.duplicateDoesNotRejectAnUnsupportedStoredStrategy` asserts duplicate returns 201 — but it builds `RestRagStore` with `mock(IRagStore.class)`, so `create` was a stub and the store's write hook never ran.
* `RagStoreValidationTest.createRejectsUnknownChunkStrategy` asserted the store rejects that exact value.

Neither test crossed the boundary, so the contradiction was invisible. **Demonstrated, not assumed:** with the store's `content.validate()` restored, the new cross-layer test fails 3 of 6 cases with the real `IllegalArgumentException`, while `RestRagStoreWriteValidationTest` stays completely green.

### Fix

The store now **normalises but never rejects**. Legacy aliases (`paragraph`, `sentence`) are still rewritten to the `recursive` they always meant — a data fix that is safe on every path, and it has to live in the store because import bypasses REST. An unsupported value is left **verbatim** rather than rewritten, so a duplicate is a faithful copy of its original. The author-facing 400 stays at `prepareForWrite`, the only layer that can tell "an author typed this" from "this document already exists" — the same layering decision made for `RestMcpCallsStore` in #619.

New `RagStoreLayeringTest` wires the **real** `RagStore` behind the **real** `RestRagStore`, mocking only the storage layer, and pins the division of labour: author input 400s at the boundary; a stored document is duplicable; a direct store write (as import performs it) does not abort; legacy aliases normalise on *both* paths. `RagStoreValidationTest`'s two rejection cases are rewritten to assert leniency, with a comment recording that their original assertion was the defect.

***

## 🩺 fix(configs): EDDI could not read the configs EDDI writes — 8 red ITs, 3 real breakages (2026-07-30)

**Repo:** EDDI (`fix/code-review-validation`)

The Integration Tests job went red on this branch with eight failures across `AgentConfigurationIT`, `AgentDeploymentComponentIT` and `AgentEngineIT`, every one of them nothing but:

```
Expected status code <201> but was <400>.
```

No field name, no constraint, no violation message anywhere in the job log. `main` was green, so the branch caused it. The cause turned out to be **two different mechanisms**, and the second one breaks production, not just tests.

### Mechanism 1 — unknown keys became fatal, and EDDI's own fixtures had three

This branch added `StrictConfigurationBodyInterceptor`, which re-parses inbound configuration bodies with `FAIL_ON_UNKNOWN_PROPERTIES` and 400s on any key the model does not declare. That is the right call for a config-driven engine — a typo'd key in `behavior.json` is a behavioural bug, not something to discard silently — and it immediately earned its keep by catching three keys EDDI had been ignoring for years:

| Key                         | Reality                                                                                                                                                                                                   |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `language` on dictionaries  | The model declares **`lang`**. Every shipped dictionary fixture set `language`, so `lang` was always null.                                                                                                |
| `type` on `outputs[]` items | `OutputConfiguration.Output` declares only `valueAlternatives`. The outer `type` was never read.                                                                                                          |
| `maxOccurrence: "ever"`     | Obsolete pre-v6 key. The current `Occurrence` reads `maxTimesOccurred`/`minTimesOccurred`, so the condition had **no bound at all** — which this branch's new `Occurrence` check now (correctly) refuses. |

All three are corrected in the fixtures. `language` is **deleted, not renamed to `lang`** — renaming looks like the fix the original author intended, but it is a behaviour change: `IDictionary.appliesToLanguage` is `isNullOrEmpty(dictLang) || dictLang.equals(userLanguage)`, so a null `lang` applies to every language while `"en"` applies only when the turn's user language is exactly `"en"`. The ITs never set one, so renaming would have silently switched off the correction dictionaries these tests assert on. Deleting preserves the asserted behaviour exactly; opting those fixtures into language scoping is a separate, deliberate change.

`maxOccurrence: "ever"` becomes `minTimesOccurred: "1"` inside its enclosing `negation` — `NOT(occurred ≥ 1)` is exactly the `maxTimesOccurred: 0` that `agentengine/rules.json` already uses to express the same "not yet welcomed" rule.

### Mechanism 2 — EDDI serialized three keys it could not read back

The setup wizard's 400 had a different cause, and finding it mattered more than the fixtures. `AgentSetupService` does not write to the stores directly: it builds each config as an object and posts it through EDDI's own internal typed REST clients. So every generated config crosses the same strict boundary — and `LlmConfiguration.Task` **serialized `agentMode`, which nothing could deserialize**. `isAgentMode()` is a derived getter (a pure function of `tools`, `enableBuiltInTools`, `a2aAgents`) with no field or setter behind it.

A sweep for the same shape found three in total, now all `@JsonIgnore`:

* `LlmConfiguration.Task.isAgentMode()` — derived from tools/builtInTools/a2aAgents
* `LlmConfiguration.Task.getSystemMessage()` — a read-through of `parameters.systemMessage`, so it also duplicated the prompt into every stored task
* `AgentConfiguration.MemoryPolicy.isEffectivelyEnabled()` — derived from `strictWriteDiscipline`

**This was never only a test problem.** Three paths hand EDDI-serialized configs straight back to EDDI's REST boundary: the setup wizard, EDDI-Manager's GET → edit → PUT, and export → ZIP import. All three would have 400'd on any LLM or agent configuration in production. Ignoring the derived keys also stops persisting values that are recomputed on every read.

### And one silently inert config, six years old

`HttpPostResponse.retryApiCallInstruction` was renamed from `retryHttpCallInstruction` in the v6 http→api sweep **with no `@JsonAlias`**. Jackson discarded the old key, so every config written before the rename lost its retry policy without a word in the logs — including EDDI's own documented reference agent, where the Agent Father's `create_agent` call declares `maxRetries: 3` on 502/503 and has been running with no retry at all. The alias is added: stored and exported JSON configs are the one backward-compatibility contract EDDI keeps, and without it those documents would now 400 outright.

### Four guards, because none of this was reachable from the unit suite

ITs need Docker and so only fail in CI; the strict boundary and the round-trip asymmetry are both checkable in seconds without it.

* **`StrictBoundaryShippedConfigsTest`** — sweeps `src/test/resources/tests` and `docs/agent-configs`, strict-parsing every config against its model (26 checked). Sweeps the tree rather than listing files, so a fixture added later is covered automatically, and prints what it skipped so "passed" can never quietly mean "looked at nothing".
* **`StrictBoundaryRoundTripTest`** — asserts no config model serializes a property it cannot deserialize.
* **`RuleSetStoreShippedRulesetsTest`** — runs every shipped ruleset, and the wizard's generated one, through the real save-time validation.
* **`SetupWizardConfigsPassStrictBoundaryTest`** — round-trips each config the wizard generates, so the wizard can never again 400 on its own output with the cause buried in `"Failed to set up agent: …"`.

**Two of these guards were wrong before they were right, and both mistakes are worth recording:**

* The round-trip test was first written by instantiating each model from `{}` and re-parsing. It **passed on `LlmConfiguration` while the `agentMode` bug sat in its nested `Task`** — a default instance has an empty `tasks` list, so the nested model was never reached. It now uses Jackson introspection to compare serializable against deserializable property sets and walks into property types, and carries a second test asserting the traversal actually reaches `Task` — otherwise the sweep could silently regress to checking only the twelve roots.
* The ruleset sweep initially reported that the setup wizard generates an invalid ruleset: `'inputmatcher' requires a non-empty 'expressions'`. That was **the test's fault** — `IExpressionProvider` was mocked, so every `expressions` value parsed to empty and every `inputmatcher` in every ruleset "failed". With a real `ExpressionProvider` the wizard's ruleset passes. Reported as a product bug it would have been a wild goose chase.

***

## 🧹 fix(llm): bound the streaming warn-suppression key set (2026-07-30)

**Repo:** EDDI (`fix/code-review-validation`)

`StreamingLegacyChatExecutor` suppressed its "configured timeout is below the default" warning by remembering keys in a static `Set`, with a comment claiming it was "bounded because a config edit re-keys the entry". That reasoning is backwards: re-keying **adds** a key and leaves the previous one behind, so config edits were precisely what made the set grow. Keyed on `(taskId, timeoutMs)`, the key space is unbounded over the lifetime of a process.

Now bounded by an explicit `MAX_WARNED_TIMEOUT_KEYS` (1 000) size check that clears on overflow.

**Two decisions worth recording, because the obvious implementations are both wrong here:**

* **Not a Caffeine cache.** This is read on the per-request path — once per streaming turn, again per cascade step — and putting cache machinery there measurably slowed it: two timing-sensitive tests failed against their 66 ms budget when it was tried. A plain key set plus an O(1) size check costs nothing per call. (Isolated by reverting: the same tests pass with the set restored.)
* **Clear on overflow, not refuse to add.** A hard cap would silently stop warning about genuinely *new* misconfigurations — the exact failure this warning exists to prevent. The worst case after a clear is that an already-warned task warns a second time: noise, not silence.

The misleading comment is replaced with the corrected reasoning rather than deleted, so the next reader does not re-derive the same wrong conclusion.

***

## 🔍 fix(all): critical re-review of the applied code-review fixes — 61 defects closed, 11 theatre tests made real (2026-07-29)

**Repo:** EDDI (`fix/code-review-validation`)

After \~120 findings from the external review had been applied across four waves, seven reviewers re-read the whole cumulative diff adversarially — explicitly assuming the previous agents got things wrong — and a mutation runner proved, empirically, which tests would actually fail if their fix were reverted.

**They found 64 defects (4 critical, 10 high) and 11 tests that pass with the fix removed.** Several of the defects were regressions introduced by the fixes themselves; one was worse than the bug it replaced. **61 closed, all 11 mutations now bite.**

### The critical four

* **`PostgresResourceStorage` — the D1 fix broke what it repaired.** The rewritten reverse-lookup emitted the jsonpath operator `@?` unescaped, and pgjdbc parses a bare `?` as a bind placeholder — so on a real PostgreSQL the queries now *threw* where before D1 they merely returned empty. This shipped in the wave-1 merge and was live on `main`.
* **`RestWorkflowStore`** — consequence of the above: cascade-delete of workflow extensions was a permanent no-op on PostgreSQL, and "which agents use this workflow" returned 500.
* **`ConverseWithAgentTool` — F18's delegation guard was inert.** The depth context was attached to `startConversation` only, while the delegated question travelled through `say()` with an empty context — so the callee always read depth 0 and the A→B→A cycle still recursed unbounded. The fix mirrors the `groupDepth` pattern groups already use: attach the context to *every* turn, including the branch that reuses an existing conversationId.
* **`Conversation` — G6 silently dropped data.** Changed-only upserts permanently lost a `longTerm` write whenever the turn that set it never reached teardown (HITL pause, error, cancel), because the next turn's baseline already contained the un-persisted value.

### The one that mattered most

**G2's secret fail-closed was incomplete.** `scrubSecretInput` only rewrote `input:initial` and the `input` output — but `InputParserTask` has already written the plaintext to `input:normalized` in the same step, and every `IData` of a step is serialized into the persisted document. Worse, the scrub was a silent no-op whenever a normalizer was configured, because `{memory.current.input}` resolves to the *normalized* text, not the raw. A secret could still persist despite the fix reporting success.

### Fixes that broke stored configs or existing deployments

* **B7's `modelID` → `modelId` rename** had no legacy fallback, so every stored `gemini-vertex` config using the previously-working spelling silently built a **nameless model**. AGENTS.md is explicit that stored MongoDB/ZIP configs must keep working. Now reads both, preferring the canonical spelling, with a deprecation warning.
* **I3 hard-rejected the MCP `sse` transport** that the config's own javadoc advertised — and in the agent path the throw was swallowed, so an existing agent silently lost *every* tool from that server and burned circuit-breaker budget each turn.
* **B14's fail-closed migration aborted forever.** It refused to proceed whenever the v6 collection merely *existed* — the normal state, since every store constructor creates its collection — and never marked itself complete, so it retried and aborted indefinitely. Now distinguishes "exists and empty" from "exists with data".
* **B9's fail-closed capability table disabled vision on Azure OpenAI**, where the model name is an operator-chosen deployment name that rarely reproduces canonical punctuation (`gpt4o-prod` failed the `gpt-4o` substring test).

### Fixes that quietly negated each other

**E19 and E8 collided.** E19's per-invocation random nonce was embedded in the Qute template *text*, and E8's compiled-template cache is keyed on that text — giving the httpcall output path a **100% cache miss rate** and unbounded churn through a bounded cache. E19's injection-safety is kept; the cache key is now stable.

**G18 and G20 collided.** An entry the ledger dropped locally had already consumed its sequence number, so a store outage or full queue permanently made verification report `BROKEN` — the ledger accusing the deployment of deleting records it had dropped itself.

### The 11 tests that were theatre

A test that passes with its fix reverted is not coverage. The mutation runner proved these did exactly that, and all 11 now fail when the fix is removed — among them: the F18 delegation-depth wiring (the guardrail tests covered the *tool* and the *resolver* but nothing pinned the call site, so hard-coding the depth to 0 left all 11 green), the C10 map-entry removal, B3's configurable drain timeout (a hardcoded 10s passed it), the E19 nonce's unguessability — the actual security property — and three separate holes in the audit ledger's drop/dead-letter accounting.

### Verification

Clean `test-compile`, full suite with **0 non-environmental failures**, and all 11 recorded mutations re-run and confirmed biting. Not verifiable here and left to CI: anything needing Docker or a bound socket — in particular a Testcontainers test executing the corrected JSONB path against a real PostgreSQL, which is what would have caught the `@?` regression in the first place.

***

## ✅ fix(configs): code-review findings wave 4a — write-time config validation (E6) and request-body validation (A11) (2026-07-29)

**Repo:** EDDI (`fix/code-review-validation`)

E6 is the highest-leverage item in the whole review: it converts most of workstream E from *silently wrong* into *loudly wrong at save time*. For a config-driven engine, silent acceptance of invalid config is the worst available failure mode — the agent author gets no feedback and the agent looks healthy while behaving wrongly.

### E6 — the prescribed fix was unsafe, and was not applied as written

The review said: "enable `FAIL_ON_UNKNOWN_PROPERTIES` on the REST mapper only (never the persistence mapper — it needs schema-evolution tolerance)". That instruction assumes a REST-only surface exists. **It does not.**

`SerializationCustomizer.configureObjectMapper` is the shared recipe behind the CDI mapper, the `@PersistenceMapper` mapper *and* the Postgres JSONB mapper. `customize()` is not REST-only either — `PersistenceModule.buildMongoClientOptions` calls `new SerializationCustomizer(false).customize(objectMapper)` on the **MongoDB BSON mapper**, so flipping the flag there would have made **Mongo document reads strict** and broken loading of any stored document written by a newer version. The CDI mapper is also injected into the MicroProfile REST client and \~15 services that parse *third-party* JSON (Slack, web search, Dream, rule deserialization), none of which control their input's shape.

So instead of the flag flip, this adds `StrictConfigurationBodyInterceptor` — a JAX-RS `ReaderInterceptor` scoped to inbound JSON bodies whose target type is a first-party configuration model. It re-parses with a strict *copy* of the REST mapper and translates only `UnrecognizedPropertyException` into a 400 naming the field, its JSON path and the known fields. Every other parse failure falls through unchanged, and an empty body keeps its original stream.

A comment in `SerializationCustomizer` now records why that flag must stay `false`, so the next person doesn't "fix" it.

### E6 — the validation hook

`AbstractResourceStore` gained `protected void validate(T)`, defaulting to a no-op and invoked from `create()`/`update()` before anything reaches storage. Read paths deliberately do **not** call it, so stored documents keep loading.

`RuleSetStore.validate` hoists the wave-1 condition checks to save time **without duplicating them**: it serializes and runs the result through the very same `IRuleDeserialization` the deploy path uses. That covers empty action/input matchers, unknown `occurrence` values, context-type mismatches, empty negations and unknown condition types — each naming the offending rule. Previously all of these surfaced only at agent-deploy time, with a message naming neither rule nor group.

`RagStore.validate` closes the I3 leftover: `RagConfiguration.validate()` already rejected unimplemented `chunkStrategy` values, but its only caller ran at *retrieval* time, so a POST with `"paragraph"` returned 201 and the field was silently ignored.

### A11 — request-body validation

There was **zero** `jakarta.validation` usage in the repo and no validator in `pom.xml`. Added `quarkus-hibernate-validator` (unpinned, via the BOM) and constrained the bodies that actually matter: `DiscussRequest.question` (unbounded free text passed straight to an LLM) and `AttachmentRef.data` (unbounded inline base64, now bounded by a validator deriving its ceiling from configuration rather than a hardcoded magic number).

### Verification

Clean `test-compile` green first attempt. Full suite: 12,688 tests, **0 real failures** — every one of the 308 failures/errors was classified programmatically from the surefire XML against the known environmental markers (this machine cannot bind loopback sockets), not eyeballed. **All six mutation checks bite.**

Because E6 makes previously-accepted input fail, the upgrade path was verified beyond the suite: the bundled `initial-agents` Agent Father config passes the new save-time validation (22 actionmatchers, 23 inputmatchers, 2 negations, all occurrences legal), and the import/export round-trip still works — `RestImportService` writes rulesets through the same store path the new hook guards.

### Known limits, stated plainly

* The interceptor's **JAX-RS provider registration** cannot be verified locally (no loopback sockets); its logic is fully unit-tested but CI is the gate for the wiring.
* `McpCallsConfiguration.validate()` and `LlmConfiguration.validate()` still run at conversation/execution time rather than save time — the same defect class, now trivially fixable via the new hook. Left for a follow-up rather than widened into this change.

## 🔑 feat(operator): the foundation for an agent that can safely write (2026-07-29)

**Repo:** EDDI (`feat/operator-write-foundation`)

Groundwork for a workspace operator agent that manages a deployment — creating and updating agents and groups — while acting as the person chatting to it. Five commits, no new capability granted: the operator's endpoint allow-list is untouched and still read-only. What changes is that widening it is now safe, where before it was neither safe nor functional.

**Generated writes did not work at all.** `McpApiToolBuilder.buildBodyTemplate` emitted Qute variables for a request body but registered none of them, and `AgentOrchestrator` builds the tool schema from `ApiCall.getParameters()` alone. So the model had no documented way to fill a body; with strict rendering off the variables rendered empty and every generated `POST`/`PUT`/`PATCH` went out structurally valid and semantically empty. Adding a write endpoint before this would have produced garbage requests that fail at the far end rather than at the config.

**The body is now one model-written variable.** A per-property template looked more helpful and was worse three ways: every variable became a *required* tool parameter (a `Map<String,String>` has nowhere to record optionality, so a `PATCH` of one field forced the model to restate all the others); values were substituted into the JSON unescaped, since the templating engine runs in `TEXT` mode, so a value containing a quote could break the body or add fields the schema never declared; and the HITL card shows tool *arguments*, so "the arguments are the request" only holds if the body is one of them. The shape a decomposed template implied now lives in the parameter description, which names each property with its type and marks which are required.

**Approval patterns can address the endpoint a tool calls.** Names come from `operationId` or a slug and drift when a spec changes, and `ToolApprovalGate` allows an unmatched call — so a renamed or newly generated write arrived ungated and silently. Method and path were available and discarded one line into registration; they now travel alongside, so a pattern may match a bare name, `source:name`, or `source.method:path`:

```json
{ "requireApproval": ["http.post:*", "http.put:*", "http.patch:*", "http.delete:*"],
  "exempt": ["http.get:*"] }
```

gates every mutation without naming a tool, while `http.post:/agentstore/agents` addresses exactly one. Both speak the same `METHOD /path` vocabulary as the endpoint allow-list, so the two can be generated from one source instead of maintained in two. Documented in [`docs/hitl.md`](/conversations-and-orchestration/hitl.md).

**Design decision — enumerate downward, never upward.** Whether something is gated stays in `requireApproval`/`exempt`; per-endpoint tuning may only *lower* friction. A missed exemption costs an approval prompt; a missed requirement is an ungated write. The same reasoning restricts the method qualifier to `http`: it is the only source whose tools record an endpoint, so `mcp.post:` is rejected at save time rather than saved as a pattern nothing could ever match.

**MCP tool calls can now run as the chatting user.** They previously ran as whatever static credential the config named, and a `${caller:token}` there passed through the global-variable and secret resolvers untouched and was sent as the literal placeholder — failing silently rather than closed. The transport supported this all along: `customHeaders` has three overloads and EDDI used the constant one. The per-request `McpHeadersSupplier` overload makes the credential per-call while the client stays cached.

**Discovery must be told apart from invocation explicitly.** The first version of this decided by asking whether a caller was bound to the thread — but discovery runs *inside* the turn, on a thread that is bound, so `initialize` and `tools/list` went out with the first caller's token. Since the client is cached, that session was then reused by everyone after them, and the tool list reflected one user's permissions while being offered to the next. langchain4j distinguishes the two — `DefaultMcpClient.listTools()` delegates with a null `InvocationContext` while `McpToolExecutor` always builds one — but it does not enforce anything; EDDI has to read `McpCallContext.invocationContext()` and act on it, which it now does. Only a tool call carries the caller; discovery goes unauthenticated on a caller-bound config and the server decides.

**Fixed in passing — a privilege bug.** MCP clients were cached by URL alone, so two agents naming the same server with *different* credentials silently shared whichever client was constructed first. The key now includes a digest of the configured credential: a digest so a literal key never becomes a map key, taken unresolved so configs sharing a vault reference still share a client. This does not multiply clients per user — a caller-bound config yields one client whose supplier reads the caller per request.

**Not solved, deliberately.** On an expired MCP session the transport retries `initialize()` on an HTTP callback thread where the caller binding does not exist. That path now sends the request unauthenticated rather than falling back to the static key under the caller's intent — a visible failure instead of the wrong authority.

**Corrections to earlier analysis, recorded because they changed decisions.** `McpCallsConfiguration.toolsWhitelist` does *not* fail closed — `AgentOrchestrator` skips filtering when the list is empty, the same shape as the approval gate. `toolApprovals` is not agent-only: `LlmConfiguration` carries a per-task override that fully replaces the agent-level block. And a `PUT` on a config does not reach a running agent, because `HistorizedResourceStore.update` creates `version + 1` while agents pin a version — so self-modification takes a chain of write, re-point, redeploy, each independently gated. The thing to guard is whatever can re-point a version reference, not `PUT` in general.

**Verification.** 566 tests pass on a clean build across the affected suites; checkstyle clean. Each fix mutation-checked by reverting it. Four tests written during this work were found vacuous by that check and rewritten — three asserted against a helper or state seeded through the very wrapper under test, one seeded a thread binding that the fix then restored, so it passed either way.

**Next:** per-endpoint approval friction (`timeoutPolicy`, `approvalTimeout` and the pause message are still single scalars for every gated tool), agent-readable documentation (EDDI's MCP client never reads MCP *resources*, so `eddi://docs/*` is reachable from a desktop client but not from an agent), widening the allow-list, and the Manager scope picker and approval surface.

***

***

## ⚙️ fix(runtime): code-review findings wave 3 — concurrency, lifecycle, cancellation, graceful shutdown, Dream wiring (2026-07-28)

**Repo:** EDDI (`fix/code-review-concurrency`)

Third wave of the 124-finding external review. **16 fixed, 1 partial.** This is the wave where the findings were hardest to fix correctly, because the bugs are non-deterministic and several of the obvious fixes are wrong.

### Pre-merge review pass (2026-07-29)

Like #618, this PR reached "approved" without CI or any review bot having seen it — a stacked base disables CodeRabbit, and a base retarget does not fire the CI trigger. A dedicated pass over the 37-file diff produced **6 findings that survived adversarial verification (12 of 18 were refuted) plus 22 from completeness/test critics**, and Copilot found four more once CI could finally run. The high refutation rate is the point: concurrency invites "this looks racy", and verifiers were required to name the interleaving or drop the claim.

* **A destructive primitive behind a missing ownership check (HIGH).** The schedule REST surface never checked `schedule.userId`, which on its own was inert. This PR's `dreamType=dream_consolidation` dispatch armed it: any `eddi-editor` could create and fire a schedule that **bulk-deletes another user's persistent memories**. `RestScheduleStore` already injected `OwnershipValidator` and simply did not use it here. Now admin-or-self on create, update (the re-point path) and `fireNow` — refusing with 403 rather than silently rewriting `userId`, which would hand back a schedule that does something other than what was asked. `system:scheduler` and blank ids stay exempt so existing stored schedules and Manager round-trips keep working.
* **Dream consolidation crossed agent boundaries.** `process()` read `getAllEntries(userId)` — userId-only, agent-unscoped — while every knob it obeyed came from *one* agent's config, so agent A's `pruneStaleAfterDays` deleted agent B's memories and A's model endpoint saw B's text. Cycles are now scoped to the firing agent's own `sourceAgentId` writes, with `crossAgentMaintenance: true` as an explicit opt-in. Newly reachable in this PR, which gave `process()` its first scheduled caller.
* **A transient LLM blip permanently disabled a schedule.** A single failure aborted the whole cycle and marked the fire FAILED, so three consecutive 429s dead-lettered the user's dream schedule. Transient classes (429/timeout/5xx) now skip the group and continue.
* **The B2 interrupt fix destroyed the bookkeeping it was protecting.** The restore in `fire()` ran *before* `logFire()`, and the sync Mongo driver throws `MongoInterruptedException` on connection checkout while the flag is set — so on exactly the interrupt the restore existed to handle, the FAILED fire log was lost and `failCount` never incremented. The flag is now parked and re-asserted in a `finally` after the store round trip, in both `fire()` and the Dream fast-path. The residual half was in `SchedulePollerService`, which ran `markFailed()` on the same still-interrupted thread: the schedule stayed CLAIMED with `nextFire` in the past, was re-claimed every lease expiry, and could never reach `maxRetries` — **an interrupt turned a failing schedule into an unbounded re-fire loop.**
* **A draining node answered 500 instead of "retry elsewhere".** `RestAgentEngine.sayInternal`'s trailing `catch (Exception)` swallowed the `RejectedExecutionException` from the new shutdown gate and rethrew it as a generic 500 — defeating the point of the graceful-shutdown work in this same PR.
* Also: the parallel-phase batch deadline was sized at one member *attempt*, so it always fired first and made the per-member RETRY/ABORT/attributed-SKIP branches unreachable; `maxSummarizationCalls` silently stopped being enforced for stored configs (now honoured as an explicit backstop, deprecated in favour of `maxCostPerRun`); `BaseRuntime` swallowed `onComplete` failures with no identifying context; and `WorkflowStoreClientLibrary` documented an invariant the code neither enforced nor detected — now the component key no longer depends on it at all.

**Docs corrected against the code, not against intent** — the third and fourth instances of that error in this stack, so every claim was re-read out of the implementation: `architecture.md` told operators to create Dream schedules with the `create_schedule` MCP tool, which has **no `metadata` parameter** and therefore cannot set the marker the dispatcher matches on, so the documented procedure produced a schedule that never consolidated (REST is the only working route today, now written out with the two gotchas that bite: a `message` is required for CRON triggers even though the Dream path ignores it, and an unset `userId` defaults to `system:scheduler`, which `DreamService` refuses). `IEventBus` and `InMemoryConversationCoordinator` both claimed the coordinator is selected at runtime via `eddi.messaging.type`; it is `@IfBuildProfile("nats")`, a **build-time** condition, and that property is read by no Java code at all.

**One disagreement adjudicated rather than deferred to severity.** The completeness critic rated the NATS C13/C10 parity gap CRITICAL; two independent verifiers refuted it because `@IfBuildProfile("nats")` keeps that class out of shipped artifacts. Both cannot be right. The code defect is real and was fixed, but the CRITICAL rating was not — it is unreachable unless someone builds with that profile, and the class now records why the two coordinators differ.

**Two tests were relabelled rather than trusted.** The critics caught that both new `GracefulShutdownService` interrupt tests pass identically with and without the fix — `sleepQuietly` restores the flag, so the old code's next `sleep` threw immediately and it exited just as fast. That fix buys accurate logs (an interrupt was being reported as a 30-second timeout), not changed behaviour, and the tests now say so instead of implying coverage they do not have.

### Cancellation that cancelled nothing (C1)

`CompletableFuture.cancel(true)` does **not** interrupt a `runAsync`/`supplyAsync` body — the JDK documents `mayInterruptIfRunning` as having no effect there. Five call sites in `GroupConversationService` relied on it, so "cancelled" agent threads **kept mutating `gc.getTaskList()`, `gc.getTranscript()` and the errors list after the orchestrator had already persisted the document**. Replaced with a cooperative `MemberTurnCancellation` token checked at the agent turn's own await points, plus a bounded drain. This is also the root cause of C7 (`resetStrandedInProgressTasks` could strand the very task it exists to rescue, because a falsely-"cancelled" thread flips state between the snapshot and the mutate).

### The \~100-turn scalability cliff (C4)

`ConversationService` submitted the inner pipeline through the **same** bounded pool as the outer coordinator callable and then blocked on `future.get()`. With no `quarkus.thread-pool.*` overrides that is the 200-thread default: at \~100 concurrent turns every thread is a waiter, no inner task can ever be scheduled, and **every turn fails at the 60s watchdog**. A cliff, not a gradual degradation.

Fixed by routing *nested* submissions to a virtual-thread executor via a `ThreadLocal` marker scoped to the callable body. Chosen over `CompletableFuture` composition deliberately: the coordinator's ordering contract is "the callable returns ⇒ the turn is done", so making the outer non-blocking would need an `IEventBus`/`IConversationCoordinator` SPI change **and** would let the next turn of the same conversation start while the previous one still ran. Virtual threads are safe here — there is not a single `@RequestScoped` bean in `src/main`, and three existing callers already drive the pipeline with no request context on virtual-thread executors. The marker is cleared before callbacks run, so `submitNext` still schedules on the managed executor exactly as before; watchdog and timeout semantics are unchanged.

### Re-execution and lost turns (C3, C9, C10, C13)

* **C9** — `onComplete` sat inside the `try` whose `catch (Throwable)` called `onFailure`, so **any unchecked throw on the completion path resubmitted the already-executed callable as a retry** — LLM calls, tool side effects and cost all running twice. Completion dispatch moved outside the guarded region *and* both callbacks gated behind a one-shot `AtomicBoolean`.
* **C13** — The coordinator retried failed turns 3×. Because `onFailure` can only be raised from inside the executor task, every retry re-ran a turn that may already have called an LLM and spent money. Retry removed entirely; genuinely pre-execution failures surface as a synchronous throw and are handled by C10's rollback.
* **C3** — A timed-out turn still persisted over a newer one, because the stale-completion guard used the interrupt flag and the pipeline cleared it via `Thread.interrupted()`. Replaced with a per-submission abandonment token set *before* delegating `cancel()`, so nothing the work itself does can clear it.
* **C10** — A throwing `submit` left the callable queued with nothing scheduled to run it, wedging that conversation **permanently** and leaking the map entry for the JVM's lifetime.

### No graceful shutdown existed at all (B3)

`grep -rn ShutdownEvent src/main` matched nothing. A rolling deploy dropped every queued and in-flight turn with no drain and no readiness flip. Added `GracefulShutdownService` + `ShutdownReadinessHealthCheck`.

### /rerun destroyed output and regenerated nothing (C5)

Selective execution passes a sublist with `startIndex=0`, so the loop index is sublist-relative — but the component-cache **key** was built from that relative index while the cache **stores** under the absolute index. The output task then ran with `component == null` and no-op'd, *after* the prior output had already been deleted. `indexOffset` was already threaded in and used only for HITL bookkeeping.

This one survived because **every existing `LifecycleManagerTest` stubs the component map empty** — the exact condition that hides it. The new test populates it at absolute indices and fails if the offset is removed.

### B2: the finding's premise was inverted

The review claimed "interrupt flags swallowed in 18 of 20 handlers". Auditing all 28 sites individually (27 explicit `catch (InterruptedException)` plus one hiding behind a broad `catch (Exception)`) found the opposite: **14 already restored the flag correctly and 9 rethrew; only 4 genuinely swallowed it.** Fixed those 4, plus:

* **`ScheduleFireExecutor`** — a broad `catch (Exception)` swallowing `InterruptedException` from `latch.await(5, MINUTES)`, so the poller kept firing schedules after being interrupted for shutdown.
* **`NatsConversationCoordinator`** — the *mirror* bug, not in the finding: it called `interrupt()` **unconditionally** on `catch (InterruptedException | TimeoutException)`, so a drain timeout left the shutdown thread flagged and would abort the `@PreDestroy` steps after it.

One restore is deliberately placed in a `finally` after the store round trips rather than at the top of the catch: the sync Mongo driver aborts with `MongoInterruptedException` when the calling thread is flagged, so an early restore would skip the very `EXECUTION_INTERRUPTED` write that branch exists to perform.

### Dream wired up (I1, G8)

Per the repo owner's decision, `DreamService` is now registered with `ScheduleFireExecutor` rather than deleted, with its ceiling switched from `maxSummarizationCalls` to the dollar-based `maxCostPerRun` the project's own guidance prescribes. `docs/architecture.md`'s claim that it performs scheduled maintenance is now true.

**G8 mattered much more once Dream actually runs**: consolidation upgraded `self` visibility to `global` whenever a group spanned multiple agents — and with `summarizeGroupBy` defaulting to `category` and `preserveAgentProvenance` defaulting to false, cross-agent grouping was the *default* path. Two agents' private memories became one entry every agent could read.

### Partial

**F6** — the REST/pipeline half is done (client disconnect now sets cancelled, and cancellation is checked at more points). The in-`modules/llm` half — cancellation checks inside the tool loop and the cascade — is deferred, since that module is owned by another workstream.

> Disconnect is detected by testing `SseEventSink.isClosed()` before and around each send — **not** by a `ConnectionCallback`, which RESTEasy Reactive does not invoke on this path, as `RestAgentEngineStreaming` documents at the call site. An earlier draft of this entry named `ConnectionCallback`: it described the approach that was tried, not the one that shipped.

### Verification

Clean compile passed **first attempt**, with no repairs needed despite three cross-workstream signature changes. Full suite **as of the original wave-3 work**: 12,633 tests, **0 non-environmental failures** (308 listed failures/errors all carry a loopback/selector/event-loop signature; this machine cannot bind sockets). All four mutation checks bite — C1, C5, C9 and B2 each fail a test when reverted, verified against whole test classes and with surefire reports checked to confirm the new classes actually executed rather than being silently skipped.

> The pre-merge review pass above re-ran the suite after its fixes: **12,912 tests**, failures confined to the same 15 known network-dependent classes. Both figures are real runs at different points — the earlier one is not superseded, it just predates \~280 added tests.

## 🔎 fix(llm): review follow-ups — workflow version parse, log sanitization (2026-07-29)

**Repo:** EDDI (`fix/review-followup-workflow-version`)

Two findings Copilot raised against #618 after it had already been approved, kept out of that PR so they arrive small enough for CodeRabbit to actually review (#618 reached 120 files, past CodeRabbit's 100-file limit, so it merged without ever getting a CodeRabbit pass).

* **`WorkflowTraversal` aborted tool discovery for a whole turn over one malformed URI.** Every other malformed-URI branch in that loop warns, marks the traversal degraded and continues. The version parse did not. `String.replaceAll` returns its input **unchanged** when the pattern does not match, so a workflow URI carrying `?version=abc` passed the `contains("version=")` guard and reached `Integer.parseInt` as the literal string `"version=abc"`. The `NumberFormatException` escaped `discoverConfigs` entirely — so a single bad workflow URI took out httpcall, mcpcall and RAG tool discovery for that turn, rather than skipping the one workflow that was broken. Now matched explicitly, with "present but unusable" treated exactly like "absent" (and a digit run too large for an `int` folded into the same path).
* **`MemoryItemConverter` logged raw exception messages** in both the prompt-snippet and global-variable catch blocks. Exception text can carry user-controlled values, so this is the same CWE-117 class CodeQL flagged five times in #618; routed through `LogSanitizer`.

## 🐳 fix(demo): the Open WebUI seeder did nothing on a second run (2026-07-29)

**Repo:** EDDI (`feat/openai-api-adapter`)

Two defects in `src/main/docker/seed-demo-agent.sh`, both found by actually re-running the stack rather than reading it. Also `.env.example` and [`docs/open-webui-integration.md`](/protocols-and-integration/open-webui-integration.md) §1.

**Adding an LLM key later silently did nothing.** The seeder guarded on *"is any model exposed"* and `exit 0`-ed if so. Because the MongoDB volume persists, the most common second run is exactly the case it broke: the rule-based agent is already there, the user has now set `EDDI_DEMO_LLM_API_KEY`, and they re-run to get an agent that can actually answer questions — and got no new model and no explanation. The guard is now **per agent**, keyed on the model-id prefix each descriptor name slugifies to (`eddi-demo-agent-`, `eddi-llm-demo-`), so each run creates only what is missing and says what it skipped. Vault storage was split into its own step that runs whenever a key is supplied, so changing the key rotates it instead of leaving the old one behind an "already exists" check.

**The final "Ready" listing could omit the agent just created.** The poll exited on `grep -q '"id"'`, which a pre-existing agent satisfies immediately — so a freshly deployed LLM agent, still inside the adapter's 30s model-cache TTL, was absent from the output and looked like a failure. Verified it was only a display problem: re-querying after the TTL showed all four models. The loop now waits for the specific prefixes expected on this run.

**Verified live, not reasoned about.** Ran the stack against a populated volume: first run created the rule-based agent, second reported `already deployed — skipping` and created no duplicate, third (with a key) added the LLM agent and listed all four models. The old code would have exited at step two.

**Also:** the closing "open this URL" line hardcoded port 3000 and was wrong whenever `OPEN_WEBUI_PORT` was remapped — compose now passes `OPEN_WEBUI_URL`. `.env.example` gained an Open WebUI demo section (the demo's `.env` is gitignored, so these variables were undiscoverable from a fresh clone). The docs gained three subsections that only exist because they bit during testing: re-running, port collisions (`Bind for 0.0.0.0:7070 failed` when another EDDI holds the port), and `down` vs `down -v`.

**Doc accuracy pass.** Cross-checked §3 against `application.properties` — all 11 `eddi.openai-compat.*` keys and all 11 defaults match; the §8 error table's 8 codes all exist in `OpenAiErrorResponse`; the documented endpoints match `RestOpenAiAdapter`'s `@Path`s. No drift found there.

***

## 🔒 fix(openai): two CodeQL alerts — logged intent, and a regex flagged as ReDoS (2026-07-29)

**Repo:** EDDI (`feat/openai-api-adapter`)

CodeQL failed the PR with 3 new alerts. Two are addressed here; the third is deliberately left.

**`java/log-injection` (medium, `PostgresUserConversationStore:139`).** The delete path logged `intent` unsanitized. That file predates this branch, but the OpenAI adapter is what makes it carry attacker-influenced data: the intent is `channel:openai:<agentId>:<chatKey>` and the chat key comes from a request header, so a newline in it could forge log entries. Routed through the existing `LogSanitizer.sanitize()`, which the bridge already uses for its own logging. The Mongo store logs nothing, so there was no counterpart to mirror.

**`java/polynomial-redos` (high, `AgentModelResolver.slugify`).** The dash trim `(^-+|-+$)` is replaced with a character walk. **This is not a fixed vulnerability, and should not be read as one.** The alert is a false positive twice over:

1. The `NON_SLUG_CHARS` pass on the line immediately above collapses every run of non-slug characters into a *single* `-`, so `-+` can never match more than one character.
2. Measured directly, the regex is linear anyway — 3ms on 400k separator characters. Java's engine anchors on `$` rather than backtracking, so the quadratic path CodeQL models does not exist here.

It was replaced regardless: a standing high-severity alert competes for attention with real ones, and character walking is no harder to read than the regex was.

**A vacuous test was written and then removed.** The first version of this change asserted `slugify` completed within 2 seconds on 400k separators. Measuring the *old* regex showed it finishes in 3ms — so that assertion would have passed against both implementations and proved nothing. It was deleted rather than shipped; a test that cannot fail is worse than no test. What remains asserts trimming *behaviour* across the implementation swap, and is mutation-checked: stubbing out the trim kills 3 tests.

**Dismissed with justification: `java/user-controlled-bypass` (high, `OpenAiAuthFilter:84`).** The filter returns early when the request path is not under `/v1`, and the path is user-controlled — which CodeQL reads as authentication being skippable.

It is not, because this filter is neither the only nor the first check in front of those paths. `application.properties` ends with `quarkus.http.auth.permission.authenticated.paths=/,/*` at policy `authenticated`, and **Quarkus HTTP authorization runs before JAX-RS request filters** — so every path the filter declines has already been required to authenticate. The one exception is `/v1/*`, which carries its own permission entry at policy `permit` precisely so the shared API key can be checked in the filter rather than rejected at the OIDC layer as a malformed JWT. That `permit` set is exactly what `isGuarded` returns `true` for. The guard therefore does not choose between *authenticated* and *anonymous*; it chooses between *the adapter's key check* and *Quarkus' own*, and declining is the safe branch.

The reasoning lives in a javadoc block on `isGuarded`, not only in the GitHub dismissal, so a reader of the code finds it where the suspicious-looking early return is.

**The dismissal is pinned by a test.** It rests entirely on two lines of configuration, and a security finding waved away on the strength of config that nobody re-checks is how a real bypass eventually ships. `quarkusStillGuardsEverythingThisFilterDeclines` reads `src/main/resources/application.properties` and asserts the catch-all paths, the catch-all policy, and the `/v1/*` exemption. Flipping the catch-all policy to `permit` fails it. Note it reads the file from *source*: the first version loaded `/application.properties` from the classpath, where `src/test/resources` shadows the production file and declares none of these keys — it asserted nothing and failed loudly on first run.

***

## ✨ feat(openai): render structured outputs, report token usage (2026-07-28)

**Repo:** EDDI (`feat/openai-api-adapter`)

Two gaps closed in the `/v1` adapter, both from the honest support assessment of the previous session. Files: new `OpenAiOutputRenderer`, new `TokenUsage` + `StreamOptions` DTOs, changes to `OpenAiConversationBridge`, `OpenAiSseWriter`, `RestOpenAiAdapter`, `ChatCompletionResponse`, `ChatCompletionChunk`, `ChatCompletionRequest`. Docs: [`docs/open-webui-integration.md`](/protocols-and-integration/open-webui-integration.md) §7.1, §7.2, §10.

**Structured outputs are no longer dropped.** An EDDI turn carries eight output types; the OpenAI protocol carries one string, and the shared `ConversationOutputExtractor` keeps only the text. Through that lens a wizard agent whose whole turn is *"Which provider?"* plus five quick replies arrived as a question with no visible answers — indistinguishable from a broken agent. `OpenAiOutputRenderer` now takes the shared extractor's text **verbatim** (a reply's wording must not depend on which channel it left through) and appends a Markdown rendering of the rest: quick replies as backticked values, images as `![alt](uri)`, application links as Markdown links, buttons as their label, input fields as a described prompt. `agentFace` and `other` are dropped deliberately — an avatar has no text equivalent, and captioning it would add a line the agent author never wrote.

* **The shared extractor is untouched.** Its other callers (`GroupConversationService`, `CreateSubAgentTool`, `ConverseWithAgentTool`) feed agent-to-agent prompts, where interaction affordances are noise. The renderer is adapter-local.
* **Quick replies render as literal values, not a numbered list.** A numbered list invites `2` as an answer, which no input matcher recognises. The `value` is what a chat UI puts on a button and therefore what a user would retype; `expressions` stays internal (asserted by test — it is an internal identifier that must not be shown).
* **Both POJO and Map item shapes are handled** — a turn that just ran yields typed items, a rehydrated conversation yields Maps.
* **Streaming needed a split.** When the model streamed the prose token by token, re-rendering the full text at `onComplete` would have sent the whole reply twice, so `renderExtras()` emits the affordances alone.

**`usage` is now reported — the previous entry's claim that EDDI does not surface token counts was wrong.** `LlmTask` writes `audit:token_usage` (`inputTokens`/`outputTokens`/`totalTokens`) into the current step, accumulated across every model call the turn made — cascade steps and tool round-trips included.

* **This required `returnDetailed=true` on both `say` and `sayStreaming`.** The filtered snapshot keeps only `input:initial`, `actions`, `output*` and `quickReplies*`, dropping every audit key — which is why the counts looked unavailable. The snapshot is read in-process and never serialized to the client, so the cost is one extra step's worth of references. This flag is load-bearing and otherwise invisible, so a test pins it: flipping it back would silently remove `usage` from every response with nothing else failing.
* **Absent, not zero, for rule-based agents.** They call no model; `0 tokens` reads in a client as a measurement rather than an absence.
* **`totalTokens` is derived when a provider omits it** but reports both parts — a usage block whose parts do not add up is worse than one that computes the sum.
* **Streaming usage is opt-in via `stream_options.include_usage`**, emitted as a trailing empty-`choices` frame after `finish_reason` and before `[DONE]`, per spec. An unrequested empty-choices frame is a protocol deviation some clients reject.

**Testing:** 183 adapter tests (up from 151) — new `OpenAiOutputRendererTest` (21) plus usage coverage in the bridge, SSE writer and wire-format suites. Two mutation checks were run rather than trusting green: stubbing `renderQuickReplies` to `null` killed 7 tests, and reverting `returnDetailed` to `false` killed the 2 flag-pinning tests.

**Not done: exposing agent groups as models.** Assessed rather than assumed — every piece exists (groups list via `readDescriptors("ai.labs.group", …)`, `discuss()` returns a `synthesizedAnswer`, `continueDiscussion()` gives multi-turn, `GroupDiscussionEventListener` gives streaming), but it needs a second bridge with its own conversation mapping, streaming path and approval surface. That is a feature-sized change, not an addition to this one, so it is recorded as gap #1 in §10 instead of half-landed here.

***

## 🧠 fix(llm): code-review findings wave 2b — LLM core, persistent memory, migration, import/export (2026-07-28)

**Repo:** EDDI (`fix/code-review-llm-memory`)

Second half of wave 2, stacked on the access-control PR. Covers the LLM tool pipeline, the persistent-memory subsystem, the v5→v6 migrations, and import/export.

### Pre-merge review pass — and corrections to the claims below (2026-07-29)

This PR reached "approved" having been reviewed by **no CI run and no review bot**: CodeRabbit reported `Review skipped: reviews are disabled for this base branch` because it was stacked on a disabled base, and a base-branch retarget fires `edited`, which is not in the default `pull_request` trigger set — so no workflow ever ran on its head SHA. A dedicated review pass over the 91-file diff found **16 findings that survived adversarial verification**, plus 21 from completeness/test critics. All are fixed here except one, named below. **Several entries further down overstated what the code delivered; those claims are corrected here rather than quietly edited away.**

Fixes with user-visible consequence:

* **F14 broke the error path it was meant to unify.** Routing rule-triggered MCP calls through `ToolExecutionService.executeToolWrapped` was right in intent, but that wrapper *catches every exception and returns an error string*. `RetryConfiguration.executeWithRetry` therefore never saw a throwable: retry never retried, `continueOnError` became dead code, and a failed MCP call was **stored as a successful response**. The metering wrapper is kept; a real failure signal is restored on top of it.
* **F18's delegation-depth guard was inert in production.** `delegationDepth` was injected only into the callee's `startConversation` context, landing on step 0; the follow-up `say` carried no context, so the turn that actually decides delegation read nothing. The claim below that F18 was "mutation-checked" was true of the *test*, not of production — the test drove the mechanism directly and never crossed the `say` boundary where the value was lost. The depth now propagates to the turn that reads it.
* **F15 was fixed on two of three merge routes.** Within-server dedupe and the `AgentOrchestrator` source merge were handled; the cross-server merge in `discoverTools()` was still `addAll`/`putAll`, so two MCP servers advertising the same tool name still shadowed silently.
* **G12 / G5 / G7 shipped on MongoDB only.** The "a turn is never silently discarded" guarantee, the `most_accessed` recency reservation, and global-entry ownership preservation were all absent from the PostgreSQL adapter — which answered benignly rather than signalling the gap. Now ported, with tests. **This is the third cross-backend gap in this stack** (after schedule `userId` and the audit `sequence`), which is no longer coincidence: each was a feature built against one backend, silently missing on the other, and invisible because the degraded answer looked like a normal one. The cross-backend conformance suite (D4/J3) is the real fix and remains outstanding.
* **A summarizer could be handed another vendor's API key.** When `conversationSummary` names a different `llmProvider` than the parent task, the parent's resolved parameters — `apiKey`, `baseUrl` — were inherited and passed to that other vendor's client. Not theoretical: the pre-PR POJO defaults serialized `llmProvider: "anthropic"` into stored configs. Credential- and endpoint-bearing keys now stop at a provider boundary; vendor-neutral tuning keys still travel.
* **Validation moved to the boundary where rejecting is safe.** `McpCallsTask.configure()` and RAG retrieval were both throwing/dropping on *stored* configs — a workflow-load failure and a silently empty knowledge base respectively. Both are now lenient on read (stored configs stay loadable, which is the one backward-compat contract that matters) and strict on write, in `RestMcpCallsStore` and `RestRagStore`.
* Also fixed: the v6 rename migration aborting permanently when a v6 collection merely *exists* (EDDI creates those itself via `createIndex`) instead of skipping; the new pre-migration backup duplicating conversation transcripts outside the reach of GDPR erasure; export cleanup recursively deleting a shared `tmp/<agentId>` it could not prove it created, reaching `tmp/import/`; and the streaming no-partials fallback overwriting the `warning` key that `responseValidation` dispatches on, silently skipping `onTruncation`/`onContentFilter`.

**Still not fixed — I5.** `AgentConfiguration.maxCheckpointsPerConversation` is *still ignored at runtime*. The test named `explicitRetentionIsHonoured` exercises an overload no production path calls, so it proved nothing; the misleading label is removed rather than left to imply coverage. Wiring the value through needs a session-scope slot on `IConversationMemory` and propagation via `IAgent`/`Agent`, which is a larger change than a review fix should smuggle in. **Any statement below that I5 is complete is wrong.**

One critic finding was investigated and **rejected**: the `ExpressionFactory.setDomain` removal was claimed to change `and(...)`/`or(...)` parsing, but domain splitting happens only in `setExpressionName(String)` and the parser builds children through constructors that assign the name verbatim — the deleted line really was a no-op.

### The tool pipeline had a second door

* **F14** — `executeToolWrapped` is genuinely well-built and has one production call site, so every one of the seven tool sources routes through it. Except **`McpCallsTask` doesn't** — a behaviour-rule-triggered lifecycle task invoking the *same* external MCP tools, which resolved a `ToolExecutor` and called it directly. It also bypassed `ToolApprovalGate`, so **`hitlConfig.toolApprovals` gated LLM-initiated calls and not rule-initiated ones** — a human-approval gate with a hole in it.
* **F18** — `converse_with_agent` had **no guardrails at all**: it never consulted `DynamicAgentConfig`, and `allowDelegation` was never checked anywhere. Agent A could call B, which calls A — and with no `conversationId` a *fresh* conversation starts, so the busy-guard never breaks the cycle. Prompt injection in a user message was sufficient to start it. Now enforces `allowDelegation`, a target allowlist, and a delegation-depth counter propagated through conversation context (reusing the mechanism groups already use for `groupDepth`).
* **F17** — `maxCreatedAgentsPerDiscussion` was enforced **per turn, not per discussion**, because `sharedCreatedIds` was created fresh in every `buildToolList` call. A 5-member × 3-phase discussion with the default cap of 5 permitted **up to 75 agents deployed to production**.
* **F15/F16** — Remote MCP tools silently **shadowed built-in tools** (specs accumulated in a `List`, executors in a `Map`, so duplicates reached the model and last-write-won), and their **descriptions entered the prompt verbatim** with no length cap or sanitisation — whitelisting operates on names, so a whitelisted tool whose description changes was ungoverned.
* **A10** — The MCP client did **no URL validation at all**, unlike its A2A sibling, while a discovery endpoint echoed the response body — a full SSRF read primitive.

### Two real bugs found by the tests, not by the review

The review's F12 fix (cache the workflow traversal that runs 3–4× per LLM task per turn) introduced two defects that only surfaced when `WorkflowTraversalTest` started returning 0 instead of 1:

1. **The cache memoized failure-derived results.** A traversal whose workflow read threw still cached its empty result for the full TTL, and replayed it to the other traversals of the same turn — an agent silently losing its httpcalls/mcpcalls/RAG configuration with nothing in the logs but one WARN. Now only complete traversals are cached.
2. **The cache key omitted the target class** while the value was cast with an unchecked `(List<StepConfig<T>>)`. Justified by a comment asserting a 1:1 mapping that nothing enforces — any future caller asking for the same step type with a different class would get another caller's entry and a `ClassCastException` from a cache hit with no connection to the calling code.

This is why the triage pass asked "stale test, bad fixture, or **real bug**?" for every failure rather than adjusting tests until green.

### F13 was fixed but unreachable

The wave-2 agent added `inheritedParameters` overloads to `SummarizationService` and tested them directly — but **no caller in `src/main` passed them**. `ConversationSummarizer` still called the 4-arg overload, so the rolling summary still could not authenticate and still silently never materialised. Threading the parent task's resolved parameters through `LlmTask → ConversationSummarizer → SummarizationService` is what actually closes it; both hops now have tests that fail if the parameters are dropped.

`DreamService` remains on the un-inherited path — it is a background job with no parent task, so it needs a credential source of its own. That is part of wiring Dream up (finding I1), scheduled for wave 3.

### Memory & properties

* **G2** — `scope: "secret"` **failed open to plaintext**. On vault failure the method returned the plaintext *before* the scrub block, persisting the secret twice: as a conversation property *and* as raw `input:initial` data. Vault-disabled is the **default** (`eddi.vault.master-key` ships empty), so this was the common path. Now fails closed, scrubbing first.
* **G1** — User-memory search and delete **crossed agent boundaries**: `getVisibleEntries` builds a proper self/group/global filter, but `filterEntries` and `getByKey` filter on `userId` alone — and the tool path used that unscoped pair.
* **G13** — Token-aware windowing could emit a prompt with **no user message at all**: the backward fill breaks on the first message that doesn't fit, and if the anchors alone exceed the budget the code only warned. The model then answered with no idea what was asked. The final user message is now reserved first; anchors get trimmed instead.
* **G12** — A turn could be **silently lost**: `replaceOne` with no upsert whose `UpdateResult` was discarded, so a conversation deleted mid-turn by erasure or a retention sweep discarded the turn while the caller got a normal response.
* **G5/G6/G7** — `most_accessed` recall did an N+1 write *inside* an open read cursor and was self-reinforcing (only already-top-N entries got incremented, so a new entry could never climb in); `storePropertiesPermanently` refreshed `updatedAt` on every longTerm property every turn, so `most_recent` degenerated to "everything is recent" and `deleteOlderThan` never expired anything for an active user; and re-upserting a recalled entry **silently flipped its owner** to the reading agent.
* **G9/G10/G11** — `ConversationProperties` broke the `Map` contract (`clear()`/`remove()` left the template map stale, so a checkpoint rollback left post-checkpoint properties visible); `scope: "step"` was persisted despite the docs; one malformed field failed the entire conversation load.

### Migration & import/export

* **B12** — Migration **irrecoverably erased typed BSON values**: it deleted the legacy `value` field unconditionally but only wrote a replacement for String/Map/Integer/Float, dropping doubles, longs, booleans and arrays with no error.
* **B13** — The template migrator rewrote **any** `{...+...}` sequence, corrupting JSON bodies and arithmetic that merely sat in a document containing Thymeleaf syntax.
* **B14** — The rename migration **skipped when the v6 collection already existed** and still marked itself complete, abandoning the v5 data.
* **D11/D12** — Import had no rollback (any failure mid-way left every already-created resource orphaned), and neither import nor export ever deleted their temp directories — `ZipResourceSource` is `AutoCloseable` and its `close()` does the cleanup, but two call sites constructed it outside try-with-resources.

### Verification

Full unit suite: 12,384 tests, **0 non-environmental failures**. G2 and G12 mutation-checked, plus a second sharper mutation for G2 that removes only the input-scrub call — both halves are independently covered. F13's new wiring is mutation-checked at the `LlmTask` hop.

## 🧵 fix(security): carry caller identity across the cascade and group dispatches (2026-07-28)

**Repo:** EDDI (`feat/caller-identity-passthrough`)

The caller binding is a `ThreadLocal`, and four more dispatch sites hand a turn to a fresh virtual thread without carrying it. A `${caller:token}` apicall reached from any of them failed closed with "the conversation turn has no authenticated caller" — safe, but invisible to the agent designer and dependent on unrelated configuration.

* `CascadingModelExecutor:577` — every cascade step runs on `TIMEOUT_EXECUTOR`. The same agent config worked or failed purely on whether `modelCascade` was set.
* `GroupConversationService:282` — the whole discussion is dispatched to a virtual thread, so *no* member agent had a caller.
* `GroupConversationService:2366` — parallel-phase speakers fan out to further threads.
* `GroupConversationService:3772` — the HITL resume path.

`CallerIdentityContext` gains `withIdentity` for `Runnable`/`Callable`, `withIdentitySupplying` for `Supplier` (needed by `CompletableFuture.supplyAsync`), and `captureOrCurrent()`, which prefers the active request and falls back to the thread's binding — the group discussion is dispatched from the REST thread, the cascade from mid-pipeline.

**Design note.** The `Supplier` variant is named apart from the `withIdentity` overloads on purpose: a value-returning lambda satisfies both `Callable` and `Supplier`, so same-named overloads are ambiguous at every call site.

**Review round two** found a fifth site and two flaws in the wrapper itself:

* `GroupConversationService:1788` — the task-force EXECUTE phase fans out again through `CompletableFuture.runAsync`, so every task wave lost the caller. Missed first time because the search pattern covered `submit` and `supplyAsync` but not `runAsync`.
* The parallel-phase fan-out captured with `current()`, which is null when `discuss()` is called synchronously on the REST thread — only `captureOrCurrent()` sees the request there.
* **A null identity was a no-op**, so work dispatched *without* a caller inherited whatever binding the pooled thread still carried from the turn before. It now binds null, masking it.
* **Nested wrappers cleared instead of restoring**, so an inner wrapper wiped the outer caller and the rest of that turn ran unauthenticated. The previous binding is now saved and restored.

**Note on scope.** A group member agent now acts as the person who started the discussion. That follows the feature's model — the operator acts as the chatting user — but it is worth stating, because a member agent's reads are now attributed to that user in the audit trail.

1204 tests pass across the affected suites. Neutering the `Runnable` wrapper fails the propagation test, so the binding is pinned rather than assumed.

***

## 🔓 fix(security): ${caller:token} never reached its resolver (2026-07-28)

**Repo:** EDDI (`feat/caller-identity-passthrough`)

A critical review of the caller-identity PR found the feature was **dead on arrival**, and the same defect silently disables two older documented features.

`ApiCallExecutor.buildRequest` runs every header value through `prePostUtils.templateValues` *before* the caller-identity, global-variable and vault resolvers. `TemplatingEngine`'s trigger regex `\{[a-zA-Z#/!]` matches `{c`, so `Bearer ${caller:token}` is handed to Qute, which parses `{caller:token}` as a namespaced expression. An unresolvable namespace is a hard failure regardless of `strictRendering`, and no `caller` resolver existed. Proven against the project's own build:

```
Bearer ${caller:token} -> THREW: No namespace resolver found for [caller]
Bearer ${vault:my-key} -> THREW: No namespace resolver found for [vault]
${vars:default-model}  -> THREW: No namespace resolver found for [vars]
```

So `${vault:...}` and `${vars:...}` in apicall headers have never worked either.

**Fix.** `CallerNamespaceResolver` returns the `caller` placeholder verbatim so it round-trips through templating to the real resolver. It resolves nothing itself.

**Deliberately `caller` only.** `vault`/`eddivault` keep failing loudly in templated positions. Letting them through would widen where a secret is substituted, and the resolved request *body* is written to conversation memory unscrubbed — a vault reference in a body template would put a plaintext API key into MongoDB. Docs claiming vault works in apicall headers were the bug, not the behaviour.

**Why no test caught it:** all three `ApiCallExecutor` suites stub templating as a pass-through, and `CallerIdentityResolverTest` calls the resolver directly. Nothing exercised header -> Qute -> resolver. `CallerNamespaceResolverTest` now does, including a guard-rail test that fails without the resolver and one asserting vault still refuses.

**Also fixed**

* `setup-api` hardcoded `null` for the LLM base URL while the manager sent Ollama's URL as `apiBaseUrl` — the *tool target* — pointing every generated tool at the model server. `CreateApiAgentRequest` gains an `llmBaseUrl` field, passed to `createLlmConfig`.
* The by-value token redaction added last round was untested through `ApiCallExecutor`; deleting the call kept every test green. Now covered with a real resolver and mutation-checked.
* Docs overclaimed: `${caller:userId}` is resolved in headers and query parameters, not "anywhere"; `rejectTokenReference`'s Javadoc claimed request bodies are checked.

***

## 🔐 feat(security): forward the caller's identity to apicall headers (2026-07-28)

**Repo:** EDDI (`feat/caller-identity-passthrough`)

An agent could only call an API with a *static* credential baked into its apicall config. That is the wrong shape whenever the API being called is EDDI's own: an OIDC token expires within the hour, cannot be least-privilege, and collapses every action to one synthetic principal in the audit trail. The EDDI-Manager "Platform Operator" needs exactly this — an agent that reads the platform on behalf of whoever is chatting with it — and could only be built by smuggling the user's bearer through the per-turn conversation `context`, which persists the token to MongoDB.

**What changed.** Apicall *headers* may now reference the authenticated caller:

* `${caller:token}` — the caller's raw bearer token
* `${caller:userId}` — the caller's principal name (not a secret)

`ApiCallExecutor` resolves these last in the header chain (after global variables and vault refs), because resolution needs the target URI.

**Files**

* `engine/security/CallerIdentity.java` — record (token, userId, origin); deliberately not part of `IConversationMemory`.
* `engine/security/CallerIdentityContext.java` — captures the identity on the REST request thread and binds it to whichever pool thread runs the turn.
* `engine/security/CallerIdentityResolver.java` — the `${caller:...}` resolver, mirroring `SecretResolver` / `GlobalVariableResolver`.
* `engine/security/OriginMatcher.java` — scheme/host/port comparison with default-port normalization.
* `engine/internal/ConversationService.java` — captures in `processConversationStep` and decorates the callable.
* `modules/apicalls/impl/ApiCallExecutor.java` — resolves in headers; rejects a token reference in a query parameter.

**Threading — the non-obvious part.** A turn is built on the request thread but executed twice removed from it: `submitInOrder` hands it to a coordinator thread, which hands the pipeline to *another* thread via `runtime.submitCallable`. Request-scoped beans (`SecurityIdentity`) resolve at none of those points. So the identity is captured while the request context is still live and travels **with the callable** (`withCallerIdentity`), not with a thread. Binding the coordinator thread would have missed the pipeline entirely — an easy and silent mistake.

**Design decisions**

1. **Same-origin only.** The token is released only when the outbound call targets the exact `scheme://host:port` the caller addressed, taken from the inbound request rather than configuration. An agent config naming a third-party host therefore cannot exfiltrate a user's token, and the feature needs no allow-list to be safe out of the box.
2. **Headers only.** `${caller:token}` in a query parameter or request body is rejected — tokens in URLs leak through access logs, proxies and browser history, and outside a header the reference is never substituted. `${caller:userId}` is resolved in headers and query parameters.
3. **Fails closed.** An unsatisfiable reference throws rather than resolving to `""`, which would silently send `Bearer` and surface far away as a puzzling 401.
4. **Never stored.** Resolution happens while building the request, and `scrubSensitiveHeaders` redacts it before the request is written to conversation memory. Header-*name* matching alone was not enough — a token placed in an unconventionally named header would have slipped through — so the resolved token is additionally matched by value.
5. **Opt-out.** `eddi.caller-identity.enabled` (default `true`) forbids the feature outright.

**Async boundaries.** Two further hand-offs lose a `ThreadLocal` binding and had to be covered explicitly, or `${caller:token}` would fail closed for no reason the config author could see: the HITL **resume** path (`runtime.submitCallable(resumeCallable, ...)`) and **fire-and-forget batch** calls in `ApiCallExecutor`. `CallerIdentityContext.propagate()` carries the binding across the latter; the resume path captures its own caller, since a resume is itself an authenticated request.

**Tests.** `CallerIdentityResolverTest` (24) and `CallerIdentityContextTest` (10) — same-origin refusals (host, port, scheme downgrade), fail-closed paths, regex-escaping of tokens containing `$`/`\`, thread isolation, and clearing on pooled threads. Disabling the same-origin guard fails 4 tests (mutation-checked). All 227 `ConversationService*Test` tests still pass.

**Note for the manager:** the operator ships this as its `caller-identity` auth mode, provisioning `apiAuth` as `Bearer ${caller:token}`; the conversation-context workaround and its token-at-rest warning are gone.

***

## 🔐 fix(security): code-review findings wave 2a — access control, A2A ownership, GDPR & audit ledger (2026-07-28)

**Repo:** EDDI (`fix/code-review-access-control`)

Second wave of the 124-finding external code review, split into two PRs so each stays under CodeRabbit's 100-file review limit (wave 1 at 151 files was skipped by it entirely). This half is the access-control and compliance surface.

### The guard triad existed — it just wasn't called everywhere

EDDI already has `OwnershipValidator`, `ConversationAccessGuard` and `HitlAccessGuard`, used correctly in `engine/internal`, `engine/hitl` and the MCP surface. Every finding here is a place that never called them.

* **A1 (critical)** — The **SSE turn endpoint had no ownership check** while its non-streaming twin did. The turn then executed under the *target* conversation's `userId`, loading that user's long-term memories into the prompt and running tool calls in their context. Fixed in two layers: the guard moved **down into `ConversationService.say`/`sayStreaming`** so no future REST adapter can omit it, *before* the memory snapshot loads — plus a REST-layer check so denial is a plain 403 rather than an error event on an already-200 SSE stream.
* **A2 (critical)** — Attachment endpoints **authorised the path parameter, not the caller**. `IAttachmentStore.load(ref, requestingConversationId)` checks that the *named* conversation owns the blob — and the caller supplies that name, so the check was self-satisfying. All five methods now require caller ownership, checked on the request thread *before* the async hop (`SecurityIdentity` is request-scoped).
* **A3** — `?deleteOlderThanDays=0` permanently deleted **every ended conversation in the deployment**, from an endpoint with no role at all. Now admin-only with a minimum of 1 day.
* **A4** — The tool control plane carried no `@RolesAllowed` at all: rate-limiter reset, cost-budget reset, and a history endpoint dumping **raw tool arguments and results** for any conversation.
* **A5** — `/propertiesstore/properties/{userId}` had neither role nor ownership check over the same `IUserMemoryStore` that `RestUserMemoryStore` guards on all nine of its methods. Writes there land in the victim's next system prompt.
* **A6** — A **conversation-id oracle**: it returned another user's live `conversationId`, which is the discovery half of A1 and A3.
* **A7** — Template preview read **any** conversation's memory and returned a flattened dump of properties/context/memory — and since the caller supplies the template, it was effectively a query language over someone else's conversation.
* **A8** — Config stores with full CRUD and no role. The review named one; **the actual inventory was six**, including two it missed (`IRestCapabilityRegistry`, `IRestWorkflowStepStore`). `IRestVersionInfo` was deliberately left alone — it has no `@Path` and is a mixin, reasoning recorded in the code.
* **A9** — A2A sat entirely outside the ownership model: caches keyed on a caller-supplied id, `contextId` as an unauthenticated read+write conversation handle, conversations created with `userId = null` (which `OwnershipValidator` treats as "legacy — allow", so permanently unowned), and raw `e.getMessage()` returned to arbitrary peers.
* **A12** — The global exception mapper returned the **raw driver message** as the 500 body — collection names, hostnames, replica-set topology. Now logged at ERROR with a correlation id.

### GDPR & audit (G14–G20)

* **G14** — Erasure was **served stale indefinitely** from a Caffeine cache with no TTL that erasure never invalidated.
* **G15** — The erasure cascade missed three stores while `docs/gdpr-compliance.md` asserted it "covers all data stores": conversation **checkpoints** (which carry `propertiesCopy` including PII, behind a javadoc claiming it was "used during GDPR erasure" when its only caller was unreachable), **group transcripts**, and **schedules** that kept firing under an erased user's id.
* **G16** — `AuditHmac.verifyHmac` had **zero production callers**; the docs told operators to "recompute the HMAC and compare it" and the product shipped no way to do so. Added admin verification endpoints.
* **G17** — GDPR pseudonymisation did `updateMany($set userId)` with no HMAC recompute, and `userId` is a *signed* field — so **every routine erasure produced rows cryptographically indistinguishable from tampered ones**. The class javadoc claiming a write-once contract was literally true and substantively false.
* **G18** — No hash chain: entries were independently signed, so **deletion and reordering were undetectable**. Added a per-conversation sequence inside the signed payload (chosen over a global chain, which would serialise all audit writes).
  * **Follow-up (review):** the first cut shipped this on **MongoDB only**. `PostgresAuditStore` persisted no sequence and left `supportsSequence()` at its `false` default, so `AuditLedgerService` skipped assigning one entirely — **every PostgreSQL deployment silently had no deletion detection**, reported as `UNAVAILABLE`, which reads like "not applicable" rather than "unprotected". Added the column, an idempotent `ALTER TABLE` defaulting old rows to the `UNSEQUENCED` sentinel, the `(conversation_id, sequence)` index verification reads, and `supportsSequence() = true`. This is the second cross-backend gap in this PR (after schedule `userId`), both in compliance code, both invisible because the degraded answer looked benign — the case for the deferred D4/J3 conformance suite.
* **G19** — `eddi.vault.master-key` ships empty, so entries were written **unsigned by default** while the docs present the ledger as evidence-grade. `ComplianceStartupChecks` had zero references to vault/HMAC/audit.
* **G20** — Unbounded audit queue that **re-offered failed batches into itself** — an OOM loop under a slow store.

### Documentation corrected (I7–I13)

Several docs described behaviour that did not exist. `docs/semantic-parser.md` documented a **stemming extension that does not exist**, four times, including inside the flagship copy-paste config — copying it throws `UnrecognizedExtensionException` and the agent will not start. The same doc's expression table claimed `number(42)` and `time(15:00)` where the code emits `integer(42)` and epoch millis, so rules written from it never fired. `docs/conversation-memory.md` and `docs/properties.md` both contradicted AGENTS.md §5.1 on the template model (`{properties.X.valueString}` fails at runtime — `MemoryItemConverter` puts raw values).

### Verification

Full unit suite: 12,384 tests, **0 non-environmental failures** (308 listed failures/errors all carry a loopback/selector/event-loop signature — this machine cannot bind sockets; CI is the gate for those). Five high-stakes fixes were **mutation-checked** — revert the fix, confirm a test actually fails, restore: G2, G12, A1, A2 and F18 all bite, verified against whole test classes (a `-Dtest=Class#method` filter silently runs 0 tests and exits 0 when the method is in a `@Nested` class, which reads exactly like a pass).

***

***

## 🐞 fix(engine): code-review findings wave 1 — parser, rules, templating, apicalls, datastore, LLM infra, deployment (2026-07-28)

**Repo:** EDDI (`fix/code-review-findings`)

First of four waves applying a 124-finding external code review. Wave 1 covers the findings whose file sets are disjoint, so they could be worked in parallel without conflicting. **53 findings: 50 fixed, 3 partial.** Every finding was verified against source before being acted on — none turned out to be a false positive.

### Crash / correctness

* **B1** `readActions(..., limit)` did `actions.subList(0, limit)` with every caller defaulting `limit=20`, so any config with fewer than 20 actions threw `IndexOutOfBoundsException` → 500. Fixed in `OutputStore`, `RuleSetStore`, `ApiCallsStore` with `Math.min(limit, actions.size())`. Invisible to tests because they all mock the store.
* **B4** `Permutation` accumulated `n!` into an `int`. From n=13 the product wraps; at n=17 it goes negative, so the iterator yielded a single permutation and parse quality silently got *worse* on longer sentences. The counter was redundant — `calculateNext()` already terminates — so it was deleted, with an explicit `length < 2` guard replacing the one case it was load-bearing for.
* **B5** `containsPunctuation` asked whether a string contains its own characters — always true. Now consults the configured `punctuationRegexPattern` (not the `PUNCTUATION` constant, which is only the default and would disagree with the replacement regex on a customised deployment).
* **B6** `isOrdinalNumber` returned `String` from an `is*` predicate; `indexOf("")` is always 0, so `"5."` returned `""` and the three callers disagreed on what that meant. Renamed to `extractOrdinalValue` → `Optional<Integer>`.
* **B7** `KEY_MODEL_ID = "modelID"` (capital D) blanked the model name for Vertex Gemini, losing capability lookup, token estimation and audit model naming.
* **B8** The Ollama builder — the default local provider — silently dropped `temperature`, `maxTokens`, `topP`/`topK`. Added, plus a shared unrecognised-key warning across all 11 builders.
* **B9** `ModelCapabilityService` was the inverse of its own javadoc: unknown models resolved to *supported*, so images were forwarded and 400'd at the provider. Now fails closed, with a per-task override so an unlisted-but-capable model can still be asserted.

### Behaviour-rule engine (E1–E5, E11)

The systemic defect was **silent acceptance of invalid config** — for a config-driven engine, the worst failure mode, because the agent designer gets no feedback and the agent looks healthy.

* **E1** `NOT_EXECUTED` was folded into `SUCCESS`, so every path yielding it made a guard rule fire unconditionally. Now treated as `FAIL` — this is the amplifier that made E2–E5 dangerous rather than merely wrong.
* **E2** Multi-child negation (the form the docs demonstrate) was ignored; now AND-combines children.
* **E3** Empty/misspelled matcher configs matched everything via `indexOfSubList(x, []) == 0`; now rejected at `configure()`.
* **E4** A typo'd `occurrence` silently became `currentStep`, converting the exact guard AGENTS.md §5.3 mandates into the globally-firing rule it warns against. Now throws, naming the bad value and the legal set.
* **E5** `sizematcher` could never match a collection (`parseInt("[a, b]")`); the documented example was unreachable.
* **E11** `ContextMatcher` NPE'd on a runtime/config context-type mismatch, killing the turn.

### Templating & output (E7–E9, E13)

* **E7** `strict-rendering` differed between dev (`false`) and prod (`true`), so a missing property rendered empty in dev and in prod shipped the **raw template literal to the end user**, while quick replies put `null` in the list. Aligned on one lenient, safe behaviour across profiles.
* **E8** Templates were re-parsed on every output, httpcall body/header and property instruction, every turn. Now a bounded Caffeine cache.
* **E9** `TemplateMode` was accepted and ignored (no escaping), and the HTML branch was unreachable because `"output:html".startsWith("output")` is always true.
* **E13** Only 3 of 8 output types were templated — `inputField`, `button`, `applicationLink`, `agentFace`, `other` shipped `{properties.x}` raw. Now an abstract `templatedCopy` on `OutputItem`, so a new type *cannot* silently miss templating.

### Parser (E14–E18)

* **E14** `appendExpressions=false` disabled the whole parser rather than just the merge — the entire storage block sat inside the flag, so no expressions and no intents reached any rule. Store/merge decisions are now separate.
* **E15** Dictionary `lang` was dead: no implementation overrode `getLanguageCode()`, so an agent with `en` and `de` dictionaries matched both against every turn.
* **E16** `PhoneticCorrection` kept only the last word per phonetic code — and the codes are lossy by construction, so night/knight/nite collapsed to one entry.
* **E17** No caps on solution enumeration (13.8s at 15 tokens with a dense dictionary). Added config-driven `maxInputTokens`/`maxSuggestions`/`maxSolutions`, a `HashSet` for the O(k²) scan, and cached `values()`.
* **E18** Damerau-Levenshtein returned the *entire* dictionary as candidates and computed `accuracy = 1.0 - distance`, yielding 0.0 and −1.0 against a documented 0..1 contract.

### API calls (E10, E19, E20)

* **E10** A `"*"` httpcall fired once **per action**, duplicating non-idempotent POSTs on a multi-action turn.
* **E19** Output JSON was built by string concatenation from upstream API response bodies — a `"` broke the JSON and a crafted value injected arbitrary output items into the agent's reply. Now built with Jackson.
* **E20** httpcalls had no timeout and no response-size cap; success bodies landed unbounded in conversation memory against Mongo's 16MB limit. Added per-call `timeoutInMillis` / `maxResponseSizeInBytes` with defaults.

### Datastore (D1–D3, D5–D10)

The two backends had silently diverged, because nothing tests them against each other.

* **D1** Postgres `data->'a.b.c'` looks up a *literal* key of that name — it does not traverse. Dotted paths returned zero rows forever while Mongo traversed correctly. Now a real JSONB path, keeping the existing injection-safety.
* **D2** The Postgres factory accepted an `indexes` argument and dropped it; real callers passed real hints and got sequential scans.
* **D3** Mongo used `$set` (merge), Postgres used `EXCLUDED.data` (replace) — so clearing a config field was a no-op on one backend and took effect on the other, both returning 200. Unified on full-document replace.
* **D5** The cascade-delete reference guard never worked: the query said `WorkflowSteps...` (capital W) against a field persisted as `workflowSteps`, and Mongo paths are case-sensitive — so shared configs were cascade-deleted while other workflows still referenced them. The guard now also fails *closed* on a query error.
* **D6** `PRIMARY KEY (id, collection_name)` put `id` first, so `WHERE collection_name = ?` could not use it, and nothing indexed `data` — every listing was a sequential scan over the single shared table. Reordered for new tables; existing tables get the equivalent index (column order can't be changed by `CREATE TABLE IF NOT EXISTS`).
* **D7** *(partial)* Restored the seven index declarations lost when the DB-agnostic `DescriptorStore` replaced the legacy one, and made `originId` a real indexed constant. **Not done:** splitting config descriptors from per-conversation descriptors into separate collections — that needs a data migration and two callers outside this workstream.
* **D8** Descriptor listings did one `read()` per id after fetching up to 10,000 ids. Added `readMany`.
* **D9** *(partial)* Fixed the `int` overflow in `ResultManipulator.limitEntities` (`index * limit` wrapped negative on deep pages). **Not done:** keyset pagination — it changes the `index`/`limit` paging contract across every REST store and the Manager UI. Deep offsets are at least index-served now.
* **D10** History and current-row writes were non-atomic; a crash between them left an archived-as-deleted row with the live row still present. Added `storeHistoryAndUpdate`/`storeHistoryAndRemove`.

### LLM infrastructure (F1–F5, F20)

* **F1** Rate limiting was a single **global** bucket per tool name, so one conversation starved every other user of that tool. Now keyed on `(conversationId, toolName)`, with an optional global bucket for provider-quota protection, and the gauge is finally tagged.
* **F2** `ChatModelRegistry` caches were unbounded *and* keyed on Qute-resolved parameters — so `"modelName": "{properties.preferredModel}"` minted a retained `ChatModel` + HTTP client per conversation-derived value. Now bounded, matching the sibling factories.
* **F3** Embedding model/store caches never evicted on credential rotation and used `expireAfterAccess`, which resets on read.
* **F4** `mongoClientCache` was unbounded, keyed on a raw connection string (a credential), and leaked `MongoClient`s on eviction.
* **F5** Cost-budget eviction claimed "oldest" but iterated `ConcurrentHashMap.keySet()` — arbitrary order — so an in-flight conversation's spend could be dropped, resetting its budget to $0.
* **F11 (part)** `ObservableChatModel` leaked a platform thread per timeout from an unbounded cached pool.
* **F20** Every wizard-created agent hardcoded `logRequests`/`logResponses` to `true`, writing all conversation content to application logs with no opt-out — a Pillar 1 violation. Now config, defaulting to `false`.

### Deployment & CI (H1–H3, H5–H10, H12–H15)

* **H1** Both k8s and Helm scaled to 2–10 replicas against a coordinator whose per-conversation serialisation is a **JVM-local** `synchronized(queue)` — so two turns of one conversation ran concurrently on different pods, both replaced the whole document, and one was silently lost. Pinned to 1 with the reasoning inline; autoscaling is now gated behind a genuinely distributed coordinator.
* **H2** *(CI half)* `always()` cancelled the implicit needs-gate and the tag branch short-circuited it, so a tag push published, signed and attested **regardless of test result**. Also added `integration-test` to `docker`'s needs, since it is the only job running the JaCoCo gate.
* **H3** The image scan ran with `exit-code: 0`, so no vulnerability could block a release.
* **H5** No container-level `securityContext` in any manifest — both deployments were rejected by a restricted Pod Security Standards namespace.
* **H6** *(partial)* Replaced mutable tag `labsai/eddi:6` with an immutable patch tag and added a digest value + `cosign verify` guidance. Pinning the literal digest remains a release-time step — inventing one would break every `kubectl apply`.
* **H7** `networkPolicy.enabled` was a dead Helm value with no template behind it; added one, plus link-local to the egress deny list so the network layer mirrors the app's own SSRF protections.
* **H8** Secrets were injected as env vars (readable via `/proc/<pid>/environ`, leaks into crash dumps). Verified Quarkus can read them from files, then switched to a mounted projected volume.
* **H9** The quick-start compose file hardcoded auth off with no override.
* **H10** Helm shipped default PostgreSQL/Keycloak credentials and rendered an empty vault key without failing; now `required`.
* **H12** The production image copied the entire docs tree — changelog, incident-response playbook, internal review standards — and served it to MCP clients.
* **H13** ZAP ran after publish, against an instance with auth deliberately disabled, passive-only, unable to fail. Dropped rather than cited as coverage it never provided.
* **H14** Fuzzing targeted vendored *copies* of the security-critical parsers with no drift check, so PRs touching the real sources fuzzed a stale duplicate.

### Verification

`./mvnw clean test-compile` green from scratch (deliberately clean, not incremental — several signature changes crossed workstream boundaries, and incremental builds reuse stale `.class` files for unedited callers). 158 targeted tests pass. E1's fix was mutation-checked: reverting it fails 2 tests in `RuleTest`.

**Deliberately deferred to later waves:** E6 (write-time config validation), E12, D4/J3 (cross-backend conformance suite — needs Testcontainers), the `pom.xml` batch (H4/H11/H15 + J1/J8/J9), and the doc corrections these fixes imply (I7, I8, I12, `semantic-parser.md`, `httpcalls.md`).

***

## ✨ feat(openai): OpenAI-compatible API adapter for Open WebUI (2026-07-27)

**Repo:** EDDI (`feat/openai-api-adapter`)

New `/v1` surface presenting deployed agents as OpenAI "models", so Open WebUI, the Python `openai` SDK, LangChain and LiteLLM can drive EDDI conversations. New package `integrations/openai/`, parallel to `integrations/slack/`. **Disabled by default.** Full guide: [`docs/open-webui-integration.md`](/protocols-and-integration/open-webui-integration.md); design rationale in [`planning/openai-api-adapter-plan.md`](https://github.com/labsai/EDDI/tree/main/planning/openai-api-adapter-plan.md).

**Four earlier drafts of this plan were built on premises that turned out to be false.** Each was verified against source before implementing; the corrections shaped the design:

* **Content negotiation cannot dispatch sync vs streaming.** The plan routed on `Accept` via two `@Produces` methods. But `openai-python` hardcodes `Accept: application/json` and never varies it by `stream`, and Open WebUI sends no `Accept` header at all (it sniffs the *response* content-type). Every streaming request would have landed on the JSON method. → **one method, dispatching on the `stream` field of the body**, which is what the OpenAI spec says and what vLLM/llama.cpp/LiteLLM/Ollama all do.
* **Open WebUI does not map `chat_id` to the `user` field.** [#27174](https://github.com/open-webui/open-webui/pull/27174) is an *issue*, closed as *not planned*; `payload['user']` is set only for pipeline models and is an object, not a chat id. The plan's entire per-chat isolation mechanism did not exist. → **key on `X-OpenWebUI-Chat-Id`** ([#15813](https://github.com/open-webui/open-webui/pull/15813), gated by `ENABLE_FORWARD_USER_INFO_HEADERS`), with `user` as a defensively-typed fallback.
* **`IRestAgentAdministration` is `@RolesAllowed({"eddi-admin","eddi-editor"})` at type level**, and Quarkus enforces it via a CDI interceptor. Injecting it for `/v1/models`, as planned, would have 403'd for every ordinary caller. → read `IAgentFactory` + `IDocumentDescriptorStore` directly. REST facades are the authorization boundary; a public surface must not reach around one.
* **The auth story did not exist.** `quarkus.http.auth.permission.authenticated.paths=/,/*` captures `/v1/*`, so an `sk-…` bearer would be rejected by OIDC before the adapter ran. → explicit `/v1/*` permission entry plus two modes (`permit` + shared key, or `authenticated` + OIDC), constant-time key compare, and a startup guard.
* **`UtilityAgentProvisioner` was cut.** It had an unauthenticated caller creating and deploying an agent, cloning another agent's credential reference and consuming `maxAgentsPerTenant` quota — blocked by the role checks above anyway, and in tension with Pillar 1 (the engine authoring agent config at runtime). → replaced by **`<model>:stateless` variants**: start, say, end. No writes, no roles, and useful beyond title generation.
* **The new-chat heuristic was cut.** `userMessageCount == 1 && hasTurns → endConversation` is unreliable (regenerate and edit-and-resend look identical to a first message), destructive (memory loss is irrecoverable), and directly contradicted the plan's own recommended Open WebUI Filter, which strips history to the last user message — under which *every* turn would have destroyed the conversation. A new chat is a new chat key, which is a new intent. No inference needed.
* **HITL was unaddressed.** `onSkipped` fires for both `AWAITING_HUMAN` and `IN_PROGRESS`; the plan mapped both to "conversation busy", which would make any agent using `PAUSE_CONVERSATION` permanently unusable with a misleading message. → sentinel-snapshot discrimination, mirroring `SlackEventHandler`, and pauses surface as **chat text with `200`**, never as an HTTP error — a 4xx makes clients discard the user's message.

**Components** (`src/main/java/ai/labs/eddi/integrations/openai/`): `RestOpenAiAdapter` (`/v1`), `OpenAiConversationBridge`, `AgentModelResolver`, `OpenAiMessageMapper`, `OpenAiSseWriter`, `OpenAiAuthFilter`, `OpenAiStartupGuard`, `OpenAiCompatConfig`, `OpenAiApiException` + `OpenAiExceptionMapper`, and 11 wire DTOs under `model/`.

**Other design decisions:**

* Model ids are `<slug>-<last 6 of agentId>`. Agent names are not unique, so a bare slug would be non-deterministic with two agents called "Support". Name and slug lookups are accepted but only when unique; an ambiguous match returns 400 listing candidates rather than guessing.
* Slugging folds accents via NFD rather than dropping them — `Übersicht` was slugging to `bersicht`, mangling every non-ASCII agent name. Caught by its own test.
* Images map to `attachment_N` context entries, so they flow through the existing `AttachmentForwarder` with its vision gating, byte caps and SSRF-guarded fetching. Zero core changes. Two parsing fixes: `data:image/png,payload` is legal and has no `;` (scanning to `;` threw), and remote URLs get a concrete MIME from the extension — `image/*` passes the forwarder's `startsWith("image/")` gate but is rejected by providers when handed to `ImageContent.from`.
* `usage` is omitted rather than zero-filled when the agent called no model. (This originally read "EDDI does not surface per-request token counts here" — that was wrong, and is corrected in the 2026-07-28 entry below.)
* In-flight completions are semaphore-bounded — each holds a worker thread, since the bridge blocks on the turn as the Slack handler does.
* **Reused, not duplicated:** `ConversationOutputExtractor.extractResponse()` already existed in `engine/memory` (added upstream) and handles more output formats than the Slack-local copy the plan proposed extracting. Phase 2 of the plan became unnecessary. Noted separately: `SlackHitlSupport.extractSlackResponseText` still duplicates it and should delegate.

**Self-review pass** (after the feature was complete) found and fixed six defects, each now covered by a test:

* **Semaphore permit leak.** The streaming path handed its permit to the `StreamingOutput` body to release. If that body never runs — a client disconnecting before serialization starts, say — the permit is never reclaimed, and after `max-concurrent-requests` such events the adapter returns 429 *permanently*, until restart. The resource now releases unconditionally and the stream body takes its own permit inside one try/finally.
* **`GET /v1/models/{id}` echoed the caller's string, not the canonical id** — a lookup by agent name returned `{"id":"Customer Support"}`, which is absent from `GET /v1/models`, so a client round-tripping the answer would ask for a model that does not exist. `ResolvedModel` now carries requested and canonical ids separately (plus the descriptor timestamp, which was hardcoded to 0).
* **`InterruptedException` was swallowed** in the turn wait, leaving the worker thread looking healthy with its shutdown signal gone.
* **A null exception message rendered to the user as the literal text "null"** in the stream error path (NPEs carry no message).
* **`hasSentContent()` lied** — it returned "the stream has started", true even after a content-free `finish()`. Renamed `hasStarted()`.
* **The `eddi.openai.requests` counter documented in the plan was never implemented.** Added with `mode` and `outcome` tags, so `paused` (reviewers behind) and `busy` (clients racing) are distinguishable from real errors.

Also verified rather than assumed: `quarkus.rest.jackson.optimization.enable-reflection-free-serializers=false` in this project, so `@JsonProperty` on record components works — `finish_reason` and `owned_by` serialize correctly. Added `OpenAiWireFormatTest` to pin that, since the non-streaming response shape had no coverage at all.

Corrected a documentation claim rather than the code: unimplemented `/v1` paths (`/v1/embeddings` etc.) return Quarkus' plain 404, not an OpenAI error envelope. A catch-all route would risk shadowing the real endpoints for a cosmetic gain.

**Tests:** 148 unit tests plus 16 integration tests, all green. Mutation-checked — reverting the model-ambiguity guard, the identity refusal, and the HITL sentinel distinction each makes the relevant tests fail. `RestOpenAiAdapterTest` (`@QuarkusTest`, binds a socket) is deferred to CI per the local-environment constraint.

**Attachment support** covers all three binary content-part types: `image_url`, `file` (inline PDFs and documents) and `input_audio`. All map to `attachment_N` context entries, so EDDI's existing forwarder does the real work — capability gating, byte caps, PDF text extraction, SSRF-guarded fetching. Three details the wire formats make easy to get wrong, each pinned by a test:

* `input_audio.data` is **raw base64 with no `data:` prefix**, unlike every other binary payload in the protocol, with the type in a separate `format` field. `mp3` maps to `audio/mpeg` — `"audio/" + format` would produce `audio/mp3`, which is not a real media type.
* `file.file_data` **is** a full data URI, and the declared `filename` beats a generic `application/octet-stream` type, since clients that base64 a file without sniffing it send exactly that.
* `file.file_id` references the OpenAI Files API, which EDDI does not implement; those parts are skipped with an actionable warning rather than becoming empty attachments.

**Stateless requests** have two routes to the same behaviour. The `:stateless` model suffix exists because a model name is the only per-request dimension a UI like Open WebUI can express — its title-generation setting is a dropdown, so a query param, header or body field could not be selected there at all. It follows the ecosystem convention for behavioural model variants (OpenRouter's `:nitro`/`:floor`/`:free`, Ollama's `llama3:8b`) and has the side benefit of appearing in `GET /v1/models`, making the capability discoverable. Alongside it, a `stateless` **body field** gives programmatic callers the explicit parameter (`extra_body={"stateless": True}` in the Python SDK). The two are OR-ed rather than letting either win: `model:"x:stateless"` plus `stateless:false` is self-contradictory, and running stateless only loses continuity while running stateful would persist a conversation the caller may not have wanted. `expose-stateless-variants=false` blocks both routes, so the switch cannot be circumvented by moving the request into the body.

**Gateway-agent recipe** documented in §11 of the integration guide, rather than building a passthrough proxy mode. A thin LLM-only agent behind this adapter already provides vaulted API keys, audit, tenant quotas and cost tracking — which is what the passthrough idea was actually after. Building a real proxy would mean entering the LLM-gateway market (LiteLLM, Portkey, Cloudflare AI Gateway) with a worse product, inheriting per-provider streaming/tool/vision passthrough maintenance for zero agent value, and shipping a feature with neither logic nor configuration — which is the opposite of Pillar 1. The recipe's limits are stated up front: single-turn only, one agent per model, no caching/fallbacks/virtual keys.

**`OpenAiCompatIT`** closes the one gap the unit tests structurally could not. The adapter serves sync and streaming from a *single* JAX-RS method, dispatching on the `stream` body field rather than the `Accept` header — a design forced by real client behaviour. A refactor toward content negotiation would look correct in every unit test and silently return JSON to every streaming client, so the guard has to live at the HTTP layer: `stream:true` with `Accept: application/json` (what openai-python sends) and with no `Accept` header at all (what Open WebUI sends) must both yield `text/event-stream`. 16 tests also cover the response wire shape, per-chat conversation isolation end to end, the error envelopes, and tolerance of the unknown request fields every client sends.

It deploys the shared minimal agent (parser/rules/output/templating), so it needs no LLM credentials and is deterministic. **Runs in CI only** — Quarkus cannot boot in the dev sandbox (`Unable to establish loopback connection`), the same environmental limit that already fails `SlackWebApiClientTest`. The test compiles and is discovered by failsafe locally; whether it passes is for CI to say.

**A real defect the integration test caught immediately.** `OpenAiCompatIT.streamingBodyIsWellFormed` failed on its first CI run (passing only on failsafe's retry) with content arriving *after* the `[DONE]` sentinel. Root cause: `ConversationService.sayStreaming` ends with `conversationCoordinator.submitInOrder(...)` and **returns immediately** — every handler callback fires later on another thread. The bridge wrote the terminator in its `finally` as soon as `sayStreaming` returned, so the response closed mid-turn and late tokens raced a closing stream. Fixed by awaiting a `CompletableFuture` completed by whichever terminal callback fires (`onComplete`/`onSkipped`/`onError`), bounded by `request-timeout-seconds`.

The unit tests had all passed because their Mockito stubs invoked the handler **synchronously**, hiding the asynchronous contract entirely — a mock that was more convenient than the real collaborator. A regression test now drives the handler from a background thread, and mutation-checking confirms removing the await makes it fail. This is precisely the class of bug an HTTP-level test exists to find, and it was found on the first run.

**A runnable demo — and two more defects it caught.** `docker-compose.openwebui.yml` brings up MongoDB, EDDI (built from the working tree, since no published image has the adapter), Open WebUI and a seeder that deploys a small rule-based agent so the model list is not empty. Standing it up and actually talking to an agent found two bugs that 146 unit tests and 16 integration tests had not:

* **`plainText` leaked into every assistant message.** Jackson treats `ChatMessage.isPlainText()` as a bean property, so responses carried a field absent from the OpenAI schema. Harmless to clients, wrong on the wire — and invisible to `OpenAiWireFormatTest`, which asserted expected fields were *present* but never that there were no extras. Fixed with `@JsonIgnore`; the tests now assert the exact field set.
* **The create-conversation race returned a bare 500.** Open WebUI issues the completion and its title request concurrently with the same chat id, so both find no mapping and both insert. The handler caught `ResourceAlreadyExistsException` — but the Mongo store lets a raw `MongoWriteException` (E11000) out, so the catch was dead code against the real store. Now caught broadly and resolved by re-reading the mapping, which is datastore-agnostic (the Postgres store reports it differently again). Verified with four concurrent requests to one chat: all 200.

Also noted while building the demo, and **not fixed here** because it is unrelated to this PR: `POST /backup/import/initialAgents` cannot import the bundled Agent Father on Linux. The ZIP's entries are correctly forward-slashed, but extraction writes them as single files with literal backslashes, so the workflow directory never exists and the import 500s. The demo seeds through the ordinary REST API instead.

**A documentation error found by using the demo, and two settings verified from Open WebUI v0.11.0's notes.**

The guide claimed Open WebUI injects RAG context into the **system message**. It does not, by default. Verified in the running container: `RAG_SYSTEM_CONTEXT` defaults to `False` (`open_webui/env.py`), and `middleware.py` then calls `add_or_update_user_message(...)` rather than `add_or_update_system_message(...)`. So dropping a PDF into a chat delivers several thousand tokens of `### Task: …`, `<context><source id="1">…</source></context>` and `<attached_files>` markup as `{memory.current.input}`, with the user's real question buried at the end — which breaks input matchers, makes property setters capture the whole blob, and stops quick replies firing. Corrected in the guide, added to the must-configure table, and `RAG_SYSTEM_CONTEXT=true` set in the demo compose. This was found by actually uploading a file, not by review.

Also verified and documented: **`AIOHTTP_CLIENT_STREAM_IDLE_TIMEOUT`** (new in v0.11.0) ends a stream when the upstream goes quiet — sized for an LLM's time-to-first-token it will cut EDDI agents short, since a rule-based agent emits nothing at all until its turn completes; and **`passthrough_params`** on a connection forwards non-standard body fields verbatim, which makes the `stateless` field settable from the Open WebUI UI rather than only via `extra_body`.

The demo agent's prompt was also reworded. A property setter bound to `{memory.current.input}` stores the *whole* turn, so answering "Gregor. What are you capable of?" stored all of it as the name and looked like a parsing bug. It now says it stores the next message verbatim — which is what a naive slot-filler does, and a fair thing for a demo to teach.

**A second injection path, and an LLM-backed demo agent.** Setting `RAG_SYSTEM_CONTEXT=true` moved the retrieved chunks to the system message as intended, but an attachment-carrying turn still arrived polluted — this time with an `<attached_files>` block. Different mechanism: Open WebUI's built-in Files tool, gated by `use_builtin_tools`, which depends on a per-model `builtin_tools` capability with **no environment override**. On an EDDI model that capability is pure cost — the adapter never returns `tool_calls`, so the tools can never fire, yet enabling them rewrites the user's message. Documented as a per-model toggle in the model editor, and added to the must-configure table alongside `RAG_SYSTEM_CONTEXT`.

The demo also gained an **optional LLM agent, with its provider key in the Secrets Vault**. The rule-based one demonstrates the transport and the state bridge without credentials, but it has no model, so it cannot answer questions *about* anything — which made "what is this pdf about?" look like an adapter failure when it was simply an agent with nothing to think with. Set `EDDI_DEMO_LLM_API_KEY` (plus optional `_TYPE`/`_MODEL`) and a second agent is deployed whose system prompt references `{context.openai_system_message}`, so it can answer about uploaded files. The key is stored via `PUT /secretstore/secrets/default/demo-llm-api-key` and the config holds only `${vault:demo-llm-api-key}` — an agent config is exported, diffed, rendered in the Manager UI and logged, so a literal key would travel with all of it.

Verified on a clean run: both agents reach `READY`, all four models are exposed, the stored config reads `"apiKey": "${vault:demo-llm-api-key}"`, and a scan of every collection in the database finds the plaintext key in none of them. The LLM *call* itself is still unverified — the run used a deliberately fake key.

Getting there also surfaced a documentation error in `AGENTS.md` §5.5: the workflow step type for LLM interaction is listed as `eddi://ai.labs.langchain`, but `LlmTask.ID` is `ai.labs.llm`, so a workflow built from the table fails to deploy with `Extension 'ai.labs.langchain' not found`. The config-store URI in that row (`eddi://ai.labs.llm/...`) was already right; only the step type was stale. Corrected.

**Not implemented (v1):** `/v1/embeddings`, `tool_calls` passthrough (would cause double-execution — EDDI's tools do not exist in Open WebUI), `n > 1`, `logprobs`. ~~Quick replies and `inputField` outputs are dropped; only text reaches OpenAI clients.~~ (Superseded by the 2026-07-28 entry above — they are rendered as Markdown now.)

***

## 🎨 Keycloak login theme matching the EDDI corporate identity (2026-07-27)

**Repo:** EDDI (`feat/keycloak-eddi-theme`)

Users redirected from the Manager or Workforce UI to Keycloak went from EDDI's amber-on-near-black design to stock Keycloak blue-on-white. This adds an `eddi` **login** theme — **no FreeMarker overrides** — plus the wiring to activate and ship it. As finally shipped the theme is eleven resources: `theme.properties`, one stylesheet, two enhancement scripts, four Noto Sans subsets, the wordmark and the favicon. The sections below are in the order the work happened, so earlier ones describe smaller inventories.

**What changed.**

* `keycloak/themes/eddi/login/theme.properties` — `parent=keycloak.v2`, `styles=css/styles.css css/eddi-login.css`, `kcHtmlClass=login-pf pf-v5-theme-dark`.
* `keycloak/themes/eddi/login/resources/css/eddi-login.css` — the palette, a block of PatternFly global-token overrides, and a short list of targeted rules (logo, page background, card border, button label, autofill).
* `.../resources/img/{logo_eddi.png,favicon.ico}` — byte-identical copies of the app's own assets.
* `keycloak/eddi-realm.json` — `loginTheme`, `displayName`, `displayNameHtml`.
* `docker-compose.auth.yml` — bind-mounts `./keycloak/themes/eddi` into `/opt/keycloak/themes/eddi`.
* `install.sh` — fetches the theme resources alongside the realm JSON (only the realm JSON is fatal; see the atomicity note below), and sets `loginTheme` through the Admin API after startup.
* `planning/keycloak-eddi-theme.md` — the full design, the verified-facts table, and the corrections below.

**How it works.** Three layers. `kcHtmlClass` adds PatternFly 5's built-in `.pf-v5-theme-dark` to `<html>`, which flips every PF component the login pages use — including ones we would never enumerate (tiles, data lists, helper text, panels). A `:root` block of `--pf-v5-global--*` overrides then recolours PF's dark defaults to EDDI's stone/amber palette. A handful of targeted rules cover what tokens cannot express. We own zero FreeMarker, so a Keycloak upgrade can only break this loudly.

**Design decisions.**

* **Pure CSS over template overrides.** Every `.ftl` we do not ship is a file we do not re-diff on every upgrade. The logo is applied to `#kc-header-wrapper` with accessible image replacement (`text-indent: -9999px`), keeping `displayNameHtml` in the accessibility tree so screen readers still announce "EDDI".
* **`styles` must name the parent stylesheet.** Theme properties merge key-by-key with the child winning, so `styles` *replaces* rather than appends; omitting `css/styles.css` silently drops all base styling. It resolves up the inheritance chain, so we ship no copy.
* **Prefer global-token overrides to component rules.** Component rules are what rot on upgrade.
* **Login theme only.** The account console, admin console and email templates remain stock; adding them later means sibling directories reusing the same palette block.

**Three things the live run corrected — none were visible from source reading alone.**

1. **`:where()` specificity 0 does not mean `:root` always wins.** PF's dark theme also sets *component* variables on *component elements* (`:where(.pf-v5-theme-dark) .pf-v5-c-login { --pf-v5-c-login__main--BackgroundColor: … }`). Custom properties resolve from the nearest declaring element, so that beats `:root` regardless of specificity.
2. **The dark theme re-points components at a different tier of globals.** The primary button reads `primary-color--300`/`--400`, the card `BackgroundColor--300`, form controls `BackgroundColor--400`, the underline `BorderColor--400`. Overriding only the `--100` tier left the Sign In button Keycloak blue (`#06c`) and the card the wrong grey. Both tiers are now set.
3. **Field-level error text reads `danger-color--200`.** Set to a red-700 (`#b91c1c`) it rendered at \~2.4:1 on the card — a WCAG failure on the most important message on a failed login. On a dark surface the higher tiers must get *lighter*, not darker; it is now red-400 at **6.40:1**.

**Verification.** Against a running `quay.io/keycloak/keycloak:26.0` on a fresh volume, reading computed styles rather than eyeballing. Realm import log clean. All assets 200, including the inherited `css/styles.css` and our favicon. Flows exercised: sign-in, failed login, update password (forced action), re-authenticate, logout confirmation, forgot password, error page. Contrast: button label 9.20:1, links 10.61:1, labels 6.91:1, error text 6.40:1 — all past AA. Mobile 375×812: no horizontal overflow. Full measurements in `planning/keycloak-eddi-theme.md` §6.1.

**Operational note — existing installations.** Realm import is one-shot: Keycloak skips realms that already exist, so editing `eddi-realm.json` does **not** reach an existing deployment. `install.sh` now applies `loginTheme` through the Admin API so upgrades pick it up; for local development the equivalent is `docker compose -f docker-compose.yml -f docker-compose.auth.yml down -v` (⚠️ destroys `keycloak-data` — all users and sessions).

**Incidental finding, now fixed.** `"temporary": true` on an imported credential does **not** create an `UPDATE_PASSWORD` required action in Keycloak 26 — the seeded users log straight through (`requiredActions` is empty after import). That made the comment at the top of `docker-compose.auth.yml` ("password change required on first login") wrong. The comment now says so explicitly, including *why* the `"temporary": true` flag is misleading, so the next reader does not "fix" the realm JSON in the wrong direction. The same block listed only two of the three seeded users and labelled them loosely; it now lists all three with their actual realm roles. Recorded as F19 in the plan.

**A colour-scheme control, and the locale picker no longer looks like a stray field.** Both came from looking at the rendered page.

* **The switcher exists now.** The earlier decision to omit it was wrong for a reason I had not checked: the Manager and Workforce show a theme switcher on *every* page, so offering one here is consistent rather than duplicative. It is a quiet icon button beside the locale picker with the same three states as the rest of the product — system (default) / light / dark — persisted per origin. It deliberately does not sync with the Manager's setting: Keycloak is a different origin, so that `localStorage` is unreadable from here. The control is a genuine enhancement — "system" removes the attribute so the CSS media query governs, which means the page follows the OS with no JavaScript at all, and the button only adds the ability to override.
* **The locale picker was my fault.** Keycloak renders it as a native `<select>` wrapped in a `.pf-v5-c-form-control`, so the field styling hit it too and gave it the recess, border, radius and 38px height of the password input — sitting next to the H1, it read as a stray form field. It is now quiet at rest and reveals its affordance on hover or focus.
* **A 16px misalignment** put the picker past the right edge of the fields, close enough to look like a mistake. The cause is the header grid's tracks overflowing its padding box by exactly the column gap. Worth recording because two plausible fixes do *not* work — `minmax(0, 1fr)` on the title track and `min-width: 0` on the title were both verified against the live page and neither moved it. Zeroing the gap and moving the spacing onto the utilities does.

Also corrected: the claim that `start-dev` randomises the resource-path hash per boot is **false** — it survived a container restart. That matters beyond testing, because it means a theme edit does not change the CSS URL, so returning browsers can hold a stale stylesheet. The response does send `Cache-Control: no-cache`, so revalidation happens, but the "bump the version to bust the cache" assumption in the plan was wrong.

**Light mode, all 30 locales, and two palette corrections.** Three things came out of questioning earlier decisions.

* **Locales — the single-locale choice was wrong.** Enabling internationalisation with `supportedLocales: ["en"]` (done purely to get `lang` on `<html>` without a switcher) silently discarded 29 translations the theme already had. Keycloak's server-info endpoint confirms the `eddi` theme inherits **30 locales** — ar, ca, cs, da, de, el, en, es, fa, fi, fr, hu, it, ja, ko, lt, lv, nl, no, pl, pt, pt-BR, ru, sk, sv, th, tr, uk, zh-CN, zh-TW — for free. All 30 are now offered; German renders as `lang="de"` with "Passwort vergessen?" / "Anmelden". The switcher Keycloak renders is a native `<select aria-label="languages">` wrapped in `.pf-v5-c-form-control`, so it inherits the field styling and is keyboard-accessible without extra work.
* **Light mode via `prefers-color-scheme`.** The Manager's own default is `defaultTheme: "system"`, so a user on a light OS already gets a light Manager while the login page stayed hard dark. Following the system preference matches that default exactly. *(At this point no toggle was added, on the reasoning that the Manager owned the explicit choice. That was revisited — see the colour-scheme control above — and `prefers-color-scheme` is now what the control's "system" state resolves to.)* Values are the Manager's own light token block. Because every PatternFly token is declared as `var(--eddi-…)`, the light block is a palette swap plus the treatments that are inherently directional — a highlight lit from above, a recess, a shadow, and the bloom, which on a light page reads as a stain rather than as light and so drops to a faint wash. The white wordmark is handled with `filter: invert(1)`; `invert()` leaves alpha alone, so no second asset is needed.
* **Two palette errors, found by reading the Manager's tokens properly.** `#0c0a09` is its `--color-primary-foreground` — the label on an amber fill — not its background; the dark background is `#09090b`. The page colour was wrong from the first commit. (The button label was coincidentally correct.) Destructive is `#dc2626`, not `#ef4444`; that is now used in light mode, while dark keeps a lighter red because `#dc2626` is only \~3.2:1 on our dark card.

Two contrast defects that only light mode exposed: amber-400 links and an amber-500 focus ring are 1.7:1 and 2.15:1 on white. Links and the ring now step down to amber-700 in light (5.02:1, clearing 4.5:1 for text and 3:1 for a focus indicator). Verified in both schemes: light gives title 21:1, field text 16:1, links 5.02:1, labels 4.8:1, button label 9.2:1; dark is unchanged.

Fonts gained the Latin-ext, Cyrillic and Greek subsets, because Latin-only would put fallback glyphs *inside* otherwise-branded words for Czech, Polish and Turkish. Measured rather than assumed: `unicode-range` normally defers those files, but the switcher lists all 30 languages under native names, so every range is in the DOM and all four are fetched (\~245 KB). Accepted — `font-display: swap` keeps it off the paint path, it caches for the session, and the page already loads a \~1.5 MB PatternFly stylesheet.

**Least-privilege default role — a privilege-escalation trap closed.** `eddi-realm.json` composited `default-roles-eddi` to `eddi-admin`, `eddi-editor` and `eddi-user`, so **any user created without explicit realm roles became an admin** — which is exactly what self-registration produces. Nothing was exposed, because `registrationAllowed: false`, but enabling registration (a one-flag change that looks innocuous) would have silently granted admin to everyone who signed up. Found while verifying the seeded users' roles for the compose comment, and independently flagged in review afterwards.

The composite is now `eddi-user` alone, in both places it is declared. Verified on a **fresh import**, since editing the file does not touch a running realm:

* `default-roles-eddi` → `eddi-user, manage-account, offline_access, uma_authorization, view-profile` — Keycloak's own built-ins survive, so refresh tokens and account access are unaffected.
* A user created with no roles → `eddi-user` and the built-ins. No admin.
* The three seeded users are untouched: `eddi` keeps admin+editor, `viewer` keeps viewer, `user` keeps user — they declare `realmRoles` explicitly, and Keycloak's import does not add the default role on top.

**Existing deployments are detected, not migrated.** Realm import is one-shot, so a realm created before this keeps the old composite. `install.sh` now checks it on every run and **warns** when the default role still grants `eddi-admin`/`eddi-editor`, printing the exact fix — but changes nothing. That asymmetry with the branding fields (which are forced) is deliberate: branding is cosmetic, whereas silently stripping elevated defaults during an upgrade could break a deployment that granted them on purpose. To apply it manually: Admin console → Realm roles → `default-roles-eddi` → Associated roles → remove them, or `DELETE /admin/realms/eddi/roles-by-id/{id}/composites`. Both branches of the check were verified against the running realm — silent once fixed, firing when the elevated roles are put back.

**A duplicated token, and the bug hiding behind it.** CodeRabbit flagged `--eddi-chevron` as declared twice in the light media query (a Stylelint error). It was worth looking at why rather than just deleting the second line: the patch that introduced the chevron had matched the same anchor twice, so it inserted **both** copies into the media query and **none** into `:root[data-eddi-theme="light"]`. The duplicate was the harmless symptom; the real defect was that choosing "light" explicitly from the toggle while the OS is dark left the chevron at the dark grey. Duplicate removed, missing declaration added, and all three paths checked: system follows the OS, explicit light `#78716c`, explicit dark `#a1a1aa`.

**The header utilities looked accidental on narrow screens.** Below PatternFly's header breakpoint the grid collapses to a single column and the utilities wrap onto their own row — which is fine — but the locale picker stretched to fill it: **268px at a 560px viewport against 82px on desktop**, so the same control looked like two different things depending on width.

The fix is to keep both utilities content-sized. Worth recording *why* it needed a newer property: `width: auto` does not size a `<select>` to its selected option (it fills), and `max-content` sizes it to the **widest option in the list** — which here is a 30-language menu, so that is worse. `field-sizing: content` is the one that does the right thing. It is Chrome/Edge-only at present, so a `max-width` cap is the fallback: capped rather than full-bleed everywhere else.

Measured at 375, 560 and 1280: the picker is **76px at every width**, the row stays right-aligned with the form fields, no horizontal overflow, and no dead selectors.

**Locale picker restored on the admin console; both realms offer the full set.**

The admin console had no language picker because I had given `master` a single locale — a deliberate trade to get `lang` on `<html>` without adding a switcher, but the wrong one once that console is EDDI-branded. Both realms now show the picker.

On which languages: they were briefly narrowed to the Manager's 14 (its bundle declares `supportedLngs` as `cs, de, es, fr, it, ja, ko, pl, pt-br, ru, tr, zh-cn, zh-tw` with `fallbackLng: "en"`), on a parity argument — a user who picks a language at login lands somewhere that speaks it. That was reverted on review: **both realms offer all 30 locales Keycloak ships**, deliberately a superset. The reasoning that won is that a localised login page is worth having even when the app behind it falls back to English; the alternative denied 16 free translations to speakers of ar, ca, da, el, fa, fi, hu, lt, lv, nl, no, pt, sk, sv, th and uk in order to avoid an inconsistency they would only notice *after* signing in.

All four font subsets are therefore shipped (\~245 KB), Greek included.

**RTL now verified, having previously only been assumed.** The superset makes Arabic and Farsi reachable, and the wordmark uses `text-indent: -9999px`, which in a right-to-left document pushes text the *other* way. Checked in Arabic: `<html lang="ar" dir="rtl">`, **no horizontal overflow** — `overflow: hidden` on the header contains it — with the logo, the 30-option picker and the theme toggle all intact.

**Admin console follow-ups — a logo collision, a missing toggle, and a shipped bug.**

* **Two logos on top of each other.** Keycloak's `master` realm ships `displayNameHtml` as `<div class="kc-logo-text"><span>Keycloak</span></div>`, and that markup lands inside `#kc-header-wrapper` where we paint the EDDI wordmark — with `keycloak.v2`'s stylesheet still carrying the rule that gives that div the Keycloak logo. **This corrects F5 in the plan:** `div.kc-logo-text` is not dead CSS in general, only for realms whose `displayNameHtml` does not contain that markup. Fixed both ways — the theme now suppresses any logo box inside the header, and `install.sh` sets master's `displayNameHtml` to plain `EDDI` so the header also has a correct accessible name. `displayName` is deliberately left as "Keycloak", because it labels the realm in the admin console's realm selector.
* **The theme toggle was missing there.** Keycloak only renders the header-utilities container when it has a locale switcher to put in it, and master has a single locale — so the control had nowhere to attach and was silently absent. `eddi-theme.js` now creates the container when it is not there.
* **A bug this branch shipped in `97c7a0d0`:** the patch that added the master-realm block wrote a literal instead of a line continuation, so the jq path read `jq '...'` — `bash -n` accepts it, but at runtime jq would take `n` as its filter and the branch would fail on any machine that has jq. Repaired, and the whole file checked for further damage.

**Keycloak manages `pf-v5-theme-dark` itself.** The template ships an inline script that adds and removes that class from `prefers-color-scheme`, so `kcHtmlClass` only sets the initial state and the class is stripped on a light OS. Nothing depends on it: because the token block assigns both the `--100` and the `--300`/`--400` tiers, components render correctly either way — verified by forcing dark on a light OS and confirming page `#09090b`, card `#18181b`, amber button with near-black label.

**Admin console login themed, and two picker defects.**

* **The admin console is now branded too.** It authenticates against Keycloak's `master` realm, which is not part of `eddi-realm.json`, so it kept the stock polygon-and-Keycloak-logo page while the EDDI realm was branded. `install.sh` now sets `loginTheme` on `master` through the Admin API — but only when master has no login theme of its own, so on a Keycloak shared with other products an operator's existing admin branding wins. It also enables internationalisation there with a single locale: master ships with i18n off, which meant no `lang`/`dir` on a page we now own the appearance of (WCAG 3.1.1). One locale adds the attributes without adding a switcher, and loses nothing, since i18n-off was English-only anyway. Verified: `<html class="login-pf pf-v5-theme-dark" lang="en" dir="ltr">` with no locale picker.
* **The locale picker had a stray underline.** PatternFly draws the field underline with `::after`; quietening the picker removed its background and border but left that line hanging under the label with no box around it.
* **And its label overlapped the caret.** PF's caret is not part of the `<select>` — it is a 32px sibling grid item overlaying the field, and the select's width comes from the wrapper's grid rather than its content, so it does not grow for a longer label. "Deutsch" therefore ran **17px underneath** the caret, and more right padding could not help because the caret is not in the select's box. Fixed by hiding PF's caret and drawing our own as a background inside the select's own padding: both are now in one box, so overflowing text clips at the content edge instead of painting over the chevron. Measured 5px clearance for "Deutsch"; the chevron colour follows the colour scheme via `--eddi-chevron`.

**Accessibility and keyboard pass.** Audited against the running page with real key events rather than assumptions.

Already correct, and left alone — Keycloak gets these right: tab order (username → password → show-password → forgot → Sign In, no `tabindex` anywhere), `<label for>` association on both fields, `autocomplete="username"`/`current-password`, `autofocus` on username (restored after a failed attempt), a single `<h1>`, a submit button inside the form so **Enter submits from any field** (verified with a real Return keypress), and an `aria-live="polite"` region that announces the error.

Three real defects fixed:

* **A focus-ring regression I had introduced.** `overflow: hidden` on the input-group — added to clip the password field and its toggle into one control — also clipped the toggle's focus ring, because that ring uses a *positive* `outline-offset` and the button sits flush against the group's edge. A keyboard user tabbing to "Show password" got a partially invisible indicator (WCAG 2.4.7). Rings inside the group now draw inward (`outline-offset: -2px`), verified fully visible.
* **No `lang` on `<html>`** — a WCAG 3.1.1 Level A failure. The template only emits `lang` when the realm has internationalisation enabled, which it did not. Fixed in **realm config, not FreeMarker**: enabling `internationalizationEnabled` emits `lang` and `dir` on `<html>`. *(This was first done with a single supported locale to avoid adding a switcher; that was the wrong trade and the realm now offers all 30 — see above.)*
* **Errors were not programmatically associated with their field.** Keycloak renders `<span id="input-error-password">` and sets `aria-invalid="true"`, but never `aria-describedby`, so a screen reader reaching the field announced "invalid entry" with no reason; the message was only spoken once, via the live region, at page load. Also, untouched fields carried `aria-invalid=""`, which is not valid ARIA and reads as *true* to some assistive tech.

That last fix required the theme's **first JavaScript** (`resources/js/eddi-a11y.js`, the accessibility helper, wired via `scripts=` in theme.properties; `eddi-theme.js` was added later for the colour-scheme control): CSS cannot set ARIA attributes. It is \~20 lines, adds no event handlers, reads no user input and makes no network calls. This is a deliberate departure from the "pure CSS, no FreeMarker" principle — the alternative was overriding templates, which is worse, or leaving the gap. `keycloak.v2` sets no `scripts` of its own, so nothing is replaced; that needs re-checking on upgrade, since `scripts` replaces like `styles` does.

**Not done, deliberately:** arrow-key navigation between fields and an Esc handler. Neither is standard for a login form, both would fight the platform conventions assistive technology relies on, and adding them would mean shipping keyboard handlers to an auth page for no real gain.

**Second design pass — after actually looking at the rendered page.** Everything up to here was measured, never seen. Seeing it made the problem obvious: the page ignored its own logo's design language. The wordmark is hairline, widely tracked, geometric — an instrument face — while the form under it was heavy 700 labels, pure-black input voids and a flat slab button. The direction taken is *a machined panel lit from a single thin source*:

* **Signature — a hairline amber light-line across the card's top edge**, brightest at centre and fading to nothing at both ends, with a matching ambient bloom behind the wordmark so the logo reads as the light source. One gesture expressed twice rather than several competing effects. It animates in once on load (620ms) and is suppressed under `prefers-reduced-motion`.
* **Depth.** The flat page fill became a layered radial wash; the card gained a 1px inset top highlight, a falloff gradient and a real shadow, so it sits *off* the page instead of *on* it.
* **The password field seam.** The field and its visibility toggle are siblings in a `pf-v5-c-input-group`, each with its own box — they met in a hard seam with mismatched corners, the most obviously unfinished detail on the page. The group is now the field; its children go flat, so the pair reads as one control.
* **Inputs** moved from pure black (which pulled more attention than the button) to a translucent recess with an inset shadow, plus an amber focus glow.
* **Type** now echoes the wordmark: title dropped to weight 300 with tracking, labels to 12px tracked caps. The title had to be set through `--pf-v5-c-title--m-3xl--FontWeight`, since `.pf-v5-c-title.pf-m-3xl` carries the weight at two-class specificity and beats a plain declaration.
* **Deliberately not done:** no gradient on the button. It is already the heaviest element; a gradient there would compete with the signature. The primary button did gain vertical padding, which incidentally takes it to a 44px touch target on mobile.

**Design pass — the page was correct but not designed.** Three fixes after looking at what the measurements implied rather than only whether they matched the palette:

* **Typography.** The page rendered in PatternFly's Red Hat Text/Display — the one EDDI surface in a different typeface, and matching neither the Manager (Noto Sans) nor this plan's own stated `system-ui` intent, because no `font-family` was ever set. Now ships the **same variable font the Manager loads**. At this point that was the Latin subset only — 35 KB for weights 100–900 — with other scripts falling through to the system stack; the Latin-ext, Cyrillic and Greek subsets were added later when the realm began offering all 30 locales (see above). Overriding `--pf-v5-global--FontFamily--text` alone is not enough — PF re-declares it as `var(--…--text--vf)` when variable fonts are present, so the `--vf` pair must be set too.
* **Inputs were invisible as inputs.** Fixing the card background had left fields at the card's own `#18181b`, separated from their container by a 1px hairline alone. Fields are now recessed to the page colour (`#0c0a09`), so they read as cut into the card rather than laid on it; the password-visibility toggle follows so the input group stays one control.
* **Three corner radii on one card** (8px card, 4px input, 3px button). Replaced with a deliberate two-step scale — 6px controls, 12px card — driven through `--pf-v5-global--BorderRadius--sm` so every PF control inherits it.

**WCAG 1.4.11 — since closed.** This was recorded as a known gap at the time: the resting input border (`#27272a` on `#18181b`) was 1.19:1 against a 3:1 requirement for UI component boundaries, and reaching it meant departing from the border colour the Manager uses throughout. It is now fixed with a dedicated `--eddi-field-border` token — zinc-500 on dark (**3.67:1**), stone-500 on light (**4.8:1**) — while the card keeps `--eddi-border`, because a panel edge is decoration and 1.4.11 does not apply to it. Fields consequently read more strongly than the Manager's do; that is the cost of the criterion.

**Review pass — four things the first commit had not established.**

1. **Graceful degradation confirmed.** Emptying the theme directory while `loginTheme: eddi` stays set — exactly what a failed `install.sh` asset download produces — yields **HTTP 200 with the built-in theme** and `ERROR ... Failed to find LOGIN theme eddi, using built-in themes` in the log. So the decision to make theme downloads non-fatal is safe **provided the fallback is all-or-nothing**: a *missing* theme cannot take authentication down. A *partial* one is a different matter — `theme.properties` would still resolve and Keycloak would render the eddi theme with its stylesheet or fonts 404ing — so `install.sh` discards the whole theme directory if any resource fails rather than leaving a half-built one mounted.
2. **The verification protocol is not vacuous.** Run against that deliberately broken state, both §6 step 2 (log grep) and step 3 (html-class and stylesheet assertions) fail as designed. Checks that cannot fail are worthless; these can.
3. **The `install.sh` realm update was executed, not just written.** Against a realm reset to `loginTheme: ""`, the GET-modify-PUT returns 204, sets all three fields, and is idempotent on a second run.
4. **A dead CSS selector removed.** `.pf-v5-c-login__main-header-desc` matches nothing and appears in no `keycloak.v2` template — the same mistake (styling an element the theme never emits) that killed the original `div.kc-logo-text` approach, reproduced at low stakes. Found by auditing every selector in our stylesheet against the live DOM. `.pf-v5-c-login__main-footer-band` was checked the same way and **kept**: it is emitted by `login.ftl` and only hidden because `registrationAllowed: false`. Also verified the keyboard focus ring — `:focus-visible` needs a real key event to match, and does then give `solid 2px #f59e0b` at 8.25:1 on the card, past WCAG 2.2's 3:1 for non-text indicators.

***

## 🧹 fix(release): retire stale deployment records, quiet the Postgres health line, revive two dead checks (2026-07-27)

**Repo:** EDDI (`fix/release-6.2-polish`) + eddi-chat-ui (`fix/release-6.2-polish`)

Findings from a full smoke test of `labsai/eddi:6.2.0-b638` against **both** datastore backends (MongoDB and PostgreSQL). Agent CRUD, multi-turn conversations, PropertySetter + Qute templating, undo/redo, group conversations (ROUND\_TABLE, followup/continue/close), MCP handshake and SSE streaming all behaved identically on both — no engine defects found. The items below are the rough edges that surfaced.

### 1. Deleting an Agent orphaned its deployment record (both datastores)

`RestAgentStore.deleteAgent` cascaded to schedules, workflows and the capability registry but never to `deployments`, and `IDeploymentStore` had no delete method to call. Every deleted-but-once-deployed Agent therefore left a row at status `deployed`; on each startup the runtime retried the redeploy, failed with `ResourceNotFoundException`, and logged a full ERROR stack trace — forever, accumulating with each deletion. Reproduced on current code; this instance had an orphan dating to 2026-04-01.

`AgentDeploymentManagement.checkDeployments` *already* carried self-heal logic for exactly this case (`isCausedByResourceNotFound` → mark `undeployed`), but it was unreachable: `AgentFactory.deployAgent` returns `void` and swallows the `ServiceException` internally, so the catch blocks never fired. That is why the April orphan survived for months despite the code being there.

* `IDeploymentStorage` / `IDeploymentStore`: added `deleteDeploymentInfos(agentId)`, implemented in `MongoDeploymentStorage` (`deleteMany`) and `PostgresDeploymentStorage` (`DELETE … WHERE AGENT_ID = ?`)
* `RestAgentStore.deleteAgent`: clears the Agent's deployment records; a failure here logs a warning and still deletes the Agent
* `AgentDeploymentManagement.checkDeployments`: checks up front whether the Agent config still exists and retires the record instead of attempting a doomed deploy. `isAgentConfigMissing` treats **only** `ResourceNotFoundException` as proof of absence — a store outage leaves the record untouched, so a transient DB failure can never mass-retire live deployments.

Three call sites delete Agents, not one. `McpAdminTools.deleteAgent` delegates to `RestAgentStore` and inherits the cascade, but `GroupConversationService`'s ephemeral cleanup and `TeardownAgentTool` call `agentStore.deleteAllPermanently` directly — and dynamic sub-agents *do* get deployment records, because `AgentSetupService.deployAndWait` goes through `RestAgentAdministration`. Every ephemeral group agent was therefore orphaning a record. Both now retire their records too (null-tolerant, non-fatal), following the existing field-injection pattern in those classes so the directly-constructed unit tests keep working.

**Ordering:** the cascade runs *after* `restVersionInfo.delete`, not before. That method validates its arguments internally and throws on a stale or unknown version — clearing the records first would strip a still-live Agent of what it needs to come back up. Pinned by a test.

**Design note:** the pre-check was chosen over making `deployAgent` rethrow — that method's dummy-agent-on-failure contract is relied on by the on-demand deploy path, and widening it for this would have been a far riskier change than a cheap existence check.

Verified against the real Postgres instance: the April orphan was retired, the ERROR + stack trace was replaced by a single `WARN … retiring stale deployment record`, and deleting a fresh Agent logged `Cascade-deleted 1 deployment record(s)` with the row gone.

> That run was observed against the first implementation, which retired a record by marking it `undeployed` — so the orphan was seen flipping `deployed` → `undeployed`. The review pass below replaced that with a scoped **delete**, so the record is now simply absent. The ERROR-to-WARN result and the cascade behaviour are unchanged.

### 2. MongoDB health check reported UP with no MongoDB (postgres profile)

Under `QUARKUS_PROFILE=postgres`, with the Mongo container stopped and its hostname unresolvable, `/q/health` still listed `"MongoDB connection health check": "UP"` — noise that would equally mask a genuine outage in Mongo mode.

The obvious `%postgres.quarkus.mongodb.health.enabled=false` **does not work**: that property is fixed at build time, so it cannot apply to one image pointed at either datastore (confirmed — even passing it as a runtime env var leaves the check in place). Disabled through smallrye-health instead, which *is* runtime-scoped:

```
%postgres.quarkus.smallrye-health.check."io.quarkus.mongodb.health.MongoHealthCheck".enabled=false
```

### 3. Two checks that could never fail

* `InfrastructureIT.openApiSpec` asserted `anyOf(200, 404)` against `/q/openapi` — a path that never serves the spec (`quarkus.smallrye-openapi.path=/openapi`). It passed whether or not OpenAPI worked. Now asserts `200` on `/openapi` plus real body content.
* `AGENTS.md` documented EDDI-Manager as served at `/chat/production`. It is served at `/manage`; `/` redirects to the `/welcome` chooser and `/workforce` is the group workspace. `/chat/production` is the standalone chat widget. The wrong path in this file is what sent an earlier review to the wrong URL.

### 4. eddi-chat-ui: dead-end landing and an agent name that never loaded

* `main.tsx` routed its catch-all to a hard-coded `/chat/production/default`. No instance has an agent literally named `default`, so this 404'd and left the visitor on a spinner that never resolved. Replaced with a new `AgentPicker` that lists the instance's actual agents, with explicit empty and error states. Reached only by deep-linking `/chat/production` without an agent id — the Manager's own chat (`/manage/chat`) was never affected.
* `fetchAgentDescriptor` called `GET /agentstore/agents/{id}` with no `version`, which answers `400`. Even fixed, that endpoint returns the Agent config and carries no name at all — so the agent name had never rendered. Now resolves the current version and reads `/descriptorstore/descriptors/{id}?version=N`.

**Note:** eddi-chat-ui builds its bundle directly into `EDDI/src/main/resources/META-INF/resources/`, so the two repos ship together. The superseded `chat-ui.p4wYUapg.js` / `chat-ui.D213XXZR.css` were removed; `chat.html` now points at the new hashes.

### Review pass on PR #611

* **CodeQL log injection (RestAgentStore).** `id` is a raw REST path parameter and reached two `log.infof`/`warnf` calls unsanitized. Now routed through `LogSanitizer.sanitize`, the pattern already used across the config REST stores. The equivalent logs in `TeardownAgentTool` and `GroupConversationService` were left raw on purpose: both only reach them after the id has passed a `createdAgentIds.contains(...)` check, so the value is a server-generated agent id, and every neighbouring log line in those classes prints it the same way.
* **Retire scoped, not agent-wide (CodeRabbit, Major).** The sweep proves only that *one* `(environment, agentId, version)` is missing, so retiring with an agent-wide `deleteDeploymentInfos(agentId)` acted far beyond its evidence — it would take out sibling records for versions nobody checked. Added `deleteDeploymentInfo(environment, agentId, version)` and use it here; the agent-wide variant stays for the delete cascade, where the whole Agent really is gone. In practice the harmful case is hard to construct — `HistorizedResourceStore.read` falls back to history, so an older version normally still resolves — but the mismatch between what is checked and what is acted on is the kind that bites once versioning semantics shift. Covered by a mixed-version regression test.
* **Pin the sibling in the mixed-version test.** The regression test asserted the missing version's record was deleted and that no agent-wide delete happened — but it would still have passed if the code had *also* removed the live sibling through the scoped API. Added the explicit negative (`never()` on version 2, in any environment) and `times(1)` on the deletion that should happen. Mutation-checked: injecting a sibling delete now fails the test, where before it slipped through. While there, the `never()` verifications in these classes moved from `anyString()`/`anyInt()` to `any()`, which also matches null — `anyString()` does not, so a null-argument regression would have verified vacuously.
* **Retire by delete, not by upsert.** `setDeploymentInfo` upserts. If an Agent were deleted between the sweep reading the deployment list and reaching the retire branch, marking it `undeployed` would *resurrect* the row the delete cascade had just removed — an inert but permanent tombstone. The sweep now calls `deleteDeploymentInfos`, which is idempotent, and an Agent that no longer exists has nothing to undeploy.
* **TOCTOU between the pre-check and the deploy (CodeRabbit, Major) — acknowledged, not fixed.** If an Agent is deleted in the window between `isAgentConfigMissing` and `deployAgent`, the record can stay `deployed` and be cached in `deploymentInfos` until the next restart. The suggested remedy is to make `deployAgent` surface a definitive missing-result — the contract change deliberately avoided above, since the dummy-agent-on-failure behaviour is relied on by the on-demand deploy path. The window is milliseconds, the consequence is one stale row, and it self-corrects on restart, so it is not worth widening a core contract on a release-polish branch.
* CodeRabbit nitpicks, all taken: `.gitattributes` marks the chat-ui bundles and Vite assets `linguist-generated` so reviewers stop being handed minified output; `MongoDeploymentStorage` gains an `agentId`-leading index, since the existing compound index starts with `deploymentStatus` and cannot serve the new agentId-only delete; and two coverage gaps closed — deployment cleanup on the cascade-success path, and the guarantee that a *failed* permanent delete leaves the records alone.
* Style nits from Copilot: imports instead of inline `java.util.concurrent.*` FQNs in `TeardownAgentTool` (per §4.7), and four comments the Eclipse formatter had reflowed into orphaned fragments (`// throws`, `// path`, `// it`, `// up.`). Comment lines are now short enough to survive `formatter:format`.

### Not fixed here (deployment config, not the release artifact)

* The stored Anthropic API key is invalid — every LLM-backed agent fails with `invalid x-api-key`. EDDI handles it correctly (non-retryable, conversation `ERROR`, no stack trace to the client), but any demo needs a working key.
* `VaultSaltManager` reports existing DEKs with no per-deployment salt and is falling back to the legacy fixed salt; the KEK migration should be run.

***

## 🔒 fix(ci): remove accidentally-committed langchain4j-mcp decompiled sources (2026-07-27)

**Repo:** EDDI (`feat/v6.2.0-prep`)

Both the **Dependency Review** and **Trivy Filesystem Scan** CI checks were failing on this branch with 3 HIGH-severity Jackson CVEs (GHSA-r7wm-3cxj-wff9, CVE-2026-54512, CVE-2026-54513).

**Root cause:** commit `702e2f79b` (`fix: serialize SSE log events as JSON instead of toString()`, 2026-07-24) accidentally committed a decompiled copy of the `langchain4j-mcp` jar into a new top-level `lc4jmcp/` directory (123 files) — leftover from decompiling the jar to debug the `toString()` logging issue, swept up by a broad `git add`. The directory sat outside `src/main/java`, was never referenced by `pom.xml` or any source file, and Maven never compiled it — but its embedded `META-INF/maven/dev.langchain4j/langchain4j-mcp/pom.xml` (the jar's own build-time manifest) declared `jackson-databind:2.21.3` / `jackson-core:2.21.3`, which both scanners picked up as if it were a real dependency manifest.

**Verification:** `./mvnw dependency:tree -Dincludes=com.fasterxml.jackson.core` confirms the actual resolved build uses `jackson-databind:2.22.0` / `jackson-core:2.22.0` (patched, via `quarkus-jackson:3.37.4` and `langchain4j:1.18.0`) — this was never a real vulnerability in the shipped app, just dead decompiled code confusing the scanners.

**Fix:** `git rm -r lc4jmcp` — no source or config referenced it, so removal is inert.

***

## 🧪 test(ui): close entity streams and assert real content in welcome/workforce tests (2026-07-26)

**Repo:** EDDI (`feat/v6.2.0-prep`)

Copilot review feedback on `RestWelcomeResourceTest` / `RestWorkforceResourceTest`: the tests left the returned `InputStream` entity open and never read it, despite a display name promising a "readable" entity.

**Changes (test-only, no production code touched):**

* Added a private `readEntity(Response)` helper to both test classes — asserts a non-null `InputStream` entity, reads it fully inside try-with-resources (closing the classpath stream), returns UTF-8 content
* `viewHtmlReturnsOkWithEntity` now asserts the body contains `<html`, matching its display name instead of only checking the entity type
* `viewDefaultAndViewHtmlServeSameShell` now compares the two bodies for equality — previously it only asserted both entities were non-null, which never actually proved both endpoints serve the same shell

**Design decision:** assert on `<html>` rather than any Vite-generated markup. `welcome.html` / `workforce.html` are SPA shells regenerated on every Manager/chat-UI asset update, so hashed asset filenames or inline styles would make content-specific assertions stale on the next `chore: update Manager UI assets` commit.

**Files:** `RestWelcomeResourceTest.java`, `RestWorkforceResourceTest.java` (6 tests, all green)

**Noted, not fixed:** `RestManagerResourceTest` wraps several calls in `try { … } catch (Exception e) { /* expected */ }`, which passes regardless of whether the code under test behaves correctly. Out of scope for this review fixup — worth tightening separately.

***

## 🐛 fix(memory): orphaned ConversationDescriptors on conversation deletion (2026-07-24)

**Repo:** EDDI (`feat/v6.2.0-prep`)

ConversationDescriptors (created at conversation start by `ConversationSetup`) were never cleaned up when a conversation was permanently deleted, causing "Memory snapshot not found — Descriptor is orphaned" warnings on every conversation listing.

**Root cause:** `ConversationDescriptorStore.deleteAllDescriptor()` existed but was never called in any deletion path. Three separate hard-delete paths all deleted the memory snapshot but left the descriptor behind.

**Fixes (all in production code):**

* **RestConversationStore.deleteConversationLog** — added `conversationDescriptorStore.deleteAllDescriptor()` after permanent snapshot deletion
* **RestConversationStore.permanentlyDeleteEndedConversationLogs** — added descriptor cleanup in both the happy path (old enough to delete) and the catch path (orphaned snapshot without DocumentDescriptor)
* **GdprComplianceService.deleteUserData** — added `IConversationDescriptorStore` dependency, hoisted conversation ID resolution before step 2 (eliminating a duplicate DB query), added step 4a to delete descriptors per-conversation before the bulk snapshot delete

**Files:**

* `RestConversationStore.java` — 3 `deleteAllDescriptor` calls added
* `GdprComplianceService.java` — new dependency, hoisted ID resolution, new step 4a, updated Javadoc
* `RestConversationStoreTest.java` — 9 verify assertions added (positive + negative)
* `GdprComplianceServiceTest.java` — mock added, constructors updated, `inOrder` verification for deletion ordering

**Design decision:** soft-delete (`deletePermanently=false`) does NOT touch the descriptor — the `DocumentDescriptorFilter` JAX-RS interceptor still handles soft-deletion via its existing mechanism. Only hard-deletes clean up the ConversationDescriptor.

***

## 🐛 fix(llm,group): empty task results, maxTokens defaults, verification display (2026-07-24)

**Repo:** EDDI (`feat/v6.2.0-prep`)

Group conversations with Anthropic thinking models (e.g. `claude-sonnet-5`) produced empty task results and raw JSON in the UI. Root cause: langchain4j's default `maxTokens=1024` was consumed entirely by thinking tokens, leaving nothing for the response text.

**Bug fixes:**

* **AnthropicLanguageModelBuilder** — added `maxTokens`, `topP`, `topK` support; default `maxTokens=16384`
* **OpenAILanguageModelBuilder** — added `maxTokens` support (was missing)
* **LlmTask** — null/blank response skips output creation (no empty bubbles)
* **GroupConversationService** — verification JSON → human-readable ✅/❌ summaries
* **ConversationLogGenerator** — null-safety on `KEY_TEXT` entries

**Documentation:** `docs/langchain.md` — Output Token Limits section, provider matrix, updated examples

**Tests:** Anthropic maxTokens, blank output guard, verification formatting, null text filtering

***

## 📝 README: add HITL section, JSON mode, tool context ceiling (2026-07-23)

**Repo:** EDDI (`feat/v6.2.0-prep`)

The README's Features section was missing Human-in-the-Loop Governance entirely — despite HITL being a Phase 9b completed feature with a 479-line dedicated doc (`docs/hitl.md`). This was the biggest gap found after a systematic comparison of the README against `AGENTS.md` completed features, `docs/changelog.md`, and all documentation files.

**Changes to `README.md`:**

* **Added `### 🛑 Human-in-the-Loop Governance` section** (8 bullets) between Smart Model Cascading and Enterprise Security — covers turn-level approval, per-tool-call gating, group phase approval, timeout policies, no-progress guard, Slack/MCP approval surfaces, and crash recovery
* **Added HITL link to Documentation table** — `docs/hitl.md` was not linked
* **Added JSON Response Mode** to LLM Provider Support — `jsonResponseFormat` policy (`auto`/`on`/`off`) with provider-aware negotiation
* **Added Tool Context Ceiling** to Memory & Context Management — `maxToolContextTokens` (default 60k) prevents provider context-window errors from tool loops
* **Added Tool Calling and Multi-Model Cascading cross-reference** under LLM Provider Support

**What was deliberately NOT added** (internal hardening, not README material):

* Constant-time HMAC, versioned audit HMAC v2, tool-cache scope fail-safe, HTTP logging guard, SSE secret redaction, budget enforcement warnings, stream resilience — these belong in release notes
* Provider count stays at 12 — verified against `docs/langchain.md` (the authoritative source per §2 rule 7); Vertex AI is a sub-type of Gemini, not a separate provider
* `docs/template-preview.md` link not added — file doesn't exist yet

***

## 🧵 Model registry: a rotation landing mid-build could re-cache a stale model (2026-07-23)

**Repo:** EDDI (`fix/chatmodel-stale-rebuild-race`)

CodeRabbit flagged this on the `fix/backlog-defect-remediation` PR (Major / "Heavy lift") and it was deliberately deferred from that branch. It is **separate** from the `C1` check-then-act race fixed there (commit `96eacacde`, which stopped a concurrent `clear()` turning a cache *hit* into a `null` return).

**The defect.** `getOrCreate`/`getOrCreateStreaming` resolve global-variable and vault-secret values, build a model from them, and then `put` it into the cache — all unsynchronised. A secret rotation (`invalidateForSecret` → `clear()`/`evictMatching`) or a global-variable edit (the invalidation listener → `clear()`) can land **between the resolve and the `put`**. The `clear()` finds nothing (the entry does not exist yet), and the in-flight build then `put`s a model constructed from the **now-stale** secret. Because the entry is present and valid-looking, every later turn keeps reusing that pre-rotation model until some *unrelated* invalidation happens to evict it — i.e. a rotation could silently fail to take effect.

**The fix — an invalidation generation.** A process-wide `AtomicLong invalidationGeneration` is bumped by every invalidation (`invalidateForSecret` for both the bulk-clear and targeted-`evictMatching` paths, and the global-variable listener). A build snapshots the generation *before* it resolves any value, and publishes through `publishIfCurrent`, which caches the model **only if the generation is unchanged**; otherwise it discards it (the racing caller still receives the instance for its own turn — a one-turn window inherent to lock-free building — but the next lookup misses and rebuilds from current values). The re-check and the `put` are held under a new `publishLock` that the invalidation paths also hold around their bump-and-clear, so "re-check then put" and "bump then clear" cannot interleave — without that lock the re-check would itself be a check-then-act with the same hole.

**Design decisions:**

* **Do-not-cache rather than rebuild-and-retry.** CodeRabbit suggested "discard and rebuild (or at least do not cache)". Not caching fully closes the reported bug (persistent staleness) with no unbounded-retry surface under an invalidation storm; the residual one-turn exposure of the racing caller is fundamental to building without holding a lock across the whole (potentially slow) provider build.
* **The generation is a coarse, registry-wide signal.** An invalidation for an unrelated secret still bumps it, so a concurrent build may be rebuilt needlessly — wasteful but always correct, and far simpler than trying to match the in-flight build against the specific rotated reference. Invalidations are rare.
* **`publishLock` scope is minimal.** Builds (the expensive part) stay fully concurrent; the lock covers only the cheap re-check+`put` and the clear/evict. Both resolvers fire their listeners outside any lock, so taking `publishLock` in the callbacks introduces no lock-ordering/deadlock risk, and no path holds `publishLock` while calling back into a resolver.
* **The counter is registry state, not conversation state.** The class stays `@ApplicationScoped` and per-conversation-stateless; the generation and lock are process-global memoization-cache concerns, which is the correct home for them.

**Tests:** `ChatModelRegistryTest` gains a nested `StaleRebuildDuringInvalidationTests` (C5) with a sync and a streaming case. Each installs a builder that fires `invalidateForSecret(null)` from inside `build()`/`buildStreaming()` — deterministically the interleaving "clear lands between build-start and publish" — then asserts the racing model is **not** served to the next caller (it rebuilds) and that caching resumes normally afterwards, with an exact build-count assertion. Mutation-checked twice: against the pre-fix code both tests fail with `expected: not same but was: <same instance>`, and reverting only the generation guard in `publishIfCurrent` (unconditional `put`) fails exactly those two tests and no others.

***

## 🧪 De-vacuum three tests and fix a doc/fixture drift (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

CodeRabbit and Copilot flagged a cluster of tests that passed for the wrong reason, plus two documentation/fixture mismatches. Each behavioural change is mutation-checked.

* **`MongoTenantQuotaStoreTest.everyUpsertIsKeyedOnTenantIdOnly` was vacuous.** It verified `findOneAndUpdate`/`updateOne` with `atLeast(0)`, which never fails, and its filter loop asserted nothing when nothing was captured. Now `atLeastOnce()` plus an explicit `upsertsChecked > 0` guard. Mutation: removing the store exercise fails with `Wanted but not invoked`; the old form passed.
* **`CacheFactoryTest.negativeLifespanIsUnlimited` read immediately** — a negative-lifespan-mapped-to-short-positive-TTL bug would slip through. Now sleeps past `PAST_TTL_MILLIS` and asserts survival. Mutation: mapping negative → 1ms fails with `expected: <value1> but was: <null>`.
* **`CacheFactoryTest.nonceCacheCapacityCoversItsTtl` used whole seconds**, so a capacity derived from `ttl.toSeconds()` (truncating) instead of `toMillis()` would pass. Now uses a fractional `390_500ms` TTL and millisecond-precision occupancy. Mutation: reverting `maximumSizeFor` to `toSeconds()` fails with `expected: <234300> but was: <234000>`.
* **`LifecycleManagerTest` audit fixture used `{input, output}`** for `llmDetail.tokenUsage`, a shape production never emits — `buildAuditEntry` copies `audit:token_usage` straight through, and that key's contract is `{inputTokens, outputTokens, totalTokens}`. Fixture and assertion aligned to the real contract, and the inline `java.util.Map` / `Consumer` FQNs replaced with imports (AGENTS.md §4.7).
* **This changelog's D5 cache-key example** still showed `arguments.length()` directly after `buildKey` gained a null-coalescing guard; the formula now shows `args = arguments == null ? "" : arguments` first.

***

## 🐛 `ToolNameResolver.canonical` could return null, contradicting its own contract (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Copilot flagged that `canonical(dispatchName, canonicalNames)` used `getOrDefault`, which returns a **stored** null when a key maps to null — so the method could hand back null even though its javadoc promises "callers never need a null check". A null slug would then flow into the price and TTL lookups. The tell was in the test itself: `nullValuedMappingDegrades` was named for degrading to the dispatch name but asserted `assertNull`, documenting the bug as the contract and leaning on `ToolInvocation` to normalise it downstream.

Fixed to treat a null-valued key the same as an absent one (`get` + null fallback), and the test now asserts what its name always said — the result is the dispatch name, never null. Mutation-checked: reverting to `getOrDefault` fails the test with `expected: not <null>`.

The map is built by `AgentOrchestrator.buildToolSetup`, which substitutes the dispatch name and never stores nulls, so this was defence-in-depth rather than a live crash — but a method whose javadoc, code and test display-name all disagreed is exactly the silent inconsistency this branch set out to remove.

***

## 🔒 Redact secrets from the tool trace before it reaches the SSE stream (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

The D3 fix earlier on this branch made the tool trace reach the `task_complete` SSE frame — the first time tool arguments and results leave the process on a channel with no redaction of its own. Those payloads are LLM- and user-controlled, so they can carry API keys or bearer tokens. Copilot flagged it HIGH on the PR; the tradeoff had been disclosed in the PR description rather than fixed.

Now fixed at the producer. `LifecycleManager.buildTaskSummary` deep-redacts the collected trace through `SecretRedactionFilter` — the same filter the audit ledger already applies via `AuditLedgerService.scrubSecrets` — before putting it on the summary, bringing the live-display path to parity with the audit path. The redaction walks maps and lists recursively and scrubs every string leaf, and it operates on a fresh copy: the trace stored in conversation memory is left intact for the owner-scoped `RestToolHistory` endpoint. Only the summary's `toolTrace` is touched; `buildAuditEntry` reads its own `audit:tool_calls` key (separately scrubbed on submit), so the audit path is unchanged and not double-scrubbed.

Regression test `LifecycleManagerTest.summaryRedactsSecretsInToolTrace` feeds an `sk-…` key and a bearer token through the trace and asserts neither reaches the summary while the non-secret structure (`type`, `tool`) survives. Mutation-checked: bypassing the redaction call fails with the raw key visible in the captured payload.

***

## 📝 Budget-enforcement docs corrected to match the shipped opt-in default (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

CodeRabbit caught that three places still described `enforceBudget` as **opt-out, default `true`** after the flag was reverted to opt-in (`false`) earlier on this branch — a config knob whose documentation stated the opposite of its behaviour, which is precisely the defect class this branch exists to remove, and it contradicted the `langchain.md` table row in the same file. Corrected:

* `LlmConfiguration.Task#enforceBudget` javadoc — "opt-out … itself `true`" → opt-in, default `false`, with the WARN rationale.
* `docs/security.md` (Cost Tracking config line and Behaviour bullet) — "default `true`" / "enforced by default" → opt-in.
* `docs/langchain.md` (Budgets prose) — "Enforcement is **opt-out** … default `true`" → opt-in.

The code was already correct (`BUDGET_ENFORCE_DEFAULT` resolves `false`); only the prose lied. Also documented that `toolPricing` accepts a dispatch name as well as a slug (dispatch name wins), and fixed two markdownlint nits (MD028 blockquote continuation in `langchain.md`, an unlabelled fence in this file).

***

## 🧹 A dead `resumeToolLoop` parameter, and a CodeQL note on a deliberately unguarded test parse (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Cluster `dead-param-and-test-nit`: one github-code-quality dead-parameter finding and one CodeQL note on a test helper. Both traced to source before touching anything — the first is real and removed, the second is a false positive left unchanged (with one clarifying javadoc line).

### A — `resumeToolLoop`'s `templateDataObjects` parameter was dead (real, removed)

Both `AgentOrchestrator#resumeToolLoop` overloads (the 7-arg delegator and the 8-arg policy-carrying body) declared `Map<String, Object> templateDataObjects`, and the javadoc claimed it fed "post-response and fallback rebuild". Reading both bodies end to end, it is referenced nowhere: the 7-arg overload only threads it into the 8-arg, the 8-arg never touches it, and the fallback path it named (`fallbackRebuildMessages(task, memory, batch)`) does not take it. `LlmTask.executeResume` still uses its *own* local `templateDataObjects` (template render + response-metadata put) — only the pass-through into `resumeToolLoop` was inert. Exactly the branch's theme: a parameter that looks load-bearing and is not.

**Fix:** dropped the parameter from both overloads, the delegating call, the `@param`/`@link` javadoc, the sole production caller (`LlmTask`), and every test call site. Because dropping an argument silently invalidates Mockito matcher lists (an unmatched stub returns null; a `verify(never())` goes vacuous — both stay green, both would be wrong), every stub/verify was re-checked: the real 7-arg calls in `AgentOrchestratorResumeToolLoopTest`/`AgentOrchestratorCoverageTest` shed their `Map.of()`; the 8-matcher `any()` lists and the `anyMap()` verifies in the `LlmTask*` tests each lost exactly the sixth matcher, keeping arity aligned to the new signature. For a behaviour-preserving removal the verification is the compile plus the suites: `test-compile` clean, and 178 tests across the 8 named classes green (`AgentOrchestratorResumeToolLoopTest` 11, `AgentOrchestratorToolPauseTest` 8, `AgentOrchestratorCoverageTest` 59, `LlmTaskResumeModeTest` 9, `LlmTaskCoverageTest` 43, `LlmTaskAgentModeMetadataTest` 7, `LlmTaskCoverage2Test` 29, `LlmTaskAuditLedgerTest` 12). The two `verify(never())` sites stay non-vacuous — in `LlmTaskResumeModeTest` the sibling positive verify binds the same 7-arg overload and passes.

### B — CodeQL "missing catch of NumberFormatException" in a test helper (false positive, one clarifying javadoc line)

`ChatModelRegistryTest.parseTimeoutLikeARealProvider` does `Duration.ofMillis(Long.parseLong(parameters.get("timeout")))` under an un-trimming `isNullOrEmpty` guard, and CodeQL wants a try/catch rethrowing `IllegalArgumentException`. The helper's whole job is to reproduce, verbatim, what the shipped provider builders do — every one of `OpenAI/Azure/Bedrock/Ollama/HuggingFace/Gemini/Anthropic/Mistral/…LanguageModelBuilder` runs `builder.timeout(Duration.ofMillis(Long.parseLong(parameters.get(KEY_TIMEOUT))))` with no guard. The C3 `TimeoutNormalisationTests` depend on that fidelity: they prove the registry's `normalizeTimeout` drops blank/non-numeric/zero values before a builder ever sees them. Wrapping the helper's parse in a try/catch would make it tolerate values the real builders reject, so a normalization regression would pass silently instead of surfacing as the `build()`-time crash the tests guard.

Demonstrated: neutralize `normalizeTimeout` (early `return`) and the C3 tests fail with `NumberFormatException` thrown straight from the helper — `"30s"`, `" "`, `"not-a-number"`, `" 5000 "` (2 failures / 4 errors); restore and `ChatModelRegistryTest` is 41 green. So the finding is a false positive and production/test behaviour is untouched. Rather than a bare suppression, the helper's existing javadoc gained one sentence stating the unguarded parse is deliberate and must stay so, naming the try/catch note as a false positive — making the intent unmistakable to the next reader without weakening the test.

***

## 🔐 Constant-time audit HMAC comparison, and a retry-cost finding that wasn't (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Cluster `retry-cost-and-hmac`: one CodeRabbit correctness claim and one CodeRabbit security nitpick. Both were traced end to end before touching anything — one is real and fixed, the other is a false positive left unchanged with the evidence recorded here.

### B — `AuditHmac.verifyHmac` compared digests with `String.equals` (real, fixed)

The compliance ledger's HMAC verification did `computeHmac(entry).equals(stored)` (and the v1 branch likewise). `String.equals` returns on the first differing character, so its running time leaks how long a prefix of a forged HMAC is correct — the classic side channel that lets an attacker reconstruct a valid tag one hex digit at a time against an endpoint that reveals verify latency.

**Fix:** decode both the recomputed and the stored digest from hex and compare the raw bytes with `java.security.MessageDigest.isEqual`, which is data-independent. The version selection is unchanged and deliberately kept non-constant-time: the `v2:` prefix only picks the canonicalizer and is not secret (per the frozen-v1 design, a v2-tagged value is *never* retried against v1). A stored value that is not valid hex — truncated, mangled, never a digest — now decodes to null and is rejected rather than throwing out of a whole-ledger verification sweep.

Because a constant-time swap is behaviour-preserving, no unit test can observe the timing difference; the added tests instead guard that the refactor did not break verification. `AuditHmacTest`:

* `bothVersionsVerifyAndBothRejectTampering` — a bare-hex v1 row and a `v2:` row both verify, and either one tampered is rejected.
* `malformedStoredHmacIsRejected` — empty / non-hex / odd-length / truncated / `v2:`-only stored values are rejected without throwing.
* `v1DigestUnderV2TagDoesNotVerify` — a v1 digest mislabelled with the `v2:` tag fails, proving the version tag is never retried against the other canonicalizer.

Mutation-checked two ways. (1) Revert the comparison to `String.equals`: all 29 tests still pass — expected, and the point, since the fix must not change verification behaviour. (2) Point the v1 legacy branch at the v2 canonicalizer: `bothVersionsVerifyAndBothRejectTampering` and the pre-existing `legacyV1EntryStillVerifies` both fail (`TooManyActualInvocations`-free, a plain assertion failure that a pre-v2 ledger row no longer verifies), proving the regression tests guard the version selection the fix preserves. Restored: `AuditHmacTest` 28, and the rest of `engine.audit` (`AuditLedgerServiceTest`, `…BranchTest`, `…ExtendedTest`, `AuditRetentionConfigTest`, `AuditStoreTest`) 61 — all green.

### A — "tool charges are not rolled back when a retry replays" (false positive, unchanged)

CodeRabbit (Major): the tool loop runs inside `AgentExecutionHelper.executeWithRetry(() -> {…})`; the lambda already resets `tokenHolder[0] = null` because a retry replays it and would double-count *tokens*; the same replay re-runs the tool calls, each of which charges an `@ApplicationScoped` per-conversation counter with no per-attempt undo, so an abandoned attempt's spend allegedly stays counted and inflates `toolCostUsd`, the audit cost and `isWithinBudget`.

Traced against source, the premise does not hold:

* **Default config (`enableToolCaching` true): the replay is free.** `ToolExecutionService.executeToolWrapped` returns at the cache-hit branch (step 2) *before* `costTracker.trackToolCall` (step 5). A replayed attempt re-issues the identical `(name, arguments)` call, hits the cache the first attempt populated, and is charged nothing. So the very configuration CodeRabbit assumes produces no double-count at all.
* **`enableToolCaching` false: the replay genuinely re-executes — and the re-charge is correct.** With no cache, the abandoned attempt's tool really ran (a real search-provider API call, a real HTTP POST), really cost money, and the successful attempt's replay runs it *again*, a second real call. `ToolCostTracker` is the authoritative real-spend ledger behind `maxBudgetPerConversation` and the audit dollar figure; billing two real executions is right, not "inflation". Rolling it back would *under*-count real spend and let an agent evade its budget by inducing retries.
* The token reset is not analogous. `tokenHolder` is a per-`ExecutionResult` reporting accumulator whose number should match the final coherent attempt; `ToolCostTracker` is a cumulative expenditure ledger. Different jobs, correctly treated differently.

So there is no phantom charge to roll back: every charge corresponds to a tool execution that really happened and really cost money. Production code left untouched. A characterization test pins the property that makes the finding moot — `AgentOrchestratorToolCostTest.retriedLoopChargesReplayedToolCallOnce`: attempt 1 executes and is charged for `searchWeb`, the next model call fails retryably, the whole lambda replays, and with a **real** `ToolCacheService` the replayed identical call is a cache hit — `searchWeb` runs once, the conversation is charged `SEARCH_PRICE` once, and `toolCostUsd` reports exactly that one charge. Mutation-check: swap the real cache for the always-miss stub (the `enableToolCaching: false` model) and the tool runs twice (`verify(times(1))` fails with `TooManyActualInvocations … But was 2 times`), demonstrating that off-cache it is real re-execution, not double-accounting. Restored: `AgentOrchestratorToolCostTest` 10, green.

***

## 🗄️ Two cache keys that threw away information (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Two Copilot findings on the caching layer, both verified against source before touching anything. Both are the branch's theme one level down: not a control that does nothing, but a **key that silently loses the distinction it was built to preserve**.

### A — `ToolCacheService.buildKey` dereferenced tool arguments that can legitimately be null

`buildKey(scopeTag, toolName, arguments)` called `arguments.length()` with no guard. Copilot claimed `ToolExecutionRequest` can be built without arguments; that is only half the question, so the live path was traced end to end:

* `dev.langchain4j.agent.tool.ToolExecutionRequest` (1.18.0) declares `private final String arguments` with **no default and no validation** — `arguments()` returns null whenever the builder never set it.
* `OpenAiUtils.toolExecutionRequest` passes the wire value straight through: `.arguments(functionCall.arguments())`, and `FunctionCall.arguments` is a plain Jackson-deserialised field. A provider that omits `function.arguments` for a zero-argument call — routine on OpenAI-compatible endpoints — yields a null. (The newer Responses API path defaults it to `"{}"` via `asText("{}")`; the classic chat-completions path does not.)
* `AgentOrchestrator` line 1614 forwards `toolRequest.arguments()` into `executeToolWrapped` with **no guard**. The only two coalescing sites on the branch (`normalizeToolCallIds`, `rebuiltRequest`) are on the HITL pause/resume path and never run for a live call.

The NPE was raised inside `executeToolWrapped`'s try block, so it never surfaced as a crash — it came back to the model as `Error executing tool: Cannot invoke "String.length()" because "arguments" is null`, **before the tool ran**. Net effect: a zero-argument tool call failed with caching on (the default) and succeeded with caching off.

**Fix:** `buildKey` coalesces null to `""`. A null and an empty argument string both mean "no arguments" and deliberately share one entry.

### B — the TTL cache key truncated to whole seconds

`CacheFactory.getCache(name, ttl)` keyed instances on `name + ":ttl=" + ttl.toSeconds()`. `toSeconds()` truncates, so every sub-second TTL collapsed onto `":ttl=0"` and any two TTLs sharing a whole-second part collapsed together. Two callers asking one cache name for two TTLs then shared **one** Caffeine instance whose expiry policy was whichever of them built it first; the other TTL was accepted and silently ignored. The same method already derived its capacity from `ttl.toMillis()` in `maximumSizeFor`, so the key was strictly coarser than the sizing it keyed.

Reachability, stated honestly: **no production caller triggers it today.** Each of the four TTL call sites (`NonceCacheService`, `ChannelTargetRouter`, and two in `SlackEventHandler`) requests its cache name exactly once, from a single `@PostConstruct`. This is a latent defect in a shared factory, not a live incident — but it is one config change away (`NonceCacheService`'s TTL is `Duration.ofMillis(maxAgeMs + clockSkewMs + buffer)` over two operator-settable millisecond properties) and the fix is a single token.

**Fix:** the key renders `ttl.toString()`. It is injective over distinct `Duration`s, identical for equal ones — `ofSeconds(60)` and `ofMinutes(1)` still share, as they should — and strictly finer-grained than the `toMillis()` the capacity is derived from, so one key can never span two capacities.

### Tests

* `ToolExecutionServiceTest.NullArgumentsTests` (3 tests) wires the **real** `ToolCacheService` over a real `CacheFactory` — a mocked cache service cannot reach `buildKey` — and drives `executeToolWrapped` with `ToolExecutionRequest.builder().id(…).name(…).build().arguments()`, i.e. the null taken from langchain4j's own type rather than a synthetic one. Asserts the tool runs, the result comes back verbatim, and a second call is served from the cache (proving the write survived too).
* `CacheFactoryTest`: two sub-second TTLs (1 ms / 999 ms) get distinct instances with distinct expiry; two TTLs sharing a whole-second part (10 001 ms / 10 900 ms) get distinct instances; and equal TTLs expressed differently still share one, so the fix does not leak an instance per caller.

Mutation-checked by reverting each fix: `expected: <2026-07-23T09:00:00Z> but was: <Error executing tool: Cannot invoke "String.length()" because "arguments" is null>`, `expected: <long> but was: <null>`, and `10001ms and 10900ms must not truncate onto one instance ==> expected: <a> but was: <b>`. Restored: `CacheFactoryTest` 19, `CacheImplTest` 51, `ToolCacheServiceTest` 79, `PaginatedResponseStoreTest` 13, `NonceCacheServiceTest` 10, `ToolExecutionServiceTest` 32, `ToolExecutionServiceExtendedTest` 8, `ToolExecutionServiceBranchTest` 2 — 214 tests, 0 failures.

***

## 🔐 Ledger integrity, replay depth, and a verification that could not fail (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Four review findings on this branch's own remediation work — the common thread is the one the branch set out to remove: **a control that looks present and does nothing.**

### D1 — three `verify(never())` assertions that matched zero invocations either way

`ToolExecutionServiceTest.nullConversationIdSkipsCostTracking` and `ToolExecutionServiceExtendedTest.skipsCostWhenNoConversation` both claimed to cover `if (enableCostTracking && conversationId != null)`. Both pass `conversationId = null`, so the only invocation a regression could produce is `trackToolCall(invocation, null)` — and **Mockito's `anyString()` does not match null**. The verification matched zero invocations whether the guard existed or not. Removing `&& conversationId != null` and running all three `ToolExecutionService*Test` classes gave `Tests run: 39, Failures: 0` — neither test noticed, while production would have hit `computeIfAbsent(null, …)` on a `ConcurrentHashMap` inside `ToolCostTracker` and returned `Error executing tool: null` to the model for every tool call.

Both now use `nullable(ToolInvocation.class)`/`nullable(String.class)`.

**Audit of the rest of the branch's tests** for the same trap — grep of every `never()` verification in the touched test files, asking for each "can production pass null at that position?" — found one more: `LlmTaskCoverage2Test.nullUserInput_ragSkipped` pins `if (userInput != null)` around `ragContextProvider.retrieveContext(memory, task, userInput)` with `anyString()` in the userInput position. Same defect, same fix. The remaining `anyString()`/`never()` pairs sit on positions production fills from literals or non-null config and are sound.

All three mutation-checked by removing the corresponding guard: `costTracker.trackToolCall(… or(isNull(), isA(String)))` / `ragContextProvider.retrieveContext(<any>, <any>, or(isNull(), isA(String)))` → `Never wanted here … But invoked here … with arguments: [ToolInvocation[…], null]`.

### D2 — the nonce replay cache was under-sized against its own TTL

`CacheFactory` pinned `nonce-replay-protection` at 100 000 with a comment computing `330s × 300rps`, but `NonceCacheService.init()` asks for `maxAge + clockSkew + TTL_BUFFER_MS` = **390s**, so steady-state occupancy at 300rps is 117 000. Worse, Caffeine's W-TinyLFU admission is frequency-based, not LRU: a nonce is written once via `putIfAbsent` and never read, so all frequency estimates tie and the filter rejects the **candidate** — the newly inserted nonce is dropped rather than the oldest. Retention collapses precisely on the most recent (most replayable) nonces; measured against real Caffeine on a fake ticker, 17.1% of nonces still inside the 330s replay window were already forgotten. A forgotten nonce is a captured signed A2A envelope that replays inside its own freshness window.

**Fix:** capacity is now *derived from the TTL the cache is asked for*, not hard-coded. `CacheFactory.RATE_SIZED_CACHES` maps a cache name to the peak write rate it must absorb (`nonce-replay-protection` → 300/s) and `maximumSizeFor(name, ttl)` returns `peakRps × ttlSeconds × 2.0`, never below the configured floor. The ×2 head-room is because eviction is not LRU: sizing at exactly the steady-state occupancy still drops the newest entries under any burst. A future change to the replay window now moves the capacity with it.

Kept `TTL_BUFFER_MS` rather than dropping it — it is a real margin against a nonce being forgotten while still replayable, and it is no longer free-floating now that the capacity tracks it.

### D3 — the audit canonical string was not injective, so a tampered entry could verify

`AuditHmac.buildCanonicalString` joins keys and values with `=`, `,`, `{}`, `[]`, `|` and escapes none of them. The map-to-string mapping is therefore not one-to-one: `{"a": "x", "b": "y"}` and `{"a": "x,b=y"}` canonicalize to the same bytes and share one valid HMAC, and `{"calls": [{"tool": "calculator"}]}` collides with the literal string `"[{tool=calculator}]"`. `String.join(",", actions)` collapses `["a","b"]` onto `["a,b"]` the same way. For an append-only ledger that is a tamper-detection hole, and this branch made it reachable: `AuditEntry.toolCalls` now carries tool-trace `arguments`/`result` strings, which the LLM and the user write, and the new recursion walks into them with the same unescaped delimiters.

**Fix — versioned canonical form.** New entries are signed over a **v2** string that escapes every delimiter inside keys and scalars and type-tags every value (`s:` scalar, `m` map, `l` list, `n` null), and the stored value carries the tag: `v2:<64 hex chars>`.

The back-compat constraint is absolute — any change to the canonical bytes invalidates the HMAC of every already-stored entry. So `verifyHmac` picks the canonicalizer **from the stored value's prefix**: tagged → v2, untagged (a bare hex digest) → the v1 canonicalizer, which is now frozen and documented as such. It deliberately never *falls back* from v2 to v1; trying both would hand the collision straight back to the attacker. `AuditHmacTest.flatMapCanonicalStringUnchangedByRecursion`, which pins the v1 string to a literal, still passes untouched.

Note that `verifyHmac` currently has no production caller — the ledger signs on write and verification is an operator/forensic operation. The fix is about what a future verifier can conclude from a stored row.

### D4 — `maxBudgetPerConversation` stopped enforcing on upgrade

The previous commit made the ceiling conditional on a new `enforceBudget` defaulting to `false`, justified as "every built-in priced at $0.00, so no stored config was ever refused". **That holds only for built-ins.** For http, MCP, A2A and dynamic tools the dispatch name *is* the configured name, so an agent with a tool called `websearch`, `webscraper` or `pdfreader` was priced from `DEFAULT_TOOL_PRICES` and *was* being refused on `main`. Those operators' cost ceiling silently ceased to exist.

**Decision — `enforceBudget` stays opt-IN, and the silence is fixed instead.** `eddi.tools.budget.enforce-by-default` remains `false`; a new `warnAboutUnenforcedBudgets` names every configured task that carries a ceiling without the flag, once per task id.

Both defaults break someone, so the choice is which failure is acceptable:

* **Enforcing by default** would make the ceilings on built-in-only agents bind *for the first time* — this release is what repaired built-in pricing — and start aborting tool calls mid-conversation on upgrade, with no warning and no way to have anticipated it.
* **Not enforcing** costs the operator whose ceiling *was* live (http/MCP/A2A/dynamic tools dispatch under their configured name, so a tool called `websearch`/`webscraper`/`pdfreader` was priced and refused on `main`) — but that loss is *detectable and announced*, and re-enabling it is one field.

An unannounced new refusal is worse than an announced lapse: the first is a production incident an operator cannot predict, the second is a log line they act on. The unacceptable part was never the default — it was that a ceiling could record without refusing and say nothing, which is precisely the "config that silently does nothing" pattern this whole release set out to remove. The WARN is what makes the opt-in honest.

An earlier revision of this branch flipped the default to `true`; that was reverted in favour of the warning. `eddi.tools.budget.enforce-by-default=true` re-enables enforcement deployment-wide for operators who want the old behaviour back in one line.

### Files

* `engine/caching/CacheFactory.java` — `NONCE_PEAK_SIGNED_RPS`, `RATE_SIZED_EVICTION_HEADROOM`, `RATE_SIZED_CACHES`, `maximumSizeFor(name, ttl)`; both `getCache` overloads size through it
* `engine/caching/CacheImpl.java` — package-private `backingCache()` so the eviction policy actually built can be asserted
* `engine/audit/AuditHmac.java` — `V2_PREFIX`, `buildCanonicalStringV2`, `canonicalValueV2`, `escape`; `verifyHmac` dispatches on the stored prefix; v1 canonicalizer frozen
* `modules/llm/impl/AgentOrchestrator.java` — `BUDGET_ENFORCE_DEFAULT` stays `false`; new `warnAboutUnenforcedBudgets` + `UNENFORCED_BUDGET_WARNED` so an unenforced ceiling is named once per task instead of passing silently
* `modules/llm/model/LlmConfiguration.java` — `enforceBudget`/`maxBudgetPerConversation` javadoc
* Docs: `audit-ledger.md` (canonical-form versioning table), `langchain.md`, `security.md`, `agent-father-langchain-tools-guide.md`

### Tests

Every behavioural test was mutation-checked by reverting the production change and confirming failure:

| Reverted                                              | Observed failure                                                                                                                                                                                                                                                                                           |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `&& conversationId != null`                           | `costTracker.trackToolCall(or(isNull(), isA(ToolInvocation)), or(isNull(), isA(String))); Never wanted here … But invoked here … with arguments: [ToolInvocation[dispatchName=testTool, …], null]` (both classes)                                                                                          |
| `if (userInput != null)`                              | `ragContextProvider.retrieveContext(<any>, <any>, or(isNull(), isA(String))); Never wanted here … with arguments: [memory, …Task@…, null]`                                                                                                                                                                 |
| `maximumSizeFor` → fixed `CACHE_SIZES` at the builder | `CacheFactoryTest.nonceCacheCapacityCoversItsTtl`: `a capacity of 100000 cannot hold the 117000 nonces written during a 390s replay window ==> expected: <true> but was: <false>`; `nonceCacheCapacityTracksTheTtl`: `expected: <200000> but was: <100000>`                                                |
| the rate-sizing rule inside `maximumSizeFor`          | `NonceCacheServiceTest.cacheCapacityCoversTheRequestedTtl`: `a 390s replay window at 300 signed requests/s holds 117000 nonces, but the cache is capped at 100000`                                                                                                                                         |
| `canonicalValueV2` → the v1 renderer                  | 4 failures: `differentStructuresDoNotCollide`, `scalarMimickingNestedStructureDoesNotCollide`, `actionListSeparatorIsNotForgeable` (`expected: not equal but was: <v2:…>`), `tamperingIntoAColludingTwinIsRejected` (`expected: <false> but was: <true>`)                                                  |
| the v1 branch of `verifyHmac`                         | `legacyV1EntryStillVerifies`: `pre-v2 ledger rows must keep verifying against the v1 canonicalizer ==> expected: <true> but was: <false>`                                                                                                                                                                  |
| `BUDGET_ENFORCE_DEFAULT` → `false`                    | `AgentOrchestratorCoverageTest.toolCall_budgetSetWithoutEnforceFlag_isStillEnforced`: `calculatorTool.calculate(<any string>); Never wanted here … But invoked here`; `AgentOrchestratorToolCostTest.ceilingWithoutFlagIsEnforced`: `webSearchTool.searchWeb("eddi", 3); Wanted 2 times … But was 5 times` |

New/changed cases: `AuditHmacTest` +7 (a `v1CanonicalStringCollides` precondition test asserts the v1 form *does* collide, so a future edit to the frozen canonicalizer is caught rather than silently invalidating the ledger), `CacheFactoryTest` +3, `NonceCacheServiceTest` +1, `AgentOrchestratorCoverageTest` +1, `AgentOrchestratorToolCostTest` +1 (and `unenforcedBudgetRefusesNothing` → `explicitlyUnenforcedBudgetRefusesNothing`, now setting the flag it names).

`./mvnw clean compile` clean; 857 tests across the audit / caching / crypto / llm-tools / orchestrator / LlmTask classes green, plus 87 in `ai.labs.eddi.engine.audit`.

***

## 🧵 Model registry and streaming executor — races and un-stripped parameters (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Four defects around `ChatModelRegistry` and `StreamingLegacyChatExecutor`, three of them introduced by D11 (the commit that stopped stripping `timeout`/`logRequests`/`logResponses` so they could join the model cache key).

**C1 — a concurrent cache clear could hand `null` to `chat(...)`.** `getOrCreate`/`getOrCreateStreaming` did `containsKey(key)` and then `get(key)`. Both caches are cleared from other threads — `invalidateForSecret(null)` on DEK/KEK rotation, and the global-variable invalidation listener on any variable edit. A clear landing between the two calls made the method return `null`; no caller null-checks it (`LlmTask` and `CascadingModelExecutor` pass the result straight to `chat(...)`), so the turn died with an NPE instead of simply rebuilding the model. On the streaming path a `null` additionally *means* "this provider cannot stream", so the race silently downgraded a streaming task to sync.

**C2 — the streaming buffer was read without a happens-before edge.** `fullResponse` is appended only from the provider's callback thread and read only from the executor thread. On the normal path `latch.countDown()`/`await()` publishes those writes; on the **timeout/interrupt** path there is no such edge — `abandoned.set(true)` is a volatile write ordering executor→callback, not callback→executor. The executor's `toString()` was therefore an unsynchronized read of a concurrently mutated `StringBuilder` and could observe a torn buffer (count advanced ahead of the char data, or a stale `value` array after a grow). The `abandoned` gate was also check-then-act: a callback already past the check could still push a token into the shared event sink after the executor believed the attempt silenced — breaking the documented "memory text matches streamed text" guarantee that D11 introduced the flag to provide.

**C3 — un-stripping `timeout` made previously tolerated values fatal.** Every provider builder parses it unguarded: `if (!isNullOrEmpty(v)) builder.timeout(Duration.ofMillis(Long.parseLong(v)))` — and `isNullOrEmpty` does not trim. Before D11 the value's only consumer was `ObservableChatModel.wrapIfNeeded`, which is deliberately lenient (guards blank, swallows `NumberFormatException`, drops zero/negative). So a stored `" "`, `"30s"` or `"0"` (historically "unlimited") went from tolerated to throwing out of `build()` on **every turn** of that agent, with no migration and no warning.

**C4 — un-stripping the logging flags turned on untruncated body logging.** Reaching the provider builders, `logRequests`/`logResponses` install langchain4j's `LoggingHttpClient`, which writes the entire request and response body at INFO with no truncation. EDDI's own `ObservableChatModel`/`ObservableStreamingChatModel` already honour both flags and deliberately truncate to 200/500 chars. Enabling the flags therefore started writing full prompts, full conversation history and full model responses to the application log.

**What changed:**

| Component                                        | Change                                                                                                                                                                                              |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`ChatModelRegistry`**                          | Both lookups are a single `get` — a miss from a concurrent clear falls through to construction, which is correct for a pure memoization cache                                                       |
| **`ChatModelRegistry.normalizeTimeout`**         | `timeout` is normalised once at the boundary that feeds both the cache key and the builders: trimmed, and dropped when blank, non-numeric or non-positive, mirroring the tolerance it used to enjoy |
| **`ChatModelRegistry.warnAboutRejectedTimeout`** | WARN naming the model type and the offending value, emitted on the **build** path only so it does not repeat on every cache hit                                                                     |
| **`ChatModelRegistry.builderParams`**            | `logRequests`/`logResponses` are removed from the map handed to a builder — they stay in the cache key, so D11's correctness fix survives                                                           |
| **`StreamingLegacyChatExecutor`**                | A per-attempt lock makes `check + append + emit` and `abandon + toString()` mutually exclusive; the pre-retry `abandoned.set(true)` takes the same lock                                             |
| **`docs/langchain.md`**                          | Documents that logging is EDDI's truncating path (not the provider's) and that an unusable `timeout` is ignored rather than fatal                                                                   |

**Design decisions:**

* **C4 chose option (a): keep the flags in the cache key, strip them from the builder input.** EDDI's decorators already honour both flags on both the sync and streaming path, including for providers whose builder has no logging switch, so nothing is lost — while provider-level logging is unbounded PII exposure that no config field advertises. `logRequestsAndResponses` (Azure OpenAI and Gemini only) is left forwarded: it predates this branch, is provider-specific, and stripping it would be an unrelated behaviour change; it is now documented as the deliberate escape hatch.
* **Normalising `timeout` before the cache key, not after.** Keeping the normalised value in the key preserves D11's fix (two genuinely different timeouts remain two models) and additionally collapses `" 5000 "` and `"5000"` into one.
* **The C2 lock is held across `eventSink.onToken`.** That is a bounded call — the SSE sink builds an event and calls a non-blocking `send()` whose `CompletionStage` is never awaited — and holding it is what actually makes "no token after abandonment" true rather than merely likely.
* **The C1 regression test replaces the cache with a map that clears itself inside `containsKey`**, reproducing the exact interleaving deterministically rather than relying on a stress loop, and asserts `containsKey` is never called at all.
* **The provider `timeout` contract is mirrored verbatim in the test builder.** Real builders open a loopback socket and cannot be constructed in the unit-test sandbox, so the `isNullOrEmpty` + `Long.parseLong` expression is reproduced in the fake — without it the C3 tests would pass whether or not the registry normalises.

**Tests:** `ChatModelRegistryTest` gains three nested classes (C1 concurrent-invalidation, C3 timeout normalisation, C4 provider logging) and its `getOrCreate_observabilityParamsReachBuilder` now asserts the flags do *not* reach the builder; `StreamingLegacyChatExecutorTimeoutTest` gains `abandonment_isMutuallyExclusiveWithTokenEmission`. All four fixes were mutation-checked: reverting C1 fails both new tests with `expected: not <null>`, C3 with `NumberFormat For input string: " "`, C4 with `logRequests must NOT reach the provider builder`, C2 with `The executor must not set 'abandoned' and read the buffer while a token emission is in flight`.

***

## 🔐 Tool-cache scope resolution honours its own invariant (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Three defects in the per-tool cache-scope surface this branch introduced with D5/D2. Two of them could put a tool on a **cross-user** partition — the exact leak D5 existed to close.

**B1 — a typo in a per-tool scope widened the tool instead of narrowing it.** `ToolCacheScope`'s javadoc (and the D5 changelog entry) promise that "a typo must never silently widen a tool's audience". The code did the opposite: `fromConfig` returns `null` for an unrecognized token, and `resolve`'s `if (scope == null)` branch could not tell "this tool has no entry" from "this tool has an entry I could not parse", so it fell through to the **task-level default**. With `defaultToolCacheScope: "global"` plus `toolCacheScopes: {"getUserProfile": "usr"}` — a typo for `"user"` in an override written to *narrow* that one tool — the tool resolved to `GLOBAL` and got the shared `"g"` tag: one partition for every authenticated user. Alice's result served to Bob.

**B2 — `toolCacheScopes` was the only per-tool map that did not speak the slug.** D2 established the canonical slug as the configuration vocabulary: `toolRateLimits` and `toolPricing` both accept the dispatch name *or* the slug. `toolCacheScopes` was looked up by dispatch name alone, so `{"websearch": "user"}` — written in the same object as the `toolRateLimits: {"websearch": 30}` next to it, in the same vocabulary as `builtInToolsWhitelist` — was silently ignored and the tool stayed on whatever the task default was, possibly `global`. Every built-in was affected; none declares `@Tool(name = ...)`.

**B3 — the dedicated `news` TTL became unreachable.** `getSmartTTL` received only the canonical slug after D2, so `searchNews` canonicalised to `websearch` and exact-matched its 1800s entry before the substring loop could reach `Map.entry("news", 600L)`. No slug contains "news", so that entry was dead for every built-in and news results were cached 3× longer than the table declares. Inert until D5b made per-entry TTLs actually take effect.

**What changed:**

| Component                                                                                 | Change                                                                                                                                                                                                                                                                                      |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`ToolCacheScope`**                                                                      | `resolve` takes both names and gains `resolvePerTool`, which returns `null` **only** when the map has no entry for a name. A present-but-unparseable entry returns `DEFAULT` (`USER`) and logs a WARN naming the tool and the bad token. An unrecognized `defaultToolCacheScope` also WARNs |
| **`ToolCacheService`**                                                                    | `resolveScopeTag(dispatchName, canonicalName, …)`; `getSmartTTL` split into a `lookupTTL` that can say "no match" plus `resolveSmartTTL(dispatch, canonical)` which tries the dispatch name first                                                                                           |
| **`AgentOrchestrator`**                                                                   | Passes the already-computed `canonicalName` into `resolveScopeTag` — the three per-tool maps on a Task now agree on key vocabulary                                                                                                                                                          |
| **`docs/langchain.md`, `docs/security.md`, `docs/agent-father-langchain-tools-guide.md`** | Scope keys documented as slug-or-dispatch-name; fail-safe parsing and dispatch-first TTL documented                                                                                                                                                                                         |

**Design decisions:**

* **Fail safe, don't fail the load.** Leniency was never the bug — the *direction* of the fallback was. Unknown tokens still never abort an agent load; they now land on `USER`, and the WARN makes the typo discoverable instead of merely inert.
* **A bad dispatch-name entry does not fall through to the slug entry either.** `{"searchNews": "usr", "websearch": "global"}` resolves `searchNews` to `USER`. Any fall-through from a broken narrowing override risks landing somewhere wider, which is the whole defect.
* **Dispatch name before slug, everywhere.** Same precedence as `resolveRateLimit`: an entry naming one operation is more specific than one naming the whole tool. That precedence is also what makes the `news` TTL reachable again, without deleting a table entry that is genuinely correct for news.
* **The cache KEY stays on the dispatch name.** Unchanged, and load-bearing: `searchWeb`/`searchNews`/`searchWikipedia` share the `websearch` slug and must not share an entry.
* **`resolveScopeTag`'s old 5-arg form was replaced, not overloaded.** A lingering dispatch-name-only overload would let future code reintroduce B2 silently.

**Operator-visible behaviour changes:**

* A `toolCacheScopes` value that does not parse now yields `user` scope instead of inheriting `defaultToolCacheScope`. Deployments relying on a typo'd key to reach a `global` default lose that (unintended) cross-user reuse — a cache-hit-rate change, never a correctness one.
* Slug-keyed `toolCacheScopes` entries now bind. An entry like `{"websearch": "global"}` that was previously inert becomes live — audit `toolCacheScopes` for slug keys that were written as documentation rather than intent.
* `searchNews` results are now cached for 10 minutes instead of 30.

**Tests:** `ToolCacheServiceTest` (79) and `AgentOrchestratorCoverageTest` (57) green. New nested suites `UnparseablePerToolEntryTests` and `DualNameScopeKeyTests`, plus TTL cases in `CanonicalPutTests` and a ticker-driven `newsTtlIsLoadBearing` against a real expiring cache. All mutation-checked: reverting each production change fails 6, 4 and 2 tests respectively.

***

## 🧾 The audit ledger stops triple-billing the turn, and starts seeing the tool spend (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Three defects in the audit / cost path, all introduced or made live by this branch's own remediation work. They share one root cause: **dollar figures that are read, copied or baselined at the wrong moment**.

**A1 — every task after the LLM task re-reported the LLM's cost and tool calls.** `LifecycleManager.buildAuditEntry` runs once per lifecycle task, and `ConversationStep`'s data is never cleared between tasks. It read `audit:cost`, `audit:tool_calls`, `audit:token_usage` and the whole `llmDetail` block with **no task-type gate**, so a workflow `[parser, behavior, langchain, output, templating]` on a turn costing $0.0042 appended **three** ledger rows at $0.0042, and recorded that the `output` and `templating` tasks made tool calls they never touched. The ledger is append-only (EU AI Act), so an auditor summing cost reads a multiple of the truth with no way to correct it. The same file already gated the SSE tool trace on the task type \~60 lines earlier, with a comment explaining exactly this hazard — the audit reader never got the same gate. Before this branch cost was a literal `0.0` and `toolCalls` a literal `null`, which is why the lingering-key mechanism was previously harmless.

**A2 — the cascade dropped the entire tool spend.** `LlmTask`'s cascade branch builds a *fresh* metadata map and hand-copies `warning`, `finishReason` and `streamingTimeout` out of the winning step's metadata. It never copied `toolCostUsd` — yet `accumulateAuditEvidence` computes the ledger cost as `cascadeCostUsd + toolCostUsd`. Since `ModelCascadeConfig.enableInAgentMode` defaults to **true**, this is the default path for any agent-mode task with a cascade: the ledger (and `responseMetadataObjectName` template data) reported token cost only. The non-cascade branches assign the agent's whole map and were always correct, so the two branches of one feature disagreed.

**A3 — HITL-approved tool calls were excluded from their own turn's cost.** `AgentOrchestrator.resumeToolLoop` snapshotted its tool-cost baseline *after* the verdict-application loop, which has already run every human-approved gated call through `executeToolWrapped` → `costTracker.trackToolCall`. Those charges were inside the baseline, so `toolCostDelta` subtracted them right back out: the cost of precisely the calls a human signed off on never reached `toolCostUsd`. The live path (`executeWithTools`) always baselined before any tool ran.

**What changed:**

| Component                    | Change                                                                                                                                                                                                                                                                                                                    |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`LifecycleManager`**       | New shared predicate `isLlmTask(ILifecycleTask)` used by **both** `buildTaskSummary` and `buildAuditEntry`, so the two readers of the same lingering keys cannot drift apart again. `llmDetail`, `toolCalls` and `cost` are gated on it; so is the summary's `confidence` (an LLM-only signal that lingered the same way) |
| **`CascadingModelExecutor`** | `CascadeResult` gains `runToolCostUsd`; the step loop's `runCostUsd`/`runTokenUsage` locals become one `RunTotals` accumulator that also sums `stepToolCost(...)` per step                                                                                                                                                |
| **`LlmTask`**                | The cascade branch now puts `toolCostUsd` (the run total) into the response metadata, matching what the non-cascade branches always carried                                                                                                                                                                               |
| **`AgentOrchestrator`**      | `resumeToolLoop` takes the tool-cost baseline at method entry, before the verdict loop; the redundant `costConversationId` local is gone                                                                                                                                                                                  |

**Design decisions:**

* **Gate, don't clear.** Clearing `audit:*` between tasks would break accumulation across an LLM config's sub-tasks (`accumulateAuditEvidence` is deliberately read-modify-write). Gating the *reader* is the narrow fix.
* **`actions`, `input` and `output` stay ungated.** `actions` is genuinely per-task, and the input/output pair describes the step context every task ran in — that is pre-existing, intended behaviour. Only the LLM-written evidence is gated.
* **Run total, not the winning step's slice (A2).** An escalating cascade re-enters `executeAgentModeStep` for every step it tries, and each of those steps really does execute (and get charged for) its tools. Copying the accepted step's `toolCostUsd` would under-bill exactly as much as the old code dropped, and `returnBestAcrossSteps`/`finalizeBest` return a *different* step's metadata anyway. `AgentOrchestrator` reports its own per-call delta of the conversation total, so the per-step values partition the run's spend and are safe to sum — the same argument that already justifies `runCostUsd` and `runTokenUsage` being run totals.
* **`RunTotals` over a fifth positional `double`.** `withRun`/`finalizeBest` would otherwise have carried two adjacent, silently swappable `double` parameters — in a cost-accounting path, on a branch whose whole theme is cost figures drifting apart. It is a plain local accumulator; the executor stays stateless.

**Operator-visible behaviour changes:**

* Ledger rows for non-LLM tasks now carry `cost = 0.0`, `toolCalls = null` and `llmDetail = null` instead of a copy of the LLM task's figures. **Summing `cost` over a turn now yields the turn's actual cost**; historical rows written before this fix remain inflated and cannot be amended (append-only).
* SSE `task_complete` no longer reports `confidence` for tasks that follow the LLM task.
* Agent-mode cascade turns now report tool spend in `audit:cost` and in `responseMetadataObjectName` — figures go **up** relative to before, because they were previously missing, not double-counted.
* A HITL-resumed turn's `toolCostUsd` now includes the approved gated calls. This also means `maxBudgetPerConversation` accounting sees them.

**Tests:** `LifecycleManagerTest.BuildAuditEntryTaskTypeGateTests` (3, driving a **real** `ConversationMemory`/`ConversationStep` so the lingering is the production mechanism rather than a per-key stub), two cascade tool-cost tests in `LlmTaskAuditLedgerTest`, one resume-path cost test in `AgentOrchestratorResumeToolLoopTest`. Every one was mutation-checked: ungating the audit reader fails with `task 'output' spent nothing … expected: <0.0> but was: <0.0042>`; ungating the summary reader fails the pre-existing trace test too; dropping the cascade's `toolCostUsd` gives `expected: <0.0095> but was: <0.002>`; turning the run total into last-wins gives `expected: <0.007> but was: <0.004>`; restoring the old baseline position gives `expected: <0.002> but was: <0.0>`.

**Files:** `LifecycleManager`, `CascadingModelExecutor`, `LlmTask`, `AgentOrchestrator`, plus their three test classes and `docs/changelog.md`.

***

## 🧮 `convertToObject` finally reaches agent mode and streaming (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D7**. `jsonMode` was computed once in `LlmTask` from `convertToObject` and handed to exactly one collaborator — `LegacyChatExecutor`, i.e. the three no-tools, non-streaming call sites. `AgentOrchestrator.executeIfToolsEnabled`, `StreamingLegacyChatExecutor.execute` and the cascade's agent-mode / live-streaming steps all built their `ChatRequest` without ever looking at it.

The visible consequence: `AgentSetupService` sets `enableBuiltInTools(true)` in the same method that sets `convertToObject=true`, so a tool-enabled agent took the agent path and got **no** API-level JSON at all. With `addToOutput=false` the quickReplies/sentiment `postResponse` then degraded **silently** — the model returned prose, `startsWith("{")` failed, and the raw text was stored as a string.

Why only mistral and azure-openai were bitten: `AgentSetupService.supportsResponseFormat` approved `openai`, `mistral` and `azure-openai` and injected a builder-level `responseFormat=json` model parameter — but `OpenAILanguageModelBuilder` is the **only** builder that reads that key. For mistral and azure-openai the parameter was dead. openai got JSON in every mode purely because the parameter was baked into its cached `ChatModel`.

**What changed:**

| Component                                                      | Change                                                                                                                                                                                                                      |
| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`JsonResponseFormatPolicy`** (new, `modules/llm/capability`) | Decides per request whether `ResponseFormat.JSON` may be set. Carries `(requested, provider, override)` and resolves against whether *that* request carries tool specifications                                             |
| **`AgentOrchestrator`**                                        | `executeIfToolsEnabled` / `resumeToolLoop` / `runToolCallLoop` take the policy; the tool loop sets `responseFormat` on each `ChatRequest`, gated on `!activeSpecs.isEmpty()`. Previous arities kept as delegating overloads |
| **`StreamingLegacyChatExecutor`**                              | `execute` / `executeCapturing` take the policy; the streamed `ChatRequest` now carries the format                                                                                                                           |
| **`LegacyChatExecutor`**                                       | The `boolean jsonMode` parameter becomes the policy, so an unsupported provider is no longer sent a format it will reject                                                                                                   |
| **`CascadingModelExecutor`**                                   | Builds a policy **per step** from that step's resolved `modelType` — a cascade routinely escalates across providers                                                                                                         |
| **`LlmTask`**                                                  | Builds one policy from `convertToObject` + the resolved provider + the task override and passes it to all three modes, plus the HITL resume path                                                                            |
| **`LlmConfiguration.Task.jsonResponseFormat`**                 | New config field: `auto` (default) / `on` / `off`                                                                                                                                                                           |
| **`AgentSetupService`**                                        | Stopped injecting `responseFormat=json`; `supportsResponseFormat` deleted (its only remaining caller was the test-compat delegate `McpSetupTools.supportsResponseFormat`, also deleted)                                     |

**The provider matrix**, read off the langchain4j 1.18.0 bindings rather than assumed:

| Provider                                 | Schemaless request JSON | With tools | Evidence                                                                                      |
| ---------------------------------------- | ----------------------- | ---------- | --------------------------------------------------------------------------------------------- |
| openai                                   | yes                     | yes        | `OpenAiUtils#toOpenAiResponseFormat` → `json_object`                                          |
| azure-openai                             | yes                     | yes        | `InternalAzureOpenAiHelper#toAzureOpenAiResponseFormat` → `ChatCompletionsJsonResponseFormat` |
| mistral                                  | yes                     | yes        | `MistralAiMapper#toMistralAiResponseFormat` → `JSON_OBJECT`                                   |
| gemini, gemini-vertex                    | yes                     | **no**     | maps to `responseMimeType=application/json`, which the API rejects alongside `tools`          |
| anthropic, bedrock                       | no                      | no         | both throw `UnsupportedFeatureException` for JSON without a schema                            |
| ollama, jlama, huggingface, oracle-genai | no                      | no         | unverified against the provider API — opt in per task with `jsonResponseFormat: "on"`         |

**Design decisions:**

* **Request level, never builder level.** The changelog entry of 2026-04-02 records a Gemini `400 Function calling with a response mime type: 'application/json' is unsupported` caused by a builder-level `responseFormat` baked into a **cached** model that was later reused with tools. A cached model is shared across turns, tasks and execution modes, so anything mode-specific must live on the request. D11 has since made the model cache key finer-grained, which does not change this: identity is not the issue, reuse across *modes* is.
* **The matrix is tools-aware, not provider-aware only.** Gemini's answer differs depending on whether the same request also carries `toolSpecifications`, so the decision is taken inside the tool loop where that is known, against `!activeSpecs.isEmpty()` rather than against "is this agent mode".
* **`supportsResponseFormat` was retired, not narrowed.** After request-level threading the builder parameter is dead for mistral/azure-openai and redundant-plus-hazardous for openai, which leaves no provider for which the matrix entry does anything. `OpenAILanguageModelBuilder` still honours a hand-written `responseFormat=json`, so existing stored configs are unaffected.
* **`on` bypasses the tools guard.** An operator forcing the format onto an unlisted provider or an OpenAI-compatible gateway is making a deliberate choice; documented as such.

**Operator-visible behaviour changes:**

* Tool-enabled and streaming agents on **openai**, **azure-openai** and **mistral** now send `response_format: json_object` (or the Azure/Mistral equivalent) whenever `convertToObject=true`. Expect *more* reliable JSON, and for azure/mistral a real behaviour change on the wire.
* **anthropic** and **bedrock** no longer receive a schemaless JSON format on the no-tools path. Previously the request was sent, rejected, and silently retried through `LegacyChatExecutor`'s fallback — one wasted round trip per turn, now avoided. Enforcement there was and remains prompt-only.
* **gemini** keeps API-level JSON when no tools are present and is never sent it together with tools.
* Newly created agents (`setup_agent`, `create_api_agent`, `POST /administration/agents/setup*`) no longer carry a `responseFormat` model parameter. Existing stored configs keep theirs and keep working.
* New optional task field `jsonResponseFormat` (`auto` | `on` | `off`).

**Tests:** `JsonResponseFormatPolicyTest` (20) and `JsonResponseFormatThreadingTest` (12) are new; three wiring tests added to `LlmTaskCoverageTest`. Every behavioural assertion was mutation-checked (strip the agent-mode format; strip the streaming format; make the matrix tools-blind; approve every provider; drop openai from the matrix; make `LlmTask` pass `DISABLED`) — each mutation fails the tests that cover it.

**Files:** 8 modified in `src/main`, 1 new (`JsonResponseFormatPolicy`), 8 test files touched (incl. 52 Mockito matcher lists widened for the new parameter — an unmatched stub returns null instead of failing, so `verify(never(...))` sites had to move too), `docs/langchain.md`, `docs/changelog.md`.

***

## ⏱️ `timeout`, `logRequests` and `logResponses` become part of a model's identity (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D11**. `docs/langchain.md` documents `timeout`, `logRequests` and `logResponses` as configuration parameters. `ChatModelRegistry.filterParams` removed all three from the map it passed to the provider builders — so **every** provider's `builder.timeout(...)` / `builder.logRequests(...)` / `builder.logResponses(...)` branch, on both the sync and the streaming builder, was unreachable dead code.

The serious half is what that did to the cache. `filterParams` produces both the builder input **and** the `ModelCacheKey`, so stripping those three keys also erased them from the model's identity:

```java
// before — same cache key for two tasks with different timeouts
returnMap.remove(KEY_TIMEOUT);
returnMap.remove(KEY_LOG_REQUESTS);
returnMap.remove(KEY_LOG_RESPONSES);
```

A task configured with `timeout: "5000"` silently received an unwrapped, timeout-free model whenever a task with otherwise identical parameters happened to be constructed first — and vice versa. **Which model a task got depended on construction order.** On the sync path `ObservableChatModel.wrapIfNeeded` was applied only to the *first* build, so the wrap was attached to whichever task won the race and then served to every other task. `getOrCreateStreaming` never called `wrapIfNeeded` at all.

### What changed

* **`ChatModelRegistry.filterParams` no longer strips the three keys.** Cache key and builder input are the same map again, which is the invariant that was broken: anything that shapes the constructed model now necessarily shapes its identity. The keys still filtered (`systemMessage`, `prompt`, `logSizeLimit`, `addToOutput`, `convertToObject`, `includeFirstAgentMessage`) are ones no builder reads.
* **`getOrCreateStreaming` honours the three settings.** `timeout` reaches the streaming builder; `logRequests`/`logResponses` additionally wrap the model in the new `ObservableStreamingChatModel`.
* **New `ObservableStreamingChatModel`** — logging-only streaming counterpart to `ObservableChatModel`, so the flags behave identically across providers, including those whose streaming builder has no logging switch.
* **`StreamingLegacyChatExecutor` resolves one backstop from both timeout fields** (`resolveTimeoutSeconds`), and stops an abandoned attempt from writing to the shared event sink.
* **`docs/langchain.md`** gains a "Timeouts and Streaming" section and drops the implication that `timeout` alone bounds a streaming turn.

### Design decisions

* **No `Future.get` bound on the streaming path.** `ObservableChatModel` bounds a sync call by submitting it to an executor and calling `future.get(timeout)`. That shape is wrong for a stream: an overall wall-clock bound truncates a healthy long answer, and cancelling the awaiting thread does not stop the provider's callback thread. The streaming-appropriate bound already exists — every streaming builder passes `timeout` to its HTTP client, and for the JDK client (`JdkHttpClient` sets it as `HttpRequest.timeout()` on an async `ofInputStream` send) that bounds the time to the provider's **first response**, not the duration of the stream. So it detects a provider that never answers without cutting off one that answers slowly. `ObservableStreamingChatModel` is therefore observability-only.
* **The two timeout fields stay two fields, with a derived default.** `timeout` (ms, model parameter, provider-level) and `streamingTimeoutSeconds` (s, task-level, EDDI's overall backstop) bound genuinely different things; collapsing them would lose a capability. What was wrong was that they were *unrelated*: a task with `timeout: "300000"` was still cut off by the undocumented 120s backstop. Resolution order is now (1) an explicit positive `streamingTimeoutSeconds` wins, (2) otherwise 120s **raised, never lowered,** to cover a longer configured `timeout`, (3) otherwise 120s. Both stored shapes keep their existing behaviour; only the previously broken combination changes.
* **`timeout` is read from the task's raw parameters when deriving the backstop.** A Qute-templated value cannot be resolved at that point and leaves the 120s default in place — the pre-existing behaviour, and never a shorter bound. The alternative (threading the processed parameter map through `execute`/`executeCapturing`) buys correctness only for templated timeouts and costs two new overloads.
* **A decorator must forward to the overload it was called on.** `ObservableStreamingChatModel` overrides both `chat(ChatRequest, handler)` and `chat(ChatRequest, ChatRequestOptions, handler)` and forwards each to the same overload on the delegate. Funnelling both through the three-argument form hits `StreamingChatModel`'s default `doChat` — `throw new RuntimeException("Not implemented")` — for any model that implements the two-argument `chat` directly. Caught by `ObservableStreamingChatModelTest`, which is why it exists.
* **R5 (`ChatModelListener` observability SPI) deliberately not implemented.** D11 is the bug; R5 is a feature and stays a separate backlog item.

### Abandoned streams no longer write to the shared event sink

`StreamingLegacyChatExecutor` retries a timed-out attempt that produced no tokens, but nothing stopped the abandoned attempt's handler: its provider callback thread stayed alive and kept calling `eventSink.onToken(...)` while the retry streamed into the *same* sink. A slow first attempt whose first token landed after the retry began therefore interleaved two answers on the SSE stream, and neither matched the text stored in memory. Each attempt now carries an `abandoned` flag its handler checks before forwarding — set on timeout, on interrupt, and before an error retry. The flag is set *before* the partial text is read, so the text returned to memory is exactly the text the client was sent. The executor still cannot cancel the provider's stream (langchain4j exposes no cancellation on this path); it can and now does make the abandoned stream silent.

### The cascade's live-stream backstop stopped assuming 120s

`CascadingModelExecutor` bounded a live-streamed step's future with a hardcoded `STREAMING_STEP_TIMEOUT_MS = 125_000L`, whose comment states the invariant it exists to hold: it *must* exceed the streaming executor's own bound, because cancelling the awaiting thread does not stop the provider's callback thread — the SSE client would keep receiving tokens for a step the cascade has already moved past. That invariant was already violable (`streamingTimeoutSeconds: 300` broke it), and deriving the bound from `timeout` adds another way in. It is now `resolveStreamingStepTimeoutMs(task)` = the executor's resolved bound + a 5s margin, so the default case is still exactly 125s and a longer configured bound raises the backstop with it.

### Operator-visible behaviour changes

| Config                                                                | Before                                                                                                                                           | After                                                                                                  |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| `timeout` on any task                                                 | Never reached the provider builder; on the sync path only the `ObservableChatModel` wrapper applied, and only if that task built the model first | Reaches the provider builder on both paths, and is part of the model cache key                         |
| `logRequests`/`logResponses` on a streaming task                      | Silently discarded                                                                                                                               | Honoured — emitted at INFO by `ObservableStreamingChatModel`, and passed to builders that support them |
| Two tasks differing only in `timeout`/`logRequests`/`logResponses`    | Shared one cached model; whichever built first won                                                                                               | Two separate cached model instances                                                                    |
| Streaming task with `timeout` > 120s and no `streamingTimeoutSeconds` | Cut off at the undocumented 120s backstop                                                                                                        | Backstop follows the configured `timeout`                                                              |
| Streaming task with only `streamingTimeoutSeconds`                    | —                                                                                                                                                | Unchanged                                                                                              |
| Retry after an empty timed-out streaming attempt                      | Late tokens from the abandoned attempt could interleave with the retry's output                                                                  | Abandoned attempt is silent                                                                            |
| Live-streamed cascade step with a bound above 125s                    | The cascade cancelled the future while the provider was still emitting                                                                           | Cascade backstop tracks the executor bound (+5s)                                                       |

Cached model instances are keyed more finely now, so a deployment whose tasks differ in these settings holds a few more model objects than before. That is the point — they were sharing an instance only one of them had configured.

### Tests

`ChatModelRegistryTest` grew an `ObservabilityCacheKeyTests` nest (different `timeout`/`logRequests`/`logResponses` ⇒ different instances, sync and streaming; construction order no longer decides which model a task gets; the settings reach the builder). Its old `getOrCreate_observabilityParamsDontAffectCacheKey` test asserted the defect and was replaced. New `StreamingLegacyChatExecutorTimeoutTest` pins the backstop resolution table including both back-compat shapes, that a short `timeout` cannot truncate a healthy stream end to end, and that a late token from an abandoned attempt never reaches the sink. New `ObservableStreamingChatModelTest` pins decorator transparency. Every behavioural test was mutation-checked by reverting the corresponding production change: 9 of the 29 `ChatModelRegistryTest` cases fail with `filterParams` restored (`expected: <5000> but was: <null>`, `expected: not same but was: <…>`), the backstop cases fail with the derivation neutralised (`expected: <300> but was: <120>`), and the abandoned-stream case fails without the flag (`NeverWantedButInvoked: conversationEventSink.onToken("LATE")`). The cascade-backstop cases fail with the hardcoded 125s restored (`Cascade backstop (125000ms) must exceed the executor bound (300000ms)`).

***

## 🧾 The audit ledger stops recording zeros and nulls (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D1**. `AuditEntry` documents `llmDetail` as carrying token usage, `toolCalls` as tool execution data and `cost` as the monetary cost of the step. Every entry the engine has ever produced carried `toolCalls = null` and `cost = 0.0` — passed as **literals** at the `new AuditEntry(...)` call site, with comments claiming an integration that did not exist:

```java
llmDetail, null,   // toolCalls — set by LlmTask in memory
actions,  0.0,     // cost — set by ToolCostTracker integration
```

Neither was true. `LlmTask` never wrote `audit:tool_calls` or `audit:cost`, and `audit:token_usage` — the key `buildAuditEntry` reads into `llmDetail.tokenUsage` — had **zero writers** in `src/main/java`. This matters beyond tidiness: the ledger is the EU AI Act Art. 17/19 traceability record, and it has been attesting that every LLM decision cost nothing and used no tools.

Same class of defect one line up: `buildTaskSummary` reads `audit:confidence` as `IData<Double>`, while `LlmTask` wrote `audit:cascade_confidence` as `String.valueOf(...)` — **wrong key and wrong type**. All four `audit:cascade_*` keys were write-only.

### What was wired

| Key                   | Written by                                | Read into                                                |
| --------------------- | ----------------------------------------- | -------------------------------------------------------- |
| `audit:token_usage`   | `LlmTask`, accumulated per turn           | `llmDetail.tokenUsage`                                   |
| `audit:tool_calls`    | `LlmTask`, accumulated per turn           | `AuditEntry.toolCalls`                                   |
| `audit:cost`          | `LlmTask`, accumulated per turn           | `AuditEntry.cost`                                        |
| `audit:confidence`    | `LlmTask` cascade branch, as a **Double** | `llmDetail.confidence` + the `task_complete` SSE summary |
| `audit:cascade_model` | `LlmTask` cascade branch                  | `llmDetail.cascadeModel`                                 |

All six audit keys moved into `MemoryKeys` with javadoc naming their producer and consumer, so the next reader/writer split is a compile-time concern rather than a silent one. `audit:cascade_confidence`, `audit:cascade_cost` and `audit:cascade_token_usage` are gone — they had no readers anywhere and never left the in-flight `ConversationStep`, so there is nothing to migrate.

### Design decisions

* **Which cost signal.** The two dollar figures that actually exist are the cascade's `runCostUsd` (from `inputPricePer1M`/`outputPricePer1M` on `ModelCascadeConfig`/`CascadeStep`) and `ToolCostTracker`'s per-conversation tool cost, which became real in D2. `cost` is the **sum of both**. No token price table was invented for non-cascade tasks — those report tool cost only, and a non-cascade turn with free tools still audits at `0.0`. Hoisting cascade pricing to task level is a config change and stays a follow-up.
* **Accumulate, never overwrite.** A turn can drive many LLM calls: one per matching config sub-task, plus every escalated cascade step and every tool-loop iteration. `getLatestData` is last-write-wins, so each contributor read-modify-writes. Only counts a provider actually reported are touched — a provider that omits `totalTokens` must not zero what earlier calls contributed.
* **`toolCalls` shape is `{"calls": [...]}`**, each entry the tool-trace record plus the `llmTaskId` that issued it; without that tag a merged list from several sub-tasks is unattributable.
* **Nested maps, and the HMAC time bomb they would have armed.** `llmDetail.tokenUsage` is a nested map and `toolCalls.calls` a nested list, but `AuditHmac.sortedMapString` flattened values with `toString()` while `AuditStore.fromDocument` copies only the top level — so an entry signed in memory with a `LinkedHashMap` would read back with an `org.bson.Document` (whose `toString()` is prefixed `Document{`) and fail to verify against its own HMAC. Latent only because `AuditHmac.verifyHmac` has no production callers yet. The canonicalizer now recurses through maps and lists; scalars still use `toString()`, so flat maps produce a byte-identical canonical string and **historical entries keep verifying** — pinned by a literal-string test plus a real BSON encode/decode round-trip test.
* **Failure-path entries deliberately unchanged.** The `AuditEntry` built when a task throws keeps `cost = 0.0` / `toolCalls = null`: the accumulators are partial at that point and the task may never have reached the model. `GdprComplianceService` and `CapabilityMatchCondition` also keep their zeros — those are administrative/rule entries where zero is correct, not defective.

### Three upstream signal defects fixed on the way

The ledger is only as honest as what feeds it:

* **`LegacyChatExecutor` built the token map with `Map.of`** over three boxed `Integer`s. Providers legitimately report only some of the three (Bedrock and Ollama commonly omit the total), so a partial report was an NPE that killed the whole turn over telemetry. Now shares `AgentOrchestrator.tokenUsageMap`, which 0-defaults.
* **The agent tool loop double-counted tokens on retry.** The accumulator was declared outside the retry lambda and `RetryConfiguration.executeWithRetry` replays that lambda, so a retried turn counted the abandoned attempt too. Reset on lambda entry.
* **The cascade under-reported tokens.** `runCostUsd` was a run total across every attempted step while `tokenUsage` reported only the accepted step, so an escalating cascade produced token counts that contradicted its own dollar figure. `CascadeResult.tokenUsage` is now the run total; per-step usage stays in the trace.

### Operator-visible behaviour changes

* LLM-task audit entries gain `llmDetail.tokenUsage`, and `llmDetail.cascadeModel` / `llmDetail.confidence` on cascade turns. Entries are larger.
* `AuditEntry.toolCalls` is non-null on any turn where a tool ran; `AuditEntry.cost` is non-zero wherever a cascade with configured prices or a priced tool ran. Cost dashboards that assumed a constant zero will start moving.
* **HITL-resumed turns gain a full `llmDetail` block.** `executeResume` never wrote `audit:compiled_prompt`, and `LifecycleManager` gates the entire block on that key — so every turn a human intervened in audited with no LLM evidence at all. Fixed.
* `ExecutionResult.responseMetadata` now carries `toolCostUsd` (the delta this model call added, not the conversation running total). Agents that surface `responseMetadataObjectName` in templates will see the extra key.
* `task_complete` SSE frames now really carry `confidence` on cascade turns.

### Known gap — kept documented, not silently swallowed

A turn that **pauses for tool approval** still loses its pre-pause token usage: `ToolApprovalRequiredException` escapes `executeWithTools` before the metadata is assembled, and `resumeToolLoop` starts a fresh accumulator. Closing it needs the usage to survive the pause, i.e. a new field on the persisted `PendingToolCallBatch` — a snapshot-format change deliberately out of scope here. The under-report is bounded to the pre-pause segment of paused turns and is called out at both code sites.

### Tests

New `LlmTaskAuditLedgerTest` backs the conversation step with a real map rather than a mock that always answers `null` — every assertion here is about accumulation, and a null-answering step makes a broken accumulator look correct. It covers agent-mode token usage, multi-sub-task summation, partial provider reports, tool-call merging with `llmTaskId`, cascade cost from configured pricing, the Double confidence key, the resumed-turn `llmDetail` block, and the audit-collector gate. `AuditHmacTest` gains the flat-map back-compat literal, nested-map/list determinism, `Document`-vs-`LinkedHashMap` equivalence and the BSON round-trip. `LifecycleManagerTest` gains toolCalls/cost/cascadeModel/confidence assertions.

Four pre-existing tests were repaired rather than extended — they passed for the wrong reason: `LifecycleManagerTest.auditEntryWithLlmDetails` asserted only `containsKey("tokenUsage")` on a stub for a key nothing wrote; `LlmTaskCoverage2Test` pinned the dead `audit:cascade_confidence` and `audit:cascade_token_usage` keys with `any()`; `LlmTaskDeepBranchTest.auditCollectorStoresData` verified `atLeast(3).createData(anyString(), any())`, which passes for any three keys at all.

***

## 🔌 The live tool trace finally reaches the SSE stream — and takes an unredacted payload with it (2026-07-23)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D3**. The `task_complete` SSE frame has advertised a `toolTrace` field since the streaming API shipped, and has never emitted one — not once, on any deployment. The writer and the reader never agreed on a key:

|        | Key                                                                                | Where                                   |
| ------ | ---------------------------------------------------------------------------------- | --------------------------------------- |
| Writer | `langchain:trace:<modelType>:<configTaskId>` (e.g. `langchain:trace:openai:taskA`) | `LlmTask.executeTask` / `executeResume` |
| Reader | `langchain:trace:` + `task.getId().name()` = `langchain:trace:ai.labs.llm`         | `LifecycleManager.buildTaskSummary`     |

`getLatestData` is a `startsWith` scan, so those two prefixes can never overlap — `summary` never got a `toolTrace` entry and the `if (summary.containsKey("toolTrace"))` branch in `RestAgentEngineStreaming.onTaskComplete` was dead code. The "live tool call display" in the UI has never received a byte.

### The fix — reader side only

`buildTaskSummary` now aggregates over `getAllElements()` instead of computing a key:

* **The writer key is untouched.** `RestToolHistory` and every `ConversationMemorySnapshot` already persisted in MongoDB scan the same `langchain:trace:` prefix with an arbitrary suffix. Changing the writer would have silently broken historical tool-history replay. The literal moved into `MemoryKeys.LANGCHAIN_TRACE_PREFIX` and all three call sites (writer ×2, `RestToolHistory`) now share it — byte-identical output, de-duplication only.
* **Aggregate, don't take the latest.** `LlmTask` writes one trace key *per LLM config task*, and `getLatestData` reverses the element list and returns only the newest match. A naive `getLatestData(LANGCHAIN_TRACE_PREFIX)` would have shipped a subtly wrong trace — the last task's calls only — which is worse than shipping nothing. `getAllElements()` is an insertion-ordered defensive copy, so write order is preserved for free.
* **The task-type gate is load-bearing.** Step data survives across tasks within a `ConversationStep`, so an ungated prefix scan would make every task executed *after* the LLM task report the LLM's trace as its own. Reads are gated on `TASK_TYPE_LANGCHAIN`.
* **Siblings are deliberately not swept in.** `langchain:cascade:trace:`, `rag:trace:` and `rag:httpcall:trace:` do not match the prefix and stay out of the frame.

### ⚠️ Security — tool arguments and results now leave the process unredacted

**Operators and downstream integrators must read this.** Until now a tool call's arguments and its result were reachable only through the owner-scoped `RestToolHistory` endpoint and the audit ledger, which runs `AuditLedgerService.scrubSecrets`. Making the trace reach the stream puts that same payload on the SSE channel, and **nothing on the `buildTaskSummary` → `onTaskComplete` path redacts anything.** Whatever a tool was called with — an API key passed as a tool argument, a token echoed back in a tool result — now streams verbatim to every client subscribed to that conversation's `sayStreaming`.

The correct place to fix that is the producer (the `tool_call` / `tool_result` maps built in `AgentOrchestrator`), so that `RestToolHistory`, the audit ledger and the stream all inherit one redaction rule. That is tracked separately (D12) and is **explicitly out of scope here** — this entry exists so the exposure is not discovered in production. Deployments that stream to untrusted clients and pass secrets through tool arguments should weigh that before taking this build.

### Other operator-visible behaviour changes

1. **The `task_complete` SSE frame gains a `toolTrace` array** on LLM tasks that executed at least one tool (including cascade turns). Purely additive — no existing field changes shape — but **a strict unknown-field-rejecting SSE parser in eddi-chat-ui or EDDI-Manager would now break on it.** \[UNVERIFIED — requires a check of both frontends' `task_complete` parsers.]
2. **The frame can get large.** One `tool_call` + one `tool_result` entry per tool call per iteration, up to `maxToolIterations`; `tool_result` carries the (truncated) tool output. There is no cap on the streamed trace — adding one would be a config field (Golden Rule 1) and is out of scope.
3. **Non-streaming `say` is unchanged.** `eventSink` is null there, and the only other consumer of `summary`, `buildAuditEntry`, reads `"actions"` and nothing else — audit output is byte-identical.

### Tests

`LifecycleManagerTest.summaryWithToolTrace` was **fully vacuous**: it hand-stubbed `getLatestData("langchain:trace:llm")` for a mock task whose id was `TaskId("llm")` — a name no real task has — so it asserted the reader against its own stub and stayed green through the entire life of the defect. Replaced with six cases that stub `getAllElements()` and never `getLatestData`: trace reaches the summary for a langchain task; multiple trace keys aggregate in write order; a non-langchain task omits it; `langchain:cascade:trace:` is ignored; a non-`List` result is ignored without a `ClassCastException`; no trace keys means no field. Each was mutation-checked against a reverted fix.

Two weak neighbours tightened: `LlmTaskCoverageTest.resume_nonEmptyTrace_stored` from a `startsWith` matcher to the exact key `langchain:trace:openai:taskA` (the writer half of the contract the reader tests now assert), and `RestAgentEngineStreamingExtendedTest.onTaskCompleteIncludesToolTrace` from `assertTrue(data.contains("toolTrace"))` — which also passes on a stringified or malformed payload — to a parsed-JSON shape assertion.

***

## 🧱 The in-turn tool context finally has a ceiling — `maxToolContextTokens` (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D6b** (the bug half of D6; the cross-turn history-lossiness half is an improvement and stays out of scope). `AgentOrchestrator.runToolCallLoop` grew ONE `currentMessages` list by an `AiMessage` plus one `ToolExecutionResultMessage` per tool call per iteration, and nothing between the loop head and `chatModel.chat` ever inspected its size, character count or token count. The only in-loop ceilings were the iteration counter (default 10) and the dollar budgets. Per-result truncation did not save it: `ToolResponseTruncator.truncateIfNeeded` returns the result unchanged when `limits == null`, and `Task.toolResponseLimits` has no default — so on an ordinary agent every tool result entered the context in full. A tool-heavy turn could therefore blow past the model context window mid-loop and hard-fail with a provider `400`, after the tool side effects had already fired.

### The fix

New per-task config field **`maxToolContextTokens`** (default **60000**) on `LlmConfiguration.Task`. Before each model call the loop meters the accumulated *tool* traffic and, while it exceeds the ceiling, evicts the **oldest complete tool exchange** — a requesting `AiMessage` **together with all of its `ToolExecutionResultMessage`s** — until the traffic fits or only the most recent exchange remains.

* **The pairing invariant is the whole reason eviction is subtle.** Dropping a result without its requesting `AiMessage` leaves a dangling `tool_call_id`; dropping the `AiMessage` without its results leaves an unanswered tool call. Either half is itself a provider `400` — the eviction would *cause* the failure it exists to prevent. Exchanges are therefore located as `[AiMessage, results…]` index ranges and removed whole. Pinned by its own test.
* **The most recent exchange is never evicted.** When it alone exceeds the ceiling the request goes through unchanged (the model asked for those results and must see them) and the overrun is logged `still_over_budget`. That case is what `toolResponseLimits` is for.
* **Only tool traffic is counted.** System / user / assistant-prose messages are never candidates — conversation history is governed by `maxContextTokens` / `conversationHistoryLimit`, and nothing in this guard may drop a history message.

### Token accounting extended, not duplicated

`TokenCounterFactory.extractText` returned `""` for both tool message shapes — an `AiMessage` announcing tool calls has a `null` `text()`, and `ToolExecutionResultMessage` hit the `default` arm — so the entire in-turn tool context weighed **zero tokens** to every caller. Extended so an `AiMessage` contributes its prose plus each requested tool name and argument JSON, and a `ToolExecutionResultMessage` contributes its tool name plus the whole payload. The SAME `TokenCounterFactory` `LlmTask` uses for history windowing is injected into `AgentOrchestrator` and reused — one accounting rule for both halves of the request, not a second estimator. `LlmTask.resolveModelName` was widened from `private` to package-private so the orchestrator picks the same estimator (tiktoken for OpenAI/Azure, chars÷4 elsewhere) rather than a drifting copy.

### Design decisions

* **Config field, not a constant** (Golden Rule 1). The ceiling is an agent-designer knob with a sensible default; `-1`/`0` disables it and restores pre-6.1 unbounded behaviour.
* **Default 60000 preserves today's behaviour.** High enough that no ordinary tool-using turn is touched — the eviction path is byte-for-byte inert below the ceiling, guarded by a test that compares the default-budget message lists against the guard-disabled (`-1`) lists — and low enough to keep a runaway loop inside a 128k window after the system prompt, history and completion are added.
* **Eviction over refusal.** Refusing the turn would strand the side effects already committed by earlier tool calls; evicting the oldest results lets the loop finish with the freshest evidence. The loss is made observable rather than prevented.
* **No gap-marker message injected.** A mid-transcript `SystemMessage` is not portable across the twelve providers (several hoist system content to a top-level field) and a `UserMessage` would fabricate a turn; the loss is surfaced through trace + metric + WARN instead.
* **Per-attempt `IdentityHashMap` token memo** on the call stack — the orchestrator is an `@ApplicationScoped` singleton and stays stateless; without the memo a 10-iteration loop retokenizes the first result ten times.
* **Estimator resolution fails safe.** An unresolved global-variable model type or an unknown model name that makes a provider tokenizer refuse to construct falls back to the approximate estimator rather than aborting the turn — a safety ceiling that throws is worse than an approximate one.

### Operator-visible behaviour changes

1. **A tool-heavy turn that used to hard-fail on a provider context-window `400` now completes**, dropping its oldest tool exchanges once in-turn tool traffic passes `maxToolContextTokens`.
2. **New trace entry `tool_context_evicted`** (token counts before/after, exchanges + messages dropped, `withinBudget`), **new counter `eddi.llm.tool_context.evictions`** (tag `outcome=within_budget|still_over_budget`), and a **`WARN`** **`llm.tool_context.evicted`** with conversation id + remediation hint. A steady stream signals: lower `maxToolIterations`, set `toolResponseLimits`, or raise `maxToolContextTokens`.
3. **Default budget is inert for normal turns.** Agents whose in-turn tool traffic stays under 60000 tokens are byte-for-byte unchanged; `-1`/`0` disables the guard entirely.

New field only; nothing removed. `FAIL_ON_UNKNOWN_PROPERTIES=false` keeps rolling deploys and ZIP imports safe both directions. No `ExtensionDescriptor` change (it exposes only `uri`), so no Manager UI work.

### Tests

`AgentOrchestratorToolContextBudgetTest` (new): the end-to-end regression drives the real loop with a mock model that requests a tool every iteration and asserts every request reaching `chatModel.chat` is within the ceiling (fails today — unbounded growth); the pairing invariant across every captured request; byte-identical message lists under the default budget vs. the guard disabled; and trace/metric observability. Plus isolation tests of `enforceToolContextBudget` for oldest-first order, whole-exchange (multi-call) eviction, history never evicted, and the unfittable-newest report. `TokenCounterFactoryTest` gains a `tool messages` nest asserting names/arguments/payloads are now counted.

Every behavioural test was mutation-checked by reverting the corresponding production change and confirming failure: disabling the guard call (`request 3 carried 636 tokens … <=500 expected: <true> but was: <false>`); shrinking exchange ranges to the `AiMessage` alone and, separately, clearing only the `AiMessage` index on removal (`orphan ToolExecutionResultMessage id=c1 … its requesting AiMessage was evicted without it`); and reverting `extractText` to the pre-fix `""` for tool messages (`the payload is what fills the context window … expected: <true> but was: <false>`, and the guard measuring 0 tokens so it never evicts).

The nine existing `AgentOrchestrator*Test` constructors and three `historyBuilder`-style call sites gained the new `TokenCounterFactory` argument.

**Files:** `AgentOrchestrator.java`, `TokenCounterFactory.java`, `LlmConfiguration.java`, `LlmTask.java`, `docs/langchain.md`, plus `AgentOrchestratorToolContextBudgetTest` (new), `TokenCounterFactoryTest`, and the nine `AgentOrchestrator*Test` constructor call sites.

**Out of scope:** the cross-turn half of D6 (tool messages absent from `ConversationHistoryBuilder`) — an improvement, not a bug; summarizing evicted tool results instead of dropping them; a provider-reported hard context limit feeding the default.

***

## 💸 `maxBudgetPerConversation` can finally bind — canonical tool names at the executor boundary (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D2**. A built-in tool has two names, and the engine only ever carried one of them past the dispatch loop.

`ToolCostTracker`'s price table and `ToolCacheService`'s smart-TTL table are keyed on the eight **whitelist slugs** (`websearch`, `pdfreader`, `calculator`, …). The only production call path passed `toolRequest.name()` — the bare `@Tool` **method** name (`searchWeb`, `extractTextFromPdf`, `calculate`), because no priced tool class declares `@Tool(name = …)`. Zero keys overlapped. Consequences, all live until now:

* **Every built-in priced at $0.00**, so `maxBudgetPerConversation` could never trip no matter how much tool work a conversation did, and `eddi.tool.costs` never carried a non-zero value.
* **`toolRateLimits` in its documented slug form was inert.** `docs/langchain.md`, `docs/security.md` and `docs/agent-father-langchain-tools-guide.md` all show `{"websearch": 30}`; the lookup only ever saw `searchWeb`, so the entry was discarded and `defaultRateLimit` (100) applied.
* **Cache TTLs mostly fell through to the flat 300s default.** `getSmartTTL` substring-matches, which is why the mismatch went unnoticed for so long: `getCurrentDateTime` happens to contain "datetime" and resolved correctly, while `calculate` does not contain "calculator" and did not.

### The fix

New `ToolNameResolver` — stateless, all-static, an **exhaustive and exact** switch from the 16 built-in tool class simple names to their whitelist slugs. No substring fallback: a lookup that is right by coincidence for some inputs is worse than one that is wrong for all of them, because it never looks broken. `AgentOrchestrator.buildToolSetup` populates a `toolCanonicalNames` map (dispatch name → slug) from the **already proxy-unwrapped** `toolClass`; using `tool.getClass()` would yield `…_ClientProxy` and reintroduce the same defect in a new coat. The map rides on `ToolSetup`, so the live loop and the HITL resume path share it.

A new `ToolInvocation(dispatchName, canonicalName, priceOverride)` record carries both names through `ToolExecutionService`. The split is load-bearing:

| Resolved from the **canonical slug** | Resolved from the **dispatch name**                                                                   |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| per-call price, cache TTL            | cache key, rate-limit bucket, metric tags, per-tool and per-conversation cost breakdown, failure logs |

Canonicalising the cache key would collapse `searchWeb`, `searchNews` and `searchWikipedia` onto one entry and serve each other's results for identical arguments — a correctness bug traded for a naming tidy-up. The legacy `String`-first overloads of `executeToolWrapped`, `trackToolCall` and `put` are retained and delegate via `ToolInvocation.of(name)`.

### Design decisions

* **`enforceBudget` is opt-in, default `false`** (deployment fallback `eddi.tools.budget.enforce-by-default`). Enforcement is deliberately *not* inferred from `maxBudgetPerConversation` being set. That ceiling has been inert for its entire shipped life, so no stored config has ever had a tool call refused by it; switching enforcement on together with the prices would newly abort tool calls on live agents. Cost tracking runs regardless of the flag. Read through `ConfigProvider` in a `static final`, not `@ConfigProperty` — `AgentOrchestrator` is constructed with `new` by `LlmTask` and is not a CDI bean, so an injection annotation would never fire while looking configurable.
* **Metric tags keep the dispatch name.** `eddi.tool.calls{tool=…}` and `eddi.tool.costs{tool=…}` could have moved to slugs to aggregate the three web searches into one series. They did not: every other `tool`-tagged meter in the module (`eddi.tool.execution.success`, `eddi.tool.cache.hits.by_tool`, `eddi.tool.execution.duration`) reports the dispatched method name, so moving these two would split the tag vocabulary in half, make the series un-joinable and break existing dashboards for a marginal analytical gain. Slug-level totals stay available as a PromQL sum.
* **`toolRateLimits` and `toolPricing` accept either name, dispatch first.** A dispatch-name entry is the more specific statement (`{"searchNews": 5}` pins one operation) and wins over the tool-wide slug entry.
* **Rate-limit&#x20;*****buckets*****&#x20;stay per dispatch name; only the&#x20;*****limit value*****&#x20;is slug-resolved.** `{"websearch": 30}` therefore yields three independent 30/min buckets rather than one shared allowance. Documented explicitly rather than left for an operator to discover.
* **Operator prices are clamped at `Math.max(0.0, …)`.** `toolPricing` values come from agent JSON; a negative one would credit the conversation and make any ceiling unreachable by construction.

### Operator-visible behaviour changes

1. **`toolRateLimits` slug keys start binding.** A config carrying `{"websearch": 30}` was previously ignored; from this release it applies — 30/min to each of `searchWeb`, `searchNews`, `searchWikipedia`. Method-name keys are unchanged. Review any agent that has been running under `defaultRateLimit` while believing it was rate-limited.
2. **Built-in cache TTLs change from a mostly-flat 300s to the intended per-tool values.** `datetime` operations **tighten**: `convertTimezone`, `addTime`, `listTimezones` and `calculateDateDifference` go 300s → 60s. Others loosen to their configured values: `calculator` 300s → 7d, `searchWeb`/`searchWikipedia` 300s → 1800s, `webscraper` 300s → 1h, `pdfreader`/`dataformatter`/`textsummarizer` 300s → 24h. One value loosens unintuitively: `searchNews` used to substring-match the `news` entry at 600s and now takes `websearch`'s 1800s. `weather`, `getCurrentDateTime` and `formatDateTime` are unchanged.
3. **`eddi.tool.costs` becomes non-zero for priced built-ins** and `GET /llm/toolhistory/costs` starts reporting real numbers: `websearch` $0.001, `webscraper` $0.002, `pdfreader` $0.001, `weather` $0.0005 per call. Tag *values* are unchanged (see above), so dashboards keep working — a series that previously only ever recorded zero now carries a value.
4. **`maxBudgetPerConversation` covers TOOL cost only** and stays inert unless `enforceBudget: true` is added. LLM token spend remains run-scoped under the cascade's `maxCostPerRun`; the two are not summed.

New config fields `enforceBudget` (Boolean) and `toolPricing` (`Map<String, Double>`) on `LlmConfiguration.Task`. No field removed; `FAIL_ON_UNKNOWN_PROPERTIES=false` keeps rolling deploys and ZIP imports safe in both directions. No `ExtensionDescriptor` change (it exposes only `uri`), so no Manager UI work.

### Tests

`ToolNameResolverTest` (new) asserts all 16 classes against their real `getSimpleName()` — renaming a tool class without updating the resolver now fails here instead of silently reverting that tool to $0.00 and a 300s TTL — plus `CalculatorTool_ClientProxy → null` and exact-match-only guards.

`AgentOrchestratorToolCostTest` (new) is the load-bearing one: a **real** `ToolCostTracker` and a **real** `ToolExecutionService` driven through the actual dispatch loop. It is the only test that can show `maxBudgetPerConversation` is reachable at all. The pre-existing budget test stubs `isWithinBudget` to `false` on a mock, so its refusal comes from the stub and it passed just as happily against the broken behaviour; it is kept for gate wiring and now says so in its javadoc.

Every new behavioural test was mutation-checked by reverting the corresponding production change and confirming failure: pricing by dispatch name (`expected: <0.001> but was: <0.0>`), dropping the slug rate-limit fallback (`expected: <7> but was: <100>`), removing the `enforceBudget` gate (`Wanted but not invoked: isWithinBudget`), removing the negative-price clamp (`expected: <0.0> but was: <-10.0>`), keying the cache on the slug (`expected: <2> but was: <1>` distinct keys), dropping the canonical map on the HITL resume path (`expected: <calculator> but was: <calculate>`), and adding a substring fallback to the resolver (`expected: <null> but was: <calculator>`).

Verifications that would have gone vacuous after the signature change were repointed rather than left to pass silently: the four `executeToolWrapped(anyString(), …)` stubs in the orchestrator tests (an unmatched stub returns `null` and nulls out tool results without failing) and `verify(costTracker, never()).trackToolCall(anyString(), anyString())`, which after the change verified an overload production no longer calls.

**Files:** `ToolNameResolver.java` and `ToolInvocation.java` (new), `ToolCostTracker.java`, `ToolCacheService.java`, `ToolExecutionService.java`, `AgentOrchestrator.java`, `LlmConfiguration.java`, `docs/langchain.md`, `docs/security.md`, `docs/agent-father-langchain-tools-guide.md`, plus `ToolNameResolverTest` and `AgentOrchestratorToolCostTest` (new) and 7 updated test classes.

**Out of scope:** folding cascade LLM-token cost into `ToolCostTracker` so `maxBudgetPerConversation` becomes a true total ceiling (separate follow-up); rate-limit bucket sharing across a tool's operations; the `<=`/pre-check budget boundary; `getSmartTTL`'s substring fallback, which still serves unmapped http/mcp/a2a tool names.

***

## ⏳ Cache entries finally expire — `CacheImpl` honours per-entry TTLs (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D5b**, the follow-up to D5. Every TTL-bearing `ICache` overload in `CacheImpl` dropped its `lifespan` argument and delegated to the untimed variant. Caffeine's standard builder has no per-entry expiry, so the wrapper simply threw the number away. Consequences, all live in production until now:

* **`ToolCacheService`'s entire smart-TTL table was decorative.** `weather` 300s, `websearch` 1800s, `calculator` 7 days — every one of those values was computed, passed to `cache.put(key, value, ttl, unit)` and discarded. Tool results were removed by 10 000-entry size eviction alone, so a stale or poisoned result could be served indefinitely. `GET /llm/tools/cache/ttl/{toolName}` reported numbers that governed nothing.
* **`PaginatedResponseStore`'s documented "15-minute TTL" never existed.** Pages lived until the cache filled.
* **A2A replay nonces had no expiry at all.** `NonceCacheService` carried a comment claiming expiry was "configured externally via Caffeine `expireAfterWrite` in `application.properties`". There is no such configuration anywhere in the repo.

### The fix

`CacheFactory` now builds **every** cache with `Caffeine.expireAfter(Expiry)` instead of leaving expiry unconfigured (size-only caches) or using `expireAfterWrite(ttl)` (TTL caches). That is what exposes Caffeine's `policy().expireVariably()` view, and `CacheImpl` writes every TTL-bearing overload through it. Two entries in one cache can now carry two different lifespans.

* **New `WriteExpiry`** (package-private, `engine/caching`) — the `Expiry` implementation. `never()` for size-only caches (an entry written without a lifespan still never expires on its own); `of(ttl)` for TTL caches (identical observable behaviour to `expireAfterWrite`). `expireAfterRead` returns `currentDuration`, so expiry stays expire-after-**write** and a read never keeps an entry alive.
* **`CacheImpl`** — `put`/`putIfAbsent` route through `VarExpiration`, which is atomic and returns the replaced value in one operation. `putAll` applies the lifespan per entry. Both `replace` overloads do the replace and then `setExpiresAfter` on success — **not atomic**, and the javadoc says so, because Caffeine has no replace-with-duration primitive. A **negative lifespan means unlimited** per the `ICache` contract and is translated into an effectively infinite duration; forwarding it would make Caffeine throw `IllegalArgumentException`. The false javadoc about `CachedResult.expiresAt` is gone.
* **`NonceCacheService`** — asks for `getCache(name, maxAge + clockSkew + 60s)` (390s with the defaults) instead of the size-only cache, and the fictional comment about `application.properties` is deleted.
* **`CacheFactory.CACHE_SIZES`** — `paginated-tool-responses` pinned at 1 000 (entries are whole oversized tool responses; the TTL is now the primary eviction path and the cap only bounds memory) and `nonce-replay-protection` raised to 100 000. The 1 000 default was a security hole: on a busy A2A endpoint a nonce could be size-evicted while its timestamp still passed the freshness check, re-opening the replay window. 100 000 covers \~300 signed requests/second sustained across the whole \~5.5-minute window.

### Design decisions

* **Variable expiry in the factory, not a per-cache workaround.** The alternative — one Caffeine instance per distinct TTL value, keyed like the existing `name:ttl=…` scheme — would have given `tool-results` nine separate caches with nine separate size budgets and no shared eviction. Per-entry expiry is what the `ICache` contract already promised.
* **`expireAfter` replaces `expireAfterWrite`, it is not added to it.** Caffeine throws `IllegalStateException` at `build()` if both are configured, which would have failed the `@PostConstruct` of `SlackEventHandler`, `ChannelTargetRouter` and `NonceCacheService` on startup. `CacheFactoryTest` pins this.
* **`maxIdleTime` on the two six-argument overloads is explicitly unsupported.** Caffeine cannot combine a per-entry write duration with a per-entry idle duration. The `lifespan` is honoured (previously both were discarded) and the javadoc states the limitation. These overloads have no callers in EDDI.
* **A `CacheImpl` over a cache with no variable expiry degrades instead of throwing.** Only reachable by constructing the class directly rather than through `CacheFactory`; the constructor logs a WARN naming the cache.
* **15 minutes is comfortably longer than a tool-calling loop.** The `PaginatedResponseStore` TTL runs from `store()`, and a loop is bounded by the LLM request timeout — seconds to a couple of minutes. `FetchToolResponsePageTool` already returns "It may have expired (15 minute TTL)" for an unresolvable `responseId`; that message is finally true.

### ⚠️ Operator-visible behaviour change

* **Tool results that were cached forever now expire on the smart-TTL table.** Cache hit rate will fall and real tool invocations — with their external API spend — will rise. The effect is sharpest for the short-TTL tools: `weather` (300s), `news` (600s), `websearch` (1800s), and anything with no table entry (300s default). `calculator`/`pdfreader`/`dataformatter`/`textsummarizer` (24h–7d) barely move. Watch `eddi_tool_cache_hits_total` / `eddi_tool_cache_misses_total`.
* **This is a correctness fix, not a regression.** Serving a 5-day-old weather reading was never intended behaviour.
* **Paginated tool responses now expire 15 minutes after they are stored.** An LLM that sits on a `responseId` past that gets the existing "may have expired" error instead of a page.
* **A2A replay nonces now expire \~6.5 minutes after first use** instead of living until size eviction. Steady-state nonce memory drops; replay protection gets *stronger*, not weaker, because the cache is also 100× deeper.
* **No configuration change is required or available.** All three TTLs were already the documented intent; nothing new is exposed.
* **⚠️ GraalVM native image — unverified.** Caffeine picks a generated `BoundedLocalCache` subclass per feature combination, and `quarkus-caffeine` registers a *fixed* list of those classes for reflection at build time. Variable expiry selects a different generated family that may not be on that list; if so, `Caffeine.build()` fails at first cache creation and every `@PostConstruct` calling `getCache(...)` dies at startup. **This cannot fail in JVM mode and is invisible to `mvnw test` and JVM-mode CI.** Logged as a Phase 3 blocker in `planning/native-image-migration.md` — the first native smoke test must exercise both `getCache(name)` and `getCache(name, ttl)`.

### Tests

`CacheImplTest`'s seven `*_delegatesToPut` cases asserted only that a value was readable immediately after a TTL put — true with or without the fix, which made them a codification of the defect. They are replaced by 26 ticker-driven cases covering all seven overloads: entry gone past the lifespan, per-entry (not cache-wide) expiry, expire-after-write rather than after-access, negative lifespan means unlimited, and the degraded no-variable-expiry path. `ToolCacheServiceTest` gains a `Smart TTL is load-bearing` group wired to a **real** `CacheImpl` over a ticker — a 60s `datetime` result dies at 61s while a 7-day `calculator` result does not — which is what finally makes `TOOL_TTL_SECONDS` more than a lookup table. `PaginatedResponseStoreTest` runs against a real cache too and winds past 901s. `CacheFactoryTest` non-vacuously protects the one expiry path that already worked. `NonceCacheServiceTest` captures the TTL and asserts it exceeds `maxAge + clockSkew`. New shared test seam: `TestCaches` (`FakeTicker` + a production-shaped cache).

Every behavioural test above was mutation-checked: with `variableExpiry` forced to `null` (the old behaviour) 10 `CacheImplTest`, 5 `ToolCacheServiceTest` and 2 `PaginatedResponseStoreTest` cases fail; forwarding a negative lifespan straight to Caffeine fails all 5 negative-lifespan cases with `IllegalArgumentException`; dropping `expireAfter` from `getCache(name)` fails `CacheFactoryTest.perEntryTtlIsHonoured`; making `expireAfterRead` reset the clock fails 11 cases; reverting `NonceCacheService` to the size-only cache fails both new nonce cases.

### Docs

`docs/security.md` and `docs/langchain.md` said per-tool TTLs were "computed but not enforced" — corrected. `planning/native-image-migration.md` gains the Caffeine variable-expiry caveat above.

***

## 🔒 Scope the tool-result cache per identity — one user's tool result no longer reaches another (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D5**, the highest-severity item in the langchain4j remediation backlog. `ToolCacheService` built its cache key as `toolName + ":" + arguments` and nothing else. That is a single global namespace: if user A asked `getAccountBalance` with `{"account":"main"}` and user B later made the byte-identical call, B was served A's cached result verbatim, with no execution and no authorization check in between. Every agent with `enableToolCaching` (default `true`) and a tool whose output depends on *who* is asking was affected.

### The fix

Every cache key now starts with a **scope tag**:

```
args = (arguments == null ? "" : arguments)
key  = scopeTag + "|" + toolName + ":" + (args.length() > 2048 ? sha256(args) : args)
```

| Scope          | Tag                                   | Reused by                                      |
| -------------- | ------------------------------------- | ---------------------------------------------- |
| `user`         | `u:<first 32 hex of SHA-256(userId)>` | only the same authenticated user (the default) |
| `conversation` | `c:<conversationId>`                  | only the conversation that produced the entry  |
| `global`       | `g`                                   | everyone — opt-in only                         |

Resolution per tool call is `task.toolCacheScopes[<tool>]` → `task.defaultToolCacheScope` → `user`. Both are new lenient-`String` fields on `LlmConfiguration.Task`; the recognized tokens live in the new `ToolCacheScope` enum (`fromConfig` / `resolve`, following the existing `CascadingStrategy` pattern). Unrecognized or misspelled tokens fall through to `user` rather than failing the agent load — a typo must never silently *widen* a tool's audience.

**Fail-closed identity handling.** `user` scope with no usable user id degrades to the narrower `c:<conversationId>` partition. If neither a user id nor a conversation id is available, `resolveScopeTag` returns `null` and the cache is **bypassed entirely** — no `get`, no `put`. `null` is never turned into `""` or `"unknown"`: that would recreate exactly one shared partition for every anonymous request, i.e. the same bug under a new name. A new `eddi.tool.cache.bypassed` counter (tagged `tool`) makes the bypass visible on `/q/metrics`.

### Changed files

* **`ToolCacheScope`** (new, `modules/llm/tools`) — the recognized scope tokens plus lenient parsing and the per-tool → task-default → `USER` resolution chain.
* **`ToolCacheService`** — `get`/`put`/`invalidate` now take the scope tag as their first argument. The old unscoped signatures were **deleted, not overloaded**: an overload lets a future caller silently reintroduce the global key. `resolveScopeTag(...)` is a static on this class so the orchestrator can build the tag without injecting the cache.
* **`ToolExecutionService.executeToolWrapped`** — takes `cacheScopeTag` (3rd parameter, next to the other cache-key inputs) and gates both the read and the write on it being non-null.
* **`AgentOrchestrator.executeSingleToolCallResult`** — a four-line insertion above the `executeToolWrapped` call resolves the tag from `task` + `memory.getUserId()` + `conversationId`. Because the live tool loop and the HITL resume path already share this one method, both are covered by that single change and no call site or method signature in the orchestrator moved.
* **`CacheFactory`** — `tool-results` was absent from `CACHE_SIZES` and silently got the 1 000-entry default. Raised to 10 000: scoping multiplies the keyspace by the number of active users, and 1 000 would thrash.
* **`CacheImpl`** — javadoc and inline comment corrected. They claimed the TTL overloads were safe because "the ToolCacheService already tracks expiry internally via `CachedResult.expiresAt`". **No such field exists** — the wrapper only records `cachedAt`, for a debug log. The comments now state plainly that the TTL argument is discarded and size-based eviction is the only eviction strategy.
* **Docs** — `security.md` (both the "SHA-256 key" and "within the same conversation" claims were factually false), `langchain.md`, `agent-father-langchain-tools-guide.md`, `metrics.md`, `monitoring/monitoring-guide.md`.

### Design decisions

* **No name-based "pure tool ⇒ GLOBAL" default table.** Agent-mode tool names are `@Tool` *method* names (`calculate`, `getCurrentWeather`), not class names — the existing `contains`-matcher in `getSmartTTL` already fails to recognize them. Guessing purity from a name would hand out cross-user reuse by accident. `global` is opt-in via config only.
* **No `TENANT` scope.** `TenantQuotaService` is still a single-tenant stub; a tenant partition today would be indistinguishable from `global`.
* **No global kill-switch flag.** The opt-out is per tool: `"toolCacheScopes": {"<tool>": "global"}`. A deployment-wide "disable scoping" switch is a foot-gun that re-opens the leak for every tool at once.
* **Scope tag first in the key.** Entries belonging to different identities cannot collide regardless of tool name or arguments.
* **The user id is hashed, not stored.** Cache keys are visible in heap dumps and debug logs; 32 hex characters of SHA-256 is enough to partition without carrying the identifier around.

### ⚠️ Operator-visible behaviour change (deploy-time)

* **Cross-user tool-result sharing stops.** This is the point of the change, and it is not configurable away except per tool.
* **Cache hit rate will drop and tool cost will rise.** Entries that were previously shared by the whole deployment are now per user. Expect more misses, more outbound tool/API calls and a higher external spend, proportional to how much cross-user reuse the agent was silently relying on. Watch `eddi_tool_cache_hits_total` / `eddi_tool_cache_misses_total` after deploying.
* **Any agent that depended on cross-user reuse must opt that tool in explicitly** with `"toolCacheScopes": {"<tool>": "global"}` — and only where the result genuinely does not depend on the caller.
* **Existing `tool-results` entries are dropped.** The cache is in-process and the key format changed, so the first requests after a restart are misses either way.
* **New meter:** `eddi_tool_cache_bypassed_total{tool="…"}`. A sustained non-zero rate means tool calls are running with neither a user id nor a conversation id and are paying full tool cost every time — fix the caller, do not widen the scope.
* **No config migration needed.** `toolCacheScopes` and `defaultToolCacheScope` are optional; stored configs without them behave as `user` scope.

### Not in scope (deliberately)

* **Cache TTLs still do not expire anything** (next item, D5b). `CacheImpl` discards the lifespan argument, so scoped entries are removed by size eviction only. Nothing in this change or its tests implies otherwise.
* **`ToolRateLimiter` remains cross-user.** Its token buckets are keyed by tool name alone, so one user can exhaust another's budget for a tool. Same class of defect, separate fix — flagged here rather than folded in.

### Tests

`ToolCacheServiceTest` gained the headline regression test `differentUsers_produceDifferentKeys` (same tool, same arguments, two identities ⇒ two distinct captured keys), a read-side twin (`get_otherUser_missesEntry`), the twelve-case scope-resolution table, and five null-scope bypass tests. Two pre-existing vacuous tests were replaced: `longArgs_sha256Key` asserted only `startsWith("calculator:")` and `length < 200` — true for any truncation — and now asserts the exact key against an independently computed SHA-256 oracle; `shortArgs_readableKey` was a byte-identical duplicate of `put_storesInCache` and now asserts the full `scopeTag|tool:args` shape. A `UnscopedApiRemovedTests` tripwire fails if the deleted global-key signatures ever come back. `ToolExecutionServiceTest` covers the bypass at the wrapper level; `AgentOrchestratorCoverageTest` captures the scope tag the orchestrator actually passes for the default, `global` and no-identity cases.

Every behavioural test was mutation-checked: reverting the key scoping, replacing `null` with a placeholder, dropping either null-guard, removing the blank-user-id degradation, dropping the userId at the orchestrator, and ignoring the per-tool scope map each produce failures (e.g. `expected: not equal but was: <getAccountBalance:{"account":"main"}>` and `expected: <fresh result> but was: <SOMEONE ELSES RESULT>`).

***

## 🧹 Delete the dead `enableParallelExecution` config and the parallel tool machinery (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D10**. Two LLM task knobs and \~250 lines of the machinery behind them were removed. Nothing in `src/main` read any of it.

### What was dead, and why wiring it was not an option

`LlmConfiguration.Task` declared `enableParallelExecution` (default `false`) and `parallelExecutionTimeoutMs` (default `30000`), with getters and setters and **zero production readers** — the only Java references outside the declaration were POJO round-trip assertions in two test classes.

The machinery they were meant to switch on was `ToolExecutionService.executeToolsParallel` / `executeToolsParallelAndWait`, which took `(Object[] toolInstances, Method[] methods, Object[][] args)` and fanned out over a ten-thread `ExecutorService`. That signature cannot be fed from the live dispatch path. Agent-mode tools are invoked by langchain4j through `ToolExecutor.execute(ToolExecutionRequest, memoryId)`, which yields a `(name, jsonArguments)` pair; for MCP, A2A and dynamic tools there is no Java `Method` behind the tool **at all**, so no `(instance, Method, Object[])` triple exists to pass. This was never a wire-up away from working — it was a second, incompatible execution model that had been left in the tree.

Concurrent tool calls remain a reasonable future feature. They belong at the `AgentOrchestrator` dispatch loop, batching `ToolExecutionRequest`s, not in a reflection path.

### Deleted

* `LlmConfiguration.Task` — both fields and their four accessors. The class javadoc now carries a historical note explaining the removal and the back-compat guarantee.
* `ToolExecutionService` — `executeToolsParallel`, `executeToolsParallelAndWait`, and **the full cascade they were the only callers of**: `executeTool(Object, Method, Object[], String, ToolExecutionTrace)`, `serializeArguments`, the `IJsonSerialization` injection, the ten-thread `ExecutorService` and its `@PreDestroy shutdown()`. `executeToolWrapped` is now the single entry point, and the class no longer allocates a thread pool per bean for nothing.
* Five meters: `eddi.tool.execution.parallel` (the only one registered eagerly, at `@PostConstruct`) plus `…parallel.count`, `…parallel.duration`, `…parallel.timeout` and `…parallel.error`, all four of which were created lazily at increment sites that could not be reached and therefore never appeared on `/q/metrics` at all.
* The corresponding rows in `docs/metrics.md` and `docs/monitoring/monitoring-guide.md`, and the **"Parallel Execution" panel** (`id: 33`) from `grafana-data/dashboards/eddi-operations.json` — an addition to the original scope, found while tracing the metric names. The two surviving panels in that dashboard row were widened from `w:8` to `w:12` to fill it.

### Design decisions

* **The cascade was taken deliberately, not stopped at the two named methods.** Half-deleting would have left `executeTool`, `serializeArguments`, an injected serializer and a thread pool alive with no production caller — the worst of both outcomes. The trade was \~15 reflection-based tests across three classes; every one of them exercised a path production cannot enter (see below).
* **`@PostConstruct init()` is kept**, now registering nothing and only logging. Every remaining meter in the class is per-tool (tagged `tool`) and created lazily on first use, so there is nothing left to pre-register; the startup log is pre-existing behaviour and removing it is an unrelated change.
* **`synchronized` stays on `ToolExecutionTrace.addToolCall` / `addFailedToolCall`.** Its javadoc justified the keyword by naming `executeToolsParallel` as the concurrent writer, so it would have become a comment pointing at deleted code. The rationale is restated on its true footing: the trace is a shared mutable accumulator that publishes no happens-before edge of its own; today's tool loop writes single-threaded, so the guard is defensive rather than load-bearing, and the cost of keeping it is nil against silent corruption of an audit artefact.
* **No `ExtensionDescriptor` change was needed** — `LlmTask.getExtensionDescriptor()` only ever exposed `uri`.

### Stored configurations stay valid

Every mapper that reads an LLM configuration — REST, Postgres JSONB, `@PersistenceMapper`, and the Mongo BSON mapper — is built from `SerializationCustomizer.configureObjectMapper`, which sets `FAIL_ON_UNKNOWN_PROPERTIES=false`. A `langchain.json` already in MongoDB carrying `"enableParallelExecution": true` still deserializes; the key is ignored on read and dropped on the next save.

New `LlmConfigurationParallelExecutionLegacyFieldsTest` is the tripwire for that invariant: it loads a legacy document through both the JSON and the Mongo BSON mapper and asserts the surrounding live fields (`defaultRateLimit`, `toolRateLimits`, `maxToolIterations`, …) still populate. Verified non-vacuous by flipping `FAIL_ON_UNKNOWN_PROPERTIES` to `true` locally — all three deserialization tests then fail with `UnrecognizedPropertyException`. The same class asserts via `java.beans.Introspector` that `Task` exposes no `enableParallelExecution` / `parallelExecutionTimeoutMs` bean property, so a getter/setter pair cannot quietly put the knob back on the REST contract.

### Tests

* **Removed:** the `ParallelTests` / `ParallelAndWaitTests` / `ParallelExec` nested classes and every `executeTool(Object, Method, …)` and `serializeArguments` test across `ToolExecutionServiceTest`, `ToolExecutionServiceBranchTest` and `ToolExecutionServiceExtendedTest`, plus the two `shutdown()` tests. **None of them covered anything still live** — they were the only callers of those methods anywhere outside the class, so they exercised a path production can never enter and stayed green whether the "feature" worked, was broken, or was disabled. Two were vacuous even on their own terms: `parallelTimeout` passed zero tools (`new Object[0]`), so `allOf()` completed immediately and the `TimeoutException` branch it claimed to cover was never taken; and `concurrentToolsShareTraceWithoutCorruption` spent 50 rounds × 32 tasks of CI wall-clock defending against a shared-trace race that no production code path can produce. The four POJO assertions on `getEnableParallelExecution() == false` / `getParallelExecutionTimeoutMs() == 30000L` pinned defaults nothing read.
* **Added (fails before the deletion, passes after):** `ToolExecutionServiceTest.ParallelMachineryRemovedTests` — no `executeToolsParallel*` and no `executeTool` on the public API, and no meter whose id starts with `eddi.tool.execution.parallel` after `init()`. Mutation-checked by restoring the old `ToolExecutionService`: all three fail, the meter one reporting `expected: <[]> but was: <[eddi.tool.execution.parallel]>`.
* `LlmConfigurationParallelExecutionLegacyFieldsTest` mutation-checked by restoring the old `LlmConfiguration`: `rewriteDropsRemovedParallelKeys` and `taskExposesNoParallelExecutionBeanProperty` both fail.

### Operator-visible changes

* **Five Prometheus series disappear** rather than reading zero: `eddi_tool_execution_parallel_total`, `…_parallel_count_total`, `…_parallel_duration_seconds`, `…_parallel_timeout_total`, `…_parallel_error_total`. Only the first was ever exported, and it could only ever be `0`. Any dashboard or alert rule referencing them outside this repo will show "no data" — no signal is lost.
* **No agent behaviour changes.** Both deleted fields were inert at every value, so no configuration executes differently.

### Follow-up required in EDDI-Manager (different repo) — the one user-visible risk

The Manager's LLM editor renders a **"Parallel Tool Execution" checkbox** (i18n key `llmEditor.parallelExecution`, `data-testid` `enable-parallel-execution`) plus a conditional `parallelExecutionTimeoutMs` input. That checkbox is now **backed by nothing on the server**.

Worse than inert: because the mapper uses `JsonInclude.NON_NULL` and the field no longer exists, the key is accepted on POST, silently dropped, and never echoed back — so the checkbox **visibly resets to unchecked on every reload**. Before this change it at least persisted while doing nothing. Removing the control in EDDI-Manager should land in the same release.

**Yes — the checked-in Manager bundles in&#x20;*****this*****&#x20;repo also carry it.** Both `src/main/resources/META-INF/resources/assets/index-B36D6B8M.js` (the one `manage.html` loads) and `index-CHgQ1fX-.js` (a second bundle graph, reachable only through chunk imports such as `cssMode-BPLGavJr.js`) contain one `enableParallelExecution` occurrence and one `data-testid="enable-parallel-execution"` each. They are **built artifacts and were deliberately not hand-patched** — the fix is to remove the control in the EDDI-Manager source, rebuild, and re-vendor both bundles. Same follow-up shape as the `injectionStrategy` `<select>` recorded in the D8 entry below.

***

## 📝 State plainly that `TenantQuotaService.recordCost` has no callers (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D9**, part 3 of 3 — documentation only, no behaviour change.

`TenantQuotaService.recordCost(tenantId, cost)` is the post-call half of the monthly cost budget. It has **zero production callers** — the only references outside its own declaration are in `TenantQuotaServiceTest`. Its pre-call twin `checkCostBudget` *is* wired (`AgentOrchestrator` calls it before each LLM turn), but it reads `ITenantQuotaStore.getMonthlyCost`, which nothing ever writes. So `eddi.tenant.quota.max-monthly-cost-usd` currently cannot deny anything, at any value, on any backend — while the code reads as if cost budgets work.

**Decision: keep the method, document the gap loudly — do not delete it, and do not wire it here.**

* *Not deleted*, because removing it would take away the seam without taking away the gap: `checkCostBudget` stays wired, `ITenantQuotaStore.tryAddCost` stays on the interface, all three stores implement it, and it is now covered by `TenantQuotaStoreParityTest`. A reader would be left with a half-system and no marker.
* *Not wired*, because there is nothing meaningful to meter yet. Built-in tool executions are priced at $0.00, and there is no token-cost metering for LLM turns at all — wiring today would add write load and record zeros. The candidate call sites are the `ChatResponse`-holding seams tracked as C5 in `planning/manager-coverage-backend-design.md`, whose own notes named the Mongo E11000 (fixed in part 1 of this item) as its blocker. That blocker is now gone.

Both `recordCost` and `checkCostBudget` carry javadoc saying this outright, including the two things that must land first.

***

## 🐛 Deny tenant quotas **at** the limit on the Postgres store (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D9**, part 2 of 3 — on PostgreSQL the daily conversation cap and the per-minute API-call cap were **never enforced at all**.

### The defect

Both increment methods had a fast path (`UPDATE … WHERE window = ? AND counter < ? RETURNING counter`) and, on a miss, a fallback `INSERT … ON CONFLICT (tenant_id) DO UPDATE SET counter = CASE WHEN window_start < ? THEN 1 ELSE counter END … RETURNING counter`, guarded by `if (rs.next() && rs.getInt(1) <= limit)`.

When the window was **not** stale, the `CASE` took its `ELSE` branch and returned the counter **unchanged** — i.e. exactly `limit`, because the fast path only ever increments while `counter < limit`. `limit <= limit` is true, so the store returned OK. And because that path never increments, the counter stayed pinned at `limit` and *every subsequent request also returned OK*. The cap leaked without bound; only the fast path's own `< limit` was ever consulted, and it had already failed.

The two unit tests covering this stubbed the fallback to return `11` against a limit of `10` (and `61` against `60`) — values production can never produce at that point — so they asserted the right verdict through an impossible fixture and stayed green while the quota did nothing.

### The fix

The `ON CONFLICT … DO UPDATE` now carries `WHERE tenant_usage.day_start < ?` (resp. `minute_start`), and resets the counter to **zero** instead of one; the increment statement is then re-run. This mirrors the Mongo store exactly:

1. conditional increment (fast path, the only statement in steady state);
2. materialise-or-roll — creates the row or resets an expired window, and is a strict no-op for a row whose window is still current;
3. retry the conditional increment.

The cap is now enforced by a single predicate, `counter < limit`, in every path. A request at the limit with a current window falls through steps 2 and 3 and is denied, and `limit = 0` denies without a special case.

`tryAddCost` got the same materialise-first treatment, which also fixes a latent bug in its `INSERT`: it seeded `day_start` / `minute_start` with a **raw wall-clock timestamp** rather than a truncated window start. A row first created by a cost write therefore had `day_start` *greater* than every truncated value the increment paths compare against — so neither `day_start = ?` nor `day_start < ?` could ever match and that tenant's conversations would have been denied until the next UTC day.

### Tests

* **`TenantQuotaStoreParityTest`** (new, Testcontainers) — runs one boundary sequence through all three `ITenantQuotaStore` implementations (in-memory, Mongo, Postgres) and asserts identical verdicts: allowed below the limit, denied at it, denied for `limit = 0`, unlimited for `limit < 0`, cost denied once spend reaches the budget, and all three counters coexisting. 18 tests. Against the reverted Postgres store exactly the three `postgres:` rows fail (`conversationsDenyAtLimit`, `apiCallsDenyAtLimit`, `zeroLimitDeniesEverything`); the Mongo and in-memory rows stay green, which is the point — this is the only shape that catches a *divergence*.
* **`PostgresTenantQuotaStoreTest`** — the two impossible fixtures are replaced with faithful ones, plus a structural guard per method asserting the materialise-or-roll SQL carries its expired-window `WHERE`. The behavioural at-limit proof cannot live in a mocked JDBC layer (whether `DO UPDATE` fires is Postgres's decision, not the code's), so the structural guard is what fails there against the old SQL; the parity test carries the behaviour.
* `PostgresTestBase.createDataSourceInstance()` widened from `protected` to `public` so the parity test can reuse the single shared container instead of starting a second one.

### Operator note — behaviour change

Deployments running `eddi.tenant.quota.enabled=true` **on PostgreSQL** were not enforcing `max-conversations-per-day` or `max-api-calls-per-minute` at all. They will now start returning quota denials (HTTP 429) once a tenant reaches its configured limit. If limits were tuned against the leaky behaviour, they will need revisiting — this reads as a regression but is the cap doing its job for the first time. MongoDB and in-memory deployments are unaffected by this part.

***

## 🐛 Keep all tenant quota counters in one `tenant_usage` document (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D9**, part 1 of 3 — `MongoTenantQuotaStore` was raising **E11000 duplicate-key errors on a live request path**, not silently mis-counting.

### The defect

`tenant_usage` holds one document per tenant carrying all three counters, and it has a `unique(true)` index on `tenantId`. But all three mutating methods issued an **upsert whose filter also pinned a rolling window**:

* `tryIncrementConversations` → `and(tenantId, dayStart >= …, conversationsToday < limit)`
* `tryIncrementApiCalls` → `and(tenantId, minuteStart >= …, apiCallsThisMinute < limit)`
* `tryAddCost` → `and(tenantId, costMonth == …)`

Whenever the extra predicate did not match — which is *always* the first time a second counter is touched, because the document written by the first method has no `minuteStart` / `costMonth` field at all — the upsert found nothing, tried to insert a second document for the same tenant, and the unique index rejected it. The exception propagates out of the store.

Concretely, with `eddi.tenant.quota.enabled=true` and both `max-conversations-per-day` and `max-api-calls-per-minute` set, `ConversationService.acquireConversationSlot()` succeeded and the very next `acquireApiCallSlot()` threw — a **500 on a user request**, not a quota denial. (Earlier notes described this as "monthly cost silently reads 0.0 forever". That was true before the unique index was added; since then the failure mode is the hard error above.)

### The design

Every write that can insert now filters on the unique-index key and nothing else. Each operation is:

1. **Fast path** — one conditional `findOneAndUpdate` (`window current AND counter < limit`), **no upsert**. In steady state this is the only round trip, so the hot path is unchanged.
2. **Materialise** — `ensureUsageDocument(tenantId)`, the single write in the class allowed to insert. Filter is `eq("tenantId", …)`; all counters are seeded together via `$setOnInsert`, so whichever operation runs first for a tenant, the other two find their fields present.
3. **Roll + retry** — reset an expired window to **zero** (conditional, no upsert), then re-run the fast path.

Steps 2–3 only run when the fast path misses (first call for a tenant, window rollover, or a real limit breach). Because the roll resets to zero and step 3 does the counting, the limit is enforced by exactly one predicate — `counter < limit` — in every path; a `limit` of `0` therefore denies without a special case, matching `InMemoryTenantQuotaStore`.

The stale-window filter is `or(exists(field, false), lt(field, windowStart))`, which doubles as repair for legacy documents written by the previous code that are missing a window field entirely.

### Migration hazard, handled

`createIndex(tenantId, unique=true)` was called unguarded in the CDI constructor. Any instance that ran the pre-index build with quotas enabled may already hold duplicate `tenantId` rows, in which case index creation fails and the **whole application fails to start**. It is now wrapped: on failure it logs an ERROR naming the collection and the remediation, and continues. Safe, because correctness no longer depends on the index — it is a safety net against upsert races, not a precondition.

### Tests

* **`MongoTenantQuotaStoreContainerTest`** (new, Testcontainers) — 13 tests. Against the reverted store, 10 of them fail with `E11000 duplicate key error collection: eddi_test.tenant_usage index: tenantId_1`. Covers: conversations→api-calls, conversations→cost, interleaved three-counter accounting, at-limit denial, zero limit, day/minute window rollover, stale cost month, and a legacy document missing its window fields.
* **`MongoTenantQuotaStoreTest.UpsertFilterDiscipline`** (new, pure unit, no Docker) — renders every captured filter to BSON and asserts each `upsert(true)` write is keyed on `tenantId` alone. Against the reverted store it fails with `expected: <[tenantId]> but was: <[$and]>`.
* `TryAddCost.staleMonth` was renamed to `usageDocumentDisappears`: its premise (both conditional updates return null) is no longer "stale month" — it is now a concurrent `resetUsage`, which must not cost the caller a denial.

### Operator note

No configuration change. The subsystem ships dormant (`eddi.tenant.quota.enabled=false`, all limits `-1`), so only deployments that opted in were affected — for those, this turns 500s back into correct allow/deny.

***

## 🧹 Delete dead RAG `injectionStrategy` / `contextTemplate` config (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D8** — two RAG configuration knobs on `LlmConfiguration` that **never had any effect**, removed rather than implemented.

### What was dead, and how it was verified

`LlmConfiguration.KnowledgeBaseReference` declared `injectionStrategy` and `contextTemplate`; `LlmConfiguration.RagDefaults` declared `injectionStrategy` (defaulting to `"system_message"`). Every one of them was write-only:

* `RagContextProvider.retrieveContext` reads exactly two override fields off a `KnowledgeBaseReference` / `RagDefaults` — `maxResults` and `minScore`. It never calls `getInjectionStrategy()` or `getContextTemplate()`.
* `LlmTask` (the provider's only production caller) unconditionally does `systemMessage += "\n\n## Relevant Context:\n" + ragContext`. There is no branch, so no strategy value could have changed anything.
* Formatting is fixed in `RagContextProvider.formatRagContext` (`### Source: <kb>` headings); no templating engine is reachable from that path, so `contextTemplate` had nowhere to be applied even in principle.
* Grep across `src/main` returns **zero** reads of either accessor. No REST resource, no `ExtensionDescriptor` (`LlmTask.getExtensionDescriptor()` registers only the `uri` `ConfigValue`), no MCP tool, no migration, no sample agent config in `src/main/resources/initial-agents` or `docs/agent-configs` referenced them. The only Java references were POJO getter/setter round-trip assertions in `LlmConfigurationTest` and `LlmConfigurationModelsTest` — tests that assert a setter stores what you set and would stay green under any behaviour whatsoever.

### Design decision — delete, not wire

Wiring `injectionStrategy` would have meant shipping user-message injection, and that is not a free "finish the feature" change: RAG context routed to the system message is **not** counted against `maxContextTokens` (`ConversationHistoryBuilder` budgets only the assembled history), while context routed into the user message **is**. Turning the knob on would therefore silently change how much conversation history survives windowing for anyone who had already saved `user_message`, with no config edit on their part. `contextTemplate` was worse still — it had zero reads *and* zero UI, and honouring it would have required introducing a templating engine into `RagContextProvider` plus inventing the `{{context}}` semantics the field name implied. Deleting a knob nothing honours is cheaper and more honest than inventing the semantics it advertised. The intended behaviour (system-message injection) is exactly what already happens, so nothing observable changes for any agent.

### ⚠️ Follow-up required in the EDDI-Manager repo (not fixable here)

The Manager UI bundles checked into this repo — `src/main/resources/META-INF/resources/assets/index-B36D6B8M.js` and `index-CHgQ1fX-.js`, loaded by `manage.html` — render an **Injection** `<select>` (System Message / User Message) bound to `injectionStrategy`, for both `knowledgeBases[]` and `ragDefaults`. Relevant strings: the i18n key `llmEditor.injectionStrategy`, the value expression `injectionStrategy??"system_message"`, and the two `onChange` writers `injectionStrategy:<e>.target.value`. `contextTemplate` has no UI presence in either bundle.

**These bundles were deliberately not edited.** They are minified Vite build artifacts (\~7 MB, single \~319 000-character lines, mangled identifiers) produced by a separate repo; hand-patching a built artifact is both unsafe and immediately undone by the next Manager release. The dropdown must be removed **in the EDDI-Manager source** and the rebuilt bundles re-committed here.

**Until that lands**, the shipped dashboard still offers the control, and its value is discarded on save — the same thing that happened before this change (the backend ignored it then too), so this is a pre-existing cosmetic defect made no worse, not a regression introduced here. Agent designers should ignore the Injection dropdown.

### Stored-configuration compatibility

**No migration, no operator action.** Every mapper that deserializes an LLM configuration is built from `SerializationCustomizer.configureObjectMapper`, which sets `FAIL_ON_UNKNOWN_PROPERTIES=false` — the REST/CDI mapper (`customize`), the `@PersistenceMapper` used for Postgres JSONB (`PersistenceMapperProducer`), and the MongoDB BSON mapper (`PersistenceModule.buildMongoClientOptions`) all share that one static recipe. Existing `langchain.json` documents in MongoDB/Postgres, and agent ZIPs exported before this change, keep loading unchanged; the leftover keys are ignored on read and dropped on the next save.

### Regression test

New `LlmConfigurationRagLegacyFieldsTest` (`src/test/java/ai/labs/eddi/modules/llm/model/`), 3 tests, pure-unit (no Testcontainers), mirroring the production mapper wiring the way `HitlTimeoutPolicySerializationTest` does. It deserializes a realistic stored `langchain.json` carrying all three removed keys across both RAG modes:

1. `storedConfigWithRemovedRagKeysDeserializesViaJsonMapper` — loads through the REST / Postgres-JSONB / `@PersistenceMapper` recipe, with the surviving fields (`name`, `maxResults`, `minScore`, `enableWorkflowRag`) asserted intact.
2. `storedConfigWithRemovedRagKeysDeserializesViaBsonMapper` — the raw document is encoded to BSON first, so the decoder sees exactly what an existing `llms` collection holds.
3. `rewriteDropsRemovedRagKeys` — re-serializing a legacy config must not resurrect either key, and the result must still round-trip (so the two `assertFalse`s cannot pass on an empty/broken rewrite).

**This class is the tripwire for the whole deletion.** (1) and (2) fail the moment anyone flips `FAIL_ON_UNKNOWN_PROPERTIES` on the shared recipe — which would break every stored configuration, not just RAG ones. All three were mutation-checked: flipping that flag to `true` fails all three with `UnrecognizedPropertyException: Unrecognized field "injectionStrategy" ... (3 known properties: "minScore", "maxResults", "name")`; re-adding the fields to `KnowledgeBaseReference` fails (3) with `AssertionFailedError: injectionStrategy was deleted because nothing honoured it; a config rewrite must not resurrect it`; re-adding only `RagDefaults.injectionStrategy` fails (3) the same way.

The pre-existing POJO round-trips in `LlmConfigurationTest` / `LlmConfigurationModelsTest` were trimmed to the surviving fields — they were the vacuous coverage that let these two fields look tested for as long as they did.

### Documentation

`docs/rag.md` never documented either field, so nothing was wrong there — but it also never said where retrieved context *goes*, which is the gap that made an unread "injection strategy" knob plausible in the first place. Added a **Context Injection** subsection stating that vector-RAG context is always appended to the system message under `## Relevant Context:` with no per-KB or per-task switch, plus a note for operators whose stored configs still carry the removed keys. The historical `docs/changelog.md` entry that listed this pair under "Feature exists but knob unwired" was left as-is — it is an append-only record of past work.

**Files:** `src/main/java/ai/labs/eddi/modules/llm/model/LlmConfiguration.java`, `src/test/java/ai/labs/eddi/modules/llm/model/LlmConfigurationRagLegacyFieldsTest.java` (new), `src/test/java/ai/labs/eddi/modules/llm/model/LlmConfigurationTest.java`, `src/test/java/ai/labs/eddi/modules/llm/model/LlmConfigurationModelsTest.java`, `docs/rag.md`

***

## 🧹 Remove dead `eddi.audit.retentionDays`; correct MCP-client and GDPR-retention docs (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Backlog item **D15** — one dead configuration property plus two pieces of documentation that described behaviour the code does not have.

### (a) `eddi.audit.retentionDays` was a silent no-op — removed

**⚠️ Operator-visible.** `src/main/resources/application.properties` shipped `eddi.audit.retentionDays=-1` (and a `%dev.` twin) since the per-category-retention work. **No Java code has ever read it.** `AuditLedgerService` declares four `eddi.audit.*` `@ConfigProperty` values — `enabled`, `flush-interval-seconds`, `dead-letter-path`, `agent-signing-enabled` — and this is not one of them. Anyone who set `eddi.audit.retentionDays=90` (or `EDDI_AUDIT_RETENTIONDAYS=90`) expecting old audit entries to be purged **was getting nothing at all**: no sweep ran, no entry was ever deleted, and no warning was logged. Drop the property from your deployment; if you need time-limited audit storage, do it at the operational/database layer (archival job, partition drop, storage-level TTL).

**Design decision — do NOT implement the sweep; delete the knob.** The audit ledger is append-only *on purpose*. `IAuditStore`'s contract states that implementations "MUST NOT provide update or delete operations", justified by EU AI Act Arts. 17/19 (immutable decision traceability) and GDPR Art. 17(3)(e); `pseudonymizeByUserId` is documented as the sole permitted mutation, and both the MongoDB and PostgreSQL stores enforce insert-only semantics. Wiring a retention sweep would have required breaking that contract to satisfy a property nobody asked for. Removing the property makes the config honest instead.

**Why this one mattered more than a typical dead key:** its sibling `eddi.usermemories.deleteOlderThanDays` **is** read and does drive a real scheduled sweep. Two adjacent, identically-shaped keys where one works and one silently does nothing is exactly the kind of asymmetry an operator cannot detect from the outside.

**Regression test:** new `AuditRetentionConfigTest` (`src/test/java/ai/labs/eddi/engine/audit/`), three assertions, no Quarkus container required:

1. Every `eddi.audit.*` key declared in `application.properties` (with any `%profile.` prefix stripped) must appear as a quoted string literal somewhere in `src/main/java` — i.e. something actually reads it. This is the general guard: it fails if *anyone* re-adds an unread `eddi.audit.*` property, not just this one. It carries a **vacuity guard** — the scan must first prove it can find `"eddi.audit.enabled"`, a key production demonstrably declares, otherwise a broken scanner would make the whole assertion pass for the wrong reason.
2. `eddi.audit.retentionDays` specifically is absent, with a failure message explaining why it can never be implemented.
3. `IAuditStore` declares no `delete*`/`remove*`/`purge*`/`drop*`/`truncate*`/`expire*` method — this pins the append-only design decision the docs now state, so a future contract change has to be deliberate and update the compliance doc alongside it.

All three were mutation-checked: re-adding the two property lines fails (1) and (2); adding a `default long deleteOlderThan(long)` to `IAuditStore` fails (3).

### (b) `docs/mcp-server.md` documented an MCP-client config shape that no longer exists

The "MCP Client — Agents as MCP Consumers" section still described an inline `mcpServers` array on a langchain task. That field and its `getMcpServers()` accessor were deleted in **43ba59811** ("Remove inline mcpServers from LlmConfiguration.Task") without a matching docs update, so the documented JSON has been silently unusable ever since. The section also named a `setup_agent(mcpServers:)` parameter; the real one is **`mcpServerUrls`** (`McpSetupTools.java`).

Rewrote the section against the code rather than the old prose. External MCP servers are configured as **`mcpcalls` workflow extensions** — a versioned configuration resource (`eddi://ai.labs.mcpcalls`, `POST /mcpcallsstore/mcpcalls`), the MCP equivalent of `httpcalls` — referenced from a workflow step ahead of the LLM step. The field table now matches `McpCallsConfiguration`: `mcpServerUrl` (not `url`), `name`, `transport` (default `"http"`, and documented honestly as *informational only* — `McpToolProviderManager.createTransport` unconditionally builds a `StreamableHttpMcpTransport` and never branches on it, so the previously documented `"streamableHttp"` default and "only `streamableHttp` supported" note were both wrong in different directions), `apiKey`, `timeoutMs`, `toolsWhitelist`, `toolsBlacklist`, `mcpCalls`. Added the dual-mode explanation the docs never had: **agent mode** (`AgentOrchestrator.discoverMcpCallTools()` traverses agent → workflow → every `mcpcalls` step at execution time and applies each config's whitelist/blacklist; gated by `enableMcpCallTools` on the LLM task, default `true`) versus **pipeline mode** (`McpCallsTask` matches behavior-rule actions against `mcpCalls[].actions` and invokes tools deterministically, no LLM involved). Corrected the `setup_agent` block to `mcpServerUrls` and documented what `AgentSetupService` actually does with it: one `mcpcalls` config per comma-separated URL (`transport: "http"`, `timeoutMs: 30000`, no whitelist/blacklist, no `mcpCalls` bindings) plus a matching `eddi://ai.labs.mcpcalls` workflow step — nothing is written inline into the LLM configuration.

Also fixed the same drift in `docs/architecture.md`, which claimed "MCP server connections are configured per LLM task".

**Left alone deliberately:** the four other `mcpServers` occurrences in `docs/mcp-server.md` (lines \~179/193/211/224) are the *client-side* `mcpServers` key in Claude Desktop / Antigravity config files — correct as written and unrelated. `HANDOFF.md` and the historical changelog entries that mention the old inline field are append-only records of past work and were not rewritten.

### (c) `docs/gdpr-compliance.md` promised deletion that cannot happen

The retention block documented `eddi.audit.retentionDays` as "delete entries older than N days" — a published compliance statement describing an operation the store forbids. Removed it and replaced it with what is actually true: the audit ledger has no retention property by design, EDDI never time-expires audit entries, and erasure requests are satisfied by pseudonymizing the `userId` via `IAuditStore.pseudonymizeByUserId` on the cascading-erasure path. Added an operator note that the old property was never read, and reworded the per-category bullet list so it no longer implies a configurable audit purge exists.

**Note for reviewers:** this narrows a published compliance claim (EDDI no longer offers even a nominal "time-limited audit retention" option). That is the honest direction — it aligns the doc with `IAuditStore`'s contract — but it is a documentation-visible policy statement and deserves a human read, not a silent patch.

**Blast radius:** zero runtime behaviour change. No Java production code was modified; the only non-doc edit is the removal of a property nothing read. `./mvnw clean compile` clean, `./mvnw validate` clean, 145 tests green across `AuditRetentionConfigTest`, `AuditLedgerServiceTest`, `AuditLedgerServiceExtendedTest`, `AuditLedgerServiceBranchTest`, `AuditHmacTest`, `McpSetupToolsTest`, `McpToolProviderManagerTest`, `McpCallsTaskTest`.

***

## 🗑️ Delete unused `EddiChatMemoryStore` — dead since Phase 6E (2026-07-22)

**Repo:** EDDI (`fix/backlog-defect-remediation`)

Removed `src/main/java/ai/labs/eddi/modules/llm/memory/EddiChatMemoryStore.java` and its two test classes (`EddiChatMemoryStoreTest`, `EddiChatMemoryStoreExtendedTest`). The package `ai.labs.eddi.modules.llm.memory` is now gone entirely — those three files were its only occupants.

**Why it was dead.** The class's own javadoc describes it as "a quarkus-langchain4j `ChatMemoryStore`", but that extension was dropped in **Phase 6E (2026-03-15)** in favour of plain `dev.langchain4j` core. Nothing in `pom.xml` pulls `io.quarkiverse.langchain4j`, so the machinery that would have discovered and used a `ChatMemoryStore` bean — `AiServices`, `ChatMemoryProvider`, `@MemoryId` — is not on the classpath at all. A repo-wide grep confirms none of those four symbols appears anywhere else in the codebase, and `ChatMemoryStore` itself occurred only in the deleted class's own `implements` clause and import. The bean was `@ApplicationScoped`, so CDI instantiated it, but nothing ever injected it and no code path called `getMessages`/`updateMessages`/`deleteMessages`.

**What actually does this job.** The production LLM history path is `ConversationHistoryBuilder` (used by `AgentOrchestrator`/`LlmTask`), which performs the same EDDI-snapshot → langchain4j-`ChatMessage` conversion via the same `ConversationLogGenerator`, but with windowing, token budgeting and multimodal content that the deleted stub lacked. If EDDI ever re-adopts `quarkus-langchain4j` (see `planning/native-image-migration.md`), the bridge would be rebuilt from `ConversationHistoryBuilder`, not from this stub — so nothing of value is lost.

**The 14 deleted tests asserted nothing about shipped behaviour.** Both test classes only pinned a bean with zero production consumers, and several were vacuous on their own terms: `getMessages_emptySnapshot_returnsEmpty` and `GetMessages#returnsMessagesFromSnapshot` both fed in an *empty* `conversationSteps` list and asserted an empty result, never once exercising the role/`ContentType` conversion loop; the two `updateMessages` "no-op" tests called `verifyNoInteractions` on a mock that a single-`LOGGER.trace` method could not possibly touch. Net effect on the suite: 14 fewer tests, and the coverage gate should move neutral-to-positive because the deleted class contributed an entirely *uncovered* conversion block.

**Design decision — the changelog is append-only.** Five earlier entries in this file (from the 2026 test-coverage pushes that created and tidied these test classes) mention `EddiChatMemoryStore`. Those were left untouched: they are accurate records of past work, and rewriting history to hide a since-deleted class would make the log unreliable. This entry is the forward record of the removal.

**Operator impact:** none. No configuration key, REST endpoint, agent JSON field, or persisted document references the class or its package, and no runtime code path could reach it. Nothing to migrate.

**Shared collaborators deliberately untouched:** `ConversationLogGenerator` (5 other production consumers), `IConversationMemoryStore`, `ConversationMemorySnapshot`.

***

## 🧪 LLM — make `AgentOrchestrator` injectable; cover agent-mode response metadata (2026-07-22)

**Repo:** EDDI (`refactor/agent-orchestrator-injectable`; branched from `fix/orphan-scan-and-quota-defects`, which has since merged to `main` — this branch now targets `main` directly)

Follow-up to the token-usage fix below, which shipped without unit coverage.

### Correction to the entry below

Its "Verification limits" paragraph states that `LlmTask` constructs its orchestrator internally, so `executeIfToolsEnabled` cannot be stubbed and "every existing `LlmTask` test therefore exercises the legacy branch". **That is wrong.** `LlmTaskCoverage2Test` and `LlmTaskResumeModeTest` already substituted a mocked `AgentOrchestrator` — via `LlmTask.class.getDeclaredField("agentOrchestrator")` + `setAccessible(true)` — and roughly 20 tests drove the agent branches through it, including the legacy-fallback case.

The branches were covered. What was never exercised was the **metadata dimension**: every stub built its result with the two-argument `ExecutionResult` convenience constructor, which hardcodes `Map.of()`. Against an always-empty map, a task that ignores `responseMetadata()` is indistinguishable from one that honours it. Branch coverage hid a data-flow gap — the more useful lesson than "the seam was missing".

### What changed

* `AgentOrchestrator` → `@ApplicationScoped` with an `@Inject` constructor. **Signature unchanged**, so the 8 `AgentOrchestrator*Test` classes that construct it directly are untouched.
* `ConversationHistoryBuilder` → `@ApplicationScoped`; it is an orchestrator constructor parameter and had to become resolvable. Stateless (only a static logger).
* `LlmTask.agentOrchestrator` is a `private final` constructor-injected collaborator. Tests pass a mock as an argument.
* **`LlmTask`'s constructor went from 41 parameters to 20.** 22 of them existed only to feed `new AgentOrchestrator(...)`, plus `attachmentStore`; all 23 left with the orchestrator, and `AgentOrchestrator` + `ConversationHistoryBuilder` came in. All 10 call sites across 9 test files were rewritten, along with the mocks, imports and locals the removal orphaned.
* **Attachment services are now injected into `AgentOrchestrator` directly** (`@Inject` fields), and `LlmTask.wireAttachmentServices()` is gone. `setAttachmentServices` survives only as a test seam for directly-constructed orchestrators.
* All three reflection hacks deleted (`LlmTaskCoverage2Test`, `LlmTaskCoverageTest`, `LlmTaskResumeModeTest` — the third was missed on the first pass and found by grepping the whole test tree rather than the two files already in hand).

### Statelessness check (AGENTS.md §4.1 rule 2)

Every `AgentOrchestrator` field is `final` except the two `volatile` attachment services, which are write-once deployment-scoped collaborators, not per-conversation state; `ToolApprovalGate` and `ChatTranscriptCodec` are stateless helpers. All conversational state travels through the `IConversationMemory` argument. Safe as a shared singleton — which it already effectively was, being owned by the singleton `LlmTask`.

### Tests

New `LlmTaskAgentModeMetadataTest` (7 tests) covering the standard agent branch, the `skipCascade` agent branch, the legacy fallback when the orchestrator declines, the HITL resume continuation (non-null and null), and a null tool trace.

Validated by mutation rather than by observing green: reverting `15b7a08a7` and re-running turns **five of the seven** red. The two that survive (`agentMode_emptyMetadata_publishesEmptyMap`, `agentReturnsNull_fallsBackToLegacyChatExecutor`) pass with and without the fix by design — they pin behaviour the fix did not alter, and are regression guards, not discriminators. Each says so at its own assertion, so the distinction is visible in the file a future reader actually opens, not only here.

> **Correction, and a lesson about arithmetic-by-inspection.** This paragraph twice carried a wrong count, and a Copilot review on PR #604 caught the inconsistency (`four` + `the other two` ≠ seven). Both the number *and* the suggested repair were wrong, in opposite directions: the review proposed listing `resumeMode_nullResult_publishesEmptyMetadata` as a third guard, but that test **does** discriminate — pre-fix, `executeResume` published no metadata at all, so its `verify` fails outright. Re-measuring gave five, not four. The stale figure came from updating the totals by hand across two changes that moved the score in opposite directions: the defensive copy demoted `agentMode_emptyMetadata_publishesEmptyMap` from discriminator to guard (identity assertion → `assertNotSame`), and the later null-trace test added a discriminator back. Mutation scores are cheap to measure and expensive to infer — re-run them.

### Review pass — what it changed

Two independent reviewers (one general, one adversarial and instructed to refute) read the diff, the full orchestrator, and `15b7a08a7`. Neither found a Critical issue; both independently refuted the sharpest hypothesis (that an immutable `Map.of()` could reach a mutation site — production always returns a mutable `HashMap`, and every post-assignment site is read-only). What the pass did produce:

* **A missing third call site.** The fix has three separately-revertable assignments — the `skipCascade` branch, the standard branch, and `executeResume` — and the original five tests reached only the last two. Deleting the `skipCascade` line left all five green. Test six closes it; surgically reverting that one line now fails exactly that one test.
* **A false claim in this entry**, retracted in place above.
* **A near-vacuous test.** `agentMode_emptyMetadata_publishesEmptyMap` asserted only emptiness, which the bug satisfies as readily as the fix.
* **A lazy-init hazard introduced by the refactor itself.** With the orchestrator injectable but its attachments still pushed in by `LlmTask`'s `@PostConstruct`, any *other* future injector of `AgentOrchestrator` would get one with null attachment services, because `@ApplicationScoped LlmTask` is created lazily and might never have run. Fixed by injecting the services into the orchestrator directly. This had been part of the agreed design and was simply not implemented on the first pass.
* **Defensive copy** at all three sites: `responseMetadata` is now `new HashMap<>(...)` rather than an alias of the orchestrator's map. Dormant today, but the published map lands in conversation memory and the `{{llmMeta}}` namespace, and the two-argument `ExecutionResult` constructor yields an immutable map — so a later metadata write would have thrown on the agent path only, in production only.
* **Fixture fidelity:** tasks now set `enableBuiltInTools`, so `isAgentMode()` is true. Previously the fixtures stubbed a non-null agent result onto a config for which the real orchestrator always returns `null`.

### Fourth review round — CodeRabbit + Copilot on PR #604

* **Assert the published metadata is&#x20;*****writable*****, not merely a distinct instance (applied).** `agentMode_emptyMetadata_publishesEmptyMap` asserted `assertNotSame` — which `Map.copyOf(...)` would also satisfy, while still throwing the instant downstream code adds a metadata key. That is exactly the failure the defensive copy exists to prevent, so the assertion was checking the wrong property. Now probes an actual `put`.
* **Inline FQNs replaced (applied).** 25 occurrences of `new io.micrometer.core.instrument.simple.SimpleMeterRegistry()` across 9 `LlmTask*Test` classes, swapped for a top-level import per AGENTS.md §4.7. Pre-existing, but carried forward by this branch when the constructor call sites were rewritten, so they belong to it now.
* **`@Inject` fields left package-private (declined, with evidence).** The suggestion was to privatise `AgentOrchestrator`'s two injected attachment fields. Declined on three counts: `src/main` contains **43 package-private `@Inject` fields and zero private ones**, so this would be the sole exception; `GroupConversationService.attachmentStore` is the identical shape, package-private and written directly by `GroupConversationServiceTest`; and Quarkus recommends package-private precisely to keep ArC from injecting reflectively, which matters for the native-image work on the roadmap. The stated risk — accidental in-package mutation — is also not removed by the change, since the package-private `setAttachmentServices` setter next to the fields exists so the eight `AgentOrchestrator*Test` classes can write them.

### Third review round — Copilot on PR #604

* **Null `trace()` guard, applied.** `executeTask` assigned `agentResult.trace()` straight into `toolTrace` while `executeResume` null-guarded it, so `!toolTrace.isEmpty()` could throw. Both agent branches now guard. Flagged by all three reviewers, and previously deferred here as "pre-existing" — a thin defence given this change edits those exact lines. Mutation-verified: removing the guard yields `NullPointerException: Cannot invoke "java.util.List.isEmpty()" because "toolTrace" is null` and fails exactly the new `agentMode_nullTrace_doesNotThrow`. Worth stating precisely, since the review overstated it: **no production path returns a null trace** — both `ExecutionResult` construction sites pass a fresh list — so this is hardening and consistency, not a live bug fix.
* **`PersistenceMapperProducer` (rated High), not applied — the finding is incorrect.** It claims the producer omits a production `WRITE_DATES_AS_TIMESTAMPS` setting. That feature defaults to `true` in Jackson, and a hand-built `new ObjectMapper()` is untouched by `quarkus.jackson.*` either way, so the suggested `configure(..., true)` is a no-op. Verified by pointing `SerializationCustomizerInstantFormatTest` at the real producer: all 7 pass, including "Instant stays NUMERIC". **But there is a real defect underneath it, in the test rather than production.** That test's helper is documented as *"Exactly what PersistenceMapperProducer builds"* and is not — it adds a `configure(WRITE_DATES_AS_TIMESTAMPS, true)` the producer never makes, so it reconstructs the mapper instead of exercising it and cannot catch producer drift. That is precisely the drift `dc117cddc` had to revert once. Fix belongs on the branch that owns the file (#603): make the helper `return new PersistenceMapperProducer().persistenceMapper();` — confirmed green.

### Verification limits

CDI wiring **is validated locally**, by `./mvnw package -DskipTests`. Augmentation completes and ArC emits `AgentOrchestrator_Bean` + `AgentOrchestrator_ClientProxy` and `ConversationHistoryBuilder_Bean` + `ConversationHistoryBuilder_ClientProxy` into `target/quarkus-app/quarkus/generated-bytecode.jar`. That answers the only genuinely uncertain part of this change: these are the repo's first package-private `@ApplicationScoped` beans, and ArC proxies them without complaint. An unsatisfied injection point anywhere in the orchestrator's 29-dependency constructor would have failed the build outright, so resolution of the whole graph is confirmed too.

> **Worth remembering, because it cost three wrong claims in this entry's drafts:** build-time augmentation binds no sockets and needs no MongoDB. The local limitation recorded elsewhere in this file is about **tests that bind a loopback socket** (`*IT.java`, HTTP-server tests) — it does not extend to `quarkus:build`. `./mvnw package -DskipTests` is the right local gate for any CDI change and takes \~13s of augmentation.

`@ApplicationScoped` was chosen over `@Singleton` to match repo convention (AGENTS.md §4.1 rule 4) and because a normal scope's client proxy is the safer way to introduce a 29-dependency bean into a dense graph — lazy resolution tolerates ordering that a pseudo-scope resolves eagerly.

> **Correction (same-day review):** an earlier revision of this entry justified the choice by claiming the chain `LlmTask → AgentOrchestrator → IConversationService → IAgentFactory → lifecycle tasks` is cyclic and that a pseudo-scope could break it. That reasoning does not hold and should not be relied on: `ILifecycleTask` is only ever consumed through `Instance<ILifecycleTask>` / `Provider<ILifecycleTask>` (`LlmModule`, `WorkflowStoreClientLibrary`, `ApiCallsModule`), which is lazy and already breaks any cycle at that edge, and `AgentFactory` injects only `IAgentStoreClientLibrary` + `IDeploymentListener`. The conclusion stands; the stated reason was wrong.

***

## 🔍 More PR review follow-ups (2026-07-22)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

* **`UsageSnapshotSerializationTest.productionMapper()` stopped mirroring the real REST mapper.** It called `SerializationCustomizer.configureObjectMapper(...)` directly — which, since the persistence/REST mapper split earlier in this branch, builds the *persistence* recipe (no `Instant` override), not the REST/CDI one. The assertions still passed, because `costMonth` is a `YearMonth` with `@JsonFormat` on the field directly and is unaffected by the `Instant` `configOverride` — but the test's stated purpose ("pins the REST wire shape") was no longer true, and it provided zero coverage of the actual production mapper. Same class of defect fixed earlier in `SerializationCustomizerInstantFormatTest` (`a186dc903`): the fixture now builds the mapper via `new SerializationCustomizer(false).customize(mapper)`, matching how Quarkus actually constructs it.
* **Inline FQN in `DescriptorStoreTest`.** `new java.util.ArrayList<>()` where `java.util.List` was already imported. Added the top-level import; this file came in via the `origin/main` merge, not authored on this branch, but the convention applies regardless of origin.

***

## 🔀 Merge `origin/main` into `fix/orphan-scan-and-quota-defects` — changelog conflict resolution (2026-07-21)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

Brought the branch up to date with `origin/main`, which had picked up the `fix/descriptor-store-limit-semantics` PR (`limit=0` now means "unlimited" in `DescriptorStore.readDescriptors`, with a `MAX_RESULT_LIMIT` safety ceiling) via a background task spawned earlier in this effort. Both that PR and this branch touch `DescriptorStore.readDescriptors`, so `docs/changelog.md` conflicted at the top (both sides prepend); `DescriptorStore.java` and its test merged automatically without conflict — this branch's `includeDeleted` inclusion-flag fix and main's `resolveDescriptorLimit`/ceiling-warning logic touch disjoint parts of the same method. Kept both changelog blocks whole, this branch's newest-first on top, main's independent entry below.

## 🔍 PR review follow-ups — a vacuous test found and fixed (2026-07-21)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

Addressing CodeRabbit and Copilot review comments. One of them exposed a test that could not fail.

**The `MAX_PAGES` ceiling test was vacuous.** CodeRabbit noted the new page ceiling had no coverage. Adding a test made it pass immediately — but mutation-checking it (disabling the ceiling so the walk truncates silently) showed it *still* passed. Cause: the fixture left `agentStore.read` unstubbed, so the traversal NPE'd on a null config, marked the scan incomplete, and produced the expected 409 **for the wrong reason**. Stubbing the agent read so the ceiling is the only possible failure source, plus asserting the refusal message names it, makes the test load-bearing — the mutant now dies with "Expected WebApplicationException to be thrown, but nothing was thrown."

**Copilot's `WRITE_DATES_AS_TIMESTAMPS` finding: premise wrong, instinct right.** It claimed the produced `@PersistenceMapper` could change the on-disk shape. It cannot — Jackson enables that feature by default, so the producer already emitted numeric. But the real defect was next door: `SerializationCustomizerInstantFormatTest` *reconstructed* the producer instead of calling it, and set the flag itself — so it would have passed even if the producer were broken. The test now builds the mapper via `new PersistenceMapperProducer().persistenceMapper()`, and the producer states the flag explicitly. Relying on a library default for a persistence-format guarantee is precisely what `dc117cddc` was reverted for. Mutation-checked: flipping the producer to ISO now fails two tests.

**TOCTOU on the agent quota — documented, not fixed, and the earlier claim corrected.** CodeRabbit correctly flagged that concurrent deploys observing `count == limit - 1` all pass. An internal note had called this "self-correcting"; that was wrong — once over, the gate merely refuses further deploys until an undeploy brings the count down. The javadoc now states the bound honestly and explains why a per-tenant lock is *not* used: it would serialize within one JVM while the count spans the shared store and every node's registry, giving the appearance of a hard guarantee exactly where it would not hold. Accepted because deploys are rare admin operations and the gate's purpose — stopping runaway growth such as an LLM creating sub-agents in a loop — survives a small transient overrun.

**Log injection.** `sanitize(conversationId)` added to the new quota-denial log line in `RestAgentEngine`, matching the rest of the class (lines 331, 368, 389, 436).

**Not actioned:** the advisory note that the orphan endpoint blocks a request thread. Pre-existing, explicitly raised as advice rather than a blocker, and moving it to `AsyncResponse` with a polling status endpoint is a separate change.

***

## 🔢 LLM — stop discarding agent-mode token usage (2026-07-21)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

`AgentOrchestrator` sums `TokenUsage` across every model call in the tool loop and returns it on `ExecutionResult.responseMetadata()` — on both the live path and the resume path. `LlmTask` then read `.response()` and `.trace()` from that result and **never `.responseMetadata()`**, so agent-mode token accounting was computed and dropped on the floor. Only the legacy-chat and cascade branches surfaced theirs.

Net effect: any agent with tools enabled reported `{}` for `responseMetadataObjectName`, and no per-turn token figure existed for the paths that dominate real usage. This is the prerequisite for monthly cost metering — there was nothing to meter.

Both agent branches now read the metadata, and `executeResume` surfaces it the same way `executeTask` does (it previously built no metadata map at all).

### Known gap, deliberately documented rather than papered over

A turn that pauses for tool approval loses its pre-pause usage: `ToolApprovalRequiredException` escapes before the metadata is assembled and carries no usage, and `resumeToolLoop` starts a fresh accumulator. So a paused turn under-reports by its pre-pause segment. Closing that requires threading the step into `runToolCallLoop` so usage is written incrementally — out of scope here.

### Safety check

`responseMetadata` also feeds `applyResponseValidation`, which branches on `warning` and `streamingTimeout`. `AgentOrchestrator` puts **only** `tokenUsage` into the map (`AgentOrchestrator.java:470,826`), so both of those keys stay absent exactly as they were with the previously-empty map — validation behaviour is unchanged.

### Verification limits

Not unit-covered: `LlmTask` constructs its `AgentOrchestrator` internally rather than receiving it injected, so `executeIfToolsEnabled` cannot be stubbed to return a non-null result at the `LlmTask` unit level, and every existing `LlmTask` test therefore exercises the legacy branch. The change was verified by reading both sides of the contract and by the safety check above; all 184 `LlmTask` tests stay green. Making this properly testable means injecting `AgentOrchestrator` — a worthwhile refactor of an already 29-argument constructor, but a separate change.

***

## 🕐 Serialization — split the persistence mapper from the REST mapper; Instant is ISO-8601 on the wire (2026-07-21)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

⚠️ **REST contract change**, and it requires a companion EDDI-Manager fix (below).

Every `java.time.Instant` on every endpoint rendered as a **1970 date** in the Manager. With `quarkus.jackson.write-dates-as-timestamps=true` (`application.properties:174`) plus `JavaTimeModule`, an `Instant` serializes as fractional epoch **seconds** (`1719964800.123`), while clients call `new Date(value)`, which expects **millis**. Affected `nextFire`, `lastFired`, `pausedAt`, `createdAt`, `updatedAt`, transcript timestamps — essentially every timestamp in the UI.

The obvious fix — a `configOverride(Instant.class)` in the shared `configureObjectMapper` — is the trap. `JsonSerialization` `@Inject`s the **same CDI `ObjectMapper`**, and it backs `DocumentBuilder` for every Mongo write, every Postgres JSONB column, the backup/export writer, and the `{json:serialize}` Qute extension available to agent authors. That would change **on-disk formats**. Concretely, `GroupConversation.lastModified` is a persisted `Instant` that `GroupConversationStore` sorts **server-side** on: Mongo's BSON cross-type ordering ranks all Doubles before all Strings, and Postgres does `ORDER BY data->>'lastModified'` lexicographically — so old numeric rows and new ISO rows would interleave wrongly, silently, with no backfill. Commit `dc117cddc` ("keep numeric date format") already reverted a broader version of this once for breaking `findDueSchedules`.

So the two mappers are now separated:

* New `@PersistenceMapper` qualifier + `PersistenceMapperProducer`, built from the same `configureObjectMapper` recipe **without** the date override. `JsonSerialization` injects that.
* The `Instant` → ISO override moved into `SerializationCustomizer.customize()` — the REST/CDI path only, never the shared static.

### Design decisions

* **`java.util.Date` deliberately untouched.** It already emits epoch millis and its consumers (e.g. `DocumentDescriptor.lastModifiedOn`) are correct today; widening the override would break working paths. The wire therefore carries two encodings by design: `Instant` = ISO-8601 string, `Date` = epoch-millis number.
* **`application.properties:174` must stay.** Quarkus's own default for `write-dates-as-timestamps` is `false`, so that line is what *prevents* Quarkus disabling the feature globally. Deleting it as "redundant documentation" would flip persistence to ISO — the exact break this change avoids.
* **A qualified CDI producer, not `new ObjectMapper()` inside `JsonSerialization`.** Keeps the persistence mapper a first-class, overridable bean and leaves the existing direct-construction test fixtures working unchanged.
* **Deserialization is unaffected in both directions** — `InstantDeserializer` dispatches on the JSON token type, not the shape hint — so rows written numerically still parse.

### Required companion change (EDDI-Manager repo)

The Schedules dashboard sorts arithmetically: `(a.nextFire ?? 0) - (b.nextFire ?? 0)`, which yields `NaN` on ISO strings, leaving the "next fire" tile showing an arbitrary schedule. Change it to `new Date(a.nextFire) - new Date(b.nextFire)`, which works with **both** encodings and can therefore land before or after this commit. Every other consumer (`new Date(x).toLocaleString()`) becomes correct automatically. The vendored bundle under `META-INF/resources/assets/` is a build artifact and was deliberately not hand-edited.

### Verification limits

CDI wiring for the new qualified producer **cannot be validated locally** — `quarkus:build` augmentation needs a loopback socket, which this environment refuses, and the repo has only one non-IT `@QuarkusTest`. CI is the gate for that specific aspect. The format behaviour itself is fully covered by unit tests.

### Tests

New `SerializationCustomizerInstantFormatTest` (7 tests) asserts both halves: the REST mapper emits ISO and keeps `Date` numeric; the persistence mapper stays numeric and **does not inherit** the REST override despite sharing `configureObjectMapper`; and both still read the old numeric form.

***

## 🎫 Tenancy — enforce `maxAgentsPerTenant` on deploy (2026-07-21)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

`TenantQuota.maxAgentsPerTenant` was persisted end-to-end by all three stores and round-tripped through the REST API, but **nothing ever read it** — `TenantQuotaService` had no agent method at all. Operators could set the limit and get silent no-enforcement.

New `TenantQuotaService.checkAgentQuota(tenantId, currentDistinctAgents)` gates `RestAgentAdministration.deployAgent`, denying with `QuotaExceededException` → **429** `{"error":"quota_exceeded"}` via the existing mapper.

### Design decisions

* **A read-only gate, not an atomic counter.** The deployed-agent count is a *stock* derived by counting current deployments, not a per-window *flow*. A stored counter would drift irrecoverably: the 10s re-deploy sweep, the 24h old-version undeploy, `TeardownAgentTool`, `GroupConversationService` and the lazy re-deploy on first use all add or remove deployments without passing any acquire/release point. Modelled on `checkCostBudget`, so no `ITenantQuotaStore` method was added and the three store implementations are untouched.
* **Placement is forced, not stylistic.** The gate sits between the null-checks and the `try`. Inside the `try`, `catch (Exception) → InternalServerErrorException` would convert the 429 into a **500**; inside the submitted `Callable` it runs off the request thread and could never produce a status code at all.
* **Counts distinct agent ids**, so redeploys and version bumps are free — required because the old-version undeploy sweep legitimately keeps two versions of one agent deployed while the previous drains.
* **The count unions persisted rows with live agents, and needs both.** `autoDeploy=false` never writes a `deployed` row, but the in-memory deploy is unconditional — so a rows-only count let a caller deploy unlimited agents with one query parameter. Those agents are genuinely live: `getLatestReadyAgent` serves them without consulting the store, and `ConversationService.getAgent` lazily re-deploys them after a restart, so the bypass is **durable**, not merely transient. Conversely, a live-agents-only count would be per-JVM and would miss other cluster nodes.
* **Fails open.** A store outage must not block deployments; the denial is logged and metered either way.
* **The two CDI callers surface the reason.** `AgentSetupService.deployAndWait` and `McpAdminTools.deployAgent` call the bean directly, so the exception mapper never runs — both now return the quota reason instead of "check server logs", which an agent designer (or a model driving `create_sub_agent`) cannot act on and would retry in a loop.
* **Scheduled sweeps stay ungated by construction** — they call `IAgentFactory` directly, so lowering the limit never undeploys anything. The limit gates *new* agents only.

### Not addressed

Single-tenant only: the gate resolves `getDefaultTenantId()`, exactly as the conversation and api-call quotas already do. This is a deployment-wide cap, not per-organisation, until the multi-tenancy plan's Phase 1 lands. Do not describe it as multi-tenant enforcement.

### Tests

New `RestAgentAdministrationQuotaTest` (8 tests): denial throws `QuotaExceededException` and submits nothing to the runtime; two versions of one agent count once; redeploy skips the quota entirely; **live agents with no `deployed` row still count** (the loophole); no double counting; non-`READY` agents skipped; null agent ids skipped; store error fails open. Mutation-checked — reverting to a rows-only count fails exactly the loophole test. `TenantQuotaServiceTest` gains 5 tests including parity with `checkCostBudget` on not inflating the `allowed` counter. Both existing `RestAgentAdministration*Test` classes updated for the new constructor arg in the same commit.

***

## 🧹 Orphan admin — `includeDeleted` becomes a true inclusion flag; purge default flipped (2026-07-21)

**Repo:** EDDI (`fix/orphan-scan-and-quota-defects`)

⚠️ **REST behaviour change on `DELETE /administration/orphans`.**

`DescriptorStore.readDescriptors` treated `includeDeleted` as an **equality** filter — `eq("deleted", includeDeleted)` — so `includeDeleted=true` matched *only* soft-deleted descriptors rather than adding them to the live ones. Two consequences:

* The parameter did not mean what its name, its `@Parameter` text, and `docs/deployment-management-of-agents.md` all said it meant.
* The shipped Manager scans with `includeDeleted=false` and purges with `includeDeleted=true`, so **the set shown to the user and the set deleted were disjoint**. The UI listed live orphans and then purged soft-deleted ones.

`true` now drops the `deleted` constraint entirely (live **and** soft-deleted); `false` constrains to live only. Every other caller in `src/main` passes a literal `false` — verified by enumerating all \~35 call sites — so their behaviour is bit-for-bit unchanged.

**`purgeOrphans`'s `@DefaultValue` flipped from `true` to `false`.** Left at `true`, the semantics fix would have made the parameterless `DELETE` dramatically *more* destructive — combined with the page-walk fix two commits earlier, from "purge ≤200 already-soft-deleted rows" to "permanently wipe every unreferenced resource, unbounded". Flipping the default makes the bare call the conservative one and matches `scanOrphans`, so a scan and a purge with no parameters now describe the same set.

### Client impact

A client that relied on the old default now purges **less**: live-but-unreferenced orphans only, not soft-deleted ones. Pass `includeDeleted=true` explicitly to also purge soft-deleted resources. EDDI-Manager should send the flag explicitly, matching whichever set it is displaying.

### Tests

`DescriptorStoreTest` gains two tests capturing the actual `QueryFilters` and asserting the `deleted` constraint is present for `false` and **absent** for `true`; mutation-checked by restoring the equality filter, which fails the second. `RestOrphanAdminSafetyTest` gains two tests pinning the flag's propagation. All 86 tests across every descriptor-store consumer green.

***

## 🔍 Review follow-ups on the orphan/tenancy fixes (2026-07-21)

**Repo:** EDDI (`claude/eddi-backend-manager-coverage-0598fe`)

Self-review of the three preceding commits found three defects, all fixed here.

* **The 409 refusal carried no body.** `purgeOrphans` threw `new WebApplicationException(String, Response.Status)`, whose response has **no entity** — so an operator saw a bare 409 and the reason existed only in the server log. Confirmed empirically by mutation: reverting to that constructor makes the new `hasEntity()` assertion fail. Now builds the `Response` explicitly with `{"error":"incomplete_scan","message":…}`.
* **Broken javadoc link.** The page-walk javadoc still referenced `buildReferencedUrisSet()`, renamed to `scanReferencedUris()` in the same commit. Also updated three now-inaccurate `@DisplayName`s in `RestOrphanAdminBranchTest`.
* **Postgres had no at-limit test.** The `>` → `>=` change was mutation-verified on Mongo but not Postgres. Added `PostgresTenantQuotaStoreTest.exactlyAtLimit` and mutation-checked it too.

Also documented why a `MAX_PAGES` trip during orphan *collection* is safely swallowed while the same failure in the *reference* scan blocks the purge: a type that cannot be enumerated yields fewer delete candidates (under-delete), whereas a missing reference promotes a live resource to "orphan" (over-delete).

**Verification:** full clean suite `tests=11273 failures=8 errors=288 skipped=3` across the same 15 classes as the pre-change baseline — all environmental (loopback sockets, network egress, model downloads). Zero regressions. Checkstyle clean on every changed file; `javadoc:javadoc` builds.

***

## 🚦 Engine — return 429 instead of 500 when the api-call quota denies a turn (2026-07-21)

**Repo:** EDDI (`claude/eddi-backend-manager-coverage-0598fe`)

`ConversationService.say`/`sayStreaming` throw `QuotaExceededException` when `acquireApiCallSlot()` denies, and `QuotaExceededExceptionMapper` maps that to **429** with `{"error":"quota_exceeded"}` and `Retry-After: 60`. But `say()` is resumed through a JAX-RS `AsyncResponse`, so the exception is caught inside `RestAgentEngine.sayInternal` and never reaches the `@Provider` mapper. Its catch chain lists `AgentMismatchException`, `AgentNotReadyException`, `ConversationEndedException`, `ConversationAwaitingApprovalException`, `ProcessingRestrictedException` and `ResourceNotFoundException` — but not `QuotaExceededException` — so the denial fell through to `catch (Exception e)` and surfaced as **500 "An internal error occurred"**.

Only the *conversation-start* quota ever produced a real 429: `startConversationWithContext` is synchronous and its narrower catch block lets the exception escape to the mapper. The per-minute API rate limit — the quota an operator is most likely to actually hit — was indistinguishable from a server fault, so clients had no way to back off correctly.

Added an explicit `catch (QuotaExceededException)` that resumes with the same status, body and `Retry-After` header as the mapper, so both quota denials look identical on the wire.

### Design decisions

* **Mirror the mapper rather than re-throw.** There is no way to route an already-captured async exception back through the provider chain, so the branch duplicates the mapper's three-line response. Kept adjacent constants and a comment so the two stay in sync.
* **Streaming is not covered here.** `sayStreaming` throws the same exception, but by then the SSE response has already committed HTTP 200, so no status code can be sent — it currently emits a generic `error` event. Giving that event a distinguishable quota type is a separate, client-visible change and is tracked, not slipped in.

### Tests

New `RestAgentEngineTest.quotaExceeded` asserts 429, the `Retry-After: 60` header and the exact entity map. Mutation-checked: replacing the new catch with an unrelated exception type makes it fail with `InternalServerError An internal error occurred` — reproducing the original bug — so the test is not vacuous. All 43 `RestAgentEngineTest` tests green.

***

## 💵 Tenancy — align the at-limit cost comparison and fix costMonth JSON shape (2026-07-21)

**Repo:** EDDI (`claude/eddi-backend-manager-coverage-0598fe`)

Two independent defects in the tenant cost-budget surface.

**1. The two production quota stores disagreed with the gate at exactly the limit.** `TenantQuotaService.checkCostBudget` denies on `currentCost >= limit`, and `InMemoryTenantQuotaStore.tryAddCost` matches it — but `MongoTenantQuotaStore` and `PostgresTenantQuotaStore` used `totalCost > limit`. At exactly the budget the pre-call gate denied while post-call accounting allowed. `docs/changelog.md` records the in-memory store being deliberately moved to `>=` for this reason; the two DB stores were never updated. Both now use `>=`.

Verified by mutation: reverting the Mongo comparison to `>` makes the new `exactlyAtLimit` test fail with `expected: <false> but was: <true>`, so the test is not vacuous.

**2. `UsageSnapshot.costMonth` serialized as a JSON array.** Under `quarkus.jackson.write-dates-as-timestamps=true` (`application.properties:174`) Jackson's `YearMonthSerializer` takes its `useTimestamp` branch and emits `[2026,7]` instead of `"2026-07"`. Both stores already persist the value as an ISO string (`YearMonth.toString()` / `YearMonth.parse`, never through Jackson), so the REST representation was the only place that disagreed. Annotated the record component with `@JsonFormat(shape = JsonFormat.Shape.STRING)`.

### Design decisions

* **Annotation, not a mapper-wide override.** A `configOverride(Instant.class)` on `SerializationCustomizer` would have fixed every temporal field at once, but that customizer's mapper is *also* the persistence mapper — `JsonSerialization` `@Inject`s the same CDI `ObjectMapper`, which backs `DocumentBuilder` for every Mongo write and Postgres JSONB column. Changing it would alter on-disk formats. Concretely, `GroupConversation.lastModified` is an `Instant` persisted through it and `GroupConversationStore` sorts *server-side* on that field, so mixed old-numeric/new-string rows would sort wrongly and silently in both backends. Commit `dc117cddc` ("keep numeric date format") already reverted a broader version of this change once, for breaking `findDueSchedules`. The wider Instant-format cleanup is therefore left out and tracked separately — it needs the persistence mapper decoupled from the REST mapper first, plus a coordinated EDDI-Manager change (the Schedules dashboard sorts `nextFire` arithmetically, which yields `NaN` on ISO strings).
* **`costMonth` is safe in isolation** precisely because neither store round-trips `YearMonth` through Jackson — verified before changing it.

### Tests

New `UsageSnapshotSerializationTest` asserts the shape on the *real* record (not a stand-in holder) through a mapper wired exactly as production is, plus a round-trip. New `MongoTenantQuotaStoreTest.exactlyAtLimit`. All 131 tenancy tests green.

***

## 🛡️ Orphan admin — fix page-walk truncation and refuse to purge on an incomplete scan (2026-07-21)

**Repo:** EDDI (`claude/eddi-backend-manager-coverage-0598fe`)

Two defects in `RestOrphanAdmin` that together could permanently delete live configuration.

**1. The descriptor page walk never advanced past page 0.** `readAllDescriptors` advanced its cursor with `index += batch.size()`, but `DescriptorStore.readDescriptors` treats that argument as a *page* index (`skip = index * effectiveLimit`, `DescriptorStore.java:61`). The second iteration therefore asked for page 200 — `skip = 40 000` — which always came back empty, so every store type was silently truncated at 200 rows. Fixed to `pageIndex++`.

The dangerous half of this was not the orphan list but `buildReferencedUrisSet`: the *referenced* set was truncated the same way, so on any deployment with more than 200 agents or 200 workflows, live in-use resources were classified as orphans — and `purgeOrphans` deletes with `permanent=true`, which is `deleteAllPermanently(id)` (current document *and* all history).

A `MAX_PAGES` ceiling (100 pages / 20 000 rows per type) now bounds the walk, since the scan is a synchronous one-read-per-descriptor traversal on a blocking JAX-RS method. Hitting the ceiling raises `ResourceStoreException` rather than truncating silently — a truncated scan must not be used to decide what to delete.

**2. The reference scan failed open.** Every read error while building the referenced-URI set was swallowed (two at `debug` level), and the partially-built set was returned as if complete. Because that set is what *protects* a resource, each swallowed error made **more** things look orphaned. One unreadable workflow could mark every rules/apicalls/output/llm/property/dictionary/parser resource in the database as an orphan.

`scanReferencedUris()` now returns a `ReferenceScan` record carrying a `complete` flag, and `purgeOrphans` refuses with **409 Conflict** when the scan is incomplete, naming the cause. `scanOrphans` (read-only) still returns its best-effort report — only the irreversible path is gated. A `ResourceNotFoundException` is explicitly *not* treated as incompleteness: a descriptor whose resource is gone is a genuine orphan.

### Design decisions

* **Fail closed on the destructive path only.** Read-only callers tolerate a partial picture; a permanent delete may not.
* **Ceiling raises rather than truncates.** Silent truncation is what made the original bug invisible.
* **No change to `includeDeleted` semantics in this commit.** It is currently an equality filter (`Filters.eq("deleted", flag)`), so `scanOrphans` (defaults to `false`) and `purgeOrphans` (defaults to `true`) operate on *disjoint* sets. Fixing that is a real behaviour change to an irreversible endpoint and is deliberately left to its own commit — with the page-walk now complete, redefining the flag without also flipping the `true` default would turn the default `DELETE /administration/orphans` from "purge ≤200 already-soft-deleted rows" into "permanently wipe every unreferenced resource, unbounded".

### Tests

New `RestOrphanAdminSafetyTest` (7 tests): page index advances by 1 and a second page is actually requested; walk stops on the first partial page; an unreadable Agent aborts the purge with 409 and deletes nothing; a *missing* Agent resource does not abort; a complete scan purges normally; `scanOrphans` tolerates an incomplete reference set and never deletes; a workflow referenced by an Agent is never purged. Fixtures use 24-char hex ids because `RestUtilities.extractResourceId` returns a null id otherwise, which would make the assertions pass vacuously. All 27 pre-existing orphan tests unchanged and green.

***

## 🐛 Fix: `readDescriptors(limit = 0)` silently returned only 20 descriptors (2026-07-21)

**Repo:** EDDI (`fix/descriptor-store-limit-semantics`)

### Summary

`DescriptorStore.readDescriptors` resolved its limit with `(limit == null || limit < 1) ? 20 : limit`, so a caller passing `0` to mean "give me everything" silently received the first 20 rows. The same `limit < 1 ? 20` fallback was duplicated in `MongoResourceStorage.findResources`, `PostgresResourceStorage.findResources`, and `ResourceFilter.readResources` — four independent copies of a magic default, none of them documented.

Three production call sites were affected in a user-visible way:

* **`PromptSnippetService.loadAllSnippets`** — a deployment with more than 20 prompt snippets never got the rest into the `{snippets.*}` template namespace. Silent: templates just rendered empty.
* **`RestExportService.exportSnippets`** — agent export dropped any referenced snippet outside the first 20, producing an incomplete ZIP that imports without error.
* **`RestAgentStore.populateCapabilityRegistry`** — only the first 20 agents were scanned for capabilities at startup, so capability-based discovery/delegation silently missed agents.

Three further call sites (`ChannelTargetRouter`, `ChannelConnectorMigration`, `RestChannelIntegrationStore`) passed a magic `1000` to dodge the cap; the channel-uniqueness check among them would have let a duplicate `channelId` through past that many configs.

### Fix — explicit sentinel plus a hard ceiling

Option (a) from the two candidates, because the storage layer supports a bounded unlimited query cleanly and paging every caller would have been six copies of `RestOrphanAdmin`'s batch loop:

* **`IDescriptorStore.NO_LIMIT` (`0`)** and **`DEFAULT_LIMIT` (`20`)** define the contract in one place, with `resolveDescriptorLimit(Integer)`: `null` → default page, `<= 0` → unlimited, `> 0` → honoured.
* **`IResourceStorage.MAX_RESULT_LIMIT` (`10_000`)** is the hard safety ceiling, applied through the shared static `IResourceStorage.resolveLimit(int)` that both backends now call — the two implementations can no longer drift apart.
* **Truncation is no longer silent**: `DescriptorStore` logs a WARN naming the descriptor type when a result set hits the ceiling. The original defect was not the number 20, it was that nothing said anything.
* All six "give me everything" call sites now pass `IDescriptorStore.NO_LIMIT` instead of `0` or `1000`, so intent is readable at the call site. `RestOrphanAdmin` keeps its explicit 200-row batch loop — it genuinely pages.

### Key Design Decisions

* **`null` and `0` deliberately mean different things.** `null` is "caller expressed no opinion" → default page; `0` is "no limit". Collapsing them would have made `?limit=` omission return everything.
* **Redefining `0` is safe for the REST API** because every REST endpoint already declares `@QueryParam("limit") @DefaultValue("20")`. JAX-RS, not the store fallback, supplies the REST default, so no endpoint's behaviour changes when the parameter is omitted. Only internal callers and an explicit `?limit=0` are affected — and `?limit=0` now means what it reads like.
* **Ceiling over true unbounded**: an unbounded query on a large `descriptors` collection is a memory risk, and `readDescriptors` issues one `read()` per descriptor. 10,000 is high enough that no realistic deployment hits it, and the WARN makes it loud if one does.
* **Legacy `mongo/ResourceFilter` updated too**, so both descriptor-store implementations obey one contract rather than diverging.
* **The ceiling warning sanitizes `type`.** Self-review caught that the new WARN logged `type` unsanitized — and `type` reaches `readDescriptors` straight from `@QueryParam("type")` on `IRestDocumentDescriptorStore`. That is the CWE-117 log-injection pattern that commit `d71de742` remediated across 13 files (the commit that also last touched this very method), so the new log line now uses the `LogSanitizer.sanitize` helper that commit introduced. Reachability is hard (10,000 matching descriptors), but CodeQL taint analysis is flow-based, not reachability-gated, and the repo's convention is to sanitize unconditionally.
* **The warning states impact, not just cause.** Its reader is an operator who cannot "page" anything — the actionable fact for them is that the returned list is incomplete and dependent features are missing entries, so the message leads with that.

### Review follow-ups (Copilot + CodeRabbit)

* **`index` javadoc overstated the contract.** It claimed any `index > 0` with `NO_LIMIT` "yields an empty list". It does not: the effective limit is the ceiling, so `index=1` returns rows 10,000–20,000. The claim was only true for collections smaller than the ceiling — the common case mistaken for the contract. Reworded to say a non-zero index pages in ceiling-sized chunks.
* **Integer overflow in the legacy `ResourceFilter` skip.** `index * limit` was int arithmetic; raising the effective limit from 20 to 10,000 dropped the overflow threshold from `index > ~107M` to `index > ~214,748`, where the skip goes negative. Now uses the same `long` cast + `Math.min` guard that `d71de742` added to `DescriptorStore.readDescriptors`, so both paths match. Reachability is low (this is the legacy store path), but the fix is three lines and removes a CodeQL-shaped pattern.
* **Removed a duplicate test.** The added `nullLimitUsesDefaultPageSize` re-tested exactly what the pre-existing `defaultsLimitWhenNull` already covered. Deleted it and updated the pre-existing test to assert against `DEFAULT_LIMIT` instead of a literal `20`, keeping coverage and dropping the magic number.

### Verification

* `./mvnw clean compile` — clean
* 976 unit tests across the datastore package and every affected call site — 0 failures, 1 pre-existing skip
* New tests pin the semantics: `null` → 20, `NO_LIMIT` → ceiling (with an explicit `assertNotEquals(DEFAULT_LIMIT)` regression guard), negative → ceiling, oversized → clamped, `skip == index * effectiveLimit`, results not post-truncated; plus helper-level tests and backend-level assertions (Mongo `iterable.limit(...)`, Postgres `LIMIT` in the generated SQL)
* Two pre-existing tests asserted the old `limit=0 → 20` behaviour (`ResourceFilterTest`, `MongoResourceStorageBranchTest`) and were updated to the new contract — they were pinning the bug
* `./mvnw formatter:format` + `./mvnw validate` clean

***

## 🔎 Auto-approve workflow — Copilot pagination review comment is a false positive (2026-07-20)

**Repo:** EDDI (`chore/auto-approve-copilot`)

Copilot flagged the CI-gate's `github.paginate(github.rest.checks.listForRef, …)` call, claiming it returns page objects rather than check-runs (so the gate would never approve) and suggested adding a `(response) => response.data.check_runs` map function. Verified against the **exact** runtime this workflow pins (`actions/github-script@v7.1.0` → `@octokit/plugin-paginate-rest@9.2.2`): the claim is backwards. Octokit auto-normalizes list responses that carry `total_count` (check-runs do), so `paginate()` already returns the flattened `check_runs` array — the current code is correct and yields `APPROVE`. The suggested map function would read `.check_runs` off that already-flattened array → `undefined` → `TypeError: Cannot read properties of undefined (reading 'name')`, crashing the job. Both outcomes were reproduced with a mocked-HTTP harness on the pinned versions. Left the call unchanged and added an inline comment documenting why no map function is used, so the concern isn't re-raised.

***

## 🔀 Merge `origin/main` into `chore/auto-approve-copilot` — conflict resolution (2026-07-20)

**Repo:** EDDI (`chore/auto-approve-copilot`)

Brought the auto-approve-workflow branch up to date with `main` to clear the open PR's merge conflict. The branch adds a single new file (`.github/workflows/auto-approve-copilot.yml`), so the merge pulled in all of main's post-branch work — including the HITL framework, multi-model cascade enterprise hardening, error-handling recovery, and group-conversation follow-ups — with a single textual conflict: `docs/changelog.md`. Both sides had prepended entry blocks; both kept whole — this merge entry plus the branch's auto-approve entry on top, main's newer history below. No source files conflicted: the branch modifies no `.java` that main also touched, so the merged tree's source is byte-identical to `origin/main`.

***

## ⚙️ Chore: Auto-approve workflow for Copilot-reviewed PRs (2026-07-08)

**Repo:** EDDI (`chore/auto-approve-copilot`)

### Summary

New GitHub Actions workflow (`.github/workflows/auto-approve-copilot.yml`) that converts a clean GitHub Copilot code review into an actual PR approval once CI is green, so PRs from trusted authors can satisfy a required-review gate without a human reviewer on hand. Built as a self-owned workflow using `actions/github-script` (SHA-pinned, v7.1.0) instead of the marketplace `strspc-pr-review` action, which was rejected for low provenance (2 stars, no contributor data) and requiring a broad classic PAT — incompatible with this repo's OpenSSF-Gold security posture.

### Key Design Decisions

* **Self-owned, not marketplace**: All logic lives in-repo; the only external dependency is GitHub's own `actions/github-script`. Bot credential is a fine-grained PAT scoped to one repo.
* **Public-repo safety**: Only PRs from authors with `author_association` in `{OWNER, MEMBER, COLLABORATOR}` are eligible — external contributors are never auto-approved.
* **`workflow_run` trigger, not `check_suite`**: GitHub suppresses `check_suite`/`check_run` events for check suites produced by GitHub Actions. Since EDDI's CI runs entirely as Actions jobs, a `check_suite: [completed]` trigger would never fire. `workflow_run: { workflows: ["CI/CD"], types: [completed] }` is used instead.
* **Single-token design**: All API calls (reads + approval) use the single `github` object authenticated via `github-token` set to the bot PAT. Avoids `getOctokit`/`require('@actions/github')` differences between `actions/github-script` versions.
* **Graceful skip when unconfigured**: `env.BOT_TOKEN` pattern avoids the `github-token: required` error when the secret doesn't exist — the step is simply skipped, no red check.

### Gates (all must pass for approval)

1. PR is open, not draft, targets `main`
2. Author is trusted (`OWNER`/`MEMBER`/`COLLABORATOR`)
3. No outstanding `CHANGES_REQUESTED` from anyone
4. Copilot has reviewed the **current** head commit (not stale)
5. Copilot's latest review has zero inline/line comments
6. All CI check-runs on head commit (excluding this workflow's own check) are completed and successful
7. Bot has not already approved at this exact head SHA (idempotency)

### Known Scope Limitations (documented, accepted)

* CI gate checks only check-runs, not legacy commit statuses. EDDI's CI only emits check-runs (confirmed via `ci.yml`); branch protection remains the actual merge gate regardless.
* "Clean Copilot review" is defined as zero inline comments; a summary-only body with concerns in prose would still pass. This is an intentional proxy.

### Verification

* YAML syntax validated (`yaml.safe_load`)
* Embedded JavaScript validated (`node --check` wrapped in async function)
* SHA pin format matches repo convention (`sha # tag` comment)
* Adversarial code review (3 dimensions × 2 agents, 7 findings, all addressed or documented)

### One-Time Manual Setup Required

A repo admin must: (1) create/designate a bot GitHub account with write access, (2) generate a fine-grained PAT scoped to this repo with `Pull requests: R/W`, `Checks: Read`, `Metadata: Read`, (3) store it as repo secret `AUTO_APPROVE_BOT_TOKEN`, (4) set repo variable `AUTO_APPROVE_BOT_LOGIN` to the bot's username.

***

## 🔁 Fix: PR-review response — interrupted streaming, FQN imports, changelog accuracy (2026-07-20)

**Repo:** EDDI (`feat/error-handling-recovery`)

Four review comments on [PR #593](https://github.com/labsai/EDDI/pull/593) (CodeRabbit + github-code-quality); all valid, all fixed. Two turned out to be broader than reported.

### 1. An interrupted streaming attempt was reported as a success (Major)

When `latch.await(...)` threw `InterruptedException`, neither `timedOut` nor `errorRef` was set, so execution fell through both guards to `break` and returned `new StreamingResult(responseText, metadata)` — an empty string presented as a completed answer, with no signal to the caller.

Corroborating evidence the review did not cite: `AgentOrchestrator` already treats a set interrupt flag as a **hard abort** (its test describes the scenario as "simulate a cascade per-step timeout cancel"). The streaming executor was therefore contradicting an established convention inside the same subsystem — a cancelled step could be accepted as a real, empty answer, and on the cascade's last step it would win outright.

Interruption is now a distinct outcome: `streamingInterrupted` is recorded in the metadata, the salvaging (`LlmTask`) path returns whatever text arrived with a `streaming_interrupted_partial` warning, and every other path throws. It is **never retried** — a retry would ignore the very cancellation being signalled. Four tests, each observed failing first.

### 2. Inline fully-qualified names in `LifecycleManagerTest` (Minor)

Flagged on one line; the file actually had **eight** — four `IAuditEntryCollector` and four `ConversationEventSink`. All replaced with two top-level imports per AGENTS.md §4.7.

### 3. The full-suite failure totals did not add up (Minor)

Correct, and the fault was in the measurement rather than the prose: the categorisation was run over `target/surefire-reports` **without a preceding `clean`**, so it also swept up XML from earlier *targeted* runs. The per-bucket figures were therefore drawn from a superset of the run they were attributed to. The numbers have been withdrawn (with an explicit correction note in the merge entry) rather than quietly adjusted, since the original claim was already pushed.

Re-measured from a genuinely clean run of the final code: **11,658 tests, 8 failures, 287 errors, and zero assertion failures.** Every failing test case in the surefire XML carries a blocked-loopback or socket-selector message; not one is a code assertion.

### 4. Useless parameter in the new error-path test helper (Note)

`executor(ChatModelRegistry, ITemplatingEngineStub)` never used its second argument, and the `ITemplatingEngineStub` marker interface existed only to make that argument look intentional — introduced in the previous commit as a "readability" device that conveyed nothing. Parameter and interface removed, all seven call sites simplified to `executor(registry)`.

One caveat, stated rather than papered over: the XML yields 301 distinct failing test cases against the console's 295 failures+errors, and that \~6 gap is unexplained (it is not reruns, not suite-level entries, and not skipped-plus-failed). The load-bearing claim deliberately does not depend on the count — "no entry in this set is an assertion failure" is a property of the set, unaffected by how its members are tallied. CI remains the source of truth for the socket- and Docker-dependent suites.

***

## 🔧 Fix: close the response-validation gaps between streaming, cascade and validation (2026-07-20)

**Repo:** EDDI (`feat/error-handling-recovery`)

Clears the four follow-ups recorded in the entry below. Each was pre-existing, and together they meant `responseValidation` — a headline feature of this PR — silently did nothing on the paths users are most likely to run it on. Every fix is TDD'd: the test was written first and observed failing.

### 1. Streaming now derives `warning` from `finishReason`

`LegacyChatExecutor` maps `LENGTH` → `truncated` and `CONTENT_FILTER` → `content_filter`; the streaming executor captured `finishReason` but never derived the warning, so `onTruncation` and `onContentFilter` could **never** fire on a streaming task. `buildMetadata` now mirrors the buffered executor. A later timeout/error warning deliberately overwrites it — a transport failure is the more urgent signal.

### 2. The cascade now carries the winning step's validation metadata

`CascadeResult` and the internal `StepResult` carried only `tokenUsage`; the producing executor's `warning`/`streamingTimeout`/`finishReason` were dropped on the floor. With `modelCascade` **and** `responseValidation` both enabled, only `onEmpty` and `onRefusal` could fire — a truncated or content-filtered cascade answer reached the user even with `action: "error"` configured. Both records gained a `responseMetadata` component, threaded from the legacy, streaming and agent-mode step paths, and merged into `responseMetadata` in `LlmTask`.

### 3. A timed-out live-streamed final step no longer wins

`executeCapturing` returns the (possibly empty) partial text on timeout instead of throwing, so a timed-out final step was accepted on the same footing as a real answer — and being last, it beat a good earlier response. A mid-stream *error* already fell back correctly; a *timeout* did not. A step whose metadata reports `streamingTimeout` is now treated as failed: it never becomes `bestSoFar`, it escalates when it is not the last step, and on the last step it falls back to `bestSoFar` (marked `streamedLive` so `LlmTask` does not re-emit different text over tokens the client already received).

Enabling this required `executeCapturing` to honour the task's `streamingTimeoutSeconds`, which it previously ignored by passing `task = null` — so the cascade was always pinned to the 120s default. The task's **retry** config is still deliberately not applied on the cascade path: the cascade owns escalation, and retrying inside a step would multiply spend against the very model it is about to escalate away from.

### 4. Restored the lapsed cascade error-path coverage

New `CascadingModelExecutorErrorPathTest` — written against the current implementation rather than restored verbatim — covering what was lost when `CascadingModelExecutorExtendedTest` was deleted and `CascadingModelExecutorCoverageTest` was rewritten on `main`: last-step timeout with and without a `bestSoFar`, timeout escalation firing `onCascadeEscalation("timeout")`, `LifecycleException` and plain-`RuntimeException` cause handling, aggregated all-steps-failed errors, and `enableInAgentMode=false` never consulting the orchestrator.

### 5. Cascade failures are now diagnosable (found while writing #4)

Two of the restored tests failed for a reason the tests were right about: the retry wrapper throws a generic `"Chat model execution failed after N attempts"`, and the cascade recorded only `e.getMessage()`. So a fully failed cascade reported `Step 0 (cheap): Chat model execution failed after 1 attempts; Step 1 (expensive): …` — byte-identical whether the cause was a rate limit, an auth failure, a malformed request or a network outage, in both the thrown message and the audit trace. A new `describeFailure()` appends the root-cause message (bounded cause-chain walk, so a cyclic chain cannot hang the error path).

### Verification

Full LLM module: 2,260 tests, **zero assertion failures** — every reported failure/error message is a blocked-loopback or socket-resource error from this sandbox, none a code assertion. All 73 cascade tests and 33 streaming-executor tests green. (Counts of each bucket are deliberately not quoted here; see the correction note in the merge entry below for why the earlier per-bucket figures were unreliable.)

### Design decisions

* **Metadata threaded as a record component, not a side channel.** `CascadeResult` is the cascade's public contract with `LlmTask`; anything the caller must validate belongs on it rather than in a mutable out-parameter.
* **A timed-out step is a failed step, not a low-confidence one.** Demoting it via confidence would still let it win when no earlier step scored higher; excluding it from `bestSoFar` outright is what makes the fallback correct.
* **Retry stays off inside cascade steps.** Escalation is the cascade's retry mechanism; stacking both multiplies cost in a way no config expresses.

***

## 🐛 Fix: two latent retry-loop defects in `StreamingLegacyChatExecutor` (2026-07-20)

**Repo:** EDDI (`feat/error-handling-recovery`)

Surfaced by an adversarial post-merge review of the streaming executor (see the merge entry below). Both defects pre-date the merge — they came in with this branch's retry loop, not with the conflict resolution — but both live inside the method the merge rewrote, and both are squarely in this PR's own subject area (error handling and recovery), so they are fixed here rather than deferred.

### 1. A failed attempt's metadata leaked into a successful retry

`metadata` is declared outside the retry loop and mutated inside it. If attempt 1 timed out with no tokens (setting `streamingTimeout=true`) and attempt 2 succeeded, the successful response was returned **with the stale `streamingTimeout` flag still attached**. `LlmTask.applyResponseValidation` reads that flag and fires `responseValidation.onStreamingTimeout` — so with `action: "fallback"` a perfectly good answer was silently replaced by the "I wasn't able to generate a complete response" string, and with `action: "error"` the turn threw. The loop body now clears the map at the start of each attempt, so each attempt reports only its own outcome.

### 2. `maxAttempts <= 0` skipped the model entirely and returned a null response

`RetryConfiguration.setMaxAttempts` does no clamping, so a config of `retry: {maxAttempts: 0}` — a natural way to write "don't retry" — made `for (attempt = 1; attempt <= 0; …)` never execute. The model was never invoked, `responseText` stayed null, and `StreamingResult(null, …)` propagated into `LlmTask` and on into a `TextOutputItem(null, 0)`: no exception, no log, a completely silent turn. Now clamped with `Math.max(1, …)` — "no retries" means one attempt.

Deliberately **not** clamped in `RetryConfiguration.setMaxAttempts` itself: the shared `executeWithRetry` already fails *loudly* on `maxAttempts=0` (throws `LifecycleException`), and changing the setter would silently alter that contract for the MCP and HTTP-call consumers and their existing tests. The clamp belongs at the streaming call site, which is the one that was failing silently.

### Tests

Three regression tests added to `StreamingLegacyChatExecutorRetryTest`, each confirmed to fail before the fix and pass after: `timeoutThenSuccess_doesNotLeakTimeoutMetadata`, `zeroMaxAttempts_stillRunsOnce`, `negativeMaxAttempts_stillRunsOnce`. A fourth, `executeCapturing_propagatesErrorDespitePartialTokens`, pins the merge's central contract at the executor level — until now it was guarded only indirectly, by a cascade-level test.

155 tests green across the streaming, cascade, orchestrator and LlmTask suites.

### Known follow-ups (not fixed here — pre-existing, out of scope for this PR)

* **Cascade drops the validation-signal metadata.** `CascadeResult` carries only `tokenUsage`, not the producing executor's `warning`/`streamingTimeout`. With `modelCascade` *and* `responseValidation` both enabled, only `onEmpty` and `onRefusal` can fire; `onTruncation`, `onContentFilter` and `onStreamingTimeout` are unreachable. Coverage was worse before the merge (the cascade path left the metadata map empty), so this is an exposure, not a regression.
* **Streaming never derives `warning` from `finishReason`.** `LegacyChatExecutor` maps `LENGTH` → `truncated` and `CONTENT_FILTER` → `content_filter`; `buildMetadata` does not, so those two policies are inert on the streaming path even without the cascade.
* **A timed-out live-streamed final cascade step is accepted as the winner** rather than falling back to `bestSoFar` — a mid-stream *error* falls back correctly, a *timeout* does not.
* **\~500 lines of cascade error-path tests lapsed on `main`** before this merge (`CascadingModelExecutorExtendedTest` deleted; 15 of 16 tests in `CascadingModelExecutorCoverageTest` replaced), covering exception-unwrapping and timeout-escalation. Worth re-adding against the current implementation.

***

## 🔀 Merge `origin/main` into `feat/error-handling-recovery` — conflict resolution (2026-07-20)

**Repo:** EDDI (`feat/error-handling-recovery`)

Second merge of `main` into the error-handling branch ([PR #593](https://github.com/labsai/EDDI/pull/593)), bringing in 43 commits (group conversations, MCP/REST ownership hardening, multi-model cascade enterprise work). Seven files conflicted; beyond those, two files were silently mis-merged and one behavioural regression was introduced by the first resolution — both classes of problem are recorded below, since neither is visible in `git status`.

### Conflicts resolved (7)

* **`RestAgentEngine`** — main refactored the descriptor-based ownership check into `ConversationAccessGuard.requireConversationOwner()`; ours added `IConversationMemoryStore` for the new admin `resetState()` endpoint. Resolution keeps **both**: `conversationMemoryStore` stays (still used by `resetState`), `conversationDescriptorStore` is dropped (main removed its last usage — verified no remaining references). Constructor is now `(conversationService, conversationMemoryStore, identity, ownershipValidator, conversationAccessGuard, hitlAccessGuard, hitlToolJournalStore, agentTimeout)`.
* **`RestAgentEngineTest` / `RestAgentEngineHitlTest` / `RestAgentEngineToolPauseDetailsTest`** — constructor-arity fallout from the above, updated to the merged signature. The HITL and tool-pause tests also swapped their inline `mock(ai.labs.…IConversationMemoryStore.class)` FQN for a top-level import per AGENTS.md §4.7.
* **`CascadingModelExecutorCoverageTest`** — ours removed the `LlmConfiguration.RetryConfiguration` shim in favour of `ai.labs.eddi.configs.shared.RetryConfiguration`; main's newer version of the file still used the nested type. Resolved onto the shared type, keeping ours' `task.setParameters(...)` line.
* **`StreamingLegacyChatExecutor`** — the substantive one; see below.
* **`docs/changelog.md`** — both sides prepended entry blocks; both kept whole, main's block (which carries the newest entry) first.

### `StreamingLegacyChatExecutor` — two overlapping features unified

Both branches rewrote the same method for different reasons:

* **ours** added a configurable timeout, a retry loop driven by `RetryConfiguration`, and `finishReason` metadata, returning `StreamingResult`;
* **main** added `executeCapturing()` returning `StreamResult` with full `ChatResponse` metadata (token usage), so the cascade keeps cost evidence when it streams the final step live.

Neither could be dropped — `LlmTask` calls the retry-aware overload, `CascadingModelExecutor` calls `executeCapturing`, and each has its own tests. The merged class keeps **one** core implementation (retry + configurable timeout) that now captures the whole `ChatResponse` and builds metadata via main's `buildMetadata()` (finishReason **and** tokenUsage); both public entry points delegate to it.

**Regression caught during merge review:** the first resolution had `executeCapturing` delegate with ours' error semantics, which *salvage* partial text on a mid-stream error instead of throwing. `CascadingModelExecutor` has no try/catch at that call — it relies on the throw to fall back to the best previous step — so a failed final step was silently accepted as successful (`CascadingModelExecutorEnterpriseTest.streamingFinalStepFailsMidStream_fallbackMarkedStreamedLive`: `expected: <0> but was: <1>`). The core now takes a `salvagePartialOnError` flag: `true` for the `LlmTask` path (keep whatever the model produced rather than fail the turn), `false` for `executeCapturing` (always propagate, so the cascade can fall back). Timeout handling is unchanged — both sides already salvaged partial text there.

### Silent auto-merge breakage (fixed)

Ours deleted the nested `LlmConfiguration.RetryConfiguration` shim while main added **new** usages of it. Git merged both sides cleanly and the result did not compile:

* `AgentOrchestratorExtendedTest` (2 usages) and `CascadingModelExecutorEnterpriseTest` (1 usage) — both moved to `ai.labs.eddi.configs.shared.RetryConfiguration`, import added.

Only a clean `test-compile` surfaced these; they were not reported as conflicts.

### Auto-merges verified, not assumed

The five remaining files touched by both sides were diffed against **each** parent to confirm neither side's changes were dropped:

* `IConversationService` / `ConversationService` / `RestAgentEngineStreaming` — ours' `onTaskFailed` callback and `resetConversationState` alongside main's `onCascadeStepStart` / `onCascadeEscalation`; orthogonal additions to the same interface and anonymous handler.
* `LlmConfiguration` — ours' `responseValidation` + `streamingTimeoutSeconds` alongside main's `judgeModel` / `heuristic` / `maxTotalDurationMs`.
* `LlmTask` — ours' retry-aware streaming call and `applyResponseValidation` alongside main's cascade wiring and `MeterRegistry`. Confirmed `applyResponseValidation` sits *outside* the `if (cascadeActive)` branch, so response validation still runs on every path including the cascade.

### Design decisions

* **Keep both streaming entry points rather than collapsing to one.** Their error contracts genuinely differ (salvage vs. propagate) and each has a real production caller; one core plus an explicit flag beats duplicating \~50 lines of streaming boilerplate or silently changing one caller's behaviour.
* **`conversationDescriptorStore` dropped, not re-plumbed.** `ConversationAccessGuard` is the single ownership-check path now; keeping a second route to the descriptor store would reintroduce exactly the drift the guard exists to remove.
* **Changelog blocks kept whole** rather than interleaved by date — the file is already organised in per-branch blocks, and re-sorting would produce a large, unreviewable diff.

### Verification

`mvnw clean test-compile` green — a *clean* build deliberately, since the `RestAgentEngine` signature change would be masked by a stale incremental one.

Full unit suite: **11,409 tests run, 8 failures and 304 errors, none of them an assertion failure.** Every failure/error message in the surefire XML falls into one of three buckets — `Unable to establish loopback connection`, Docker/Testcontainers unavailable (the Mongo + Postgres store tests), and socket-selector/event-loop creation errors — i.e. this sandbox's inability to open loopback sockets or run Docker. CI remains the source of truth for those suites.

> **Correction (post-review):** an earlier draft of this entry gave a per-bucket breakdown (297/16/5 = 318) that did not reconcile with the run's 312 failures+errors. The categorisation had been run over `target/surefire-reports` without a preceding `clean`, so it also counted XML left behind by earlier *targeted* runs — a superset of the full run. The conclusion is unaffected (a superset containing zero assertion failures still contains zero), but the counts were not defensible as stated and have been removed rather than quietly adjusted.

The 56 tests directly covering the merged paths pass locally: `StreamingLegacyChatExecutor{,Retry,Coverage}Test`, `CascadingModelExecutor{Coverage,Enterprise}Test`, `RestAgentEngine{,Hitl,ToolPauseDetails}Test`, `AgentOrchestratorExtendedTest`.

### Next

Merge-ready for PR #593. Remaining follow-up unrelated to the merge: ours' `onTaskFailed` SSE handler in `RestAgentEngineStreaming` still hand-builds its JSON via `String.format` + `escapeJson`, while main introduced a `sendJsonEvent(...)` Jackson helper on the same class and documents it as preferred. Correct as-is (the payload is escaped), but worth converting for consistency.

***

## 🔀 Merge `origin/main` into `feat/group-followups` — conflict resolution (2026-07-20)

**Repo:** EDDI (`feat/group-followups`)

Merged `origin/main` (multi-model cascade enterprise hardening + MCP/REST conversation-ownership security hardening) to resolve PR #595's merge conflicts against `main`. Two files conflicted, both non-substantive:

* **`ToolExecutionTrace.java`:** both branches independently documented the same `synchronized` rationale on `addToolCall`/`addFailedToolCall` — ours as a Javadoc block above each method, `origin/main`'s as an inline comment restating it. Kept the existing Javadoc, dropped the redundant inline comment.
* **`docs/changelog.md`:** both branches appended new entries directly below the file header. Resolved as a union, newest-first per branch: this branch's own entries (group follow-ups work, 2026-07-08 to 2026-07-15) kept first, followed by `origin/main`'s entries (MCP/REST conversation-ownership hardening + multi-model cascade, 2026-07-03 to 2026-07-15).

***

## 🧵 ToolExecutionTrace — thread-safe recording (fixes CI flake) (2026-07-15)

**Repo:** EDDI (`feat/group-followups`) — a **pre-existing** bug on `main`, unrelated to the group work, that surfaced as an intermittent full-suite CI failure on this branch: `ToolExecutionServiceBranchTest.executeMultipleInParallel` → `expected <Hello, Alice!> but was <Error executing tool: ConcurrentModificationException>`. Committed here to unblock this branch's CI (per decision), clearly scoped as an independent fix.

**Root cause (systematic-debugging, root-caused before any fix):** `ToolExecutionService.executeToolsParallel` shares **one** `ToolExecutionTrace` across every concurrent task by design (it's the accumulator). But the trace's `addToolCall` / `addFailedToolCall` mutated a plain `ArrayList`, a plain `HashMap` (`toolMetrics`), and non-atomic counters with no synchronization. Under real parallelism, `HashMap.computeIfAbsent` detects concurrent structural modification and throws `ConcurrentModificationException`; `executeTool` catches it and returns `"Error executing tool: " + e.getClass().getSimpleName()` (CME's message is null) — the exact observed string. This is a genuine production bug: `executeToolsParallelAndWait` is a live API.

**Fix:** `synchronized` on both trace mutators, serializing concurrent recording (covers the collection adds, the `computeIfAbsent`, and the primitive accumulations). `updateMetrics` is private and only called under those locks. Reads happen after `CompletableFuture.allOf(...).join()` (a happens-before edge), so writer synchronization is sufficient.

**Coverage:** new `concurrentToolsShareTraceWithoutCorruption` stress test — 50 rounds × 32 parallel tasks sharing one trace, asserting no task returns an error string and every call is recorded exactly once (no lost `ArrayList` updates). It fails reliably on the unsynchronized version (reproduced the CME at round 41) and passes 5/5 with the fix. The original 2-task test had too small a race window to be a reliable regression guard.

***

## 🔍 Group conversations — Copilot PR-review response (2026-07-15)

**Repo:** EDDI (`feat/group-followups`)

Copilot flagged 5 items on the pushed branch. Each was adversarially verified against the actual code (a 6-agent verification workflow plus independent reading) before implementing — external review is evaluated, not rubber-stamped. **All 5 confirmed**, all fixed:

* **MCP raw-exception leak (×3, Medium)** — `followup_with_member`, `continue_group_discussion`, `close_group_conversation` returned `errorJson(e.getMessage())`, forwarding raw internal exception text to MCP callers (information exposure). Now each logs the full throwable server-side (`LOGGER.error(msg, e)`) and returns a stable curated `errorJson("Failed to …", "INTERNAL", null)` — matching the hardened `McpHitlTools` convention. (Verified the 3-arg `errorJson` overload exists, so this compiles; Copilot's suggested form was correct despite its own hedge.)
* **REST 400 missing `TEXT_PLAIN` (Medium)** — `rejectAttachmentsOnContinue()` was the sole error response in `RestGroupConversation` not setting `.type(TEXT_PLAIN)`. Added, for content-type consistency with every sibling 400/404/409/504.
* **Stale concurrency comment (Low)** — the `operationsInProgress` field comment called `compareAndSetState` a "best-effort read-check-update". That is now false: `GroupConversationStore.compareAndSetState` does a fast-path read then an **atomic** storage-layer conditional write (`storeIfFieldEquals` → single Mongo `updateOne` / Postgres `UPDATE` filtered on the current `state`). Corrected the comment: the in-memory `Set` is a single-node fast-fail optimization; the cluster-wide guard is the storage CAS.

**Coverage (mutation-verified):** three new `McpGroupToolsTest` cases drive each tool's generic `catch` (service throws a recognizable `boom-internal-detail-42`) and assert the response does **not** contain the raw text and **does** contain the curated message + `"errorCode":"INTERNAL"`. Reverting all three curations fails exactly those three tests (and nothing else) — proving they pin the non-leak contract. 46 `McpGroupToolsTest` cases pass (was 43); full group/MCP suite green; Checkstyle clean.

**Deliberately out of scope:** the same `errorJson(e.getMessage())` pattern exists in \~10 pre-existing catch blocks in `McpGroupTools` (and other MCP tool classes), and two existing tests *depend* on those messages (`start_group_discussion` → "Group not found", `delete` → "Not found"). A blanket sweep would break tested behaviour and expand well beyond this PR, so it is left as a separate follow-up rather than bundled here.

***

## 🌐 Group follow-up/continue — status-split completion: mid-round timeout → 504 (2026-07-15)

**Repo:** EDDI (`feat/group-followups`)

An adversarial re-review of the status-code split found one path the split had missed. `executeAgentTurn` (the per-member turn used by the *initial* discussion and re-run by every continuation) still threw a **base** `GroupDiscussionException` on an ABORT-policy member timeout — so a genuine member-agent timeout mid-round mapped to `502 Bad Gateway`, not the `504 Gateway Timeout` the split documents. Every other timeout site was already `GroupTimeoutException`; this was the last one.

* **Fix:** `GroupConversationService.executeAgentTurn` now throws `GroupTimeoutException` on the `onAgentFailure=ABORT` timeout branch. `executeDiscussion`'s re-wrap preserves the subtype, so it surfaces as `504`.
* **REST body hedge:** `continueDiscussion`'s `502` (`GroupExecutionException`) body previously claimed only "a member agent could not be reached". That catch also covers unrunnable-config failures, so the body now reads "a member agent or a required dependency could not be reached, or the group is misconfigured" — no longer asserting a single cause it cannot verify.
* **Coverage (mutation-verified):** new `GroupConversationServiceExtendedTest.FailurePolicies#abortPolicy_agentTimesOut_throwsGroupTimeoutException` drives a real member timeout (a `doNothing()` say stub so the future never completes) under `ABORT` and asserts `GroupTimeoutException`. Reverting the fix to the parent `GroupExecutionException` flips exactly this one test to failing ("expected GroupTimeoutException but was GroupExecutionException") — proving it pins the 504-vs-502 distinction, not just "some 5xx".

250 group/MCP unit tests pass (52 extended, 82 service, 39 REST, 3 routing, 31 HITL, 43 MCP); Checkstyle clean.

***

## 🌐 Group follow-up/continue — HTTP status-code split (2026-07-14)

**Repo:** EDDI (`feat/group-followups`)

A final pre-review pass flagged that `followUpWithMember` / `continueDiscussion` mapped **every** `GroupDiscussionException` to `409 Conflict` — including an unknown target agent (a client error) and mid-round server/upstream failures (LLM/DB down, agent timeout). A `409` tells the client "retryable conflict", which is wrong for a typo'd agent or a provider outage. This was pre-existing feature behaviour, not a regression, but it is a real API-semantics issue a reviewer would raise, so it was fixed properly.

* **Cause-differentiated exception subtypes** (all extend `GroupDiscussionException`, so existing `catch (GroupDiscussionException)` in the MCP tools and tests keeps working): `GroupMemberNotFoundException` (→ 404), `GroupExecutionException` (→ 502), `GroupTimeoutException extends GroupExecutionException` (→ 504). The base type now means **only** a state/concurrency conflict (→ 409).
* **Single interception point** for execution failures: `executeDiscussion` re-throws every failure caught in its phase loop as `GroupExecutionException` (preserving `GroupTimeoutException`), so the many deep agent/quota/config throw sites did not each need editing. `followUpWithMember`'s own agent-call/timeout throws are re-typed directly.
* **REST mapping** (most-specific catch first): unknown member → `404`, agent timeout → `504`, agent/model failure → `502` (Bad Gateway — an upstream dependency failed, logged with its stack trace), state/concurrency → `409`. The `@APIResponse` annotations now list the full set. `close` is unchanged (only ever a state conflict → 409).
* Nits swept: the new `400` bodies now set `.type(TEXT_PLAIN)` for consistency; the follow-up `InterruptedException` path restores the interrupt flag.

Coverage: added REST tests (`followUp`/`continue` → 404/502/504, and still-409 for a genuine conflict) and service tests asserting the specific subtypes are thrown (unknown member → `GroupMemberNotFoundException`, agent failure → `GroupExecutionException`, and the `executeDiscussion` failure-policy tests now assert `GroupExecutionException`). **Each new mapping was mutation-verified**: reverting the split makes the corresponding test fail (404/502/504 → 409; specific subtype → base). 654 group/MCP unit tests pass; Checkstyle clean.

***

## 🧬 Group conversations — mutation-audited regression coverage (2026-07-14)

**Repo:** EDDI (`feat/group-followups`)

The question "do we have coverage for everything we fixed?" was answered with **mutation testing** rather than by reading test names: each fix was reverted in turn and the suite re-run. A fix whose mutant *survives* (suite still green) has no coverage and can silently regress.

### Mutants that survived — i.e. bugs with ZERO coverage

| Fix                                                                                              | Why the tests could not see it                                                                                                                                                                                                                                       |
| ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **JAX-RS routing** — the `/{groupId}/conversations` prefix on the four post-discussion endpoints | The unit tests invoke resource methods **directly** and never exercise JAX-RS path binding. This was the single highest-severity bug of the whole effort (every follow-up/continue/close would have 404'd), and it could be reintroduced with the suite still green. |
| **SSE error curation** — the *service* pushing raw `e.getMessage()` into `GroupErrorEvent`       | Nothing asserted what actually goes out over the wire to the browser.                                                                                                                                                                                                |
| **Malformed-id reflected value** — Mongo's `ObjectId` parser echoing the raw caller string       | No test drove an unparseable id.                                                                                                                                                                                                                                     |
| **Curated exception messages** (store + `loadInGroup`)                                           | The existing test asserted the *response body*, which the REST layer curates anyway — so the message itself (which `read`/`delete` surface through the global mapper) was unguarded.                                                                                 |
| **Continuation `startPhaseIndex = 0`**                                                           | Nothing asserted that a continuation re-runs from the FIRST phase; a mutant that skipped phase 0 passed.                                                                                                                                                             |
| **`finally` cleanup condition** (defer `COMPLETED`, reclaim on `FAILED`/`CANCELLED`)             | Nothing asserted that a COMPLETED round keeps its ephemeral agents for follow-ups.                                                                                                                                                                                   |
| **`resumeQuestion` read side**                                                                   | Only the *write* was tested; nothing asserted that `resumeDiscussion` actually uses it.                                                                                                                                                                              |
| **The three new metrics counters**                                                               | Never asserted.                                                                                                                                                                                                                                                      |

### Coverage added (each verified to KILL its mutant)

* **`IRestGroupConversationRoutingTest`** (new) — reflective assertions on the JAX-RS annotations: every `@PathParam` must have a matching `{template}` segment (a mismatch binds `null` — the exact production failure), every per-conversation route stays under `/groups/{groupId}/conversations/{groupConversationId}`, and the four endpoints resolve to their documented URLs. This is the invariant a direct-invocation test structurally cannot check.
* **`GroupConversationServiceExtendedTest.MergeRegressionGuards`** (new) — a failed discussion streams a *curated* error (never the raw exception text); a continuation re-runs from phase 0 and emits `round_start` (not `group_start`); a COMPLETED round keeps its ephemeral agents.
* **`GroupConversationServiceHitlTest`** — a paused *continuation* resumes with the follow-up question, not the stale round-1 one.
* **`GroupConversationStoreTest` / `RestGroupConversationTest`** — the not-found message never embeds the caller id; a malformed id is answered 404 without reflecting the payload; the group-mismatch exception *message* is curated.
* **`GroupConversationServiceTest`** — the three operation counters, and the failure counter incrementing even when the CAS is lost.

Mutants **already killed** before this pass (genuinely covered): `failConversation`'s upsert, the MCP ownership gate, `CLOSED`-blindness in `persistedTerminalOverride`, `continueDiscussion`'s conditional write, the control-token pre-registration, `cancelDiscussion`'s `CLOSED` guard, `availableActions` for `CANCELLED`, and the MCP list owner-filter.

647 group/MCP unit tests pass; Checkstyle clean. No production code changed in this commit.

***

## 🛡️ Group conversations — terminal-state integrity + error-body hardening (2026-07-14)

**Repo:** EDDI (`feat/group-followups`). Found by an adversarial review of the previous PR-review-response commit — which had hardened the *success* write in `continueDiscussion` while leaving the *failure* write, and the rest of the reflected-value surface, wide open.

### Terminal states are now irreversible

* **`failConversation` was an unconditional whole-document write — i.e. an UPSERT.** It could **re-create a group conversation another pod had deleted**, and could overwrite a terminal `CANCELLED` (committed by a cross-pod cancel) with `FAILED`, clobbering that writer's transcript. It is now a conditional write.
* **The CAS expectation is taken from the PERSISTED state, not the in-memory one.** A first attempt CASed on `gc.getState()` and was itself a blocker: `executeDiscussion` flips the conversation to `SYNTHESIZING` **in memory** before the synthesis phase runs and only persists it afterwards, so a failure inside a synthesis phase would have CASed `SYNTHESIZING` against a persisted `IN_PROGRESS`, lost the race, skipped the write, and **stranded the conversation `IN_PROGRESS` forever** — worse than the bug being fixed. `failConversation` now re-reads the persisted state, skips the write when it is already terminal (aligning the in-memory state to it, so the `finally` makes the right ephemeral-agent decision), and counts the failure metric unconditionally so a lost race can never hide a failure from operators.

### No exception text reaches the client (CodeQL: information exposure / reflected value)

The previous commit curated one handler. This closes the class:

* **Throw sites:** `GroupConversationStore` and `GroupConversationService` no longer embed caller-supplied ids in exception messages (`"Group conversation not found: {id}"`, `"Group not found: {groupId}"`, `"No phases defined for group: {groupId}"`).
* **Sinks:** `RestGroupConversation` returns **no raw exception text in any body** — every 400/404/409 is a curated, deliberately non-committal message (these exceptions cover several causes, so the body must not assert one), with the detail logged via `LogSanitizer`.
* **SSE:** the *service* was pushing raw `e.getMessage()` into `GroupErrorEvent`, which the streaming listener forwards to the browser — so LLM/DB/driver detail (and the caller's own input) reached the client even though the REST catch sites were curated. Those events are now curated too, and the raw cause is logged with its stack trace.
* **Malformed ids:** a non-hex id reaches Mongo's `ObjectId` parser, whose message embeds the **raw caller string** — the most exploitable sink. It is caught **inside `loadInGroup`, scoped to the id lookup only**, and answered with a curated 404. It is deliberately *not* a blanket `catch (IllegalArgumentException)` around the whole operation: that would mask a genuine internal bug as a false "not found" and hide its stack trace.

Verified: 636 group/MCP unit tests pass (incl. new regression tests for the persisted-state CAS and the already-terminal skip); Checkstyle clean. Remaining full-suite failures are environmental only (Testcontainers/Docker + loopback-socket suites).

***

## 🤖 Group follow-ups — automated PR review response (CodeQL / Copilot / CodeRabbit) (2026-07-14)

**Repo:** EDDI (`feat/group-followups`) — responses to the bot reviews on [PR #595](https://github.com/labsai/EDDI/pull/595).

### Correctness

* **Terminal-state resurrection in `continueDiscussion`** (Copilot, *High*): after the `COMPLETED → IN_PROGRESS` CAS, the round/question mutation was persisted with an **unconditional** whole-document `update()`. A cancel/close/delete winning the window would be overwritten and the conversation resurrected as `IN_PROGRESS`. Now a conditional write (`updateIfState(gc, IN_PROGRESS)`) → `409` on conflict. This is the same defect class already fixed in `followUpWithMember`; the continue path had been missed.

### Security — reflected input & error exposure

* **Reflected `groupId` in 404 bodies** (Copilot, *Medium*): `loadInGroup()` embedded the caller-supplied `groupId` in its exception message, which follow-up / continue / close echo verbatim into the `404` body (CodeQL reflected-value/XSS). Now a curated body; both ids are logged server-side via `LogSanitizer`.
* **Reflected `targetAgentId` in 409 bodies**: `followUpWithMember`'s "not a member" message echoed the caller-supplied agent id into the `409`. The id is no longer reflected (the available-member list — server data — is kept).
* **Information exposure through an error message** (CodeQL, *Medium*): the delete-conflict `409` returned the raw `e.getMessage()`. Now a curated "busy, please retry" body with the detail logged server-side.

### Observability & docs

* **Metrics for the new operations** (CodeRabbit): `followUpWithMember` / `continueDiscussion` / `closeGroupConversation` were uninstrumented, contrary to AGENTS.md. Added `eddi_group_followup_count`, `eddi_group_continue_count`, `eddi_group_close_count`. Instrumented in the **service** (not the MCP tools, as suggested) so the counters cover the REST *and* MCP surfaces.
* **Authorization denials now log at WARN** (CodeRabbit) — a security-relevant event should be alertable.
* **`getAvailableActions()` javadoc corrected** (Copilot, *Low*): it claimed "not persisted", but Jackson serializes it into stored documents. It is `READ_ONLY`, so the value is never read back and is always recomputed from `state` — the javadoc now says so rather than making a false claim.

### Tests

Post-CAS `IN_PROGRESS` read modelled so the `FAILED` recovery path is actually exercised; `close` now asserts a `CLOSED` result rather than `assertSame` on a stale instance; added the concurrent-terminal-transition conflict test, the SSE `cancelled` callback test, the admin-bypass tests (the other half of `requireOwnerOrAdmin` and the list-filter exemption), and assertions that neither the raw exception text nor the caller-supplied `groupId` reaches the client.

### Not actioned (false positive)

* CodeRabbit (*Major*) claimed `updateIfState` wrapping `ResourceNotFoundException` in the unchecked `GroupConversationGoneException` bypasses the REST layer's 404 handling and yields a 500. It does not: `RestGroupConversation` explicitly catches `GroupConversationGoneException` alongside `ResourceNotFoundException` in a multi-catch on every surface that exposes the operation (cancel / approve / approve-stream) and maps it to `404`. The unchecked type is a deliberate design from the HITL work (documented on the exception) so existing CAS call sites keep compiling; `compareAndSetState` converts it to `ResourceNotFoundException` for its own callers. No change made.

***

## 🔐 Group merge — third-pass review: MCP authz, CLOSED-blindness, atomic CAS (2026-07-14)

**Repo:** EDDI (`feat/group-followups`)

A third critical review targeted what the earlier passes never looked at — the **auto-merged** files (git merged them without conflict, so nobody had reviewed them), the whole-branch PR surface, test coverage of the fixes, and security. 14 findings survived adversarial verification. All fixed:

### Security — MCP was an authorization bypass (IDOR)

* `McpGroupTools` gated the conversation-scoped tools on a **role check only** (`eddi-viewer` for `followup_with_member` / `continue_group_discussion` / `read_group_conversation`, `eddi-editor` for `close`/`delete`) with **no ownership check**, while the equivalent REST endpoints all enforce `requireOwnerOrAdmin` (403). Any authenticated viewer could read another user's full transcript, append to it, re-run every phase against their conversation (burning their LLM budget), and an editor could close or delete it. Injected `OwnershipValidator` and added a `requireConversationOwner()` gate to all five tools, with a uniform non-leaking denial. Main's own HITL MCP tools already enforced this via `HitlAccessGuard` — MCP is now consistent with REST.
* **Owner resolution on creation** (found reviewing the gate above): MCP recorded the owner as the literal `"mcp-client"` (or any caller-supplied `userId`), so the new gate would have **locked the creator out of their own conversation** whenever auth was enabled — and let a caller create a conversation owned by someone else. `discuss_with_group` / `start_group_discussion` now resolve the owner via `validateAndResolveUserId` (the calling principal; impersonation rejected), falling back to `"mcp-client"` only when auth is off.
* **`list_group_conversations` is now owner-filtered** (mirroring REST). It returns full conversation documents, so without this the per-conversation gate was pointless — a non-owner could simply list the group and read everyone's transcripts.

### Correctness — a systemic `CLOSED`-blindness

Ours introduced `CLOSED` as a new terminal state; theirs' HITL/cancel code predates it. A sweep of every terminal-state check found exactly two blind spots:

* **`persistedTerminalOverride`** treated only `{CANCELLED, FAILED, COMPLETED}` as terminal, so a running leg that found the conversation `CLOSED` did **not** stop — it fell through to an unconditional whole-document write and **resurrected the closed conversation** to `IN_PROGRESS`→`COMPLETED` after its member conversations were ended and its ephemeral agents deleted. (The previous round's fix F — close-of-`CANCELLED` — made this materially more reachable.)
* **`cancelDiscussion`** likewise ignored `CLOSED`, so a cancel could CAS `CLOSED → CANCELLED` and un-terminalize an irreversible state.

### Correctness — the cross-process guard wasn't atomic

* **`compareAndSetState` was a read-check-write**, not a CAS: it read, compared in Java, then wrote unconditionally, so two racing callers could both pass the check and both write. It is the *only* cross-process guard behind follow-up/continue/close (the original changelog admitted "single-node only… would require a conditional update at the storage layer"). The merge made the fix available — it now uses theirs' `storeIfFieldEquals`, returning `false` on a lost race.

### Contract / robustness

* `followUpWithMember` with a null/blank `targetAgentId` NPE'd into a **500**; now validated → **400** (service throws `IllegalArgumentException`, REST rejects up front). Same for a blank `question` on follow-up and continue.
* `POST /continue` advertised `attachments` on its body but **silently dropped them**. Now **rejected with 400** rather than silently ignored. (A first attempt to *honour* them was reverted after review: attachments are granted and injected to a member only on its first-ever turn, and on a continuation every member conversation already exists — so the "fix" was a no-op that stored an orphaned blob and still returned 200. Actually sharing new files mid-conversation needs the attachment fan-out reworked — see *What's next*.)
* `DELETE` during an in-flight follow-up/continue returned **500**; now a `GroupDiscussionException` → **409**.
* OpenAPI updated for the new 400/409 responses and the `CANCELLED`-closeable state.

### Tests

Backfilled the previously untested fixes (they could each have been reverted with the suite still green): `resumeQuestion` persistence, control-token pre-registration + removal, `persistedTerminalOverride` state alignment across all four terminal states, cancel-of-`CLOSED`, conditional-CAS + lost-race, MCP ownership denial/allow, blank-input 400s, continue-attachment forwarding, delete 409, and the streaming listener forwarding HITL + `round_start` events.

Also fixed a **latent broken test the merge introduced**: `GroupConversationHitlTest` still asserted 7 enum states (the merge made it 8 with `CLOSED`) — it was never in the narrower test selections and had been failing since the merge commit.

### What's next (deliberately not done here)

* **Attachments on a continuation round.** `grantAndInjectAttachments` runs only on a member's *first-ever* turn (`privateConvId == null`), so a continuation cannot share new files. Supporting it means reworking the fan-out to grant/inject per *round* (e.g. tracking which member conversations have been granted the current attachment set) — a change to shared attachment code, out of scope for a merge-response fix. Until then `/continue` rejects attachments with 400.
* **Follow-up to a dynamically recruited agent.** `GroupConversation.dynamicMembers` has no production writer (sub-agent creation records only `createdAgentIds`), so recruited agents are not addressable as follow-up targets at all. A first attempt to resolve them by display name was reverted as dead code; the real fix is to register recruited agents as members (roster + `memberConversationIds`).

Verified: `mvnw test` green — **758 group/MCP unit tests pass** (0 failures). The remaining full-suite failures are environmental only (Testcontainers/Docker and loopback-socket suites — Mongo/Postgres stores, HTTP tool tests); no group or MCP test fails. CI covers those.

***

## 🩹 Group merge — cross-feature review-response fixes (2026-07-14)

**Repo:** EDDI (`feat/group-followups`)

A deep adversarial review of the merge (7 dimensions, 58 agents, each finding cross-examined by 3 skeptics) confirmed the conflict resolution was sound but surfaced **cross-feature interaction bugs** between *ours* (continue/follow-up/close) and *theirs* (HITL pause/cancel/resume) that neither branch could have had alone — none caught by the impl-level unit tests. Fixed:

* **A — stale question on continuation resume:** a continuation round that paused at an HITL gate resumed with the round-1 question (`resumeDiscussion` read `originalQuestion`, which `continueDiscussion` never updated) — silent wrong multi-agent output. Added a dedicated `GroupConversation.resumeQuestion` field: `continueDiscussion` sets it, `resumeDiscussion` reads it (falling back to `originalQuestion` for round 1 / legacy docs). Kept separate from `originalQuestion` so the Manager conversation-list title (which renders `originalQuestion`) is not rewritten by continuations.
* **B — continue/stream dropped HITL events:** `continueDiscussionStreaming` used a hand-rolled inline SSE listener predating theirs' HITL callbacks, so a continuation that paused/cancelled emitted no event and hung the client + leaked the sink. Unified it on the shared `createStreamingListener` (added an `onRoundStart` override there); removed \~55 lines of duplication.
* **C — ephemeral-agent leak on cross-pod terminal race:** the merged `finally` cleans up only on `FAILED`/`CANCELLED` (to defer `COMPLETED` for follow-up reuse), but the cross-pod terminal-override and lost-completion-CAS exits left in-memory state stale (running/optimistic-`COMPLETED`), skipping cleanup. Both exits now align in-memory `state` to the actual persisted terminal value so the `finally` decides correctly.
* **D — follow-up clobbered a racing cancel:** `followUpWithMember`'s success path used an unconditional `update()` that could overwrite a concurrent `CANCELLED`. Switched to `updateIfState(gc, IN_PROGRESS)` (matching its own error path); a concurrent cancel/delete now yields a `409` instead of resurrecting the conversation.
* **E — cancel race + latency on continuation:** `continueDiscussion` didn't pre-register a `DiscussionControlToken`, so a cancel racing the CAS→`executeDiscussion` window took the DB branch and was overwritten, and cancel latency was a whole phase worse. Now pre-registers the token right after the CAS (mirrors `startAndDiscussAsync`/`resumeDiscussion`); removed on the pre-exec failure path.
* **F — `CANCELLED` had no reclaim path:** a cancel landing in the follow-up/continue pre-exec window could reach `CANCELLED` with orphaned ephemeral agents and no recovery. `closeGroupConversation` now accepts `CANCELLED → CLOSED` and `getAvailableActions()` returns `["close"]` for `CANCELLED`, giving operators a reclaim path.

Added regression tests (`CANCELLED` available-actions, close-of-`CANCELLED`); updated the follow-up success-write assertion. Verified: `mvnw test` green — 189 group-conversation unit tests pass (0 failures); each fix re-verified by an adversarial pass (5/6 clean first time; A refined from overloading `originalQuestion` to the dedicated field per that review).

***

## 🔀 Merge `origin/main` into `feat/group-followups` — conflict resolution (2026-07-14)

**Repo:** EDDI (`feat/group-followups`)

Merged 170 commits of `origin/main` (HITL framework + multimodal-attachments group parity) into the group follow-up/continuation/close branch. Both sides evolved the group-conversation subsystem in parallel, so all 23 conflict hunks across 12 files were resolved as a **union** of the two feature sets. Key decisions:

* **`GroupConversationService.executeDiscussion` setup (conflict):** interleaved ours' member-display-name population + round-aware start events (`onGroupStart` on round 1, `onRoundStart` on continuation rounds) with theirs' attachment re-hydration, resume-seeded turn counter (`pausedTurnCount`), HITL granularity, and control-token registration. The start-event now fires only on fresh execution (`startPhaseIndex == 0`), branching round-1 vs continuation.
* **`executeDiscussion` finally-block cleanup (conflict):** reconciled ours' "defer ephemeral cleanup for COMPLETED rounds so follow-ups can reuse dynamic agents" with theirs' "keep agents alive while `AWAITING_APPROVAL`". Result: always remove the control token; drop the verification cursor unless paused; clean up ephemeral agents only on terminal states with no follow-up/close path (`FAILED`, `CANCELLED`). `COMPLETED` cleanup stays deferred to `close`/`delete`.
* **`GroupConversationState` is now 8 values** (ours' `CLOSED` + theirs' `CANCELLED`). `getAvailableActions()` gained a `CANCELLED` arm (terminal, no actions — updated the exhaustive switch); the enum-count guard test was corrected 7 → 8.
* **`executeDiscussion` signature:** theirs added `int startPhaseIndex`; ours' `continueDiscussion` call site now passes `0` (a continuation re-runs the full protocol from phase 0).
* **Group-path guard unified on `loadInGroup()`:** all six endpoints (read, delete, followup, continue, continue/stream, close) route through ours' `loadInGroup()`; theirs' parallel `requireGroupMembership()` helper was removed as dead code. HITL endpoints keep theirs' `validateGroupConversationOwnership`.
* **JAX-RS routing fix (would-be regression):** theirs flattened the class-level `@Path` from `/groups/{groupId}/conversations` to `/groups`, moving the `{groupId}/conversations` prefix onto each method. Ours' four methods (`followup`/`continue`/`continue/stream`/`close`) carried their old class-relative `@Path("/{groupConversationId}/…")`, so post-flatten they lost the `{groupId}` template segment while still declaring `@PathParam("groupId")` — `groupId` would bind `null` and every call would 404 (invisible to the unit tests, which call the impl directly). Each of the four method paths was prefixed with `/{groupId}/conversations`, restoring the original external URLs. Caught by an adversarial merge review.
* **Both feature APIs preserved:** ours' `followUpWithMember` / `continueDiscussion` / `closeGroupConversation` / `compareAndSetState` + theirs' `cancelDiscussion` / `resumeDiscussion` / `approveGroupPhase(/Streaming)` / `getGroupApprovalStatus` / `listGroup(All)PendingApprovals` / `updateIfState` / `findByState`; all SSE events and listener callbacks from both sides retained.
* **Review nitpick:** `RestGroupConversationExtendedTest` now has an `@AfterEach` that invokes the package-private `RestGroupConversation.shutdown()`, so a full suite run no longer accumulates un-terminated virtual-thread executors.

Verified: `mvnw clean test-compile` green; \~187 group-conversation unit tests pass (GroupConversation 25, Store 19, Rest 25, RestExtended 21, Service 68, Hitl 29 — 0 failures/errors).

***

## 🔒 Group Conversation Follow-Ups — review-response hardening (2026-07-13)

**Repo:** EDDI (`feat/group-followups`)

Addresses static-analysis and code-review findings on [PR #595](https://github.com/labsai/EDDI/pull/595) (CodeQL, Copilot, CodeRabbit, GitHub Code Quality).

### Security & correctness

* **Log injection (CWE-117)**: sanitized user-controlled `groupConversationId` in log statements via `LogSanitizer.sanitize()` (`GroupConversationService` follow-up recovery, close, delete-not-found, and timeout-resolution paths).
* **Group-path validation**: `followup` / `continue` / `continue/stream` / `close` now verify the conversation belongs to the `{groupId}` in the path — mismatches return `404` (SSE `group_error` for the stream) via a shared `loadInGroup()` helper. Closes a "wrong group path" access gap **and** resolves the unused-`groupId` findings.
* **403 on streaming continue**: `continueDiscussionStreaming` now rethrows `ForbiddenException` so ownership failures map to HTTP `403` instead of a `200` SSE error event.

### Concurrency

* **Per-conversation operation guard**: `followUpWithMember` / `continueDiscussion` / `closeGroupConversation` acquire a fail-fast in-process guard (`ConcurrentHashMap.newKeySet()`) keyed by conversation ID; a second concurrent operation on the same conversation is rejected (`409`) rather than racing the `compareAndSetState` read-check-update. **NOTE:** single-node only — cluster-wide atomicity would require a conditional update at the storage layer (documented as future hardening, not built here).

### Resource lifecycle

* **`@PreDestroy` on `RestGroupConversation`**: the virtual-thread executor is now shut down on bean destroy.
* **Ephemeral cleanup on delete**: `deleteGroupConversation` now reclaims dynamically-created agents. Deferred cleanup previously ran only on `close`, so deleting a `COMPLETED` conversation orphaned them — this regression is introduced-and-fixed within the same feature branch. Extracted `cleanupEphemeralAgentsForGroup()` shared by close + delete.
* **Known limitations** (ephemeral-agent reclamation): (a) a `COMPLETED` conversation that is never closed *or* deleted still retains its ephemeral agents; (b) if cleanup fails on `delete` (group config already gone, or a transient undeploy error), the record is still hard-deleted, so the `createdAgentIds` mapping is lost and a future reaper cannot reclaim those agents (they remain operator-recoverable via the agent store). A TTL reaper (built on `ScheduleFireExecutor`) is the planned mitigation for (a) — tracked as a separate item.

### API cleanup

* **Removed dead `userId` param** from `followUpWithMember` / `continueDiscussion` (service interface, impl, REST, MCP tools). Ownership is validated via the stored conversation owner; the param was never used downstream.
* **Configurable follow-up timeout**: the follow-up agent call now uses the group's `protocol.agentTimeoutSeconds()` (default 60, consistent with discussion turns) via `resolveAgentTimeoutSeconds()`, instead of a hardcoded 120s.
* **Model encapsulation**: `GroupConversation.getMemberDisplayNames()` returns an unmodifiable view; population goes through the new `addMemberDisplayName()` method (the getter can no longer be mutated); the setter defensively copies.
* **OpenAPI**: added `404` responses to `followup` / `continue`, and `200` / `404` + event listing (incl. `round_start`) to `continue/stream`.

### Round 2 — multi-agent adversarial review + CI (2026-07-13)

Second pass after an adversarial multi-dimension review (concurrency / security / REST / lifecycle / test-coverage) plus the CI test run.

* **CI green**: updated the `GroupConversationState` (6→7, `CLOSED`) and `TranscriptEntryType` (14→15, `FOLLOW_UP`) enum-count guard tests. Added the three follow-up MCP tools (`followup_with_member`, `continue_group_discussion`, `close_group_conversation`) to `McpToolFilter` — they were **filtered out entirely** (never exposed to MCP clients) because the whitelist was never updated, so this is a functional fix, not just a test tweak.
* **Delete race** (flagged by two review dimensions): `deleteGroupConversation` now participates in the per-conversation guard. Because deferred cleanup made delete a *terminal* operation, an unguarded delete could tear down member conversations / ephemeral agents while a `continue`/`follow-up` was mid-run and then be resurrected as a "zombie" document via the store's upsert-by-id `update()`.
* **`close` status codes**: business conflicts (in-progress / wrong-state) now throw `GroupDiscussionException` → `409`; a genuine `ResourceStoreException` (DB failure) falls through to `500` via the global mapper, instead of every store error mapping to `409`. Aligns `close` with `followup`/`continue`.
* **Consistent group-scoping**: `readGroupConversation` and `deleteGroupConversation` now also route through `loadInGroup()`, so *every* endpoint under `/groups/{groupId}/conversations/{id}` verifies the conversation belongs to the path group (404 on mismatch). Previously only the new endpoints did.
* **CWE-117 in MCP tools**: the three new MCP follow-up tools now sanitize `e.getMessage()` before logging (`targetAgentId` is user-controlled and flows into exception messages).
* **Test coverage**: added unit tests for `getAvailableActions()` (per state), `memberDisplayNames` encapsulation, the `round` default, `compareAndSetState()` (all branches), `followUpWithMember` / `continueDiscussion` / `closeGroupConversation` service logic (display-name resolution, wrong-state, concurrency guard, state restore, round increment), and the new REST endpoints including the `loadInGroup` 404 guard.

### Deliberately not done

* **SSE listener factory extraction** (CodeRabbit nitpick): skipped — a pure DRY refactor of working, integration-only streaming code with no behavior gain.

### What's next

* TTL reaper for abandoned `COMPLETED` conversations (ephemeral-agent reclamation).
* Optional cluster-wide atomic state transition at the storage layer if concurrent group operations become a real deployment concern.
* Optional: sweep the remaining pre-existing `McpGroupTools` catch blocks for the same `LogSanitizer` treatment (≈11 older tools still use the unsanitized `errorf(..., e.getMessage())` pattern — lower priority, outside this PR's scope).

***

## ✨ Group Conversation Follow-Ups — member follow-up, continuation rounds, explicit close (2026-07-08)

**Repo:** EDDI (`feat/group-followups`)

### Summary

Adds three new interaction patterns for completed group conversations:

1. **Follow up with any member** — ask a specific agent (including the moderator) a question; both the question and response are appended to the group transcript as `FOLLOW_UP` entries
2. **Continue the full group** — re-run all discussion phases with a new question; agents retain conversation memory from prior rounds via reused private conversations; round counter increments
3. **Explicit close** — end member conversations, run ephemeral agent cleanup, lock the conversation permanently (`CLOSED` state)

### Changes

**Model layer:**

* `GroupConversation.java`: Added `round` field (1-based counter), `CLOSED` to `GroupConversationState`, `FOLLOW_UP` to `TranscriptEntryType`
* `IGroupConversationStore.java`: Added `compareAndSetState()` for optimistic concurrency control
* `GroupConversationStore.java`: Implemented `compareAndSetState()` (read-check-update pattern)

**Service layer:**

* `IGroupConversationService.java`: Added `followUpWithMember()`, `continueDiscussion()`, `closeGroupConversation()` + `onRoundStart()` listener method
* `GroupConversationService.java`: Implemented all three methods; modified `executeDiscussion()` to emit `round_start` SSE event (instead of `group_start`) for continuation rounds; deferred ephemeral agent cleanup to `closeGroupConversation()` for successful rounds (only immediate cleanup on failure)
* `GroupConversationEventSink.java`: Added `EVENT_ROUND_START` constant and `RoundStartEvent` record

**REST + MCP layer:**

* `IRestGroupConversation.java`: Added 4 endpoints (`POST /{gcId}/followup`, `POST /{gcId}/continue`, `POST /{gcId}/continue/stream`, `POST /{gcId}/close`) + `FollowUpRequest` record
* `RestGroupConversation.java`: Implemented all 4 endpoints with ownership validation and SSE streaming support for continuation
* `McpGroupTools.java`: Added `followup_with_member`, `continue_group_discussion`, `close_group_conversation` tools

### Design decisions

* **Concurrency**: `compareAndSetState(COMPLETED → IN_PROGRESS)` is a best-effort guard against overlapping follow-ups; state restored to `COMPLETED` on error. (Hardened on 2026-07-13 with a per-conversation in-process guard — see that entry.)
* **Deferred cleanup**: Ephemeral agents survive until explicit close so follow-ups can use dynamically-created agents; immediate cleanup only on failure
* **No TranscriptEntry.round field**: Round boundaries are inferred from `QUESTION` entries in the transcript — avoids churn on the 13-field record with 4 constructors
* **Plain-text follow-up&#x20;*****input***: The follow-up **input** is the plain question; the full transcript is still provided via the `groupTranscript` **context** variable (not re-injected into the input text), and the agent's private conversation retains prior turns

### Client experience improvements (follow-up commit)

* **Consistent response shapes**: All endpoints (`followup`, `continue`, `close`) now return the full `GroupConversation` — same shape as the initial `discuss` endpoint
* **Display name resolution**: `followUpWithMember` accepts either an agent ID or a display name (case-insensitive). Error messages list available members if target not found
* **`memberDisplayNames` map**: New field on `GroupConversation` maps agentId → displayName, populated at discussion start from group config. Eliminates client-side transcript scanning
* **`availableActions` computed property**: JSON response includes `["followup", "continue", "close"]` when COMPLETED, `["close"]` when FAILED, `[]` otherwise. Clients can discover available operations without reading docs
* **Close returns body**: `/close` now returns the closed `GroupConversation` with `state: CLOSED` and `availableActions: []`
* **Lifecycle documented in OpenAPI**: Close endpoint description includes `discuss → COMPLETED → [followup|continue]* → close → CLOSED (terminal)`

***

## 🔭 Security — conversation-listing scan: enforce the budget per-descriptor + changelog accuracy (2026-07-15)

**Repo:** EDDI (`fix/mcp-conversation-ownership`)

Second CodeRabbit pass on this branch (its first pass predated the metrics/doc commit and was already moot). Two valid, current items:

* **Enforce `MAX_OWNER_SCAN` per-descriptor, not per-page.** The budget was only checked in the `do-while` condition (after a full page), so a non-admin scan could reach `MAX_OWNER_SCAN + limit - 1` before stopping — over the documented bound. Added an in-loop `break`. Impact is small (the overrun rows are within an already-fetched page, and foreign rows skip the snapshot load either way), but it makes the bound exact and the `owner_scan_exhausted` metric fire at 500 rather than up to a page late. Not separately unit-tested: with all-foreign pages the store returns the same empty list and the same page-read count with or without the break, so the tightening isn't observable through the store interface; the existing bounded-scan tests guard against regression.
* **Changelog accuracy.** The `009ca0f20` entry's "Fix" paragraph said the ownership check "runs **after** `populateDataToDescriptor`" — stale since `8bb304b4c` split it (common case decides *before* the snapshot load; only a legacy null-owner row is re-checked after). Corrected. Also dropped a second stale "mirroring the MCP twin's budget" reference (MCP has no scan cap since `009ca0f20`).

***

## 🔭 Security — MCP/REST conversation ownership: PR-review response (metrics + doc accuracy) (2026-07-15)

**Repo:** EDDI (`fix/mcp-conversation-ownership`)

Triaged the Copilot + CodeRabbit review of this branch. Most bot findings targeted the MCP-side `list_conversations` over-fetch/scan loop that `009ca0f20` already **deleted** (its page-index bug was the reason for the delete), so they were moot against current HEAD. The substantive, still-valid items:

* **Observability (Micrometer).** Per the project convention "always add metrics to new features", the new authorization paths were operationally invisible. Added two counters via field-injected `MeterRegistry` (AGENTS.md metrics pattern, `SimpleMeterRegistry` default so unit tests that construct the bean directly stay non-null): `eddi.mcp.conversation.access.denied{tool}` incremented on every MCP ownership denial (the six gated read/drive tools via `accessDenied`, plus `chat_managed`'s impersonation denial — MCP denials return a 200 error-body, so unlike REST 403s they are not visible in `http.server.requests`); and `eddi.conversations.listing.owner_scan_exhausted` incremented when a non-admin listing stops on the `MAX_OWNER_SCAN` budget with fewer than `limit` results, so a persistently-truncated user is not invisible.
* **Fail-open on a missing descriptor — kept, documented.** CodeRabbit flagged that `requireConversationOwner` returns `null` (operation proceeds → 404) rather than denying when a descriptor is absent. Kept deliberately: a missing descriptor means the conversation is genuinely not found, and 404 is correct; flipping the *shared* guard to deny would change REST 404→403 and contradict its documented "let the operation handle the 404" contract. The only residual is orphaned memory with no descriptor — a deletion-path integrity concern, not something the read gate should mask. Softened `accessDenied`'s javadoc, which had over-claimed that a denial is indistinguishable from "does not exist".
* **Doc accuracy.** `RestConversationStore.MAX_OWNER_SCAN` javadoc no longer says it "mirrors the MCP owner-scan cap" (MCP has none since `009ca0f20`; this is now the sole budget); the previous entry's "never starved" line is qualified to "within the scan budget".

**Declined:** the suggestion to take `userId` out of `chat_managed`. The cited rule exempts "external interfaces (MCP, REST) that operate outside a conversation", which is exactly what `chat_managed` is (it routes to a per-`intent+userId` managed conversation rather than running inside one); `resolveOwnerUserId` already rejects impersonation.

***

## 🔒 Security — `RestConversationStore` listing: owner-filter the conversation store enumeration (2026-07-15)

**Repo:** EDDI (`fix/mcp-conversation-ownership`)

**Closes the first residual gap filed two entries below.** `RestConversationStore.readConversationDescriptors` — the `GET /conversationstore/conversations` endpoint declared on `IRestConversationStore` — carried no `@RolesAllowed` and no ownership filter, so it fell through to the global `authenticated` policy. With `authorization.enabled=true`, any authenticated caller could enumerate **every** user's conversation descriptors (id, agent, state, and the descriptor's `userId`). It is the REST twin of the MCP `list_conversations` gap fixed in the entry two below.

**Fix (`RestConversationStore`):** inject the existing `ConversationAccessGuard` and filter the listing inside the endpoint's existing paging `do-while`. `seesAllConversations()` is resolved once up front (admins, and any caller when authorization is disabled, skip filtering entirely); otherwise each descriptor is dropped unless `canAccessConversation(descriptor.getUserId())` admits it. The check runs **before** `populateDataToDescriptor` for the common case (every conversation since v5.1.6 records its owner on the descriptor), and only **after** populate for a legacy null-owner row, where populate resolves the owner from the snapshot (the pre-v5.1.6 fallback) — see the *bounded back-fill* note below. An unowned/legacy conversation stays visible, matching `OwnershipValidator.requireOwnerOrAdmin`.

**Design decisions**

* **Owner-filter, not admin-only.** The endpoint backs the **EDDI-Manager** conversation views (its bundled UI calls `conversationstore/conversations`). Owner-filtering keeps admins' full visibility and still lets a non-admin Manager user see *their own* conversations; a blanket `@RolesAllowed("eddi-admin")` would 403 the listing for every non-admin and diverge from the MCP twin, which chose owner-filtering. The `ConversationAccessGuard` was purpose-built for a listing (`canAccessConversation` / `seesAllConversations`), so this reuses it rather than adding a check.
* **No starvation, no dedup needed.** The store's `readDescriptors` treats its `index` as a **page number** (`ResourceFilter`: `skip = index * limit`), and the endpoint's `do-while` already re-pages (`index++`) until it fills `limit` or the store is exhausted. So a filtered-out row is naturally back-filled from a later, non-overlapping page — a caller's own conversations are not starved just because newer pages belong to others (bounded by the scan budget in the next bullet), and (unlike the MCP over-fetch) no resource-URI dedup is required.
* **Bounded, cheap back-fill (self-review fix).** Two costs had to be contained before that back-fill was safe on a large multi-tenant store, since this is the default EDDI-Manager list view: (1) the ownership check is split around `populateDataToDescriptor` — for the common case (every conversation since v5.1.6 records its owner on the descriptor) the decision is made on `descriptor.getUserId()` **before** the snapshot load, so a foreign row is skipped without loading its full memory document; only a legacy null-owner row falls through to the post-populate re-check that resolves the owner from the snapshot. (2) the back-fill is capped at `MAX_OWNER_SCAN = 500` descriptors — the sole owner-scan budget in the system, since the MCP listing delegates here rather than scanning itself — so a caller who owns few/none of a huge store cannot force a full-collection scan. Admins / auth-disabled callers are never filtered and never reach the budget. **Tradeoff:** for a non-admin, a very old owned conversation buried beyond the 500-descriptor (most-recent-first) window may not appear; the `List` return type can't signal truncation the way the MCP tool's `incomplete` flag did. The proper long-term fix is an owner-scoped descriptor query (index on `userId`) rather than in-memory filtering of a global scan — filed as a follow-up; the same unbounded-scan shape pre-existed for the `agentId`/`state`/`viewState` filters.

**MCP `list_conversations` simplified (same branch).** `McpConversationTools.listConversations` now issues a **single** store call and relays the result, dropping its per-caller `seesAll`/over-fetch branch, the 100-row chunking, the resource-URI dedup, and the `incomplete`/`note` signal. This is safe once the reality of the internal hop is accounted for — and that reality is why the removed loop was effectively dead code:

* **The MCP→store call is an unauthenticated loopback.** `RestInterfaceFactory` builds a bare REST client to `http://127.0.0.1:<port>` with no `ClientHeadersFactory` and no header propagation anywhere, so the caller's identity does **not** cross that hop.
* **Auth off (the default):** `DisabledAuthController` reports authorization disabled, so the loopback is allowed and the store's `seesAllConversations()` is `true` — it returns everything and the tool relays it. This matches the prior behavior exactly (the tool's own guard also admitted everything with auth off), so the simplification is **behavior-neutral** here.
* **Auth on:** the `authenticated` HTTP policy rejects the token-less loopback with **401 before the endpoint method runs**, so the tool's internal listing is non-functional under `authorization.enabled=true` — and was **already** so, independently of this change. That is also why the removed scan-loop is safe to delete: its only runtime path (auth-on, non-admin) 401s upstream and never executes. (For the record, that loop also carried a latent bug — it passed `scanned += page.size()`, a row count, as the store's page `index`, so a second page would `skip = 100 * 100`; the ownership unit test masked it by mocking `IRestConversationStore` directly.)

**Tests**

* New `RestConversationStoreOwnershipTest` (real guard over a mocked `SecurityIdentity`, `authorization.enabled=true`): a caller sees only their own conversations and never another user's; a non-owner enumerating the store gets nothing of the owner's; an admin sees all; an unowned/legacy (null-owner) conversation stays visible; a personal list is **back-filled across foreign pages** (first page all-foreign, the caller's own on the next) rather than starved; a foreign row is **skipped without loading its memory snapshot** (guards the cheap pre-check); a legacy null-owner descriptor whose snapshot resolves to a **foreign** owner is filtered out (guards the post-populate ordering — added per self-review, so a reorder that moved the check above `populateDataToDescriptor` would fail); and a sparse owner's scan is **bounded at `MAX_OWNER_SCAN`** (5 page reads, no snapshot loads) rather than scanning the whole store.
* `RestConversationStoreTest` and `RestConversationStoreFilterTest` updated to construct with the guard (stubbed `seesAllConversations() → true`, so their filter/paging assertions are unchanged).
* `McpConversationToolsOwnershipTest.ListConversations` reduced to a single delegation test (one store call, no scan loop); the ownership-filtering assertions moved to `RestConversationStoreOwnershipTest`. `McpConversationToolsExtendedTest`'s `list_conversations` cases already exercised the single-call path (its guard has auth disabled) and are unchanged.

**Remaining residual gaps** (deliberately out of scope): the single-conversation REST reads (`readRawConversationLog` / `readSimpleConversationLog`) and `getActiveConversations` carry no ownership check; **internal loopback REST calls via `RestInterfaceFactory` do not authenticate**, so MCP tools that call them are non-functional under `authorization.enabled=true` (a pre-existing, cross-cutting gap affecting every internal REST caller and the auth model, not just this endpoint — filed as a follow-up); `read_agent_logs` without a `conversationId` was closed in the entry immediately below.

***

## 🔒 Security — `read_agent_logs`: admin-gate unscoped/agent-only log reads (2026-07-15)

**Repo:** EDDI (`fix/mcp-conversation-ownership`)

**Closes the residual gap filed by the entry below.** The ownership pass gated `read_agent_logs` only when a `conversationId` was supplied; **without** one (unfiltered, or filtered by `agentId` alone) it still returned `BoundedLogStore` entries — workflow execution logs, LLM provider errors, internal diagnostics that can quote other users' conversation data — to any caller holding the coarse `eddi-viewer` role. That unscoped read pulls from a single shared server-side ring buffer that mixes every user's activity: an **operator surface**, not one user's data.

**Why this is the same class of bug the ownership pass fixed:** the REST equivalent, `IRestLogAdmin` (`/administration/logs` — recent, `/history`, and `/stream`), is `@RolesAllowed("eddi-admin")` at the interface level. Every log read over REST already requires admin; the MCP tool was the more permissive door. This aligns MCP with REST.

**Fix (`McpConversationTools.readAgentLogs`):** require `eddi-admin` when no `conversationId` filter is present; the conversation-scoped path is unchanged (`ConversationAccessGuard.requireConversationOwner` → owner-or-admin). The admin check is placed **before** the `try` so a role denial surfaces as an honest role error, not the ownership `accessDenied(...)` "you do not own this conversation" message (there is no conversation to own).

**Design decisions**

* **Owner-or-admin kept for the conversation-scoped path, rather than admin-only parity with REST.** `BoundedLogStore.getEntries` filters by **exact** `conversationId`, so a scoped read returns only that one conversation's log lines — no cross-user leakage. That preserves the self-service diagnostics the ownership pass deliberately added for `read_conversation` / `read_audit_trail`. The cross-user exposure was only ever in the *unscoped* path, and that is what is now closed. (An admin-only-for-all-log-reads posture was considered and rejected as an unnecessary regression of that self-service capability.)
* **`agentId`-only counts as unscoped.** An agent filter still spans every user of that agent, so it is admin-gated too — only a `conversationId` narrows the read to a single owner's data.

**Tests:** extended `McpConversationToolsOwnershipTest.ReadAgentLogs` — a viewer is denied the unscoped buffer and the agent-only buffer (`ForbiddenException`, and `BoundedLogStore` is never reached); owning one conversation does **not** grant the unscoped firehose; an admin may read both the unscoped and agent-scoped buffers; the existing owner/non-owner conversation-scoped cases still pass.

***

## 🔒 Security — MCP conversation tools had no ownership check (2026-07-14)

**Repo:** EDDI (`fix/mcp-conversation-ownership`, from `main`)

**Gap (pre-existing on `main`):** every conversation-scoped tool in `McpConversationTools` was gated on the coarse `eddi-viewer` role and nothing else, while the equivalent REST endpoints on `RestAgentEngine` all enforce `requireOwnerOrAdmin` (403). With `authorization.enabled=true`, any caller holding `eddi-viewer` could — over MCP — read **any** user's conversation memory (`read_conversation`) and transcript (`read_conversation_log`), enumerate conversations across all users (`list_conversations`), read another conversation's prompts/tool-calls/costs (`read_audit_trail`, whose REST surface is `@RolesAllowed("eddi-admin")`) and its server logs (`read_agent_logs`), **inject turns into** someone else's conversation and read the agent's reply (`talk_to_agent`, `chat_with_agent` with a foreign `conversationId`), and take over another user's managed conversation by simply naming their `userId` (`chat_managed`). The read half also defeats the group-conversation ownership gate: group members' conversations are ordinary conversations, so `list_conversations` + `read_conversation_log` reached transcripts that `read_group_conversation` denies.

**Two findings that shaped the fix:**

1. **A naive ownership gate would have broken MCP outright.** MCP created conversations with `userId = null`, and `ConversationSetup.computeAnonymousUserIdIfEmpty` turns that into a generated `anonymous-<uuid>` — a *non-blank* owner matching no principal. Gating reads on `requireOwnerOrAdmin` alone would therefore have locked every MCP-created conversation away from its own creator (admins only). REST never had this problem because it resolves the owner at creation. So the fix has to **stamp the caller as owner at MCP conversation creation** — that is what makes the gate both effective and non-regressive.
2. **The read gap had a write-side twin** (`talk_to_agent` / `chat_with_agent` / `chat_managed`), which is strictly worse than reading and lives in the same file, so it is closed here too.

**Fix — new `ConversationAccessGuard` (`engine.security`), the non-HITL sibling of `HitlAccessGuard`:**

* `requireConversationOwner(conversationId)` — owner-or-admin via the conversation descriptor; skips when the descriptor is absent (the operation itself 404s); fail-closed on a store error. This is `RestAgentEngine`'s private `validateConversationOwnership` lifted out verbatim.
* `canAccessConversation(ownerId)` / `seesAllConversations()` — non-throwing predicates for listings; admit exactly what the read gate admits (admin, owner, or unowned legacy data), so a caller never lists what they cannot read, nor reads what they cannot list.
* `resolveOwnerUserId(requestedUserId)` — delegates to `validateAndResolveUserId`, stamping the caller and rejecting impersonation.

`RestAgentEngine` now delegates to the guard (behavior identical; its `IConversationDescriptorStore` dependency became dead and was dropped). `McpConversationTools` gates all eight conversation-scoped tools, each catching `ForbiddenException` ahead of its generic catch and returning a uniform, non-leaking `accessDenied(...)` that never distinguishes "not yours" from "does not exist".

**`list_conversations` owner-filtering (reworked after self-review).** The first cut filtered a single 100-row page, which silently starves a personal list: on a shared agent the newest page is often entirely other users' conversations, so the caller would get `count: 0` — indistinguishable from "you have none" — and the requested `limit` stopped meaning anything. It now scans forward page by page until the limit is filled or a 500-descriptor budget is spent, dedupes by resource URI (the store's own paging skips deleted rows, so its cursor can outrun the rows it returns and an offset scan can re-read one), and sets `incomplete: true` with a note when it stops on the budget rather than on the store running out — per AGENTS.md "no silent caps".

**Design decisions**

* **Shared guard, not a third copy.** `McpGroupTools` (on `feat/group-followups`) had already duplicated this logic once; a third copy in the MCP conversation tools would guarantee drift. One `@ApplicationScoped` guard is now the single answer to "who may read or drive a conversation", exactly as `HitlAccessGuard` is for "who may decide an approval".
* **No auth-disabled short-circuit inside the guard.** It reads the descriptor unconditionally, as `RestAgentEngine` always has; `OwnershipValidator` already no-ops when authorization is off. Short-circuiting would have quietly changed REST semantics (and its tests mock `OwnershipValidator`).
* **`read_audit_trail` gets the ownership gate, not admin-only.** REST's audit surface is admin-only; matching that would have been a role-policy change beyond this fix. The ownership gate is a strict tightening either way.

**Behavior change (auth on only).** With `authorization.enabled=false` — the default — nothing changes: every check no-ops and new conversations still get their `anonymous-*` id. With auth on, pre-existing `anonymous-*` conversations become invisible and unreadable to non-admins over MCP: they provably belong to nobody. That is the intended tightening.

**Residual gaps, deliberately out of scope** (filed as follow-ups): `RestConversationStore.readConversationDescriptors` — the REST store listing — is unfiltered for every caller, and `read_agent_logs` *without* a conversationId still returns cross-user server logs to any viewer.

**Tests:** new `ConversationAccessGuardTest` (owner / non-owner / admin / missing descriptor / store error / unowned / auth-off, plus the listing predicates) and `McpConversationToolsOwnershipTest`, which asserts for every gated tool that a non-owner is denied **and** the underlying service is never reached — no data leaves, not even inside an error message — while owner and admin pass. Existing `McpConversationTools*Test` and `RestAgentEngine*Test` constructors updated to wire a real guard from the same mocks.

***

## 🔬 Multi-Model Cascade — Merge-readiness review: fixes + coverage backfill (2026-07-14)

**Repo:** EDDI (`feat/model-cascade-enterprise-hardening`)

A critical whole-branch review (6 parallel high-effort reviewers, then adversarial confirm/refute on each finding) declared the branch **merge-ready** — every unit "ready-with-nits", no blockers. Acted on the confirmed nits and backfilled test coverage for new/adapted paths the existing suite missed.

### Fixes

* **Validator ↔ runtime parity (`CascadeConfigValidator`).** The `convertToObject`-incompatibility warning now uses `EvaluationStrategy.fromConfigOrDefault`, so an *unknown* `evaluationStrategy` — which `resolveEffectiveStrategy` also resolves to `structured_output` and then downgrades at runtime — warns too (previously only `null`/`structured_output` warned).
* **Live-stream mid-failure de-dup (`CascadingModelExecutor`).** If the live-streamed final step fails *after* emitting partial tokens, the fallback to the buffered best is now marked `streamedLive=true`, so `LlmTask` does not re-emit the best's (different) text as a duplicate token stream after the partial tokens the client already received — the correct full response still arrives via the final `done` snapshot. Added a `withRun(…, streamedLive)` overload and a per-step `stepStreamedLive` flag read in the catch.

### Coverage backfill (new tests; all green)

* **AgentOrchestrator:** token accumulation into `ExecutionResult.responseMetadata` (the cascade-cost feed — previously 0% exercised because every test mocked null response metadata), direct `sumTokens`/`tokenUsageMap` unit tests (helpers made package-private), and the before-tool cooperative-cancellation check.
* **CascadingModelExecutor:** step-param templating + credential skip (`TEMPLATE_SKIP_PARAMS` — previously 0%, all tests used a null templating engine), a deterministic duration-ceiling test (replacing a timing-flaky one), and the streaming mid-failure de-dup above.
* **LlmTask:** the skipCascade legacy-fallback SSE emit and cascade token-usage surfacing (`responseMetadata` + `audit:cascade_token_usage`).
* **ConfidenceEvaluator:** `stripJsonWrapper` fallback, `extractFirstBalancedObject` backslash-escaped-quote handling, and the judge-model readTree-throw → regex-fallback path.
* **CascadeConfigValidator:** cascade-level negative-pricing hard-fail and the `convertToObject` + unknown-strategy warn path.

### Flagged (pre-existing, out of scope)

* A HITL tool-approval pause originating *inside* an agent-mode cascade step resumes on the base model, not the cascade step's (cheaper) model — misattributing cost/audit. Confirmed real but pre-existing (baseline already threaded the tool-approval params; the resume path predates cascade-step pauses and stores only the outer task's model). Tracked as a follow-up.

***

## 🧊 Multi-Model Cascade — PR-review follow-ups: type-safe SSE events + strategy enums (2026-07-14)

**Repo:** EDDI (`feat/model-cascade-enterprise-hardening`)

Addressed two @niedch review comments on PR #587, after merging `origin/main` (tool-level HITL) into the branch.

* **Typed SSE cascade events (comment #1).** `RestAgentEngineStreaming` built the `cascade_step_start` / `cascade_escalation` SSE payloads with hand-written `String.format` JSON (manual escaping, `%.4f` formatting). Replaced with two `record` payloads serialized through the existing Jackson `MAPPER` via a new `sendJsonEvent` helper (graceful `{}` fallback on the unexpected serialization failure). Non-finite `confidence`/`threshold` are still sanitized via `finite()` before serialization. The `escapeJson`/`finite` helpers remain (still used by the task/error events).
* **Strategy enums (comment #2).** Introduced `EvaluationStrategy` (`structured_output` / `heuristic` / `judge_model` / `none`) and `CascadingStrategy` (`cascade` / `parallel`) as the **single source of truth** for the recognized strategy tokens. `ConfidenceEvaluator` (exhaustive enum switch), `CascadingModelExecutor` (`resolveEffectiveStrategy` + gating checks), and `CascadeConfigValidator` (valid-set + warn logic) now resolve to these enums instead of scattered magic strings.
  * **Design note (answers "is there a reason it's a String?"):** the config *wire* fields (`ModelCascadeConfig.strategy` / `.evaluationStrategy`) deliberately stay lenient `String`s. An unrecognized value (a typo, or one written by a newer engine) still loads, the validator warns, the runtime falls back to the enum `DEFAULT`, and the original token round-trips unchanged through export/import — behavior a strict enum field would regress. Parsing to the enum happens at the boundary via `fromConfig` / `fromConfigOrDefault`. If the field type itself should become an enum, that's a separate contract decision (see the HITL enums for the pattern).
  * Behavior is byte-for-byte preserved (verified: `ConfidenceEvaluator*Test`, `CascadeConfigValidatorTest`, `CascadingModelExecutor*Test`, `LlmTask*Test` — 324 tests green); new `StrategyEnumsTest` locks the lenient `fromConfig` contract (case-insensitive, trimmed, unknown→null, default fallback).

***

## 🚀 Multi-Model Cascade — Enterprise Hardening (2026-07-03)

**Repo:** EDDI (`feat/model-cascade-enterprise-hardening`) **What changed:** Full enterprise pass over the multi-model cascading feature. A review found two documented-but-dead capabilities (SSE events, judge model), a compliance bug (audit recorded the wrong model), and discarded token/cost/metrics that made the cost-savings pitch unmeasurable. This lands all of it. Plan: [`planning/model-cascade-enterprise-hardening-plan.md`](https://github.com/labsai/EDDI/tree/main/planning/model-cascade-enterprise-hardening-plan.md).

### Correctness / compliance

* **Audit records the real model (#5).** `LlmTask` now writes `audit:model_name` and `audit:cascade_model` from the cascade-selected step (`provider/model (step N)`), not the task-level default. Added `audit:cascade_cost` and `audit:cascade_token_usage`. An auditor can now reconstruct which model produced an answer.
* **Agent-mode confidence (#6).** `structured_output` cannot be injected around the tool-loop, so agent mode auto-routes to `judge_model` (if configured) else `heuristic`. A single deploy-time warning replaces the previous per-turn WARN.
* **convertToObject + cascade (#7).** The cascade now honors native `jsonMode`, and forces a non-wrapper confidence strategy when `convertToObject=true` (the wrapper contradicts the raw-JSON instruction).
* **Global-var / Qute consistency (#8).** Step `type` is resolved through `GlobalVariableResolver` and step param values are run through the template engine — parity with the standard path.

### Broken promises made real

* **SSE cascade events (#1).** `StreamingResponseHandler` gained default `onCascadeStepStart`/`onCascadeEscalation`; the anonymous sink in `ConversationService.sayStreaming` forwards them; `RestAgentEngineStreaming` emits `cascade_step_start` / `cascade_escalation` SSE events. The plumbing is now live end-to-end.
* **judge\_model implemented (#2).** New `judgeModel: {type, parameters}` config block, built once via `ChatModelRegistry` (vault + global-var resolution), passed into `ConfidenceEvaluator`. `evaluationStrategy: judge_model` without a judge logs a deploy-time warning and falls back to heuristic at runtime.
* **`strategy: parallel` (#3) / budget javadoc (#4).** Unknown/`parallel` strategy logs a deploy-time warning and runs sequentially; the false "budget exhausted" javadoc replaced with real ceiling docs.

### Observability & guardrails

* **Token + cost evidence.** Per-step `tokenUsage` and `costUsd` in the trace; aggregate run cost + token usage surfaced via `responseMetadataObjectName` (was `{}`). Agent-mode token usage is accumulated across tool-loop iterations.
* **Micrometer metrics** under `eddi.llm.cascade.*`: executions, escalations (tag `reason`), accepted step, step latency, confidence distribution, step errors (tags `provider`,`type`), tokens, cost, ceiling exceeded (tag `kind`).
* **Cascade ceilings.** `maxTotalDurationMs` (wall-clock) and `maxCostPerRun` (dollars, from configurable per-step `inputPricePer1M`/`outputPricePer1M`) stop escalation and return the best response so far. Per-step timeout is capped by the remaining duration budget for **buffered** steps only — a live-streamed step is exempt (see the "Live-stream timeout" fix below).
* **Configure-time validation** (`CascadeConfigValidator`): invalid *new* numeric fields (negative pricing, non-positive `maxTotalDurationMs`, negative `maxCostPerRun`) fail fast at deploy; legacy conditions (empty steps, unknown `evaluationStrategy`/`strategy`, `judge_model` without a judge, thresholds ∉ \[0,1], dead non-last null thresholds, non-positive `timeoutMs`) emit deploy-time warnings but still load (backward-compatible).

### Robustness

* **Confidence parsing** tries a real Jackson parse first (only reads `confidence` from an identified wrapper object, so a stray `"confidence":` in answer content is ignored), regex as fallback.
* **Heuristic i18n.** `heuristicConfig` makes phrases/thresholds config-driven (English defaults); the no-phrase-match fallback is language-agnostic (keeps the default score rather than mis-scoring).
* **Cancellation safety (#9).** `AgentOrchestrator` checks interruption between tool-loop iterations and before each tool, so a timed-out cascade step stops launching further side-effectful tools. Residual risk: a tool already in-flight when the timeout fires may complete.
* **Streaming the final step live.** The always-accepted final step (legacy mode, non-wrapper strategy, streaming-capable provider) streams token-by-token via the event sink instead of buffering. `StreamingLegacyChatExecutor.executeCapturing` preserves token usage while streaming.
* **`returnBestAcrossSteps`** (opt-in): return an earlier step's response if it scored strictly higher than the finally-accepted step.
* **Base-model laziness.** The base `ChatModel` is no longer built when the active-cascade branch owns the request.

### Architecture

* `CascadingModelExecutor` converted from a static utility to an instance (constructed by `LlmTask`) holding `ChatModelRegistry`, `GlobalVariableResolver`, `ITemplatingEngine`, `LegacyChatExecutor`, `StreamingLegacyChatExecutor`, and `MeterRegistry`. `AgentOrchestrator.ExecutionResult` gained a `responseMetadata` field (2-arg constructor retained for compatibility).
* Backward compatible: all new config fields optional with today's defaults; configs without `modelCascade` and `enabled:false` are unaffected; `StreamingResponseHandler` cascade methods are `default`.

### Cross-provider credentials

* Because step/judge parameters are merged **over** the task parameters, a step (or judge) targeting a **different provider** than the task would silently inherit the task's `apiKey` — wrong for that provider, failing at runtime as a 401 that looks like an escalation. `CascadeConfigValidator` now emits a **deploy-time warning** for a different-provider step/judge that omits its own `apiKey`. Not a hard error (Ollama/Bedrock don't use `apiKey`); documented in `docs/model-cascade.md`.

### Tests & coverage

* Updated the 3 executor test classes to the instance API and the 6 `LlmTask` test classes to the new constructor. Removed the backward-incompatible `languageAgnosticScore` band that regressed the default heuristic score.
* New coverage: `CascadingModelExecutorEnterpriseTest`, `CascadingModelExecutorCoverageTest` (agent mode, live streaming, cost/duration ceilings, timeout + retryable escalation, convertToObject downgrade), `ConfidenceEvaluatorEnterpriseTest`, `StreamingLegacyChatExecutorCoverageTest`, and expanded `CascadeConfigValidatorTest`. New-code coverage ≈ **92% instruction / 78% branch** (residual branches are the 120s streaming-timeout guard and typed-exception variants); the project aggregate stays above the 90%/80% gate.
* `CascadingModelExecutor.isRetryableError` message matching collapsed to a single regex (fewer branches, same behavior).

### Adversarial-review fixes

A multi-lens adversarial review (7 reviewers → independent skeptics) surfaced several real defects, now fixed:

* **`returnBestAcrossSteps` vs. live streaming (high):** a final step already streamed live is no longer superseded by an earlier higher-scoring step — that would have replaced text the client had already received. The trace marks the superseded step accordingly.
* **Agent-mode cascade streaming (medium):** the cascade now emits the agent-mode final response to the SSE stream as a single chunk, matching the standard (non-cascade) agent path (it was silently dropped before); docs corrected.
* **Validator backward-compat (medium):** `CascadeConfigValidator` now **warns** (instead of hard-failing) for conditions older releases tolerated at load — unknown strategy/evaluationStrategy, out-of-range thresholds, dead non-last steps, judge\_model without a judge, empty steps — so upgrading cannot stop a previously-loading agent from deploying. Only the new pricing/ceiling fields hard-fail on an invalid value.
* **Heuristic clamping (medium):** config-supplied heuristic scores are clamped to \[0,1] so a mis-set value can't produce an out-of-range confidence.
* **`unescapeJsonString` (low):** rewritten as a single-pass scanner so an escaped backslash is consumed before the following char (chained `replace` corrupted `\\n`). Judge regex fallback scoped to the extracted object.
* **Streaming-timeout caveat** documented (partial tokens of an abandoned final step).
* New regression tests for all of the above, plus the previously-missing SSE-forwarding and cooperative-cancellation tests. New-code coverage ≈ **92% instruction / 79% branch**.

### Second-pass review fixes

A lean second adversarial pass (5 reviewers → synthesizer) found five more real issues, now fixed:

* **Live-stream timeout (high):** a live-streamed step is no longer subject to the per-step/duration timeout — cancelling it couldn't stop the provider's callback thread, so tokens leaked to the client while the cascade re-emitted a different response (concurrent SSE writes). A streamed step now runs under the streaming executor's own \~120 s bound and its result (even if partial) is the accepted answer; no re-emit, no mid-stream cancel. `streamLive` also tightened to *guaranteed-accept* steps only (last, null-threshold, or `none`≤1.0).
* **Judge confidence regression (medium):** the judge regex fallback runs over the full judge text again (scoping it to the first balanced object dropped the score when a reasoning object preceded the rating).
* **Docs vs. validator (medium):** the Configure-time Validation section now states the real two tiers (hard-error only for new pricing/ceiling fields; warnings for legacy conditions).
* **Single-line code fence (low):** `stripCodeFences` now unwraps ` ```{...}``` ` (no newline), which was being discarded.
* **`returnBestAcrossSteps` trace (low):** the earlier winning step's trace entry is relabeled `accepted_as_best` so the trace agrees with `stepUsed`.

Regression tests added for each. Full touched-area suite green.

### PR-review fixes (bots)

CodeRabbit + Copilot + github-code-quality on PR #587 flagged further items, now addressed: retry token usage accumulated across *all* attempts (not just the last); the cancellation interrupt flag is cleared (`Thread.interrupted()`) so it can't leak; the `accepted.step` metric + trace status name the *actual* returned step under `returnBestAcrossSteps`; unknown `evaluationStrategy` normalized to `structured_output` at runtime (matches the validator + evaluator default); SSE `cascade_escalation` guards non-finite `confidence`/`threshold`; the structured-output regex fallback uses the fence-stripped text; the cascade-disabled agent path forwards its buffered response to the stream; an unused parameter removed; a dead `@Disabled` test deleted; and the docs/changelog/plan corrected to say the validator *warns* (not "rejects/fails fast") on legacy conditions.

### Status

Complete and merged-ready on `feat/model-cascade-enterprise-hardening` (PR #587). No open items; the branch is the terminal state of this feature — next planned work is unrelated (see Section 3 of AGENTS.md).

***

## 🔒 Fix: CodeRabbit review — LifecycleManager failure-path hardening (2026-07-16)

**Repo:** EDDI (`feat/error-handling-recovery`)

### Summary

Addressed four CodeRabbit findings on the error-handling PR's own `LifecycleManager` failure path (PR #593). All four verified as valid against the code; each fix reuses existing infrastructure rather than adding a new utility.

### Key Changes

* **Audit must not bypass strict-write recovery (Major).** If `auditCollector.collect()` threw, the strict-write rollback was skipped — leaving the partial task writes it exists to remove — *and* the audit exception propagated out of the catch, replacing the original task failure. Strict-write recovery now runs **first** (it is integrity-critical), and audit collection is shielded in a try/catch that attaches any reporting error to the original exception via `addSuppressed` instead of masking it.
* **Redact credentials from audit/SSE summaries (Major).** `summarizeForAudit()` only truncated; its output is persisted to the audit ledger and streamed to admins over `task_failed` SSE. It now applies the existing `SecretRedactionFilter.redact()` **before** truncating — cutting first can split a secret so the pattern no longer matches, leaving a fragment behind. URLs and class names are deliberately retained: the audience is privileged and needs them to diagnose (this is what distinguishes it from `summarizeException`, the LLM-facing path).
* **Typed causes outrank wrapper messages (Minor).** `classifyError()` checked each level's message before descending, so a `"429"` wrapper around a `SocketTimeoutException` classified as `rate_limit`. It now scans the whole chain for typed causes first (authoritative), and only then falls back to message heuristics — substring matching is easily fooled (e.g. `"failed after 429ms"`).
* **SSE failure logging (Minor).** The `task_failed` emission catch logged at DEBUG and dropped the throwable plus all context. Now WARN, with the throwable, the sanitized conversation id (`LogSanitizer`, CWE-117) and the task id.

### Tests

Four regression tests added, each of which fails under the previous behavior: typed-cause precedence, credential redaction, redact-before-truncate ordering, and audit-failure-does-not-mask-the-original.

***

## 🐛 Fix: two stale tests red since the error-handling PR (2026-07-16)

**Repo:** EDDI (`feat/error-handling-recovery`)

### Summary

CI on this branch had been red since 2026-07-08 (commit `c054b430`, "Tests run: 9776, Failures: 1, Errors: 1") — both failures pre-date the `origin/main` merge and were surfaced again by it. Each is a stale test asserting a contract the error-handling PR itself deliberately superseded. Test-only changes; no production behavior altered.

### Key Changes

* **`StreamingLegacyChatExecutorTest.execute_error_throwsRuntimeException`** — asserted that a streaming error *always* throws. The PR intentionally changed this: an error arriving *after* partial tokens now returns the partial text with a `streaming_error_partial` warning, and only a zero-content error throws. Retargeted the test at the zero-content case (matching its name) and added `execute_errorAfterPartial_returnsPartialContent` to cover the partial contract, preserving the original token-forwarding assertion.
* **`ConversationExtendedTest.saySucceeds`** — stubbed `getConversationState()` with a call-count-sensitive consecutive-return sequence (`READY`, then `IN_PROGRESS`). The PR's `EXECUTION_INTERRUPTED` auto-recovery added a state read at the top of `runStep`, consuming the `READY`, so the in-progress guard saw `IN_PROGRESS` and threw `ConversationNotReadyException`. The mock now tracks state like real memory (returns whatever was last set), making it robust to how often production reads it.

### Design Decisions

* Fixed the **tests**, not the production code: both behaviors (partial-response salvage, interrupted-state auto-recovery) are intentional, documented features of this PR and are covered by `StreamingLegacyChatExecutorRetryTest`. The tests simply encoded the pre-feature contract.

***

## 🧹 Refactor: Remove duplicate `RetryConfiguration` shim in LlmConfiguration (2026-07-16)

**Repo:** EDDI (`feat/error-handling-recovery`)

### Summary

Follow-up to the error-handling PR (#593): resolved a code-quality finding (nested class with the same simple name as its superclass). The `LlmConfiguration.RetryConfiguration` nested class was an empty subclass of the extracted `ai.labs.eddi.configs.shared.RetryConfiguration`, kept as a backward-compat shim. It overrode nothing and shadowed the imported shared type within `LlmConfiguration`'s body.

### Key Changes

* **`LlmConfiguration.java`**: Deleted the empty nested `RetryConfiguration` subclass. The `retry` field, getter, and setter now bind directly to the imported shared `RetryConfiguration` (import already present).
* **9 test files**: Replaced `new LlmConfiguration.RetryConfiguration()` with the shared `RetryConfiguration` (added the `configs.shared` import). Includes `LlmConfigurationTest`, which had pulled the nested type in via a `LlmConfiguration.*` wildcard import.

### Design Decisions

* **Deleted the shim rather than renaming it** (a reviewer suggested `LegacyRetryConfiguration`). The subclass added zero fields/overrides, so a rename would keep a misleadingly-named dead class for the same test churn. Per project philosophy, internal-API removal is safe — the only backward-compat concern is stored JSON, and because the removed subclass added no fields, the `retry` JSON structure is byte-for-byte identical (existing MongoDB/ZIP configs deserialize unchanged).

### Verification

* `mvnw clean test-compile` clean; 123 affected unit tests pass (0 failures).
* Repo-wide grep confirms zero remaining references to the nested type (dotted, JVM binary-name, reflection strings, or wildcard imports).

***

## ⚡ Feat: Holistic Error Handling and Recovery Infrastructure (2026-07-07)

**Repo:** EDDI (`feat/error-handling-recovery`)

### Summary

Complete overhaul of error handling across LLM, HTTP, and MCP call subsystems plus cross-cutting infrastructure for admin visibility, recovery, and monitoring. 19 source files changed, 7 test files added (83+ new tests). Code-reviewed and all review findings addressed before commit.

### Key Changes

* **Shared RetryConfiguration**: Extracted reusable retry logic with exponential backoff, retryable error classification, configurable per subsystem.
* **LifecycleManager**: Error classification, failure audit entries, SSE `task_failed` events, Micrometer counters tagged by `error.type`.
* **Admin state reset**: `PATCH /{conversationId}/state` endpoint to recover stuck conversations.
* **HTTP error body storage**: 4xx/5xx response bodies stored in memory; JSON parse softened.
* **MCP continueOnError + retry + circuit breaker**: Config-driven error resilience per MCP call.
* **LLM ResponseValidation**: Config-driven policies for empty/truncated/filtered responses.
* **Streaming retry**: Zero-token failures retried; partial responses returned with metadata.

### Design Decisions

* Retry at call site (not pipeline level) per user directive.
* Strict-write default kept as `false` (opt-in) to avoid breaking existing agents.
* No `"retry"` validation action — retry handled by RetryConfiguration at call level.

***

## 🧹 Multimodal Attachments Completion — Remove dead config knob `reattachTurns` (2026-07-13)

**Repo:** EDDI (`feat/multimodal-attachments-completion`)

`LlmConfiguration.Task.reattachTurns` (`@since 6.1.0`, added on this branch) was a no-op: `getReattachTurns()` is called nowhere in `src/main`, so setting it changed nothing at runtime. Past-turn PDFs/docs already reach the model via text-extract stitching (`attachments:extracts`), never native re-attachment. Removed the field, getter/setter, and its round-trip test.

Surfaced by a codebase-wide dead-config audit (adversarial multi-agent sweep). The audit flagged \~26 other candidate no-op knobs; rather than mass-delete, they were **triaged** and tracked as follow-ups:

* **Genuinely dead** — `ModelCascadeConfig.strategy` ("parallel = future", never built), `dream.batchSize`.
* **Feature exists but knob unwired** — `enableParallelExecution` + `parallelExecutionTimeoutMs` (orphaned `ToolExecutionService` parallel machinery), RAG `injectionStrategy`/`contextTemplate`, `McpServerConfig.transport`, `autoRecallCategories`, `dream.schedule`/`maxUsersPerRun`.
* **⚠️ Unenforced guardrails** — `DynamicAgentConfig.allowRecruitment`/`allowDelegation`/`maxRecruitedAgentsPerDiscussion`/`maxDelegationsPerTask`/`inheritParentModel` are read nowhere; the guardrails silently don't apply (tracked as its own security/cost fix).
* **Roadmap scaffolding — keep** — `sessionManagement`/`autoSnapshot`/`maxCheckpointsPerConversation` (Session Forking is in-progress per roadmap).
* **Audit blind spot** — operator knobs selected via Quarkus `@IfBuildProfile`/`@LookupIfProperty` (e.g. `eddi.messaging.type`) are *not* dead; a getter-grep can't see build-time bean selection. Those need per-item verification, not deletion.

***

## 🔍 Multimodal Attachments Completion — PR #588 review-comment fixes (2026-07-13)

**Repo:** EDDI (`feat/multimodal-attachments-completion`)

Addressed CodeRabbit + Copilot review comments.

**Correctness**

* **Download 404-vs-500 (High):** `IAttachmentStore.load`/`getMetadata` threw a bare `AttachmentStoreException` for *both* a missing blob and an internal store failure, so `RestAttachmentUpload.downloadAttachment` mapped SQL/backend errors to 404 (at DEBUG) — hiding outages. Added a typed `AttachmentNotFoundException` (symmetric with `AttachmentAccessDeniedException`); both stores throw it for genuinely-missing blobs; the endpoint returns 404 for it and 500 (ERROR log, `ATTACHMENT_STORE_ERROR`) for any other store exception. +regression test.
* **GDPR export isolation (Major):** the attachment-metadata export wrapped the whole conversation loop in one try/catch, so one failing `listByConversation` truncated the export for every remaining conversation. Each conversation is now isolated (mirrors the conversation-snapshot block above it).
* **URL group attachment without mimeType (Medium):** `RestGroupConversation.toAttachments` kept URL refs with null/blank mimeType that `AttachmentContextExtractor` silently drops later; now skipped up front so the loss is explicit.

**Observability**

* `AttachmentForwarder`: reusable `Counter`s initialized once (in the constructor — the registry is constructor-injected, so `@PostConstruct` wouldn't fire in the direct-construction unit tests) instead of resolved per `forward()`; `MeterRegistry`/`Counter` imported.
* `AttachmentTextExtractor`: per-extraction PDF logs lowered INFO → DEBUG (they run on every user turn / tool call).
* `Conversation`: the attachment-issue warning now includes the conversation id.

**Style** (the import guideline just added to AGENTS.md §4.7)

* `LlmTask` (`@jakarta.inject.Inject` → `@Inject`), `GroupConversation` (`Attachment` imported), `GroupConversationServiceTest` (`Context` imported), and the FQN `MeterRegistry` in `AttachmentForwarder`.
* `GridFsAttachmentStoreTest.whenFindIterate` generalized to any file count (was hardcoded to the 0/1/2-file cases).

**Declined / documented**

* `LlmConfiguration.MultimodalOverride` kept as a mutable Jackson POJO (not a record) for consistency with every sibling nested config type in the file — converting only one would be inconsistent and need `@JsonCreator` wiring.
* URL-only group attachments still aren't recovered after a HITL resume — a deliberate, documented limitation (the blob store is the durable source; URLs aren't blob-backed). The PR description should note this.

All affected unit tests green.

***

## 🐛 Multimodal Attachments Completion — Fix: group attachments lost on HITL resume (2026-07-13)

**Repo:** EDDI (`feat/multimodal-attachments-completion`)

Found by a critical adversarial re-review of the `origin/main` merge (10-dimension workflow + per-finding refutation). A **merge-emergent** bug — neither parent could exhibit it alone: our branch added group-shared attachments; `origin/main` added group HITL pause/resume; combined, they interact badly.

**Bug:** `GroupConversation.attachments` is `@JsonIgnore` transient (the durable copy is the blob store). `resumeDiscussion()` reloads a fresh GC from the store, so `getAttachments()` is null; `executeDiscussion()` re-seeded the sibling transient field `dynamicAgentConfig` but **not** `attachments`. Result: a member speaking for the first time *after* a HITL resume got neither the blob-store grant nor the `attachment_*` context — blind to the shared files. Compiles cleanly; runtime-only.

**Fix:** new package-private `rehydrateAttachmentsFromStore(gc)`, called in `executeDiscussion` right after the `dynamicAgentConfig` re-seed (so the two transient fields are handled symmetrically in one place). It rebuilds the metadata list from `IAttachmentStore.listByConversation(gc.getId())` when the in-memory list is empty — keeping the blob store as the single source of truth (no dangling refs after erasure) with **no persistence-schema change**. Guarded by null/empty (not `startPhaseIndex`, since a task-level pause in phase 0 resumes at index 0). **Known limitation:** URL-only attachments are not blob-backed and are not recovered on resume (documented in code; a follow-up can persist those if it becomes a real need).

4 unit tests added (`rehydrate_*`); `GroupConversationServiceTest` + `RestGroupConversationTest` green. The rest of the merge review came back clean — 9/10 dimensions no findings, and the integration sweep confirmed the conflict resolutions themselves are correct (clean unions, no mis-picked sides, consistent call sites).

***

## 📎 Multimodal Attachments Completion — Human review fixes: FQN → imports (2026-07-13)

**Repo:** EDDI (`feat/multimodal-attachments-completion`)

Addressed @niedch's human review comments on PR #588:

1. **`GroupConversationService`** — the field-injected attachment store used a fully-qualified `@jakarta.inject.Inject` and `ai.labs.eddi.engine.attachments.IAttachmentStore` type. `jakarta.inject.Inject` was already imported, so the annotation is now `@Inject`; added an `IAttachmentStore` import and the field reads `IAttachmentStore attachmentStore;`.
2. **`ConversationService`** — same FQN smell on the injected field **and** the anonymous `getAttachmentStore()` override (reviewer flagged the override; the field had it too). Added the `IAttachmentStore` import and simplified both usages.

Compile clean (`mvnw compile` → exit 0). No behavior change — pure import hygiene.

Also codified the convention in `AGENTS.md` §4.7 (new **Imports** subsection): always import types/annotations and reference them by simple name; the only acceptable inline FQN is disambiguating two same-named classes used in one file. Prevents this review comment from recurring.

**Deferred (tracked separately):** @niedch also suggested a "general solution for the authorization to avoid duplicating it in multiple places" on `PostgresAttachmentStore.authorize`. Verified as a real duplication — the access policy is copy-pasted across **4 sites** (read owner-or-grant + delete owner-only, in both the Postgres and GridFS stores) with an identical denial message, and the **read** path has already drifted for the null-owner edge case (Postgres denies, Mongo allows; the delete path stays consistent). Because the reviewer framed it as future work and unifying the read path is a security-behavior change that deserves its own tested PR, it was **not** folded into this PR — spun off as a dedicated follow-up (extract a shared `AttachmentAccessPolicy`, standardize null-owner reads to deny-by-default, add a two-backend regression test).

***

## 📎 Multimodal Attachments Completion — Automated review fixes (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`)

Addressed the GitHub code-quality / Copilot review of PR #588:

1. **(High) `readAttachment` couldn't see group-shared blobs.** `listByConversation` returns only *owned* blobs, so a group member — whose shared attachments are owned by the group conversation and merely *granted* to it — got an empty list and couldn't recall them via the tool. Added `IAttachmentStore.listAccessible(conversationId)` (owned **OR** granted) in both backends (GridFS `metadata.grants` array match / Postgres `? = ANY(grants)`), and `ReadAttachmentTool` now lists/resolves through it.
2. **(Medium) URL attachments dropped when no store configured.** `GroupConversationService.materializeAttachments` returned early on a null store, discarding hosted-`url` attachments that don't need a store. Restructured to skip only the inline-base64 (store-requiring) path.
3. **(Medium) Brittle access-denied detection.** The download endpoint keyed 403-vs-404 off `message.contains("denied")`. Added a typed `IAttachmentStore.AttachmentAccessDeniedException` (thrown by both backends' authz/delete paths); the REST layer catches it for 403 and treats other store exceptions as 404/500.
4. **(Note) Unused local variable** removed from a GridFS test.

Tests updated + added (grant-aware listing, url-without-store materialize, typed-exception 403 paths); 277 green across the affected classes, coverage gate still met.

***

## 📎 Multimodal Attachments Completion — Adversarial review + fixes (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`)

A multi-agent adversarial review of the whole implementation surfaced **two real high-severity defects** (both verified by an independent refutation pass, both missed by the unit tests because they stubbed `getLatestData` directly and used single-turn memories):

1. **Prefix-collision silent data loss.** `IConversationStep.getLatestData` is a *prefix* scan, and the `ATTACHMENTS` key `"attachments"` is a prefix of the `attachments:extracts` / `attachments:errors` keys the forwarder `persist()`s. A second forwarder (or `readAttachment` auto-add, or `ContentTypeMatcher`) read in the same step reverse-scanned and returned a `List<String>` instead of the `List<Attachment>` → **zero attachments forwarded, no error note**. Reachable with two langchain tasks sharing an action or two langchain workflow steps. Fixed by reading the exact key via `getData(MemoryKey)` in `AttachmentForwarder`, `AgentOrchestrator`, and `ContentTypeMatcher`.
2. **Mirror-inverted history stitching.** `ConversationLogGenerator.withAttachmentExtracts` passed the *forward* conversation-output index into `IConversationStepStack.get()`, which is *reverse*-ordered (`get(0)` = newest). In a 3-turn conversation, turn 1's extract surfaced on turn 3 and turn 1 lost it; only the middle turn aligned. Fixed by converting the forward index to the reverse accessor index (`size-1-index`).

Regression tests added for both (a real `ConversationMemory` with persisted extract/error keys proving the forwarder still forwards; a 3-turn stitching test proving extracts land on the correct turn). All new/changed classes remain above the >90% instruction / >80% branch gate; 654 tests green across the touched surface.

***

## 📎 Multimodal Attachments Completion — Phase 6 (partial): Metrics + GDPR portability (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`) **Plan:** `planning/multimodal-attachments-completion-plan.md` (Phase 6 of 6, partial).

### What changed

* **Forwarder metrics** — `AttachmentForwarder` now takes a `MeterRegistry` and records `eddi.attachment.forwarded` (content items sent to the LLM) and `eddi.attachment.errors` (dropped/gated/failed) per turn, satisfying AGENTS.md's "always add metrics" rule for the multimodal hot path.
* **GDPR portability** — `UserDataExport` gains an `attachments` list (`AttachmentExportEntry` = conversationId/storageRef/fileName/mimeType/sizeBytes, **metadata only, never bytes**) plus a backward-compatible constructor. `GdprComplianceService.exportUserData` collects attachment metadata across the user's conversations via `IAttachmentStore.listByConversation`, and the compliance audit event records `attachmentsExported`.

### Deferred (documented follow-ups)

Still open in Phase 6: nightly reaper (orphaned blobs / stale grants via `ScheduleFireExecutor`), `CostTracker` multimodal token estimates, and an `attachmentsForwarded` audit-ledger entry. Phase 5 (multipart 1:1 `say`, SSE/output chips, and the EDDI-Manager / eddi-chat-ui frontend in their own repos) is likewise a follow-up — the two-step upload→say flow already works end-to-end.

### Tests

Forwarder metrics assertion + GDPR attachment-metadata export test. Both green.

***

## 📎 Multimodal Attachments Completion — Phase 3: Group parity (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`) **Plan:** `planning/multimodal-attachments-completion-plan.md` (Phase 3 of 6).

### What changed

* **`DiscussRequest` carries attachments** — `IRestGroupConversation.DiscussRequest` gains an optional `List<AttachmentRef> attachments` (`AttachmentRef = {mimeType, data, url, fileName}`) plus a two-argument compat constructor, so existing JSON clients and call sites are unaffected. `IGroupConversationService.discuss(...)` and `startAndDiscussAsync(...)` gain attachment-carrying overloads (default methods → real impl overrides).
* **Materialize + bind at fan-out** — `GroupConversationService.materializeAttachments` stores inline base64 files in `IAttachmentStore` **bound to the group conversation id** (so they can be granted and reaped with it) and passes hosted `url` refs through, stashing the result on the (transient) `GroupConversation.attachments`.
* **Grant + inject per member** — on each member's **first** turn, `grantAndInjectAttachments` calls `IAttachmentStore.grantAccess(storageRef, memberConversationId)` (the only place grants are minted — trusted server code, D2) and injects `attachment_*` context into the member's `InputData`. Stored refs are granted; URL refs are forwarded without a grant. Later phases rely on the Phase-2 extract-stitching and the Phase-4 `readAttachment` tool. Nested groups receive the parent's attachments and re-grant down the chain.
* **REST routing** — `RestGroupConversation` converts `AttachmentRef → Attachment` and routes through the attachment overload only when attachments are present (so the no-attachment path — and its existing mock-based tests — is untouched).

### Design note

Group members run in their **own** conversations, so strict per-conversation ownership would block them from reading a group-uploaded blob — grants are exactly the primitive that makes this safe without opening cross-conversation access generally. Transport is JSON inline (base64/url); a multipart file-part variant of the endpoint is a thin follow-up (the capability and service path are complete).

### Tests

7 service tests (materialize base64/url/no-store/empty; grant+inject stored-ref/url/grant-failure/none) + 2 REST routing tests (attachment overload vs plain). Group ITs (member observes content, grant-before-turn, nested) stay CI-only.

***

## 📎 Multimodal Attachments Completion — Phase 4: readAttachment tool (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`) **Plan:** `planning/multimodal-attachments-completion-plan.md` (Phase 4 of 6).

### What changed

* **`ReadAttachmentTool`** (`modules/llm/tools/impl`, `@Vetoed`) — the multi-turn recall path. Two `@Tool`s: `listAttachments()` (name/type/size/ref of every attachment in the conversation) and `readAttachment(nameOrRef, page)` (loads one attachment, extracts text — 1-based PDF page or 0 for whole doc — else a "no extractable text" note). The conversation id is implicit (constructor-injected), so the LLM never supplies a userId/conversationId and can only reach its own (or granted) attachments — enforced by `IAttachmentStore`.
* **Auto-add wiring** — `AgentOrchestrator` gains `setAttachmentServices(store, extractor)` (wired by `LlmTask` in a new `@PostConstruct`, after CDI injection, so the long constructor + its six direct-construction tests are untouched). `addReadAttachmentToolIfEnabled` adds the tool in the no-whitelist branch when the turn has attachments, and in the whitelist branch under key `readattachment`; skipped when the services are unset (isolated tests) or the turn has no attachments. The forwarder's fallback notes already point the model at this tool.

### Tests

`ReadAttachmentToolTest` (11 — list/read by name & ref, case-insensitive, PDF page, not-found, non-extractable, denied load, empty text, blank ref) + 5 orchestrator auto-add branch tests (no-whitelist, whitelisted, whitelist-excluded, services-unset, no-attachments). Existing orchestrator/LlmTask tests unchanged.

***

## 📎 Multimodal Attachments Completion — Phase 2: Unified forwarder (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`) **Plan:** `planning/multimodal-attachments-completion-plan.md` (Phase 2 of 6). Forwarder core.

### What changed

* **`AttachmentForwarder`** (`modules/llm/impl`, new) — the single place attachments become langchain4j `Content` on the outgoing user message. Replaces the image-only `MultimodalMessageEnhancer` (deleted, with its tests). Per attachment it resolves bytes from any source (stored blob → `store.load`, URL → `SafeHttpClient` download, base64 → decode) under **uniform per-file (10 MB) and aggregate (20 MB) byte caps across all sources** (the base64 path was previously unguarded), gates on `ModelCapabilityService(provider, model)`, and emits:
  * `image/*` → `ImageContent` when vision-capable (URL passed through when the provider fetches URLs, else **downloaded and inlined** — provider URL normalization, D7), else a note;
  * `application/pdf` → **hybrid**: native `PdfFileContent` when the model supports documents, else PDFBox text extraction inlined as `TextContent`;
  * text-like (`text/*`, JSON, XML, CSV, YAML) → decoded + inlined, **no capability required** (always works);
  * `audio/*` → `AudioContent` when supported, else a note;
  * everything else → a metadata note pointing at the (Phase 4) `readAttachment` tool.
* Extracted text is persisted to `attachments:extracts` (for Phase-2 history stitching) and every drop/skip/gate is appended to `attachments:errors` — **never silent**; each also leaves a note the LLM can relay.
* **`LlmTask`** now calls the forwarder with the resolved `(provider, model)` instead of the static enhancer (field-injected + null-guarded so the six direct-construction `LlmTask` tests are untouched).

### Design decisions

* **Capability service uses the real defaults, not mocks, in tests** — the forwarder test drives the true `ModelCapabilityService` matrix (OpenAI URL-image fast path, Gemini download-and-inline, Anthropic native PDF, OpenAI PDF text-fallback, jlama no-vision note).
* **Skip ≠ silence** — a per-file/aggregate cap hit, store-load failure, or download failure records to `attachments:errors` *and* emits a `TextContent` note so the model can tell the user, rather than dropping the attachment invisibly.

### Tests

`AttachmentForwarderTest` (18) covers the full branch matrix incl. URL-passthrough vs download-inline, base64/stored images, PDF native vs text-fallback (with extract persistence), text inline, audio on/off, unsupported note, per-file cap, store-load failure, and no-source skip. Enhancer tests removed.

### Phase 2 tail (completed same branch)

* **Per-task multimodal override + reattachTurns** — `LlmConfiguration.Task` gains an optional `multimodal { vision|documents|audio: auto|on|off }` block and `reattachTurns` (default 0). Old JSON configs deserialize cleanly (`FAIL_ON_UNKNOWN_PROPERTIES=false`). `AttachmentForwarder.forward` gains a `Support`-parameterized overload; `LlmTask` parses the task block and passes the overrides (per-task > deployment > default precedence).
* **History stitching** — `ConversationLogGenerator.generate` gains an opt-in `stitchAttachmentExtracts` flag (only the LLM-facing `ConversationHistoryBuilder` path passes `true`, so the visible transcript stays clean). Per turn it appends that step's `attachments:extracts` to the rebuilt user message; verified aligned 1:1 with conversation outputs and that non-public step data survives snapshot persistence/reload, so a turn-2 follow-up sees turn-1's PDF/text extracts. `reattachTurns` is schema-ready; extract-stitching + the `readAttachment` tool (Phase 4) are the primary multi-turn continuity mechanisms.

### What's next (Phases 3–6)

Phase 3 (group parity), 4 (`readAttachment` tool), 5 (UX), 6 (ops).

***

## 📎 Multimodal Attachments Completion — Phase 1: Storage unification + secure upload (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`) **Plan:** `planning/multimodal-attachments-completion-plan.md` (Phase 1 of 6).

### What changed

1. **One blob store.** Collapsed the two parallel abstractions onto `IAttachmentStore`. Uploads already wrote to it (GridFS / Postgres `*Store`), but conversation-deletion and GDPR erasure cascaded through a *different* store (`IAttachmentStorage` → `Mongo`/`PostgresAttachmentStorage`), so uploaded blobs were never actually deleted. Ported both consumers (`RestConversationStore` delete cascade, `GdprComplianceService` erasure) to `IAttachmentStore`, then deleted `IAttachmentStorage` + both impls + their 4 tests (verified write-dead — only the delete cascades referenced them).
2. **Grants + owner-or-grant authz.** New `IAttachmentStore.getMetadata()` (server-validated metadata, no bytes), `grantAccess()` (trusted-caller-only cross-conversation read grant), single-item `delete()` (owner-only). `load()`/`getMetadata()` authorize owner **OR** an explicit grant; grants die with the blob. This is what lets group members read a blob uploaded to the group conversation (Phase 3) without opening cross-conversation access generally.
3. **UUID ref hardening (open decision #4).** GridFS now returns an unguessable random-UUID `storageRef` held in file metadata (legacy ObjectId-hex refs still resolve); Postgres already used UUIDs. Both backends unified on one opaque ref format.
4. **Quotas.** Per-conversation count + total-byte caps enforced in `store()` (`eddi.attachments.max-per-conversation` = 50, `eddi.attachments.max-total-bytes-per-conversation` = 100 MB; non-positive disables).
5. **`storageRef` extraction branch (defect #2 — upload was orphaned).** `AttachmentContextExtractor` now parses `{storageRef}` (precedence storageRef > url > data) and `resolveAndGuard()` resolves each stored ref's authoritative MIME/size via `getMetadata` (owner/grant authorized) **before** behavior rules run, enforces the per-turn cap (`eddi.attachments.max-per-turn` = 5), and records every drop/failure to `attachments:errors` — never silent. Wired into `Conversation` init via `IPropertiesHandler.getAttachmentStore()`/`getMaxAttachmentsPerTurn()` (populated by `ConversationService`).
6. **Secure REST surface.** `RestAttachmentUpload` gains a `forwardableInline` hint on upload (upload cap 20 MB > forward cap 10 MB — warn at upload, not silently at forward), a single-item download endpoint (`GET /conversations/{id}/attachments/{storageRef}`, owner/grant-checked, Content-Disposition sanitized) and single-item `DELETE`.

### Design decisions

* **Auth model fits EDDI's anonymous-capable conversations.** No other conversation endpoint uses `@RolesAllowed` (only admin endpoints do), and anonymous deployments must keep working (D2). Enforcement is therefore store-level owner-or-grant authorization on every `load`/`getMetadata`/`delete`, plus unguessable UUID refs — not an OIDC role gate. `@RolesAllowed` can be layered on when a deployment makes OIDC mandatory. `tenantId` stays advisory (sanitized, not an access boundary).
* **Field injection for the two new `ConversationService` deps** (attachment store + per-turn cap) so the numerous direct-construction unit tests need no change.

### Tests

161 unit tests across the affected classes: `GridFsAttachmentStoreTest` rewritten for UUID refs + grants + quota (26), `AttachmentContextExtractorTest` +storageRef/resolveAndGuard (27), `RestAttachmentUploadTest` +download/delete-one/forwardableInline/CD-sanitization (21), re-typed consumer tests. Postgres store IT and full ITs stay CI-only.

### What's next

Phase 2 — the unified `AttachmentForwarder` (replaces `MultimodalMessageEnhancer` + `convertMessage`): hybrid PDF (native `PdfFileContent` vs PDFBox text), universal text inline, uniform per-file/aggregate caps across all sources, provider image-URL normalization, capability gating via `ModelCapabilityService`, and extracts-in-history stitching.

***

## 📎 Multimodal Attachments Completion — Phase 0: Foundations & bug fixes (2026-07-03)

**Repo:** EDDI (`feat/multimodal-attachments-completion`) **Plan:** `planning/multimodal-attachments-completion-plan.md` (Phase 0 of 6). Low-risk foundations that ship alone.

### What changed

1. **`@JsonIgnore` on `Attachment.getBase64Data()`** (`engine/memory/model/Attachment.java`) — the `transient` keyword did **not** stop Jackson (getter-based serialization, no `PROPAGATE_TRANSIENT_MARKER`), so inline base64 payloads were being serialized into Mongo conversation documents. Now excluded; metadata still persists. Serialization tests prove the payload never reaches persisted JSON.
2. **Scrub inline base64 from persisted context copies** (`engine/memory/AttachmentContextExtractor.java` + `engine/runtime/internal/Conversation.java`) — new `AttachmentContextExtractor.scrubInlinePayload()` returns a metadata-only copy of an `attachment_*` context when it carries a `data` payload. `Conversation.createContextData()` builds the persisted copy (step data + `context.*` conversation output) through it, so the raw base64 (\~1.33× file size/turn against the 16 MB doc limit) never lands in Mongo and is never exposed via `{context.attachment_*.data}`. The live payload still rides ATTACHMENTS memory for the turn. Mirrors secret-input scrubbing.
3. **`AttachmentTextExtractor`** (`modules/llm/tools/impl/`, new) — shared PDFBox + plain-text extraction behind a uniform, configurable cap (`eddi.attachments.extraction.max-chars`, default 10k). `extractText(bytes, mime[, maxChars])` dispatches PDF + text-like (text/\*, JSON, XML, CSV, YAML); PDF full/page-range/info methods; `canExtractText()`. `PdfReaderTool` now delegates all extraction to it (download/SSRF/formatting unchanged). Reused by the Phase 2 forwarder and Phase 4 readAttachment tool.
4. **`ModelCapabilityService`** (`modules/llm/capability/`, new) — resolves vision/documents/audio/image-by-URL support for a `(provider, model)` pair. Precedence: per-task override > deployment override (`eddi.multimodal.<provider>.<cap>` then `eddi.multimodal.<cap>`) > conservative model-aware defaults (plan §5). Unknown ⇒ unsupported ⇒ fallback. Injectable via MicroProfile Config; Function-based constructor keeps it unit-testable.
5. **Body-size alignment** (`application.properties`) — added `quarkus.http.limits.max-body-size=25M` (was Quarkus' 10 MB default, below the 20 MB attachment cap → 10–20 MB uploads died with a bare 413), plus documented `eddi.attachments.max-size-bytes` and `eddi.attachments.extraction.max-chars`.

### Design decisions

* **Scrub is a copy, not a mutation** — the original context map keeps its payload so the current turn's extraction/forwarding is unaffected; only the persisted derivative is stripped.
* **Extractor owns extraction, tool owns presentation** — `PdfReaderTool.getPdfInfo` still formats the human-readable string; the extractor returns a structured `PdfInfo`, so the shared service stays presentation-free and reusable by the forwarder.
* **Capability defaults are conservative and model-aware** — vision-first providers (OpenAI/Anthropic/Gemini/Mistral) default on but downgrade for known text-only models; model-dependent providers (Ollama/Bedrock/Oracle) default off but upgrade for known vision models; image-by-URL only for OpenAI/Azure (everything else inlines).

### Tests

146 new/covered unit tests: `AttachmentTest` (serialization no-payload), `AttachmentContextExtractorTest` (scrub matrix), `AttachmentTextExtractorTest` (PDF/text/caps/corrupt), `ModelCapabilityServiceTest` (74 — default matrix across 11 providers + override precedence). `PdfReaderToolTest` remains CI-only (SafeHttpClient opens a loopback selector local JVMs may block).

### What's next

Phase 1 — storage unification (collapse `IAttachmentStorage` into `IAttachmentStore`, port conversation-delete + GDPR cascades), grants (`grantAccess`/grant-aware `load`), authenticated upload/list/download/delete, quotas, `storageRef` extraction branch, UUID ref hardening.

***

## 🐛 schedule — correct poll-batch-size comment (at-least-once, not exactly-once) (2026-07-13)

**Repo:** EDDI (`feat/hitl-framework`)

Copilot PR review flagged that the `eddi.schedule.poll-batch-size` comment in `application.properties` claimed "cluster-wide CAS still guarantees exactly-once" — which **contradicts `IScheduleStore`'s documented contract**: firing is *at-least-once*, not exactly-once (an expired lease can be stolen, so schedule targets must be idempotent). Corrected the comment to state that per-lease CAS gives a single claimant but delivery is at-least-once, and the HITL timeout handler (resumes/cancels via CAS on conversation state) is idempotent.

Two other Copilot nits were **declined** as inconsistent with established codebase convention: (a) the exact `GroupConversationState.values().length == 7` assertion is a deliberate tripwire matching its sibling `TranscriptEntryType` test — a lower-bound guard would lose the "did you mean to change the state set?" protection; (b) the `// MINOR-2:` label on `OwnershipValidator` is consistent with a pervasive plan-reference convention (9 `MINOR-/MAJOR-` labels plus hundreds of `#NN`/`Hn`/`Task N` markers) — a one-off removal would be inconsistent.

***

## 📝 HITL enum refactor — documentation audit (2026-07-13)

**Repo:** EDDI (`feat/hitl-framework`)

Audited all documentation for the enum refactor (changelog accuracy, user-doc coverage, code comments). **User-facing docs correctly need no change**: `README.md`, `AGENTS.md`, and `docs/hitl.md` reference `timeoutPolicy` only at the config/REST layer (the JSON string values `AUTO_APPROVE`/`AUTO_REJECT`/`ABORT`/`WAIT_INDEFINITELY`), which the internal `String → enum` retype leaves byte-identical — no config-schema or wire-format change to document. Two accuracy fixes made:

* **`HitlCrashRecoveryObserver` comment** (group re-arm site): the comment claimed the inline null-default avoids "the String overload the regular surface shares" — stale after the regular surface also became an enum. Corrected to state that **both** bookmarks are now enum and `parsePolicy(String)` survives only for the `PendingApprovalSummary` projection scan (still `String`).
* **Changelog** (regular-surface entry): it listed the `McpHitlTools` regular read site among the sites updated to `.name()`, but that site was the one **missed** in that commit and fixed in the follow-up — corrected to say so.

Also re-verified `HitlTimeoutPolicySerializationTest` passes directly (`Tests run: 15, Failures: 0`).

***

## ✅ HITL enum refactor — round-2 review clean + serialization regression guard (2026-07-13)

**Repo:** EDDI (`feat/hitl-framework`)

After the round-1 review caught `McpHitlTools:185`, ran a **second, deeper adversarial review** (4 orthogonal angles: complete call-site re-inventory, runtime serialization across both stores, end-to-end timeout-fire path, and an explicit "find one more bug" hunt — each finding verify-gated, plus a completeness critic). Result: **zero findings**, verdict `CORRECT_AND_COMPLETE`. Every remaining `String` touchpoint is a deliberate guarded boundary conversion (`PendingApprovalSummary` stays `String` via guarded `.name()`; schedule metadata stays `String` and `HitlTimeoutHandler` parses it back via `valueOf`; REST/MCP/Slack summaries via `.name()`); crash-recovery null-defaults faithfully mirror the old `parsePolicy`; `parsePolicy(String)` remains live for the projection-scan path.

Added **`HitlTimeoutPolicySerializationTest`** (pure-unit, no Testcontainers) as a permanent regression guard for the invariant the whole refactor rests on. It replicates BOTH production mappers — the JSON mapper (Postgres JSONB + REST) and the BSON mapper (MongoDB, built like `PersistenceModule`) — across BOTH surfaces (`ConversationMemorySnapshot`, `GroupConversation`) and asserts: enum ⇄ `name()`-string round-trip for all four values; BSON encodes a **string, not an ordinal**; a null policy is omitted (NON\_NULL) and round-trips to null; and **legacy pre-refactor documents** (policy as a bare JSON string) still deserialize into the enum. 15/15 pass locally — closing the residual runtime/persistence risk that the CI-only Testcontainer store tests would otherwise be the sole coverage for.

***

## 🐛 HITL enum refactor — fix missed McpHitlTools read site (clean-compile break) (2026-07-13)

**Repo:** EDDI (`feat/hitl-framework`)

A thorough adversarial code review (5 dimensions + verify + completeness critic) of the two enum-refactor commits below found **one real, CRITICAL defect**: the regular-surface MCP read site `McpHitlTools.getApprovalStatus` (`McpHitlTools.java:185`) still put `snapshot.getHitlTimeoutPolicy()` (now the enum) into a `Map<String,String>` **without `.name()`**. Because that map's value type is `String` (unlike the `Map<String,Object>` sibling at `:359`), the mixed `enum : ""` ternary is a **hard javac error** (`bad type in conditional expression: HitlTimeoutPolicy cannot be converted to String`). The group twin (`:359`) and `RestAgentEngine:407` were fixed; this regular twin was missed.

**Why it slipped past verification:** the earlier `./mvnw test` runs reported BUILD SUCCESS because Maven **incremental compilation reused a stale `McpHitlTools.class`** — the source file wasn't edited, so it wasn't recompiled even though its dependency (`ConversationMemorySnapshot`) changed type. A `./mvnw clean compile` fails. The prior changelog claim that "the full main + test tree compiles" was therefore based on a false pass and is corrected here. **Lesson: verify type-signature refactors with `clean compile`, not incremental.**

**Fix:** append the guarded `.name()` to match its three siblings — `summary.put("timeoutPolicy", paused && snapshot.getHitlTimeoutPolicy() != null ? snapshot.getHitlTimeoutPolicy().name() : "")`. Wire output is byte-identical (`"AUTO_REJECT"` / `""`).

Verified with a **clean** build: `./mvnw clean test` compiles the whole main + test tree from scratch and the affected suites pass (`Tests run: 258, Failures: 0, Errors: 0`). The review's other four dimensions (serialization/persistence, null-safety, behavior-preservation, test-fidelity) and the completeness critic returned **no other defects** — the refactor is otherwise correct and complete.

***

## 🎯 Regular surface — type hitlTimeoutPolicy as the HitlTimeoutPolicy enum (2026-07-13)

**Repo:** EDDI (`feat/hitl-framework`)

Follow-up to the group-surface enum change (below): applied the same `String → HitlTimeoutPolicy` retype to the **regular (agent) conversation surface** so both surfaces are consistent. The `hitlApprovalTimeout` field stays `String` on both, for the same reasons documented in the group entry (uniform convention + `Duration` would serialize as a number under `write-dates-as-timestamps=true`).

**Model layer** (`IConversationMemory` default methods, `ConversationMemory` impl field + accessors, `ConversationMemorySnapshot` field + accessors) now carry the enum. `ConversationMemoryUtilities` copies memory ↔ snapshot unchanged (both enum). **Consumer** (`ConversationService`): the four bookmark set-sites drop `.name()` (the source `AgentConfiguration.HitlConfig.getTimeoutPolicy()` / `ToolApprovalsConfig.getTimeoutPolicy()` / the computed `effectivePolicy` are all already the enum); `scheduleHitlTimeout` compares `== WAIT_INDEFINITELY` and emits `.name()` only into the `Map<String,Object>` schedule metadata. **Read/display sites** call `.name()`: the `RestAgentEngine` summary map, `ConversationMemoryStore.collectPendingSummaries` (feeds the `String`-typed `PendingApprovalSummary`), and `SlackEventHandler.formatTimeoutInfo`. (The parity `McpHitlTools:185` regular read site was **missed here** and fixed in the follow-up above.) **Crash recovery** (`HitlCrashRecoveryObserver`): the regular `IN_PROGRESS`-recovery site inlines the `null → WAIT_INDEFINITELY` default; `parsePolicy(String)` stays intact for its remaining caller (the `PendingApprovalSummary` projection, still `String`).

**Persistence — verified wire-safe.** `ConversationMemorySnapshot` is stored as a JSONB/BSON blob (Jackson serializes the enum as its `name()`), so already-persisted `AWAITING_HUMAN` bookmarks deserialize unchanged. The Postgres bounded projection (`data->>'hitlTimeoutPolicy' AS timeout_policy` → `rs.getString(...)` → `PendingApprovalSummary`) reads the raw JSON name string and is unaffected by the model type change. The REST `awaitingApproval` summary and Manager UI contract are byte-identical.

Scope: 9 main files + 12 test files (setters/asserts moved to enum constants; `String`-param helpers convert via `valueOf`; store-test call-sites retyped). `SimpleConversationMemorySnapshot` does not carry this field, so it is untouched. Verified: `Tests run: 258, Failures: 0, Errors: 0` across the regular + group HITL unit suites; full main + test tree compiles (the Testcontainer store tests compile and run in CI). `IRestAgentEngine.java` (formatter oscillation) restored/excluded again. Nothing pushed.

***

## 🎯 GroupConversation — type hitlTimeoutPolicy as the HitlTimeoutPolicy enum (2026-07-13)

**Repo:** EDDI (`feat/hitl-framework`)

A PR review comment (@niedch) on `GroupConversation`'s HITL bookmark getters/setters said: *"I would prefer to go with the actual enum type and Duration for this."* Analyzed both halves (4-way investigation + adversarial verify) and **split the decision**: did the enum, deliberately skipped the `Duration`.

**Enum — done.** `hitlTimeoutPolicy` was a raw `String` copied from config at pause time, but the `HitlTimeoutPolicy` enum (`WAIT_INDEFINITELY, AUTO_APPROVE, AUTO_REJECT, ABORT`) already exists and is the *declared* type in all three HITL config POJOs (`AgentConfiguration.HitlConfig`, `AgentGroupConfiguration.HitlConfig`, `ToolApprovalsConfig.timeoutPolicy`) — `GroupConversation` was the lone raw-`String` outlier for the policy. The field is control-flow-relevant (gates/arms `scheduleGroupHitlTimeout`, drives crash re-arm in `HitlCrashRecoveryObserver`), so typing it removes stringly-typed `valueOf`/`.name()`/`parsePolicy` juggling. It is **wire-safe**: Jackson serializes enums by `name()`, so `"AUTO_REJECT"` round-trips identically in Mongo/Postgres JSON, over REST, and to the Manager UI — exactly how the sibling `GroupConversationState state` field already persists. Every set-site only ever wrote a valid `name()` or `null`, so deserializing already-persisted `AWAITING_APPROVAL` transcripts cannot throw.

Files:

* **`GroupConversation.java`** — field + getter/setter → `HitlTimeoutPolicy` (import added).
* **`GroupConversationService.java`** — `commitPause` / `restoreGroupPause` pass the enum directly (dropped `.name()`); `restoreGroupPause`'s `fallbackTimeoutPolicy` param + `resumeDiscussion`'s `savedTimeoutPolicy` local retyped to the enum; `scheduleGroupHitlTimeout` compares `== WAIT_INDEFINITELY` and calls `.name()` only when writing the schedule-metadata `Map<String,Object>`; `listGroupPendingApprovals` null-guards `.name()` for the `String`-typed `PendingApprovalSummary`.
* **`HitlCrashRecoveryObserver.java`** — the group site inlines the `null → WAIT_INDEFINITELY` default instead of routing through the shared `parsePolicy(String)`, which stays intact for its two **regular-surface** callers (`PendingApprovalSummary`, `ConversationMemorySnapshot`).
* **`RestGroupConversation.java` / `McpHitlTools.java`** — the summary map puts `.name()` (identical wire value, keeps the value a `String`).
* Tests (`GroupConversationServiceHitlCoverage2Test`, `…CoverageTest`, `HitlCrashRecoveryObserverTest`, `…CoverageTest`) — reflective `restoreGroupPause` signature + args updated to the enum, assertions compare the enum, `gc.` helpers convert.

**Duration — skipped (deliberate).** `hitlApprovalTimeout` stays `String`. Unlike the policy, `approvalTimeout` is uniformly `String` across every carrier (all three configs, the memory bookmark, `PendingApprovalSummary`, the transcript), so the convention favors `String`. And it is **not** wire-safe: the deployment sets `quarkus.jackson.write-dates-as-timestamps=true` with no `WRITE_DURATIONS_AS_TIMESTAMPS` override, so a `java.time.Duration` would serialize to a bare number (`900.0`) instead of `"PT15M"`, breaking the raw-over-REST OpenAPI/Manager-UI contract (the frontend reads it as an ISO-8601 string with a `PT15M` placeholder). A correct migration would need a whole-surface change + a custom ISO-8601 serializer + coordinated frontend work — a separate cross-cutting effort, out of scope for a review nit.

**The `ConversationMemorySnapshot` regular surface keeps `String`** for both fields — it is a separate class with its own `parsePolicy(String)` path; only the group transcript was retyped. (A parallel enum change there is a possible follow-up but was left out to keep this diff scoped.)

Verified: the four affected unit-test classes pass (`Tests run: 149, Failures: 0, Errors: 0`); surefire compiled the entire main + test tree first, so all call-sites type-check. Neither change is a correctness bug and the comment was a *"prefer"*, so this does not block the branch. `IRestAgentEngine.java` (reformatted by `formatter-maven-plugin` during the build, unrelated to this task) was restored and excluded. Nothing pushed.

***

## 🧹 ChannelTargetRouter — drop dead getPlatformConfig() null-checks + duplicate allocations (2026-07-06)

**Repo:** EDDI (`feat/hitl-framework`)

A Copilot review of the HITL PR flagged one site in `ChannelTargetRouter.getIntegrationByApprovalChannel` where `getPlatformConfig()` was called twice behind a redundant `!= null` guard. Verified against `ChannelIntegrationConfiguration.getPlatformConfig()`: it returns `new HashMap<>(platformConfig)` — a fresh defensive copy, with the field initialized non-null and the setter null-guarded — so it **never returns null**. That makes every `getPlatformConfig() != null` sub-check dead code, and each doubled call allocates a throwaway map per invocation.

The same pattern existed in **five other methods** Copilot did not flag; fixed all six for consistency:

* `getIntegrationByApprovalChannel` (the flagged one) — cache the copy once, drop the dead guard
* `getBotToken`, the config-load loop, `ResolvedTarget.botToken`, `ResolvedTarget.signingSecret` — drop the redundant `&& getPlatformConfig() != null` clause
* `deepCopyConfig` — drop the always-true `if` wrapper, single call

Every **genuine** guard is preserved (`integration != null`, `config != null`, `getChannelType() != null`); only the provably-dead `getPlatformConfig() != null` sub-checks and the duplicate allocations were removed. Behavior is identical because the getter cannot return null.

**Copilot's second comment — rejected (verified against tooling):** it wanted `IRestAgentEngine.listPendingApprovals` collapsed to one line. The `formatter-maven-plugin` (Eclipse formatter, bound to the build) produces exactly the two-line split it objects to and auto-reverts the single-line form on every `mvnw compile`, so the nit conflicts with the project's enforced format — no change made.

Verified: `./mvnw compile` exits 0 (checkstyle at `validate`, formatter at `process-sources`, javac all clean). Change isolated to `ChannelTargetRouter.java`. Nothing pushed.

***

## 🐛 mcpcalls — register McpCallsTask via startup module (2026-07-06)

**Repo:** EDDI (`feat/hitl-framework`)

`McpCallsTask` (the MCP-client httpcall lifecycle task) was tracked and committed, but its bootstrap registration was not — so on a fresh checkout the task was never inserted into the lifecycle-task provider map (`@LifecycleExtensions`) and the mcpcalls feature silently failed to wire into the pipeline. Added `McpCallsModule` (`@Startup(1000)` + `@PostConstruct`), mirroring the seven sibling bootstraps (`ApiCallsModule`, `LlmModule`, `OutputGenerationModule`, `PropertySetterModule`, `RulesModule`, `SemanticParserModule`, `TemplateEngineModule`), which registers `McpCallsTask.ID`. The file existed but was untracked in the working tree; surfaced while auditing the tree during the MCP-whitelist review and committed here as a wiring bug relevant to this branch. Compiles clean.

***

## 🔌 MCP tool filter — expose HITL/memory/GDPR tools + build-time regression guard (2026-07-06)

**Repo:** EDDI (`feat/hitl-framework`)

A client reported that new HITL MCP tools were "implemented but not available." Confirmed: `McpToolFilter` is a **name whitelist** (`ToolFilter` SPI — quarkus-MCP only surfaces a tool's *name*, not its declaring class/annotation, so filtering must be by name), and it exposes only the intended MCP tools while hiding the langchain4j built-in agent tools (calculator, websearch, etc.) that leak into the same scan. Three `Mcp*Tools` classes had shipped `@Tool`s that were **never added to the whitelist**, making them unreachable dead code — a quarkus-MCP `@Tool` has no other invocation path:

* **`McpHitlTools`** (9): `list_pending_approvals`, `get_approval_status`, `resume_conversation`, `cancel_conversation`, `list_group_pending_approvals`, `list_all_group_pending_approvals`, `get_group_approval_status`, `approve_group_phase`, `cancel_group_discussion` — documented as the MCP HITL surface in `docs/hitl.md` but invisible.
* **`McpMemoryTools`** (8): `list_user_memories`, `get_visible_memories`, `search_user_memories`, `get_memory_by_key`, `upsert_user_memory`, `delete_user_memory`, `delete_all_user_memories`, `count_user_memories`.
* **`McpGdprTools`** (2): `delete_user_data`, `export_user_data`.

**Why it slipped through:** the existing regression test (`McpToolFilterTest.test_allMcpToolMethods_areWhitelisted`) scanned only a **hardcoded array of 4** `Mcp*Tools` classes — HITL, memory, and GDPR were not in it, so CI stayed green.

**Fix:**

* **`McpToolFilter.java`** — added all 19 names to `MCP_TOOLS` (whitelist 55 → 74 = every declared quarkus-MCP `@Tool`). Verified there is **no name collision** with any langchain4j built-in tool (effective names cross-checked), so whitelisting a name cannot accidentally expose an internal agent tool. All three classes already enforce their own authz (`requireRole` viewer/admin + per-user `OwnershipValidator`; GDPR delete is admin-only + `CONFIRM` arg), identical to their REST counterparts — MCP is a transport, not new authority.
* **`McpToolFilterTest.java`** — rewrote the guard to **auto-discover** every class in the `ai.labs.eddi.engine.mcp` package by scanning the compiled-classes directory (no hardcoded class list), resolve each `@Tool`'s effective name (explicit `name`, else method name — the `McpGroupTools` convention), and fail the build if any is not whitelisted. Anchor tools (one per `Mcp*Tools` class) guard against a broken scan passing vacuously. Any *future* MCP tool that isn't whitelisted now turns CI red.
* **`docs/mcp-server.md`** — corrected the stale tool count (63 → 74), documented the name-only `ToolFilter` constraint and the new build-time guard.

**Decision:** whitelist (not delete) memory/GDPR — they were intended MCP tools (Phase 11a persistent memory, GDPR/CCPA framework) that were simply never wired into the filter; the annotation encodes intent to expose.

**Follow-up — adversarial code review + fixes:** the commit was then put through a 4-dimension adversarial review (whitelist-correctness, test-robustness, security/authz, docs-completeness), each finding skeptic-verified. Whitelist-correctness and test-robustness came back **clean**; 3 low-severity findings survived and 2 were addressed here:

* **Doc role-name fix** (`docs/mcp-server.md`): the "Recommended Role Mapping" table named non-existent roles `mcp-user`/`mcp-admin` and cited `@RolesAllowed`; the code actually enforces `eddi-viewer`/`eddi-editor`/`eddi-admin` (via `requireRole`) and `eddi-approver`/owner (via `HitlAccessGuard`). Rewrote the section with the real role strings and mechanism; this became load-bearing now that 10 role-guarded memory/GDPR tools are reachable.
* **Collision-guard test** (`McpToolFilterTest.test_noLangchain4jBuiltinToolIsWhitelisted`): the "no name collision with langchain4j built-ins" property was a one-time manual check. Added the **inverse** build-time guard — auto-discovers every `dev.langchain4j.agent.tool.Tool` under `modules.llm.tools` and fails if any effective name is whitelisted (would leak an internal agent tool to MCP). Also hardened both discovery helpers to load classes **without static init** (`Class.forName(name, false, …)`).
* **Not fixed (decision deferred):** the memory/GDPR mutation tools lack an independent MCP mutation kill-switch like HITL's `eddi.mcp.hitl.mutations.enabled` — flagged low, consistent with the pre-existing posture of other whitelisted destructive tools (`delete_agent`, etc.); left for the maintainer to decide whether to add symmetric kill-switches across the MCP mutation surface.

**Method:** verified the whole diagnosis against source (annotation imports, `ToolInfo`/langchain4j `@Tool` APIs via `javap`, collision analysis) before changing anything; both regression guards were proven to fail on an injected regression, then restored. `./mvnw -o test -Dtest=McpToolFilterTest` → 90 green; `./mvnw -o validate` clean. **Nothing pushed** — that stays the maintainer's call.

***

## 🔧 Dependency bumps — Quarkus 3.37.1, quarkus-mcp-server 1.13.1 (2026-07-06)

**Repo:** EDDI (`feat/hitl-framework`)

Patch bumps in `pom.xml`: `quarkus.platform.version` `3.37.0` → `3.37.1` and `quarkus-mcp-server.version` `1.13.0` → `1.13.1` (used by `io.quarkiverse.mcp:quarkus-mcp-server-http`). Both are single-property changes; the version is defined only in `pom.xml`, so no other current-state reference needed updating (historical changelog/release-note mentions left as-is). Verified locally with `./mvnw -B compile` — BUILD SUCCESS against the new BOM (`quarkus:3.37.1:generate-code` ran) and `quarkus-mcp-server-http:1.13.1` resolved into the local repo; full test suite runs in CI.

***

## 📝 AI-agent docs audit — AGENTS.md overhaul + linked-doc consistency fixes (2026-07-06)

**Repo:** EDDI (`feat/hitl-framework`)

Audited `AGENTS.md` (the instruction file AI coding assistants load; `CLAUDE.md` just delegates to it) against every authoritative source it relies on, then rewrote it for correctness and frictionless cold-checkout onboarding. Verification ran as five parallel research agents cross-checking claims against `pom.xml`, `ci.yml`, `Dockerfile`, `README.md`, `docs/project-philosophy.md`, `docs/architecture.md`, `docs/hitl.md`, and the Agent Father config, plus an independent "fresh contributor" friction review.

**AGENTS.md — factual fixes:** Quarkus `3.34.1` de-pinned (versions now reference `pom.xml` as the single source of truth, per the file's own rule 7); MCP `33 tools` → `60+`; `CostTracker` → `ToolCostTracker` (real class name); `SafeMathParser` clarified as a static inner class of `CalculatorTool`; HITL tool-source count `8` → `7` (verified against `ToolApprovalPatterns.KNOWN_SOURCES`); §5.6 corrected (Agent Father uses `scope: "conversation"`, not `secret`); HITL moved Upcoming → Completed with residual work (Manager approvals UI, `inGroupTurns: INBOX`) kept in Upcoming; Multi-Channel narrowed to Teams (Slack ships via HITL); five broken `docs/planning/` → `planning/` links.

**AGENTS.md — onboarding & policy:** added a table of contents, a Build & Test Commands section (Windows `.\mvnw.cmd` note, sandbox/IT caveat, `mise.toml` toolchain, prerequisites → README, pre-push hook activation), an external-contributor fork-model pointer, and a pillar cross-reference. Codified two team policies: **no AI co-authorship trailers or tool-advertising footers on commits/PRs** (§2 rule 5) and **ask before pushing** (§2 rule 4).

**Linked-doc consistency fixes:** `docs/hitl.md` `eight` → `seven` tool sources; `docs/architecture.md` retired stale v5 `package` terminology across the Agent Composition section (`packagestore` → `workflowstore`, `.package.json` → `.workflow.json`, `packageExtensions`/`WorkflowExtension` → `workflowSteps`/`WorkflowStep`, `configs.packages.model` → `configs.workflows.model`, `"packages"` → `"workflows"`) and fixed the LLM URI `llmstore/llmconfigs` → `llmstore/llms` — all verified against `WorkflowConfiguration.java` and the Agent Father config; `CONTRIBUTING.md` reconciled "squash fixup commits" with the never-rewrite-pushed-history rule.

**Decisions:** kept all content inline in AGENTS.md (no extraction to new files) per maintainer preference and because the architecture audit confirmed AGENTS.md's prescriptive content is high-value and, on workflow-vs-package naming, *more* current than `architecture.md` was; prefer referencing canonical sources over restating drift-prone numbers/versions; committed to `feat/hitl-framework` rather than a new branch off `origin/main` because `docs/hitl.md` exists only on this branch.

**Method:** six delegated research/critique agents, each finding verified against source before acceptance. **Nothing pushed** — that stays the maintainer's call.

***

## 🧭 HITL — whole-branch merge review (round 2) + all 22 findings fixed + fix-batch review (2026-07-05)

**Repo:** EDDI (`feat/hitl-framework`)

The **entire** branch (111 commits, 242 files, base `6f5f5dd68` → `a5df6afd2`) — including the \~90 commits of MCP-HITL pre-work the earlier 15-commit review never covered, plus the composition of the five follow-up fixes below — was put through a **second whole-branch adversarial review** (11 dimension reviewers → per-finding skeptic verification → gap round). It surfaced **25 confirmed defects (3 high, 6 medium, 13 low after de-dup)** that the narrower per-task and 15-commit reviews had missed because they are cross-cutting. The two most safety-critical dimensions (**durability/at-most-once**, **backward-compat**) again came back clean. Verdict: **not merge-ready until the 3 highs were fixed**; the user chose to **fix all 22**. All are now fixed across **10 commits** (`b9a6c1263..499095fa4`), each build+test-gated:

**Blockers (high):**

* **`b9a6c1263` — `EXECUTION_INTERRUPTED` no longer bricks input (H1).** The queued-say guard skipped `EXECUTION_INTERRUPTED`, but nothing returns that state to READY except a running turn — so an ordinary 60s `agentTimeout` watchdog expiry or a `HitlCrashRecoveryObserver` "unlock say()" recovery permanently locked the conversation's input (a non-HITL-scoped regression). It is a *recoverable* marker, not terminal: dropped from the guard so a fresh say re-runs and self-heals. Same commit fixes **M1** (pause-commit CAS now from the actual pre-turn state, not hard-coded READY, so an ERROR/interrupted-retry pause commits), **M2** (post-commit `isCancelled()` re-check converts a cancel-raced pause to `EXECUTION_INTERRUPTED` instead of stranding it with an armed timer), and two lows (undo/redo now CAS from the loaded state; the 2-min re-arm grace clamps only past-due deadlines, honoring sub-2min `approvalTimeout`).
* **`056a02e13` — tool-approval gate honored on the `CONVERSATION_START` init turn (H2).** The agent-level `toolApprovals` carrier was populated only on say/resume, never at conversation start, so a gated tool invoked by a greeting-turn LLM task executed **without approval** (fail-open). The `Agent` now carries the config (like `memoryPolicy`) and sets it on memory before `init()`; also covers scheduled and group-member conversations.
* **`6e4474552` — PostgreSQL journal store + working TTL + GDPR erasure (H3, M4, M5, M6).** `IHitlToolJournalStore` had no Postgres impl, so on `eddi.datastore.type=postgres` every tool-approval resume/approval-status read dialed a nonexistent Mongo — tool-level HITL was unusable on a first-class backend. Added `PostgresHitlToolJournalStore` (`INSERT … ON CONFLICT DO NOTHING` preserves at-most-once) + a `DataStoreProducers` selector matching the 16 sibling stores. **M4:** the Mongo TTL was inert (`executedAt` stored as int64, which the TTL monitor ignores) — now `claimedAt`/`executedAt` are BSON Dates, the TTL is anchored on `claimedAt` (so orphaned EXECUTING claims also expire), with `IndexOptionsConflict` drop+recreate. **M5:** GDPR erasure now cascades to the journal (before conversation deletion, so ids resolve). **M6:** the vacuous unique-index test and tautological `differentPauseEpoch` test are now genuine assertions.

**Medium/low (other commits):** `dcfe2c2f7` (M3 — the raw conversation-read REST endpoint no longer leaks `argumentsRaw`+transcript; names-only projection reused from fix #4), `e7da12f8f` (gate lows: null-tool-name NPE, resume kill-switch threading, null-id callId normalization at AiMessage reception, cascade/watchdog abandoned-thread guard), `a7d0df601` (RULE-pause `hitl:status` output marker), `c6745b6ae` (MCP `approve_group_phase` returns BAD\_REQUEST not INTERNAL for non-string values), `d76b84869` (group cancel-signal remove-window + sub-2min timeout parity), `50c97387f` (doc: `outcome_unknown` is WARN-logged not audited; errorCode set += CONFLICT/INTERNAL), `499095fa4` (LlmTask→orchestrator transcript-cap threading test).

**Fix-batch critical review.** The 10 fix commits were then themselves put through an adversarial review (4 concern reviewers — state-machine, gate, journal, cross-cutting — each finding skeptic-verified). The **journal batch came back fully clean** (CDI producer pattern verified byte-identical to the 16 siblings; Postgres at-most-once and Mongo TTL-anchor correctness confirmed). The gate abandoned-thread concern was **refuted** (fail-safe holds). Two low, fix-introduced defects were **confirmed and fixed** in **`8ebf5b691`**: (a) `storeConversationMemorySnapshotIfState` could NPE on a null `expectedState` (M1/undo/redo now pass a live-looked-up state that is null if the conversation was deleted concurrently) — guarded to a clean CAS-miss in both stores; (b) the RULE-pause `hitl:status` marker was never cleared on resume, so a resolved turn kept advertising "awaiting approval" — now removed on resume via a new `removeConversationOutput` step API, with the key/value extracted to shared constants.

**Method:** two deterministic multi-agent review workflows (11 + 4 reviewers, each finding adversarially verified before acceptance); the large Postgres-journal batch was implemented by a delegated agent whose diff was reviewed against spec before commit. Full clean compile green; every batch's targeted tests green. **Nothing pushed** — that stays the maintainer's call.

***

## 🔬 Tool-level HITL — adversarial final review + follow-up fixes (2026-07-04)

**Repo:** EDDI (`feat/hitl-framework`)

After Tasks 5–17 landed (see the entry below), the whole tool-level HITL change (15 commits, `3e5da4345..dda7c644e`) was put through a **whole-branch adversarial review**: six independent dimension reviewers (correctness, concurrency, security, backward-compat, durability/at-most-once, spec-completeness), each finding then handed to an independent skeptic instructed to *refute* it, then a synthesis pass. Verdict: **merge-ready, no blockers** — the two most safety-critical dimensions, **durability/at-most-once** and **backward-compat**, came back clean (no double-execution of an approved tool across crash/re-approval; null-config / RULE-pause / legacy-snapshot paths byte-identical). Five fail-safe defects survived verification and are now all fixed (each via a TDD fix + independent review gate):

* **`fix(hitl)`** **`6413db97a` — generic read surface no longer leaks raw tool args + transcript.** `SimpleConversationMemorySnapshot` (the generic conversation-read DTO behind e.g. MCP `read_conversation` and the REST simple log) carried the full `PendingToolCallBatch` including `argumentsRaw` and `chatTranscriptJson` with no `@JsonIgnore`, so any `eddi-viewer` could read unredacted tool arguments + the whole transcript of a paused conversation — broader than the deliberately approver-only `detail=full` gate. Fixed with a **names-only projection** at the Simple-snapshot boundary (fresh `PendingToolCall` objects carrying only `callId`/`toolName`/`source`/`gateReason`/`argsTruncated`); the persisted full `ConversationMemorySnapshot` is untouched, so the at-most-once resume path still round-trips. (Introduced by the Task-13 Simple-snapshot extension.)
* **`fix(hitl)`** **`1d7ca72e7` — say-path pause commit guarded by a state-CAS.** The say-path fresh-pause persistence was an unconditional full-document store guarded only by an up-front `isCancelled()` check; a concurrent `end`/`cancel` landing in the TOCTOU window could be lost, **resurrecting an ENDED/cancelled conversation** as `AWAITING_HUMAN` with an armed timeout. Now uses `storeConversationMemorySnapshotIfState` (compare-and-store from the running `READY` state); a miss discards the pause (no store, no counter, no schedule), mirroring the resume path's existing guard. Covers both RULE and TOOL\_CALL pauses.
* **`fix(hitl)`** **`30e495b88` — tool-pause policy resolved from the task-scoped effective config.** Post-pause resolution of timeout policy, no-progress policy, auto-approval cap, and pending message read only the *agent-level* `hitlConfig.toolApprovals`, ignoring the per-task `LlmConfiguration.Task.toolApprovals` override that the gate itself honors — so a task-scoped finite `timeoutPolicy=AUTO_REJECT`/`approvalTimeout` silently degraded to `WAIT_INDEFINITELY` (waited forever instead of auto-rejecting). The gate's resolved effective config is now stamped onto the `PendingToolCallBatch` and read back by all four resolvers (agent-level fallback for legacy/RULE/null batches); Task 10's inherit-from-outer + `AUTO_APPROVE`-demotion semantics are unchanged.
* **`fix(hitl)`** **`69143a799` — resumed tool-pause turn renders only the final answer.** On a same-index TOOL\_CALL resume the final answer was appended to the same step's `"output"` list that still held the "awaiting approval" placeholder, so the turn rendered both stacked. Resume now removes exactly the deterministic placeholder (identified via `resolvePendingMessage`, stable across pause→resume through the persisted batch) plus its mirror `Data<>`; the placeholder still shows while `AWAITING_HUMAN`, earlier multi-task output is preserved, and the RULE path is unchanged.
* **`fix(hitl)`** **`9d07c525d` — `eddi.hitl.tool.transcript-max-bytes` wired.** The plan-mandated transcript-cap override property was never wired (hard-coded 2 MB). Now injected in `LlmTask` (the CDI seam, like `eddi.hitl.tool.enabled`) and threaded through the standard + cascade branches to `buildPendingBatch`; an absent property reproduces the unchanged 2 MB default.

**Method:** the review ran as a deterministic multi-agent workflow (12 agents); every finding was adversarially verified against HEAD before acceptance, and every fix was independently re-reviewed before landing. The full local suite is \~10,500 tests green with no logic regression (only the pre-existing Docker/Testcontainers/loopback-socket classes fail in the sandbox — they run in CI).

**Tracked follow-ups (fail-safe, non-blocking):** (a) `resumeToolLoop`'s rare re-pause-during-continuation still caps the transcript at the 2 MB default rather than the configured value (commented at the call site); (b) no validation of a pathological `0`/negative `transcript-max-bytes` (fail-safe always-omit). The Testcontainers ITs and the BSON/JSONB round-trip of the new `effectiveToolApprovals` batch field are CI-verified only.

***

## 🛠️ Tool-level HITL — complete feature + documentation (Tasks 5–17 of 17) (2026-07-04)

**Repo:** EDDI (`feat/hitl-framework`)

Completes and documents **tool-level HITL approval gating**: the conversation pauses for human approval when the LLM invokes a *gated tool* (any of the 8 sources — built-in `@Tool`, `http`, `mcp`, `a2a`, `dynamic`, `memory`, `recall`), gated *before* the tool executes (fail-safe), configured via allow/exempt glob patterns, coexisting with the behavior-rule `PAUSE_CONVERSATION` turn gate. Builds on the Tasks 1–4 foundation (see the earlier 2026-07-03 entry — do not duplicate). Full plan: [`planning/hitl-tool-approval-plan.md`](https://github.com/labsai/EDDI/tree/main/planning/hitl-tool-approval-plan.md).

**Task 17 — documentation (this entry).** No production code changed. Docs/markdown only:

* **`docs/hitl.md`** — removed the now-false "Tool-level HITL … is deferred" from *Known Limitations* (it contradicted the rest of the same doc) and replaced it with an *implemented* note; scoped the Slack data-minimization sentence to **RULE** pauses and documented the **TOOL\_CALL** exception (the approval channel renders redacted, 300-char-truncated tool arguments so a reviewer can see what they are approving; the in-thread notice stays pause-reason-only); added a **Tool-Level Approval Gating** section (config schema + defaults, pattern language, precedence, effective-timeout-policy rule, per-call verdict/amendment REST bodies with a JSON example, `pauseDetails` reference, the write-ahead journal + outcome-unknown contract, Slack all-or-nothing buttons, group-member REJECT, frozen-transcript semantics, and the `eddi.hitl.tool.enabled` rolling-upgrade note). Reconciled with the existing tool-level content earlier tasks had already added (the `pauseDetails` shapes, MCP Surface table, `eddi.mcp.hitl.mutations.enabled` kill-switch) — no duplicate sections.
* **`AGENTS.md` §5.3** — extended the HITL note: behavior rules gate *turns* (`PAUSE_CONVERSATION`); `hitlConfig.toolApprovals` gates *individual LLM tool calls* (`hitlPauseType: "TOOL_CALL"`); both share the same pause/timeout/audit/Slack machinery; link to `docs/hitl.md`.
* **`planning/hitl-framework-plan.md`** — flipped the decision-table line "Tool-level HITL: Deferred" to "Implemented — see `planning/hitl-tool-approval-plan.md`".

**The 5 product decisions (from the plan's decision record), now locked and documented:**

1. **`AUTO_APPROVE` never applies to tool pauses implicitly** — explicit opt-in only. Agent-level `AUTO_APPROVE` covers RULE pauses; for a tool pause it is demoted to `WAIT_INDEFINITELY` unless `toolApprovals.timeoutPolicy` sets `AUTO_APPROVE` explicitly (a silent timeout must never auto-execute a gated tool).
2. **Crash inside an approved tool yields an honest `EXECUTION_OUTCOME_UNKNOWN`** — never silent re-execution. The write-ahead journal (`IHitlToolJournalStore`, keyed by `conversationId + pauseEpoch + callId`) replays `EXECUTED` results and reports `EXECUTING` (crashed mid-tool) as genuinely unknown, audited and surfaced in `pauseDetails.outcomeUnknown`.
3. **Group-member tool pauses auto-reject gracefully** (`system:group`) — a group has no reviewer, so the member's gated call is REJECTED through the normal resume path and its LLM produces a coherent tool-less contribution (fallback: SKIP + auto-cancel). `inGroupTurns: "INBOX"` is reserved (400 in v1).
4. **Ungated calls in a mixed batch execute before the human sees the pause** — the approver is then shown which ones already ran via `pauseDetails.executedUngatedCalls`.
5. **A multi-day pause resumes against pause-time prompt state** — the exact in-flight langchain4j transcript is frozen at pause time and replayed on resume (same task index), never rebuilt from current memory.

**Cross-checked every documented fact against source** (`ToolApprovalsConfig`, `ToolApprovalPatterns`, `ToolApprovalGate`, `HitlDecision`/`ToolCallDecision`, `HitlConfigValidation`, `ConversationService.applyEffectiveToolTimeoutPolicy` + `validateToolDecisions`, `RestAgentEngine.buildToolCallPauseDetails`, `AgentOrchestrator.resumeToolLoop`, `IHitlToolJournalStore`, `SlackHitlSupport`, `GroupConversationService.tryResolveMemberToolPause`, and the `eddi.hitl.tool.enabled` flag in `LlmTask`/`application.properties`) — field names, defaults (`maxPausesPerTurn` 3/1..10, `maxAutoApprovalsPerTurn` 2/0..10), ranges, note caps (top-level 4096, per-call 1024), the 300-char Slack display truncation, and the precedence/timeout rules all match the implementation.

**Next:** the tool-level HITL feature (Tasks 1–17) is complete on `feat/hitl-framework`. Remaining HITL roadmap items: EDDI-Manager approvals UI (separate repo/PR) and the reserved `inGroupTurns: "INBOX"` mode.

***

## 🔐 MCP HITL surface — resolve approval gates over MCP (2026-07-03)

**Repo:** EDDI (`feat/hitl-framework`)

Exposes the HITL approval operations over the MCP server so an external MCP client (agent / orchestrator / ops console) can list, read, resume/approve, and cancel paused conversations and group discussions — at full parity with the REST endpoints, for both the regular (1:1) and group surfaces. Closes the loop `chat_managed`/`talk_to_agent` already open: they return `PAUSED_FOR_APPROVAL` but, until now, the client had to drop to REST to resolve it. Full plan: [`planning/mcp-hitl-surface-plan.md`](https://github.com/labsai/EDDI/tree/main/planning/mcp-hitl-surface-plan.md).

**Design — human authority preserved:** MCP is a transport, not a new authority; no tool lets an agent approve its own gate. Authorization mirrors REST exactly via a new shared `HitlAccessGuard` (extracted from `RestAgentEngine`/`RestGroupConversation`, so *who may decide* lives in exactly one place): per-conversation owner / `eddi-admin` / `eddi-approver`, owner-scoped listings, fail-closed on a missing descriptor. Decisions are attributed server-side as `mcp:<principal>` (mirroring the existing `system:timeout` convention). A global kill-switch `eddi.mcp.hitl.mutations.enabled` (default `true`) can make MCP a read-only HITL surface without touching REST.

**Method:** brainstorm → adversarial design critique (four subagent workflows: verify-assumptions + security + architecture + completeness; 28/28 code assumptions verified, \~26 findings triaged — folded in owner-scoped **group** listings, structured error codes, the discoverability hint, and metrics; **rejected** agent-level config, dev-mode fail-closed mutations, and a by-intent resume variant, each with reasons) → TDD execution, one commit per task. All new tests are plain Mockito (locally runnable, no Quarkus boot).

**Landed (each task = its own commit, all tests green locally):**

* `McpToolUtils.errorJson(msg, code, details)` — structured error JSON (`errorCode` ∈ `NOT_FOUND | WRONG_STATE | FORBIDDEN | DISABLED | BAD_REQUEST`), manual construction so it never throws on the error path.
* `HitlAccessGuard` — shared HITL ownership check + owner-scoped pending-approval listing (regular + group). `RestAgentEngine`/`RestGroupConversation` refactored to delegate the `hitlOperation=true` path (non-HITL paths untouched); existing REST HITL tests pass unchanged via a real guard wired with the same mocks.
* `McpHitlTools` — 9 `@Tool`s: `list_pending_approvals`, `get_approval_status`, `resume_conversation`, `cancel_conversation`, `list_group_pending_approvals`, `list_all_group_pending_approvals`, `get_group_approval_status`, `approve_group_phase` (optional `taskApprovals` JSON for TASK granularity), `cancel_group_discussion`. `@Blocking`, JSON returns, `eddi.mcp.hitl.*` metrics (verdict-tagged).
* `chat_managed`/`talk_to_agent` `PAUSED_FOR_APPROVAL` payload now names `"suggestNextTool": "resume_conversation"` so an LLM client can chain the approval over MCP.

**Pause-type-agnostic:** because the tool-level HITL layer (see below) reuses the same `AWAITING_HUMAN` state + `/resume` + single-verdict `HitlDecision`, the regular tools resolve **both** `RULE` and `TOOL_CALL` pauses unchanged; `get_approval_status` reports `pauseType` and exposes the pending tool-call batch via `detail=full`. No SSE/streaming variant over MCP, no autonomous approver, no new realm role.

**Docs:** `docs/hitl.md` (new *MCP Surface* section), `docs/mcp-server.md` (new *HITL Tools* category).

***

## 🛠️ Tool-level HITL — foundational layer (Tasks 1–4 of 17) (2026-07-03)

**Repo:** EDDI (`feat/hitl-framework`)

Implementing **tool-level HITL approval gating**: pausing a conversation for human approval when the LLM invokes a *gated tool* (any source — built-in `@Tool`, MCP, A2A, httpcall, dynamic-agent, memory, recall), configured via allow/disallow pattern lists, co-existing with the behavior-rule `PAUSE_CONVERSATION` mechanism. Full plan: [`planning/hitl-tool-approval-plan.md`](https://github.com/labsai/EDDI/tree/main/planning/hitl-tool-approval-plan.md). This closes the deferred "Tool-level HITL" limitation (`docs/hitl.md` Known Limitations).

**Architecture — Durable Re-entry (ToolGate-DR):** a batch gate in `AgentOrchestrator.executeWithTools()` intercepts gated calls *before execution* (fail-safe), serializes the exact in-flight langchain4j message list, persists it + pending-call metadata on `ConversationMemorySnapshot`, and aborts the LLM loop with an unchecked `ToolApprovalRequiredException` that `LifecycleManager` converts into the existing `ConversationPauseException` (new `pauseOrigin=TOOL_CALL`). Resume re-enters the **same** task index, replays the transcript, applies verdicts (write-ahead journal → at-most-once), and continues the loop. Chosen over a turn-completing "pending result" design after a 3-way design panel + 2 adversarial judges: durable replay uniquely preserves *exactly the state the human is approving against* (a multi-iteration turn that rebuilds from memory loses intermediate tool results).

**Method:** two adversarial subagent workflows (design-judge, then fact-verification against the codebase); 20 verification findings (1 blocker, 5 major) folded back into the plan before execution.

**Landed (each task = its own commit, TDD, all tests green locally):**

* **Task 1 — the architectural gate.** `ChatTranscriptCodec` wraps langchain4j 1.17.0 `ChatMessageSerializer`/`Deserializer` with a size cap + typed failure. Its test **empirically proves** the round-trip the whole design depends on (`AiMessage`+`ToolExecutionRequest`+`ToolExecutionResultMessage`+multimodal content survive serialize→deserialize). 6 tests. *If this had failed, execution was gated to stop-and-escalate — it passed.*
* **Task 2 — pattern engine + gate.** `ToolApprovalPatterns` (ReDoS-safe `*`-only glob, source-prefix validation with typo suggestions), `ToolApprovalGate` (batch classify; precedence exempt-beats-require; `source:name` then bare-name matching = fail-safe), `ToolApprovalsConfig` POJO. 9 tests.
* **Task 3 — config homes + validation.** `toolApprovals` on `AgentConfiguration.HitlConfig` (agent-level default) and `LlmConfiguration.Task` (per-task full-replace override). `HitlConfigValidation.validateToolApprovals` (actionable 400s: bad pattern w/ index, both-lists conflict, duplicates, exempt-without-require, range checks, reserved `INBOX`, timeout/reason length); wired into `LlmStore` create/update; agent-level `AUTO_APPROVE`-inheritance WARN. 15 tests + all existing validation suites still green.
* **Task 4 — memory model.** `PendingToolCallBatch` (+ `PendingToolCall`, size caps); `ConversationPauseException.PauseOrigin` (RULE default / TOOL\_CALL, backward-compatible 3-arg ctor); transient `hitlPauseType`/`hitlPendingToolCalls`/`agentToolApprovalsConfig`/`hitlResumeDecision` on `ConversationMemory`+`IConversationMemory`; persisted mirrors on `ConversationMemorySnapshot`; both-directions copy in `ConversationMemoryUtilities`. Round-trips through Jackson; legacy documents (null pauseType) treated as RULE. 4 tests + all existing HITL/resume/lifecycle suites green.

**Design decisions locked for the remaining tasks (flag if wrong):** (1) `AUTO_APPROVE` never applies to tool pauses implicitly — explicit per-gate opt-in only; (2) crash-inside-an-approved-tool yields honest `EXECUTION_OUTCOME_UNKNOWN`, never silent re-execution; (3) group-member tool pauses auto-reject gracefully (`system:group`); (4) ungated calls in a mixed batch execute before the human sees the pause (approver is shown which ran); (5) a multi-day pause resumes against pause-time prompt state.

**Adversarial review of the foundation (5 dimensions → per-finding verification):** 14 raw findings → 9 refuted, 5 survived, all minor. Fixes applied: (a) reject leading/trailing-colon patterns (`:foo`, `mcp:`) at save time — previously accepted but inert; (b) terminal cleanup (`ConversationMemoryStore` + `PostgresConversationMemoryStore` `clearHitlBookmark`) now also drops `hitlPauseType`/`hitlPendingToolCalls` so no stale tool-pause state lingers on ended/cancelled docs; (c) `ConversationMemoryUtilitiesHitlTest` extended to assert the two new fields round-trip both directions; (d) `PendingToolCallBatchSnapshotTest` doc clarified as a *structural* proxy (production uses BSON-backed `JacksonCodec`) with the real BSON round-trip routed to a CI Testcontainers IT (added to Task 14). No blockers, no majors.

**Next:** Task 5 (the gate hook + signal plumbing + pause commit in `AgentOrchestrator`/`LifecycleManager`/`Conversation` — the heaviest single task), then 6 (journal store), 7 (per-call verdict REST model), 8 (same-index re-entry), 9 (`resumeToolLoop`), 10 (timeout/no-progress), 11–13 (approver surfaces, Slack, delegated/group parity), 14 (crash recovery), 15 (lints), 16 (ITs), 17 (docs). Tasks 8→9 share a `resumeToolLoop` stub to keep each commit building.

***

## 🐰 CodeRabbit review triage — 21 fixes, adversarially verified (2026-07-03)

**Repo:** EDDI (`feat/hitl-framework`, PR #585)

CodeRabbit posted 23 actionable findings (16 inline + 7 outside-diff) plus observability nitpicks. Each was **adversarially re-verified against HEAD** (two parallel review passes) before any change — several overlapped fixes already made, some were stale/unreachable, and one CRITICAL-tagged item turned out already-mitigated-or-worse than described. Fixed the 21 that survived verification; skipped 2 as INVALID; deferred 1 as a scoped follow-up. **No PR threads were replied to or resolved** (standing instruction).

**Scheduler / crash-recovery:**

* **(HIGH) Lease-expired `CLAIMED` schedules were never reclaimable.** `findDueSchedules` returns lease-expired `CLAIMED` rows, but both `tryClaim` impls only matched `PENDING`/`FAILED` — so a crashed/wedged pod's claim was fetched every poll and never re-fired. `tryClaim` now takes a `leaseExpiry` and steals a `CLAIMED` row with `claimedAt <= leaseExpiry` (Mongo + Postgres, mirroring `findDueSchedules`).
* **(MEDIUM) `dispatchClaimed` per-future timeout could stack to N×leaseTimeout.** Now bounded by one shared batch deadline.
* **(MEDIUM) Retention sweep ignored group conversations.** `HitlCrashRecoveryObserver.sweepExpiredPendingApprovals` now also cancels expired group `AWAITING_APPROVAL` pauses (new `IGroupConversationService` dependency).
* **(MEDIUM) Crash-recovery re-arm could keep a stale timeout across pause→resume→pause.** The re-arm re-check now compares the pause bookmark (`pausedAt`), not just the awaiting state, at all three sites.
* **(HIGH) `PostgresScheduleStore` metadata (de)serialization failed open** (returned `null`, silently stripping the HITL contract). Now fails closed with `ResourceStoreException`.
* **(HIGH) `RestScheduleStore.requireAdminForHitl` failed open** on any read error. Now only `ResourceNotFoundException` falls through; other failures return 500 and stop the mutation.
* **(doc) "exactly-once" was an overclaim** — the design is at-least-once with idempotent HITL fire targets (the lease-steal above makes this explicit). Corrected `SchedulePollerService`/`IScheduleStore` javadoc + `docs/hitl.md`.

**Engine / conversation:**

* **(LOW) `endConversation` only disarmed the timeout when `AWAITING_HUMAN`** — a resume-in-flight `IN_PROGRESS` window could leave a stale timer. Now disarms unconditionally (idempotent).
* **(nitpick) `ConversationMemoryStore.compareAndSetState`** now uses `getMatchedCount()` (consistency with `storeConversationMemorySnapshotIfState`; avoids a no-op-CAS false negative).

**Group surface:**

* **(MEDIUM) Synchronous member-pause exception stranded the member approval.** `executeAgentTurn` now catches `ConversationAwaitingApprovalException` and routes to `handleMemberPause` (cancel + SKIPPED) instead of `handleAgentFailure`.
* **(MEDIUM) Cancel window between the resume CAS and control-token registration.** The `DiscussionControlToken` is now registered immediately after the CAS, so a concurrent cancel takes the signal path and stops before any phase runs.
* **(LOW) `RestGroupConversation` reflected raw ids** in `requireGroupMembership`/`validateGroupConversationOwnership` `NotFoundException` messages, and **(MEDIUM)** the streaming approve endpoint echoed raw exception text over SSE. Both now curated (generic message + sanitized server-side log), matching the non-streaming hardening.
* **(MEDIUM) `GroupConversationStore.findByState` aborted the whole batch** on one record's `ResourceStoreException`. Now logged-and-skipped per record (mirrors `listByGroupId`).

**Slack / MCP tools:**

* **(HIGH) Slack approval-notification idempotency was too coarse** (keyed by `conversationId`, marked before the post). Now keyed per-pause (`hitlPausedAt`) and cleared on failed delivery, so retries deliver and a second distinct pause is not suppressed.
* **(HIGH) Slack HITL `resolveOwningIntegration` fell back to a by-approval-channel lookup** for unbindable (bare) action values, reintroducing shared-channel cross-integration ambiguity. Removed — bare values now resolve to empty and are rejected (403).
* **(HIGH) MCP `talk_to_agent`/`chat_with_agent` reported a deliberate `AWAITING_HUMAN` pause as BUSY** (and `chat_with_agent` lost a freshly-created conversation id on skip). Both now return a structured `PAUSED_FOR_APPROVAL` and preserve the created id.
* **(MEDIUM) `CreateSubAgentTool`** treated a skipped initial turn as a real reply. Now mirrors `ConverseWithAgentTool`'s `onSkipped` handling.
* **(LOW) `RestSlackWebhook`** malformed percent-encoding threw → 500 before signature check. Now caught → 400.
* **(HIGH) `SecretRedactionFilter`** Bearer rule only matched dotted JWTs; opaque tokens leaked. Now redacts opaque tokens too (possessive, ReDoS-safe).

**Skipped/deferred (with reason):**

* **INVALID:** F9 (fractional-second read compat) — moot for unreleased/disposable schedule rows and would reintroduce the seconds-heuristic the epoch-millis fix removed; F14 (`ConverseWithAgentTool` ERROR-skip) — `onSkipped` provably never receives `ERROR`.
* **DEFERRED (follow-up task):** owner-scoped group pending-approvals query (owner filter applied after the limit → possible starvation). The safe fix needs a DB-agnostic exact-match for `userId` (the query layer treats string filters as regex on both backends; `Pattern.quote` is Postgres-incompatible), so it warrants its own focused change rather than a rushed regex that could over-match.
* Observability nitpicks (Micrometer counters, `SafeHttpClient` for the fixed Slack host, managed executor for one-shot startup recovery, `CREATE INDEX CONCURRENTLY`, test-style suggestions) — intentionally out of scope for this correctness/security pass.

Every fix has a regression test; all touched unit suites are green locally (Testcontainers ITs run in CI).

***

## 🔬 Critical adversarial re-review — 7 findings fixed (2026-07-03)

**Repo:** EDDI (`feat/hitl-framework`, PR #585)

A final 6-reviewer / adversarial-verify pass over the HITL branch surfaced seven confirmed defects (1 CRITICAL, 1 HIGH, 4 MEDIUM, 1 LOW). All fixed with regression tests:

* **CRITICAL — `MongoScheduleStore` truncated every `Instant` to epoch-SECONDS (1000× too small).** `toDocument()` round-trips through the shared Jackson mapper, which (with `write-dates-as-timestamps=true` + `JavaTimeModule`, default `WRITE_DATE_TIMESTAMPS_AS_NANOSECONDS`) serializes an `Instant` as fractional **seconds** (`1719964800.123`). Deserialized into a `Document` that's a `Double`; `convertInstantField`'s `num.longValue()` then stored epoch **seconds** (\~1.7e9), not millis (\~1.7e12). Since `findDueSchedules` compares `nextFire <= nowMs` (millis), **every future-armed schedule looked immediately due** — HITL approval timeouts (and Dream/maintenance) fired on the very next poll instead of after their configured duration, and it diverged from `PostgresScheduleStore` (which stores millis). This is why the earlier ISO-string "defence in depth" (commit `dc117cddc`) never actually fixed it: the numeric `Number` branch's millis assumption was the real bug, and no test asserted a *future* `nextFire`. Fix: `MongoScheduleStore` now writes every date field as an epoch-millis `Long` straight from the getters (`writeScheduleInstants`/`writeFireLogInstants`) and reads them back via `readEpochMillis` (`Instant.ofEpochMilli`), stripping them before the Jackson build — **both directions are now independent of the mapper's date format** (the seconds-based `convertInstantField` and its ISO branch are gone), mirroring `PostgresScheduleStore.setNullableEpoch`/`instantFromEpoch`. New `MongoScheduleStoreInstantRoundTripTest` exercises the REAL serialization (the sibling test mocks it, which is why the bug hid) and asserts a future `nextFire` is stored as millis and round-trips; the Mongo/Postgres store ITs gain an epoch-millis assertion; branch-coverage tests rewritten for the new helpers.
* **HIGH — `SchedulePollerService.dispatchClaimed` could still pin the poll thread forever.** The per-fire `future.get(leaseTimeout)` bound was undermined by the try-with-resources `ExecutorService.close()`, which awaits termination indefinitely; `future.cancel(true)` only unblocks tasks that honor interruption, but the real fire path can stall on a NON-interruptible synchronous DB socket read. Replaced try-with-resources with an explicit `finally { executor.shutdownNow(); }` (no `awaitTermination`) so a wedged fire leaks a single cheap virtual thread instead of freezing all scheduling. New regression test uses a fire stub that swallows interruption and asserts the poll cycle still returns.
* **MEDIUM — resume `onComplete` could clobber a terminal state (lost update / resurrection).** The resume persist was an unconditional full-document store guarded only by a single up-front `isCancelled()` check; a concurrent `end`/`cancel` landing between the check and the store overwrote `ENDED`/`EXECUTION_INTERRUPTED` with `READY`, resurrecting a terminated conversation that then accepted new `say()` input. Added `IConversationMemoryStore.storeConversationMemorySnapshotIfState(snapshot, expectedState)` — an atomic compare-and-store (Mongo: `replaceOne` with a state predicate; Postgres: `UPDATE … WHERE conversation_state = ?`) — and the resume `onComplete` now persists only while it still owns `IN_PROGRESS`, discarding its outcome (no schedule/notify) when a terminal writer won.
* **MEDIUM — a transient snapshot-load failure during resume permanently dropped the finite-timeout schedule.** `resumeConversation` deleted the HITL timeout schedule *before* loading the snapshot; a transient load failure then restored `AWAITING_HUMAN` without re-arming, so an `AUTO_REJECT`/`AUTO_APPROVE`/`ABORT` policy silently degraded to wait-forever until the next restart. The delete now runs only *after* the snapshot loads and the agent is confirmed deployed — a pre-execution failure leaves the original timer armed. (The `AWAITING_HUMAN→IN_PROGRESS` CAS already prevents the timeout firing concurrently, so nothing races on the deferred window.)
* **MEDIUM — Slack group HITL pause leaked a parked virtual thread per paused discussion.** `SlackGroupDiscussionListener.completionLatch` was counted down only in the terminal callbacks; `onHitlPause` (a pause is terminal for *this* listener — resume flows through a different instance) did not, so `SlackEventHandler.registerAgentThreadMappings` parked on `awaitCompletion(300s)` for every paused expanded-mode discussion and follow-up thread routing was unavailable for that window. `onHitlPause` now counts the latch down in a `finally`.
* **MEDIUM — dead-letter endpoints bypassed the HITL admin guard.** `RestScheduleStore.dismissDeadLetter`/`retryDeadLetter` were the only schedule-by-id mutations lacking `requireAdminForHitl()`, so a non-admin editor could disarm an `ABORT`/`AUTO_REJECT` safety timeout (`dismissDeadLetter` on a one-shot → `markCompleted(id, null)` → disabled). Both now carry the guard; regression tests assert 403 for a non-admin on a HITL timeout schedule.
* **LOW — group HITL REST error bodies reflected raw ids/exception text.** The CodeQL XSS/info-exposure hardening applied to `RestAgentEngine` was not mirrored on `RestGroupConversation`; `cancelDiscussion`/`approveGroupPhase`/`getGroupApprovalStatus` echoed `e.getMessage()` (which embeds the caller-supplied `gcId` for the gone/404 case). Now curated `text/plain` bodies with the detail logged server-side (id sanitized), same HTTP status codes.

***

## 🔍 Copilot PR review — 5 findings (2026-07-03, post-push)

**Repo:** EDDI (`feat/hitl-framework`, PR #585)

Automated PR review (`copilot-pull-request-reviewer`) flagged 5 issues in the already-pushed HITL work; all fixed:

* **`SchedulePollerService.dispatchClaimed`**: each claimed fire's `Future#get()` had no timeout — one stalled downstream call (e.g. a hung LLM call inside a fired schedule) could block the `@Scheduled` poll loop forever, stopping this instance from claiming or firing *any* further schedule. Now bounded by `leaseTimeout` (the same window after which another instance may reclaim the schedule anyway) — a timeout cancels the future (best-effort interrupt) and logs, instead of hanging. New regression test asserts the poll cycle returns promptly under a stalled fire.
* **`SchedulePollerService.claimSchedule`**: after a successful CAS claim, the in-memory `ScheduleConfiguration` still carried its pre-claim `fireId` (often `null`) — both `tryClaim()` implementations (Mongo, Postgres) persist a fresh `fireId` (`scheduleId + "_" + now`) but only return a `boolean`, so the caller never saw it. `ScheduleFireExecutor` uses this field for fire-log correlation and injects it into the agent context, so every claimed fire's correlation id was wrong. The poller now derives the identical value and sets it on the in-memory object after a successful claim. New regression test asserts the fired schedule carries a non-null, correctly-derived `fireId`.
* **`ConversationPauseException`**: the exception message was built as `"Conversation paused: " + pauseReason`, producing the confusing `"Conversation paused: null"` in logs/clients when no `pauseReason` was configured (the common case, since `pauseReason` is optional). Now falls back to `"human approval required"` for the message only — `getPauseReason()` still returns the raw (possibly null) value for callers that need it.
* **`GroupConversationStore.listByGroupId`**: called `SAFE_ID.matcher(groupId)` without a null guard — a `null` groupId (a defensive REST layer, or an internal caller) threw NPE instead of returning an honest empty list. Fixed; regression test added. (The sibling `findByState` already guarded this correctly.)
* **`UserConversationStore`/`PostgresUserConversationStore`**: `readUserConversationByConversationId` (added by the Slack HITL continuation-push work) queried by `conversationId` with no supporting index — a full collection/table scan on every reverse lookup (which happens on every HITL resume that needs to notify a Slack thread). Added a dedicated index on both backends (Mongo: ascending index; Postgres: expression index on the JSONB field) and a regression test asserting the Mongo index is created on construction.

***

## 🛡️ CI gate fixes — CodeQL (XSS/ReDoS) + Postgres Instant serialization (2026-07-03, final gate)

**Repo:** EDDI (`feat/hitl-framework`, PR #585)

**CodeQL (high-severity) — both blockers fixed:**

* `RestAgentEngine` HITL error bodies reflected the raw `conversationId` path param (`java/xss`) and echoed internal exception messages to the client — now `text/plain` with curated, non-reflecting messages; detail logged server-side (id sanitized). New-in-PR, HITL-introduced.
* `SecretRedactionFilter` ReDoS (`java/polynomial-redos`, pre-existing on `main`): the redaction patterns used ambiguous/unbounded backtracking quantifiers, quadratic on adversarial inputs (long `${vault:` repetitions). Made the quantifiers **possessive** (`++`/`*+`/`{n,}+`) — behavior-preserving (every quantified class is followed by a literal outside the class, so no backtrack is ever needed for a correct match) and linear-time. `SecretRedactionFilterTest` 6/6 still green. Addressed here because it was the sole remaining CodeQL gate blocker.

Final full-suite verification surfaced a latent Postgres-only defect: the JSONB-backed resource storage serializes snapshots through the shared Jackson mapper (`SerializationCustomizer.configureObjectMapper`), which did **not** explicitly register `JavaTimeModule` — it relied on Quarkus auto-registration. The HITL bookmark carries an `Instant` (`hitlPausedAt`); a **non-null** `Instant` (i.e. an actual pause) would fail serialization ("Java 8 date/time type not supported") on the Postgres backend, so HITL pause persistence was one module-registration-order change away from breaking on Postgres (Mongo was unaffected — its BSON codec handles `Instant`; null `Instant`s serialize fine, which is why it stayed latent). `SerializationCustomizer` now registers `JavaTimeModule` explicitly — the date **format is deliberately left at the default numeric timestamps** (an initial attempt to also switch to ISO-8601 strings was reverted: `MongoScheduleStore` normalizes date fields to epoch-millis for numeric range queries and expects numbers, so an ISO string silently broke `findDueSchedules`). As defence in depth, `MongoScheduleStore.convertInstantField` now also parses ISO-8601 strings → epoch-millis, so schedule storage is correct regardless of the mapper's date format. The Postgres store tests (and the MCP HITL test) now build their mapper via `configureObjectMapper` instead of a bare `new ObjectMapper()`, so they exercise the production serialization path — `PostgresConversationMemoryStoreTest` goes 20/22 → 22/22.

***

## 🛠️ HITL Round-2 Engine-Core Remediation (2026-07-03, WS-G)

**Repo:** EDDI (`fix/hitl-r2-engine`, branched from `feat/hitl-framework`) **Scope:** Confirmed findings from a second adversarial review of the HITL remediation, engine-core only (no `integrations/slack/**`, no tool/mcp bridges).

* **G1 (HIGH) — queued say resurrecting a terminated conversation:** `ConversationService.processConversationStep`'s queued-turn skip set only covered AWAITING\_HUMAN/IN\_PROGRESS. Added ENDED and EXECUTION\_INTERRUPTED so a say queued behind a running turn that terminates (endConversation / cancel) before it runs is routed through `onSkipped` with the persisted terminal state instead of executing the pipeline and persisting READY over the terminal state.
* **G2 (HIGH) — say-path onComplete ignoring cooperative-cancel (group member-pause stranded approval):** `runGuardedConversationStep`'s onComplete now checks `memory.isCancelled()` (parity with the resume path). A concurrent cancel/end (e.g. group `handleMemberPause → cancelConversation` that loses both state CAS races because the pause isn't persisted yet) makes the completing turn skip pause persistence/schedule/counter and CAS the running state to EXECUTION\_INTERRUPTED — the approval is never stranded.
* **G3 (HIGH) — approval gate bypass via schedule create/update:** `RestScheduleStore.createSchedule`/`updateSchedule` now reject any request BODY whose metadata is a `hitl_timeout` (via `HitlSchedules.isHitlTimeout`) for EVERYONE (even admin) with 400 — these schedules are minted internally only. Closes the forge/convert path that let an editor mint a timeout schedule the poller would fire to force-resume/abort a victim's approval unauthenticated.
* **G4 (MEDIUM) — endConversation terminating a pause with no audit/actor:** added `endConversation(String, String endedBy)`; the AWAITING\_HUMAN branch now writes the `hitl.approval` cancellation audit with the actor. Callers attribute: RestAgentEngine → principal, RestConversationStore delete/bulk-end paths → `system:admin-end`, 1-arg overload → `system:end`. AgentDeploymentManagement already SKIPs paused conversations (verified — no change).
* **G5 (MEDIUM) — cancel/end never fired HitlResumeCompletedEvent:** cancelConversation's pauseCancelled branch and endConversation's AWAITING\_HUMAN branch now fire `HitlResumeCompletedEvent` with `verdict=null`, the cancelling/ending actor, and the terminal snapshot (new `fireHitlResumeCompletedTerminal` helper, async + fully isolated). The Slack observer renders these; the event's fields/signature are unchanged.
* **G6 (MEDIUM) — retention sweep attributed to "unknown":** `HitlCrashRecoveryObserver.sweepExpiredPendingApprovals` now calls the 3-arg `cancelConversation(id, CANCEL_GRACEFUL, "system:retention")`.
* **G7 (LOW) — restored pause re-armed at now+timeout:** `ConversationService.scheduleHitlTimeout` and `GroupConversationService.scheduleGroupHitlTimeout` now anchor `fireAt` to `pausedAt + timeout` (clamped to `now + 2m` grace, mirroring crash recovery) so restore-after-failed-resume re-arms at the original deadline instead of extending it.
* **G8 (test) — vacuous Postgres zombie regression:** added `loadReportsColumnStateOverForgedDivergentDocument` (Testcontainers, CI-only) that forges TRUE document/column divergence via raw SQL and asserts both `loadConversationMemorySnapshot` and `loadActiveConversationMemorySnapshot` report the COLUMN state — deleting `applyStateColumn` now fails a test.
* **Cheap extras:** `RestAgentEngine.resumeConversation` null-guards `identity.getPrincipal()` (parity with cancel/end — no NPE for anonymous). The resume `catch (IllegalStateException)` carve-out is narrowed via a private `AgentNotDeployedForResumeException` sentinel so ONLY the deliberate agent-not-deployed ISE re-throws without a double restore; any other ISE (e.g. from continueConversation) now restores the pause and maps to 500.

Verification: `./mvnw -q compile test-compile` clean. Plain-Mockito tests for all touched classes run green locally (G1/G2 in ConversationServiceSayHitlTest, G4/G5 end in ConversationServiceTest, G5 cancel in ConversationServiceHitlTest, G3 in RestScheduleStoreTest, G6 in HitlCrashRecoveryObserverTest, plus reconciled RestAgentEngineTest/RestConversationStoreTest). The G8 Postgres test is Testcontainers → CI-only.

***

## 🔐 HITL Slack round-2 remediation — IDOR fix + approval-flow correctness (2026-07-03, WS-H r2)

**Repo:** EDDI (`fix/hitl-r2-slack-impl`, branched from `feat/hitl-framework`) **Scope:** Ten confirmed re-review findings (H1–H10) plus cancellation-rendering (H-consume) on the new Slack HITL surface + tool bridges. Ownership: `integrations/slack/**`, `integrations/channels/ChannelTargetRouter.java`, `modules/llm/tools/ConverseWithAgentTool.java`, `engine/mcp/McpConversationTools.java`, and their tests. No `engine/internal/*` or `engine/api/*` touched.

### H1 (HIGH, security) — cross-integration IDOR on `/interactive`

`/interactive` previously verified the raw-body signature against the **pooled** set of all Slack signing secrets (incl. legacy per-agent ChannelConnector secrets) and only checked authz afterward — so a holder of ANY one Slack secret could forge an approval on another integration's paused conversation. **Fix:** the decision is now **bound to the integration that owns it**, carried explicitly in the button value. New `SlackSignatureVerifier.verifyWithSecret(ts, body, sig, secret)` verifies against exactly ONE secret. `RestSlackWebhook.handleInteractive` resolves the owning integration's secret from the payload first (`SlackInteractivityHandler.resolveSigningSecretForDecision`), then verifies against only that; an unbindable decision (legacy/unknown → no owning new-style integration) is rejected. `/events` keeps pooled verification (unchanged).

### H2 (MEDIUM) — shared-approval-channel nondeterminism

The approval button `value` format changed from `<subject>` to **`<integrationName>|<subject>`** (`<integrationName>|<conversationId>` and `<integrationName>|group:<gcId>`). The handler resolves the owning integration by NAME (`ChannelTargetRouter.getIntegrationByName`), not by an arbitrary-first channel lookup — so authz + verification are deterministic even when integrations share one `hitlApprovalChannel`. New `SlackHitlSupport.buildActionValue`/`parseActionValue` (+ `ActionValue` record). Legacy bare values (no name) parse with a null integration and are rejected up-front (acceptable per spec).

### H3 (MEDIUM) — group double-click idempotency

`SlackInteractivityHandler.resolveGroup` caught only `IllegalStateException`; `resumeDiscussion` signals a non-paused group with the CHECKED `GroupDiscussionException`, a race with `ResourceModifiedException`, and a deleted group with the unchecked `GroupConversationGoneException` — all fell into the generic catch (warn-spam + live buttons). Now all four route to `finalizeAlreadyResolved`.

### H4/H7 — dropped-turn (onSkipped) discrimination

`SlackEventHandler.sendAndWait` now uses a full `ConversationResponseHandler` overriding `onSkipped`, mapping a skip to sentinel snapshots: `AWAITING_HUMAN` → STILL\_AWAITING notice (no second approval card, H4); else → new `CONVERSATION_NOT_ACTIVE_NOTICE` (H7). `ConverseWithAgentTool` and `McpConversationTools` (talk\_to\_agent/chat\_with\_agent/chat\_managed) detect `onSkipped` and return busy/not-active (chat\_managed: AWAITING\_HUMAN-skip → PAUSED\_FOR\_APPROVAL) instead of replaying the previous turn's output (mirrors RestAgentEngine's 409 discrimination).

### H5 + H-consume — resume ERROR / cancellation rendering (defensive)

`SlackHitlResumeObserver.decisionSummary` now takes the snapshot: an approved resume that ended in `ERROR` renders a failure ("continuation failed…") not "continuing" (H5), and a null-verdict event (cancel/timeout-abort/end, terminal `EXECUTION_INTERRUPTED`/`ENDED`) renders "⛔ cancelled or expired" — the previously-dead branch is now live and correct (H-consume). Implemented **defensively**: works whether or not the engine's parallel change to fire `HitlResumeCompletedEvent` with `verdict==null` on cancel/end has landed. (Approval-card button-removal on cancel is not done — the approval card's message ts is not resolvable from the conversationId in the current data model; the thread message, which IS resolvable, is delivered.)

### H6 — "Bearer null" guard

`notifyApprovers` now skips the call and logs an explicit "no bot token — HITL approval notification NOT delivered for " error when neither the resolved integration token nor the approval-channel lookup yields a non-blank token (mirrors `postMessage`'s guard).

### H8 — init-turn (CONVERSATION\_START) pause never notified approvers

An init-turn pause happens inside `getOrCreateConversation → startConversation`; the first user say then throws `ConversationAwaitingApprovalException` and no approval card was ever posted. The exception branch now calls `notifyApprovers`, made idempotent via a `slack-hitl-approval-notified` cache (`putIfAbsent`) so re-message-while-paused never posts a second card.

### H9 — follow-up conversations get the resume continuation push

`SlackHitlResumeObserver` now recognizes the `channel:followup:<channelId>:<parentTs>` intent shape (in the prefix guard and `parseIntent`) in addition to `channel:slack:...`, so agent-thread follow-ups receive the verdict/continuation in their thread.

### H10 — pause card read the bookmark before it was persisted

The say callback completes before `ConversationService` persists the HITL bookmark, so `getConversationMemorySnapshot` re-reads returned the previous turn (null pause reason/timeout). New `loadHitlBookmark` retries the read (5×100ms) until state==AWAITING\_HUMAN and is loaded ONCE per pause, shared by the in-thread notice and the approval card.

### New contracts

* `SlackSignatureVerifier.verifyWithSecret(String ts, String body, String sig, String secret)` — single-secret (integration-bound) verification for `/interactive`.
* `SlackInteractivityHandler.resolveSigningSecretForDecision(String payloadJson)` — resolves the owning integration's signing secret from the button value; null → endpoint rejects.
* `ChannelTargetRouter.getIntegrationByName(String channelType, String name)` — deterministic by-name lookup.
* Approval button `value` format: **`<integrationName>|<conversationId>`** / **`<integrationName>|group:<gcId>`** (was bare `<conversationId>` / `group:<gcId>`).

### Tests

Plain JUnit/Mockito (compile + test-compile pass; touched suites run green locally): `SlackSignatureVerifierTest` (+verifyWithSecret), `SlackHitlSupportTest` (+buildActionValue/parseActionValue), `SlackInteractivityHandlerTest` (rewritten for value-binding + H1 cross-integration + H3 group double-click + resolveSigningSecretForDecision), `SlackHitlResumeObserverTest` (3-arg decisionSummary, ERROR/cancellation/followup delivery), `RestSlackWebhookTest` (new interactivity flow), `SlackGroupDiscussionListenerTest` (integration-bound group value), `ConverseWithAgentToolHitlTest`/`McpConversationToolsHitlTest` (H7 skip), `ChannelTargetRouterRefreshTest` (getIntegrationByName).

***

## 🧪 HITL Coverage Closure + Schedule-Contract Consolidation (2026-07-03, WS-F + merge)

**Repo:** EDDI (`feat/hitl-framework`, PR #585)

**Coverage (WS-F, findings 10/22/23/24/41/43):** 13 test files, +1469 lines, tests only. Queued-say guard + say fast-fail (finding 10 — previously zero coverage), zombie-pause discard guard, the entire finite-timeout leg (schedule creation/metadata routing/fire-log parity/error isolation/delete-on-resume+cancel, initial say-path arming), regular-surface endpoint authz incl. fail-closed missing-descriptor, resume robustness against REAL workflow lists (config drift → ERROR, multi-workflow continuation order), HitlConfigValidation wiring at AgentStore/AgentGroupStore CRUD + the import seam, storage regressions (Postgres zombie: post-CAS load must report the column state; `jsonb_set` convergence; owner-filtered summaries; `storeIfFieldEquals` deleted-404 vs mismatch-409 on both backends; anchored group filters + SAFE\_ID rejection), case-insensitive verdict round-trip, REJECTED-path `ConversationOutput` visibility + ACTIONS strip. Testcontainers classes (`PostgresConversationMemoryStoreTest`, `MongoConversationMemoryStoreTest`) execute in CI; everything else ran green locally (202 tests). Known residual gaps documented in the test agent's report: no wall-clock end-to-end timeout IT (every seam unit-covered), full ZIP pipeline (validation seam covered).

**Consolidation:** new `ai.labs.eddi.engine.hitl.HitlSchedules` — single source of truth for the HITL timeout-schedule contract (names `hitl-timeout-*`/`hitl-timeout-group-*`; metadata keys `hitlType`/`policy`/`surface`/`conversationId`; `isHitlTimeout` predicate) — adopted by ConversationService, GroupConversationService, ScheduleFireExecutor, HitlTimeoutHandler, HitlCrashRecoveryObserver, RestScheduleStore. Closes the "HITL lifecycle glued by magic strings across five classes" review finding.

***

## 🔌 HITL — Slack integration + nested-consumer bridges (2026-07-03, WS-E)

**Repo:** EDDI (`feat/hitl-ws-e-slack`, branched from `feat/hitl-framework`) **Scope:** Human-in-the-loop support in the Slack channel adapter, plus finding 25 (nested/managed/delegated conversation consumers stranded by a pause).

### Slack HITL surface

* **In-thread pause notice (`SlackEventHandler`):** when a say returns an `AWAITING_HUMAN` snapshot, the output-so-far is posted followed by a pause notice ("⏸️ This conversation is awaiting human approval" + pause reason from the HITL bookmark). A `ConversationAwaitingApprovalException` on subsequent messages posts "Still awaiting approval — a reviewer must decide…" instead of the generic error. Follow-up (group-thread) replies get the same handling.
* **Approver notification + Approve/Reject buttons (config-driven, fail-closed):** two new **optional** `ChannelIntegrationConfiguration.platformConfig` keys — `hitlApprovalChannel` (Slack channel id for approval notifications) and `hitlApproverUserIds` (comma-separated Slack user ids allowed to decide). On pause, an interactive Block Kit message is posted to the approval channel (conversationId, agent, reason, timeout policy/deadline). Buttons are rendered **only** when `hitlApproverUserIds` is set (otherwise notification-only). Data minimization: only the pause reason is included, never the user's message.
* **Interactivity endpoint (`RestSlackWebhook`):** new `POST /integrations/slack/interactive` (form-urlencoded, `payload` param). Verifies the Slack signature over the RAW body with the existing `SlackSignatureVerifier` + router signing secrets, acks 200 within 3s, processes async on a virtual thread (`SlackInteractivityHandler`). Handles `block_actions` (`hitl_approve`/`hitl_reject`). AUTHZ is fail-closed against the owning integration's approver list; `decidedBy` is always derived server-side (`slack:<userId>`). On success it `chat.update`s the message ("✅ Approved by …" / "⛔ Rejected"), removing buttons; an already-decided/timed-out click resolves to the `IllegalStateException` path and updates the message without error-spam (idempotent double-click).
* **Continuation push after resume (CDI event):** `ConversationService` fires a new async CDI event `ai.labs.eddi.engine.events.HitlResumeCompletedEvent` when a resume settles to a non-paused state (in `resumeFinished.onComplete`, after `storeConversationMemory`, `fireAsync` so observers never block the engine; failures isolated). `SlackHitlResumeObserver` observes it, resolves the conversation's Slack routing via the new reverse-lookup `IUserConversationStore.readUserConversationByConversationId` (implemented on both Mongo + Postgres for parity), and posts the verdict + continuation output to the originating channel/thread. Timeout (`system:timeout`) and cancellation outcomes flow through the same event.
* **Group discussions (`SlackGroupDiscussionListener`):** implemented the HITL listener callbacks — `onHitlPause` (thread notice + approval-channel buttons whose action value carries `group:<groupConversationId>`, routed to `resumeDiscussion`), `onHitlResume` (verdict), `onMemberPauseSkipped`, and `onCancelled`.
* **Shared helpers:** `SlackHitlSupport` (config keys, Block Kit builders, approver authz, and the Slack-friendly response-text extraction refactored out of `SlackEventHandler`); `SlackWebApiClient` gained `postBlocksMessage` + `updateMessage` (chat.update), reusing a shared send/parse helper with the existing retry/backoff classification.

### Finding 25 — nested/managed/delegated consumers

* `ConverseWithAgentTool`, `McpConversationTools#chat_managed`, and `CreateSubAgentTool` now detect the delegated conversation pausing (AWAITING\_HUMAN snapshot **or** `ConversationAwaitingApprovalException`) and return a structured, actionable `PAUSED_FOR_APPROVAL` result (with the conversationId and the `/resume` instruction) instead of "\[no response]" or a 60s hang. The nested pause is **not** auto-cancelled (a delegated approval may be intended) and managed mappings are preserved so re-invoking after approval continues the same conversation.

### Tests

Plain JUnit/Mockito (no Quarkus boot): `SlackHitlSupportTest`, `SlackInteractivityHandlerTest`, `SlackHitlResumeObserverTest`, `ConverseWithAgentToolHitlTest`, `McpConversationToolsHitlTest`, plus additions to `SlackEventHandlerTest`, `SlackGroupDiscussionListenerTest`, `RestSlackWebhookTest`. Covers signature rejection on `/interactive`, unauthorized user cannot decide, authorized resume with `decidedBy=slack:…`, double-click no error-spam, buttons omitted without approvers, observer posts to the right channel/thread and ignores non-Slack conversations, and the bridges' PAUSED\_FOR\_APPROVAL result. `docs/hitl.md` needs a new "Slack" section (docs phase — not edited here).

***

## 🔧 HITL Production-Readiness Remediation — storage parity + say-path contract (2026-07-03, session 5)

**Repo:** EDDI (`feat/hitl-framework`, PR #585) **Trigger:** A 92-agent adversarial review of the branch confirmed 46 findings (1 CRITICAL, 7 HIGH, 18 MEDIUM, 20 LOW). This entry covers the storage-layer and regular-surface batches; parallel batches (schedule security/sweeps, group surface) land as separate commits from their own branches.

### CRITICAL — PostgreSQL conversation-state duality (the Postgres zombie)

State was persisted twice on Postgres: in the indexed `conversation_state` column (updated by CAS/cancel) AND inside the JSONB snapshot (read by loads). A cancelled or ABORT-timed-out pause kept reporting `AWAITING_HUMAN` from the stale document — wedging `say()`, showing phantom pauses in approval-status, and letting the next user message resurrect the terminated approval as a zombie (full-document store flips the column back, re-arms a timeout; a later approve fails into ERROR). Fixes, defense in depth:

* **Column wins on load** — `loadConversationMemorySnapshot`/`loadActiveConversationMemorySnapshot` overwrite the deserialized state with the `conversation_state` column (`applyStateColumn`).
* **Writers converge the document** — `setConversationState` and `compareAndSetState` also `jsonb_set` the JSONB `conversationState` in the same statement.
* **Say-path zombie guard** — `runGuardedConversationStep.onComplete` discards a turn result whose `AWAITING_HUMAN` state was already present at submit time (a stale pause the turn did not produce is never re-persisted or re-armed).

### HIGH — say() into a paused conversation: honest 409 instead of a 60s hang

`say()`/`sayStreaming()` now **fast-fail** with a new `ConversationAwaitingApprovalException` → REST 409 with an actionable body (matches the docs' "say() is rejected" promise, mirrors ENDED→410). The queued-say race backstop now completes the response via a new `ConversationResponseHandler.onSkipped(snapshot)` (default: delegates to `onComplete`) instead of dropping the turn into the 408 watchdog — and the `processingConversationReferences` gauge entry is removed on every exit path (was a permanent leak per dropped request). `RestAgentEngine` maps skipped turns to 409 ("awaiting approval" vs "busy — retry"). Callback consumers (group, Slack, MCP) are unaffected: the default `onSkipped` delivers the snapshot whose state they already inspect.

### Storage-layer fixes (DB-agnostic parity)

| Fix                         | Detail                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Postgres regex 500s         | `GroupConversationStore` built filters with `Pattern.quote` (`\Q…\E`) — valid in Mongo's PCRE, **rejected by PostgreSQL's regex engine** → group listing + pending-approvals 500'd on PG. Replaced with charset-validated plain anchoring (`^id$`; ids are hex/UUID, no metacharacters). Non-id input → honest empty result.                                                                                                                                                                                                    |
| Projected pending summaries | Postgres now runs ONE projected query (JSONB field extraction, `hitlPausedAt` round-tripped through the same Jackson mapper) instead of `1+limit` full-document deserializations; Mongo now runs ONE projected query instead of N+1 point-reads.                                                                                                                                                                                                                                                                                |
| Owner-scoped inbox          | `findPendingApprovalSummaries(ownerUserId, limit)` implemented on both backends — the owner filter is INSIDE the query, so the limit applies after the restriction (a non-admin's inbox can no longer be starved by other users' backlog). New Mongo compound index `(conversationState, userId)`. `RestAgentEngine` uses it for non-admin/non-approver callers.                                                                                                                                                                |
| CAS 404-vs-409              | `storeIfFieldEquals` (both backends) now distinguishes "document deleted" (`ResourceNotFoundException`) from "field mismatch" (`ResourceModifiedException`) via an existence check on zero-match. `GroupConversationStore.updateIfState` surfaces deletion as unchecked `GroupConversationGoneException` (kept unchecked so existing CAS call sites compile; surfaces map it to 404). The `IResourceStorage` default no longer silently degrades the CAS to an unconditional store — it throws `UnsupportedOperationException`. |
| Truncation visibility       | `findByState` WARNs when it hits its limit (pending listings / crash recovery must never truncate silently).                                                                                                                                                                                                                                                                                                                                                                                                                    |

### Regular-surface fixes

* **End-vs-resume race:** the resume pre-execution guard now also aborts on persisted `ENDED` (previously only `EXECUTION_INTERRUPTED`) — an accepted resume can no longer resurrect an ended conversation.
* **Cooperative-cancel integrity:** all `inFlightConversations.remove(key)` calls are now value-conditional `remove(key, memory)` — a finishing leg can no longer evict a newer execution's registration.
* **Cancel attribution (EU AI Act):** `cancelConversation(id, mode, cancelledBy)` threads the actor into the `hitl.approval` audit entry (`decidedBy` + `automated`); REST passes the principal, `HitlTimeoutHandler` passes `system:timeout`. Old 2-arg signature delegates (`unknown`).
* **Configurable pause reason:** new optional `hitlConfig.pauseReason` (agent-level, ≤500 chars, validated at save/import) flows into the bookmark → pending-approvals/approval-status answer "what am I approving?". Falls back to the generic constant.
* **approval-status payload** now includes `approvalTimeout` so UIs can render the auto-decision deadline.
* **Fail-closed HITL authz:** resume/cancel/approval-status on a conversation whose descriptor is missing now require admin/approver (was: ownership check silently skipped).
* **REJECTED-path visibility:** the rejection message is now written to `ConversationOutput["output"]` (UIs/log generator actually render it) and the stale `PAUSE_CONVERSATION` action is stripped on the REJECTED path (as on APPROVED).
* **Verdict parsing** is case-insensitive on all surfaces (`HitlVerdict.fromString` @JsonCreator); note-length cap single-sourced as `HitlDecision.MAX_NOTE_LENGTH`.
* **`IConversation.resume(decision)`** — dead `contexts` parameter removed (CodeQL).
* Misleading "transient — not serialized" comment on the HITL bookmark fields corrected (they ARE persisted via the snapshot).

### Deliberately deferred

* Bookmark value-object refactor (6 flat fields → 1 object): cosmetic, touches the persisted snapshot shape late in the branch — deferred.
* `RestGroupConversation`'s duplicated note-length constant: consolidation phase (group-surface files owned by a parallel batch).

***

## 🔒 HITL Schedule Security, Sweeps & Retention — WS-C (2026-07-03)

**Repo:** EDDI (`fix/hitl-ws-c-schedules`, branched from `feat/hitl-framework`) **Trigger:** Confirmed code-review findings (5, 7, 17, 26, 32, 44) on schedule security, idle/undeploy sweeps, poller scalability, and pause retention.

| Finding                   | Fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| #5 (HIGH, security)       | **Editor could bypass the HITL approval gate via the schedule REST surface.** `RestScheduleStore` now: (a) `fireNow` **refuses** any schedule with `metadata.hitlType=="hitl_timeout"` — for **everyone** including admins — returning **409 Conflict** directing to `/agents/{id}/resume` or `/cancel` (manual firing side-steps the /resume owner/admin/approver audit gate); (b) `updateSchedule`/`deleteSchedule`/`enableSchedule`/`disableSchedule` on a HITL schedule require `eddi-admin`, else **403 Forbidden** (detected via the STORED schedule so a doctored request body can't hide the marker); (c) `readAllSchedules` **redacts** HITL schedules for non-admins so they can't be enumerated. Internal firing via `SchedulePollerService` is unaffected (it bypasses REST). **Parity enabler:** `PostgresScheduleStore` never persisted `metadata` (Mongo did via full-doc serialization) — added a `metadata JSONB` column (+ idempotent `ADD COLUMN IF NOT EXISTS` upgrade, `IJsonSerialization` round-trip). Without this the HITL timeout fast-path never fired on Postgres AND the security guard couldn't recognize HITL schedules there. |
| #7 (HIGH)                 | `AgentDeploymentManagement.endOldConversationsWithOldAgents` now **skips AWAITING\_HUMAN** conversations (mirrors the deliberate `getActiveConversationCount` exclusion) instead of force-ENDing them with a raw non-CAS write (which left armed schedules, stale bookmarks, and no audit). Logs at INFO how many paused conversations were spared. The pre-existing agent-document-age heuristic for non-paused conversations is left unchanged with an explanatory comment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| #17 (MEDIUM, scalability) | Poll batch size is now configurable (`eddi.schedule.poll-batch-size`, default 100) in **both** stores; `SchedulePollerService` **claims all due schedules on the poll thread (CAS before dispatch)** then **fires the claimed ones concurrently on virtual threads** (`newVirtualThreadPerTaskExecutor`) with **per-fire error isolation** — a mass HITL-timeout burst no longer serializes behind one thread and starves Dream/maintenance schedules. Exactly-once cluster semantics preserved.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| #26 (MEDIUM)              | `RestConversationStore.endActiveConversations` routes AWAITING\_HUMAN conversations through the HITL-aware `IConversationService.endConversation` (schedule disarm + bookmark clear + audit + in-flight-resume signal) instead of a raw ENDED write; non-paused conversations keep the raw path.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| #44 (LOW)                 | `RestConversationStore.deleteConversationLog(deletePermanently=true)` now calls `endConversation` for an AWAITING\_HUMAN conversation **before** deleting the document — disarming the leaked one-shot schedule, clearing the bookmark, auditing, and invalidating the cached state — via the existing public service method.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| #32 (LOW)                 | New **optional** pause-retention sweep in `HitlCrashRecoveryObserver` (`@Scheduled`): `eddi.hitl.pending.max-age` (ISO-8601, default empty=OFF) auto-cancels pauses older than the threshold via `cancelConversation` (audited, schedule-disarmed); `eddi.hitl.pending.sweep-interval` (default 6h). Reuses the existing poller/scheduling infra — no new scheduler.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| CodeQL                    | `HitlCrashRecoveryObserver.onStartup(@Observes StartupEvent event)` — silenced the "unused parameter" alert with `@SuppressWarnings("unused")` + comment; the param is the required CDI observer trigger.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

**REST status codes chosen:** manual fire of a HITL schedule → **409 Conflict** (operation not permitted for anyone; directs to /resume|/cancel). Non-admin mutate/disable/delete of a HITL schedule → **403 Forbidden**.

**New config properties:** `eddi.schedule.poll-batch-size` (default 100), `eddi.hitl.pending.max-age` (default empty/OFF), `eddi.hitl.pending.sweep-interval` (default 6h) — documented in `application.properties`. `docs/hitl.md` (owned by a later phase) should document the retention property for operators.

**Tests (pure JUnit/Mockito):** RestScheduleStore HITL guards (editor/admin fire+mutate+redact); AgentDeploymentManagement paused-skip; SchedulePollerService concurrent dispatch + per-fire error isolation + claim-before-dispatch; RestConversationStore paused end/delete routing; HitlCrashRecoveryObserver retention sweep (OFF-by-default, cancels-expired, non-positive=OFF); PostgresScheduleStore metadata round-trip.

***

## 🔧 HITL PR Review Response — Copilot + CodeRabbit (2026-07-02, session 4)

**Repo:** EDDI (`feat/hitl-framework`, PR #585) **Trigger:** Automated review on the open PR (GitHub Copilot + CodeRabbit) surfaced \~24 findings. Each was independently, adversarially re-verified against the actual code (not the bot's paraphrase); the confirmed ones are fixed here, deliberate skips are recorded with rationale.

| Area                           | Fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cross-group leak (MAJOR)       | `GroupConversationStore.findByState(state, groupId, limit)` passed `groupId` as a raw filter value; `MongoResourceStorage.findResources` turns String filters into **unanchored** regexes, so a group id that is a substring of another matched across groups. Now anchored + `Pattern.quote`d (`^\Q…\E$`) in both `findByState` and the pre-existing `listByGroupId`.                                                                                                                                                                                                                                                                                                       |
| Stuck resume (MAJOR)           | `resumeConversation`'s critical section (CAS → `submitInOrder`) only caught `ServiceException`/`InstantiationException`/`IllegalAccessException`; an unchecked exception from `continueConversation` escaped, leaving the conversation stuck `IN_PROGRESS` with a leaked `inFlightConversations` entry. The catch now also covers `RuntimeException` (restores the pause + drops the registry entry), while the deliberate agent-not-deployed `IllegalStateException` is re-thrown as-is so it still maps to 409.                                                                                                                                                            |
| Props on failed resume (MAJOR) | `Conversation.resume()`'s finally ran `postConversationLifecycleTasks()` whenever the state was not `AWAITING_HUMAN` — including `ERROR`, so a failed resume persisted long-term properties, unlike the say path. Now also skipped on `ERROR`.                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Resume bookmark index (MAJOR)  | `LifecycleManager` selective execution (`rerun`) ran a suffix sublist but stored the sublist-relative loop index as the "absolute" pause bookmark. Added an `indexOffset` threaded only into the pause-index computation (component-cache/telemetry indices unchanged), so a pause during selective execution records a true absolute index. Also added bounds validation to `executeLifecycleFromIndex` (negative → `LifecycleException`; strictly-past-end from a redeployed workflow → warn + skip; exactly `size()` remains valid).                                                                                                                                      |
| End-vs-resume race (MAJOR)     | `endConversation` wrote `ENDED` without signalling `inFlightConversations`, so a resume past the CAS could persist its snapshot back over the terminal state. It now sets the cooperative-cancel flag on any in-flight memory (mirrors `cancelConversation`), so the resume's `onComplete` skips persistence and `ENDED` wins.                                                                                                                                                                                                                                                                                                                                               |
| Timeout fire-log parity        | `ScheduleFireExecutor`'s HITL fast-path returned before `logFire()` and had no exception isolation. It now records a `ScheduleFireLog` (with the conversationId + FAILED status on error) and wraps `handleTimeout` in try/catch, matching the normal path.                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Consistency / hardening        | `OwnershipValidator.isAdmin` null-guards `identity` (matches `isApprover`/`isOwner`); `AgentGroupConfiguration.HitlConfig` setters null-coalesce to their defaults (a JSON `null` can no longer wipe `timeoutPolicy`/`granularity`/`onTaskRejection`, mirroring `setLifecyclePolicy`); `DiscussionControlToken` wave loop re-checks `isCancelled()` after `setActiveFuture` so a `CANCEL_IMMEDIATE` landing mid-registration still cancels the new future; `MongoScheduleStore` uses a `NAME` constant; `ConversationMemory` drops redundant `java.time.Instant` FQNs; `SharedTaskList.TaskStatus` Javadoc corrected (AWAITING\_APPROVAL is fully wired, not a placeholder). |
| Tests                          | New `resetFromAnyToAssigned` coverage (7 cases: valid transitions, ASSIGNED no-op, rejected states); new resume test for an unexpected `RuntimeException` from `continueConversation` (asserts pause restored + `ResourceStoreException` + never submitted); `turnCounterSeedsFromPausedTurnCount` de-flaked by capturing `pausedTurnCount` at the synchronous resume CAS instead of racing the async completion; bogus timeout-policy strings in memory round-trip tests replaced with real enum names.                                                                                                                                                                     |
| Deliberately skipped           | `listPendingApprovals` "max 1000 not enforced" — FALSE POSITIVE, the service already clamps `Math.min(limit, 1000)`. Crash-recovery pagination (10k scan cap) — 10k concurrent pauses is unrealistic and the code already warns. Pending-summary N+1 (Mongo/Postgres) and `findByState` bulk-read — perf-only on bounded/one-time paths, a pre-existing `IResourceStorage` API limitation; deferred. `HitlPauseType` vs `HitlGranularity` unification — intentional pause-record vs config separation. Duplicated cancel-check refactor — declined to churn just-hardened control flow.                                                                                      |

***

## 🔧 HITL Merge-Readiness Hardening (2026-07-02, session 2)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** Second full-branch review after the phase 1–7 fixes; verified 22 tracked findings (10 FIXED, 7 PARTIAL, 5 UNFIXED) plus new regressions in the fix commits themselves. This session closes the remainder before PR.

| Step      | Fix                                                                                                                                                                                                                                                                                                                                                       |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Red tests | `HitlTimeoutHandlerTest` — inject `SimpleMeterRegistry` (new metrics field NPE'd all 7 tests); `RestGroupConversationHitlTest.approveDenied` — body validation runs before authz, so the test now sends a valid decision (plus a new test pinning the 400-before-authz ordering). The suite was red at HEAD; these commits restore green as the baseline. |

\| Crash recovery (C-A/F4/F6) | **`HitlCrashRecoveryObserver` reworked from destroyer to repairer.** It no longer transitions old paused conversations to ERROR/FAILED (the previous behavior destroyed legitimately-paused WAIT\_INDEFINITELY conversations — the default policy — on every restart). New behavior: finite-policy pauses get their one-shot timeout schedule idempotently re-armed at the original due time (applies the configured policy through the normal handler, even after a crash); WAIT\_INDEFINITELY pauses are never touched; regular conversations stuck IN\_PROGRESS with an intact HITL bookmark (pod died mid-resume) are CAS-restored to AWAITING\_HUMAN; IN\_PROGRESS without bookmark → EXECUTION\_INTERRUPTED. Stale-threshold config dropped (no longer meaningful); added `eddi.hitl.crash-recovery.recover-in-progress` (default true, multi-pod caveat documented). Prereq: the regular pause commit now populates `hitlTimeoutPolicy`/`hitlApprovalTimeout` bookmark fields (F6) — `approval-status`/`pending-approvals` finally report the effective policy on both surfaces. | | Regular cancel (F2) | `cancelConversation` is no longer a silent no-op: in-flight registry (`inFlightConversations`) lets cancel set the cooperative `setCancelled` flag the LifecycleManager checks at task boundaries (mode param now read; IMMEDIATE degrades to graceful, documented); returns typed `CancelOutcome` — REST maps CANCELLED→200, NOT\_FOUND→404, NOTHING\_TO\_CANCEL→409 (was unconditional 200). | | Resume robustness (F4/F7) | Resume now: pre-checks existence → 404 (was 409 with misleading message); reports the current state in the 409 body; **restores the pause** (CAS back to AWAITING\_HUMAN + re-arms timeout) when the agent is undeployed or a service error occurs, instead of destroying the approval with ERROR; wraps execution with the same watchdog as the say path (hung LLM → EXECUTION\_INTERRUPTED, never stuck IN\_PROGRESS). | | Undo/redo gate (F5/C-C) | Gate now reads the DB-loaded state (was per-pod cache — silently bypassed after restart/cross-pod) and returns `false` → 409 CONFLICT (was 500). |

\| Group cancel (F9) | No-token branch now uses the `updateIfState` state-CAS (was plain read-modify-write racing approve/resume); `CancelledEvent` is finally emitted via new `onCancelled` listener method and the SSE stream closes on cancel (was: `/discuss/stream` clients hung forever after a cancel). | | Group approvals (F13/C-D) | `taskApprovals` validated up front (unknown taskId / task not awaiting → 400, no partial in-memory mutation, schedule untouched); RETRY rejection now passes the reviewer's note into the re-queued task and `buildTaskExecutionInput` surfaces it as feedback (was: blind retry loop); `resetFromAnyToAssigned` enforces its documented status contract. | | Group robustness (F11/5f) | Stranded IN\_PROGRESS tasks are reset to ASSIGNED in ALL wave-abort branches (was: timeout only — cancellation/error still stranded tasks forever); config-drift guard now also fails when phases were REMOVED (bookmarked index out of range no longer silently skips the check); nested-group sub-pauses are cancelled instead of stranded with an armed schedule; `commitPause` reuses the in-scope config (no duplicate store read per pause). | | Group pending list (C-B) | `GET /groups/{groupId}/conversations/pending-approvals` now scopes to the path's group and applies the same ownership filter as the regular listing (admin/approver see the group's items, others only their own, anonymous nothing) — was a global unfiltered dump of all users' paused conversations incl. transcripts. | | Approver role | New `eddi-approver` role: `OwnershipValidator.isApprover`, added to `@RolesAllowed` on all HITL endpoints (approver-only accounts were blocked at RBAC), and approvers now see pending listings on both surfaces (they could approve but not list). SSE error events now serialize through the JSON serializer (no string-concatenation injection). |

\| Audit (F15) | `hitl.approval` AuditEntry submitted on BOTH surfaces for every decision (verdict, decidedBy, automated flag, note) — covers human and `system:timeout` decisions; `GroupConversationService` now injects `AuditLedgerService`. Combined with the earlier resume audit-collector wiring, the EU AI Act human-oversight trail is complete. | | Quota (F16, plan §10a) | `getActiveConversationCount` excludes AWAITING\_HUMAN on Mongo AND Postgres — paused conversations no longer block undeploy/old-version GC forever. Undeployed-while-paused conversations keep their pause; resume reports 409 and restores it. | | Pending listing scale (F17) | New `findPendingApprovalSummaries(limit)` store method: Mongo uses POJO-codec projection (never deserializes step data), Postgres a LIMIT-bounded loop; REST takes `?limit` (default 200, max 1000); `PendingApprovalSummary` gains `userId` so the ownership filter no longer does N+1 descriptor reads. | | Config safety (F20/C-E/C-F) | Duplicate `engine.lifecycle.model.HitlTimeoutPolicy` enum deleted — `AgentGroupConfiguration.HitlTimeoutPolicy` is the single source (no more constants-drift between schedule metadata writer and parser). New `HitlConfigValidation` enforced in `AgentStore`/`AgentGroupStore` create+update: finite policy requires a valid positive ISO-8601 `approvalTimeout`, actionable 400 messages via the existing `IllegalArgumentExceptionMapper`. | | Input bounds | HITL decision `note` capped at 4 KB on both surfaces (400 on overflow). Dead `counterHitlTimeout` field removed (handler owns the timeout metric). |

\| Test hardening (F22) | R2 cancel tests: `else assertNotNull` escape hatches removed — reverting the R2 fix now fails them. NEW R1 test: cancel racing a `requiresApproval` phase asserts CANCELLED + commitPause never ran. B1 test rebuilt: snapshot reflects post-CAS reality (IN\_PROGRESS) and the memory state is captured at `continueConversation` time — deleting the B1 fix now fails it. New behavioral tests: `stripPauseAction` (stale action removed, others preserved), decision visibility (`hitlDecision` output + `hitlVerdict` property), Invariant 9 asserted via step-property survival (pause) vs purge (normal turn). | | Integration test (F22) | **New `HitlPauseResumeIT`** — full end-to-end with zero mocked seams: real behavior rule emits PAUSE\_CONVERSATION → real Mongo persistence → REST resume completes the remaining pipeline tasks (the original BLOCKER's fail-on-revert), plus REJECTED path, cancel path, 400-does-not-consume-pause, 404 unknown id, 409 not-paused, and undo-blocked-while-paused. New `tests/hitl/*.json` agent fixtures. Runs in CI via `mvnw verify -DskipITs=false` (ci.yml:179); local Docker daemon was unavailable during this session, so first execution happens in CI — same as any IT change. | | Docs (F21) | New [docs/hitl.md](/conversations-and-orchestration/hitl.md): both surfaces, config reference incl. `onTaskRejection`, real REST paths, template access (`hitlDecision` / `{properties.hitlVerdict}`), timeout policies, approver role, crash recovery config, operations notes (metrics, undeploy semantics, cancel matrix), known v1 limitations, `requiresApproval` upgrade note. AGENTS.md reserved-action list updated with PAUSE\_CONVERSATION. `planning/hitl-framework-plan.md` committed. |

\| Final sweep | Full 9,761-test suite executed: the only genuine branch defect was `GroupConversationTest.groupConversationStates` still asserting 6 enum values after the branch added CANCELLED (fixed: 7 + CANCELLED assertion). All other local failures are environmental (Docker unavailable for Testcontainers classes, sandbox-blocked loopback sockets for HTTP-server-based tool tests) — these classes are untouched by this branch and run in CI. Mongo `findPendingApprovalSummaries` reworked to bounded projected point-reads with explicit id mapping (the bulk POJO-codec projection could not be guaranteed to populate `_id`→conversationId). |

*(End of session 2.)*

***

## 🔧 HITL Final-Review Fixes — Round 3 (2026-07-02, session 3)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** Third full-branch multi-agent review (70 agents, adversarial verification completed): 43 confirmed findings (12 MAJOR). This session fixes all of them.

| Area                            | Fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Queued-say race (MAJOR)         | `processConversationStep` re-reads the persisted state at execution time and DROPS a queued turn when the conversation is AWAITING\_HUMAN/IN\_PROGRESS — a stale pre-pause memory copy can no longer execute and full-document-overwrite a just-committed pause.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Zombie resume (MAJOR)           | Resume persistence moved from the callable's `finally` into `IFinishedExecution.onComplete` — BaseRuntime's completed-after-cancellation discard now protects the resume path like the say path; a timed-out resume can never clobber state written after its watchdog fired.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Cancel-vs-resume window (MAJOR) | The live memory is registered in `inFlightConversations` synchronously after the resume CAS; the resume callable re-checks `isCancelled()`/persisted EXECUTION\_INTERRUPTED before executing and skips persistence when cancelled. Cancel now also wins over a pause committed by the very task it interrupted (`Conversation` treats pause-while-cancelled as stop), on both say and resume paths.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Init pause (MAJOR)              | `startConversation` performs the same HITL bookkeeping as the say path — a CONVERSATION\_START pause now gets its policy bookmark, pause counter, and timeout schedule.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Resume rollback (MAJOR)         | Every post-CAS failure restores the pause: snapshot-load failures, `RejectedExecutionException` from a saturated coordinator, and service exceptions. Wrong-state/agent-undeployed now throw `IllegalStateException` → 409; infrastructure failures throw `ResourceStoreException` → 500 (was: everything 409). Audit + resume counter moved AFTER the successful submit — rolled-back resumes no longer pollute the compliance trail or metrics; undeployed-agent restores skip schedule re-arm (kills the infinite timeout→restore→re-arm loop).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Bookmark hygiene                | New `clearHitlBookmark` store op (Mongo `$unset` / Postgres jsonb key-removal) called when a pause is terminally resolved outside resume (cancel, end-while-paused) — stale bookmarks no longer round-trip forever, mislead approval-status (now also state-gated), or trick crash recovery into resurrecting dead pauses. Cancel of a pending approval writes an `hitl.approval` audit entry (verdict CANCELLED). `endConversation` on a paused conversation disarms the schedule and clears the bookmark (round-1 leftover).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Config resolution               | HITL timeout config is read ONCE per pause at the conversation's PINNED agentVersion (fallback to latest); `scheduleHitlTimeout` derives from the memory bookmark — bookmark and schedule can no longer diverge, and draft config edits no longer change paused conversations' behavior. Re-pauses now increment the pause counter (metric parity with the group surface). Undo/redo additionally rejected during IN\_PROGRESS (protects the resume-CAS invariant crash recovery relies on).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Group cancel window (MAJOR)     | Control tokens are registered BEFORE the executor submit in `startAndDiscussAsync` and `resumeDiscussion` (`executeDiscussion` uses `computeIfAbsent` so a signalled pre-registered token is never wiped) — a cancel landing between the resume CAS (or async start) and thread startup now finds a signalable token instead of being overwritten by the running leg's unconditional updates. New `convertPauseToCancelIfSignalled`: a cancel that lands while `commitPause` is writing converts the just-committed pause to CANCELLED (CAS), disarms the schedule, audits, and emits the cancelled SSE event — cancel can no longer report success while the pause survives.                                                                                                                                                                                                                                                                                                                                                         |
| Group TASK-gate bypass (MAJOR)  | The `requiresApproval` EXECUTE gate now also pauses when an aborted wave (timeout/error) left executable tasks behind with nothing awaiting approval — previously the phase loop fell through to VERIFY/synthesis over unexecuted work and silently skipped the remaining tasks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Group resume resilience (MAJOR) | Config-drift aborts and pre-`executeDiscussion` failures now RESTORE the pause (`restoreGroupPause`: bookmark re-set, CAS IN\_PROGRESS→AWAITING\_APPROVAL, schedule re-armed) and fire `group_error` so SSE clients terminate — previously they persisted terminal FAILED, destroying the approval, without ever notifying the stream. `executorService.submit` failures roll back the same way (resume) or fail the conversation honestly (async start) instead of leaving IN\_PROGRESS zombies. Failures INSIDE `executeDiscussion` are not double-handled (it owns its terminal states + events). REJECTED verdict now fires `group_complete` so streams close.                                                                                                                                                                                                                                                                                                                                                                    |
| Group terminal cleanup (MAJOR)  | New `cleanupAfterTerminalState`: ephemeral dynamic agents and `lastVerifiedIndex` entries are released when a paused discussion reaches a terminal state OUTSIDE the execution loop (cancel-of-paused, REJECTED resume) — previously they leaked forever because the in-loop finally only runs while a thread is executing.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Group cancel outcome            | `cancelDiscussion` returns boolean (`false` = already terminal / lost CAS race) → REST maps to 409 (was unconditional 200); paused-cancels emit an `hitl.approval` audit entry (verdict CANCELLED) — parity with the regular surface. Timeout handler logs skipped aborts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Group approvals                 | `taskApprovals` VALUES validated up front (only APPROVED/REJECTED, case-insensitive → else 400 before any mutation); an explicit `{}` map is treated as the approve-all shortcut instead of approving nothing and instantly re-pausing. `approveGroupPhase` returns a freshly-read copy — the HTTP layer no longer serializes the live object being mutated by the background thread.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Group timeout source            | `scheduleGroupHitlTimeout` reads the pause bookmark on the conversation (set by `commitPause`/`restoreGroupPause`) instead of re-reading the group config — schedule and approval-status can no longer diverge after a config edit.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Group pending listing           | `listGroupPendingApprovals(groupId, limit)` returns bounded `PendingApprovalSummary` objects (query-level group filter + limit, new `findByState(state, groupId, limit)` store variant) instead of unbounded full transcripts; summary gains `groupId`; REST takes `?limit` (default 100).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Approver read scope (MAJOR)     | `detail=full` on both approval-status endpoints is now gated for approver-only callers (not owner, not admin): full content is readable ONLY while the conversation is actually awaiting approval → 403 otherwise. The approver role exists to decide pending approvals, not as a universal read-everything grant over all conversations/transcripts. New `OwnershipValidator.isOwner` helper.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Group approval-status detail    | `GET .../approval-status` on the group surface finally honors `detail`: default is a summary projection (state, pausedAt, phase, pauseType, reason, timeoutPolicy, awaiting task ids — stale fields suppressed outside AWAITING\_APPROVAL, mirroring the regular surface) instead of always dumping the full conversation incl. transcript; `detail=full` returns the conversation, subject to the read-scope gate.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Crash recovery scale (MAJOR)    | The recovery sweep runs on a background virtual thread — application readiness no longer blocks on repairing thousands of paused conversations. The paused-regular sweep reads bounded PROJECTED summaries (10k cap, logged if hit) instead of full multi-MB documents, and skips WAIT\_INDEFINITELY pauses without any further read (`PendingApprovalSummary` gains `approvalTimeout`, projected on Mongo + Postgres and exposed on both listing surfaces). `rearmSchedule` re-checks the pause state AFTER creating the schedule and withdraws it if a resume/cancel landed in the window — no armed timeout on a no-longer-paused conversation.                                                                                                                                                                                                                                                                                                                                                                                    |
| Schedule + listing bounds       | New `name` index on schedules (Mongo + Postgres) — HITL timeout delete/re-arm by name was a collection scan on every pause/resume/cancel. Mongo `findPendingApprovalSummaries` bounds the ids query with `.limit()` at the DB instead of materializing every paused id first.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| HITL enum home                  | `HitlTimeoutPolicy`/`HitlGranularity`/`HitlRejectionPolicy` moved from nested types in `AgentGroupConfiguration` to the neutral `ai.labs.eddi.configs.hitl` package (`HitlConfigValidation` moved there too) — the regular-surface agent config and the whole engine no longer depend on the GROUP config class for shared HITL vocabulary. JSON compatibility unchanged (enum names serialize identically).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Dead surface removed            | `ControlSignal.PAUSE` and `DiscussionControlToken.isPaused()/shouldStop()` deleted — HITL pauses are committed by the execution loop at the gates, never signalled through the token; the dead PAUSE path only invited misuse (a signalled "pause" would have been persisted as CANCELLED). Token checks now read `isCancelled()` explicitly.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| hitl\_resume event wired        | `HitlResumeEvent`/`EVENT_HITL_RESUME` existed but were never fired. New `onHitlResume` listener method fires after the resume CAS commits; the SSE streaming endpoint forwards `hitl_resume` WITHOUT closing, so `/approve/stream` clients see an explicit resume marker before the resumed discussion's events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ZIP import validation           | `RestImportService` validates `hitlConfig` right after deserializing the agent file — an invalid config now fails the import up front with 400 (via `IllegalArgumentExceptionMapper`) instead of importing all workflows/extensions first and then failing agent creation with a 500 (partial import).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Javadoc drift                   | `IConversationMemory.getHitlTimeoutPolicy` no longer documents non-existent policy values ("expire"); `AgentGroupConfiguration.HitlConfig` no longer claims a "per turn / per discussion" granularity that never existed (actual: PHASE or TASK). Unused Mongo `Updates` import removed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| Test hardening (round 3)        | New `HitlConfigValidationTest` (both surfaces, all rejection branches). `taskApprovals` validation tests: unknown id / wrong state / bad value → IAE with NOTHING mutated and no CAS; case-insensitive values; `{}` = approve-all. Audit emission tests (hitl.approval on resume with automated flag; verdict CANCELLED on cancel-of-paused; silent when ledger disabled). Note-cap tests on both surfaces (4097 → 400, 4096 → OK). Undo/redo HITL gate unit tests (AWAITING\_HUMAN + IN\_PROGRESS → false, nothing stored) + redo-409 IT. Group pending listing filter tests on summaries (admin/approver/owner/anonymous + default limit). Crash-observer tests rewritten for the projected sweep incl. schedule-withdraw races on both surfaces. Postgres container IT now covers the HITL store primitives (CAS, state query, projected summaries incl. approvalTimeout + limit, bookmark clearing). `HitlPauseResumeIT.waitForState` polls the DB-backed approval-status instead of the per-pod `/status` cache (flakiness fix). |
| Docs                            | [docs/hitl.md](/conversations-and-orchestration/hitl.md) updated: approver read-scope gate (403 semantics), group approval-status summary/full, pending summaries + `?limit`, group cancel 409, `hitl_resume` SSE event + every-terminal-path-closes guarantee, drift-restores-pause behavior, cancel audit entries, async + projected crash recovery.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| Group end-to-end IT             | **New `GroupHitlIT`** — full group-surface HITL path with zero mocked seams: a `requiresApproval` phase commits a real pause through the store, `/approve` applies the decision, and the background resume re-enters the phase loop and runs the post-gate phase to completion (transcript asserted). Also: approval-status summary vs `detail=full`, pending-approvals summary listing (conversationId + groupId), REJECTED-is-terminal (later approve/cancel → 409), cancel-of-paused → CANCELLED (second cancel/late approve → 409), 400s that do NOT consume the pause (missing verdict, >4 KB note), 404 for unknown ids. Like all ITs, first execution happens in CI (`-DskipITs=false`); local Docker unavailable.                                                                                                                                                                                                                                                                                                             |

*(End of session 3.)*

***

## 🔧 HITL Review Fixes — Phases 1–5 (partial) (2026-07-02)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** 7-phase implementation plan from code review (1 BLOCKER + 21 MAJORs).

### Fixes Implemented

| ID  | Severity    | Phase | Fix                                                                                                                                                                                                                                                                                                                   |
| --- | ----------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| #1  | **BLOCKER** | 1a    | **Resume re-pause loop**: `checkIfPauseConversationAction` is now delta-based — only throws if the just-executed task *added* `PAUSE_CONVERSATION` (not if it was stale from the prior turn). Belt-and-braces: `Conversation.resume()` strips `PAUSE_CONVERSATION` from step ACTIONS before re-entering the pipeline. |
| #1b | MAJOR       | 1b    | **Decision visibility**: Verdict stored as conversation output (`hitlDecision`) and conversation-scoped property (`hitlVerdict`) for template/behavior-rule access. REJECTED emits public output.                                                                                                                     |
| #8  | MAJOR       | 2a    | **Request body validation**: Null/missing verdict → 400 on both REST surfaces (regular + group + streaming).                                                                                                                                                                                                          |
| #12 | MAJOR       | 5g    | **Double-approve → 409**: `GroupDiscussionException` caught and mapped to 409 Conflict (was falling through to 500).                                                                                                                                                                                                  |
| #9  | MAJOR       | 3a    | **Group cancel state guard**: No-token branch validates state before writing CANCELLED — terminal states (COMPLETED/CANCELLED/FAILED) cannot be overwritten.                                                                                                                                                          |
| #3  | MAJOR       | 3b    | **Timeout rescheduling on re-pause**: Resume callable's `finally` block arms a new HITL timeout if the conversation re-paused to AWAITING\_HUMAN.                                                                                                                                                                     |
| #5  | MAJOR       | 4a    | **Undo/redo gate**: Undo and redo blocked during AWAITING\_HUMAN state (would corrupt the HITL bookmark).                                                                                                                                                                                                             |

### Test Changes

* `pauseActionThrowsPause`: Updated for delta-based semantics (sequential mock: null → actionData)
* `fromIndexDetectsPause` → split into `fromIndexIgnoresStaleAction` (stale action = no re-pause) + `fromIndexDetectsNewPause` (new action = re-pause)
* New: `executeLifecycleDetectsFreshPause` — verifies fresh pause on `executeLifecycle`

### Files Changed

* `LifecycleManager.java` — Delta-based `checkIfPauseConversationAction`, unconditional `actionsBefore` snapshot
* `Conversation.java` — `stripPauseAction` helper, decision visibility, rejection output
* `ConversationService.java` — Undo/redo gate, timeout rescheduling on re-pause
* `RestAgentEngine.java` — Resume body validation
* `RestGroupConversation.java` — Approve body validation, `GroupDiscussionException` → 409
* `GroupConversationService.java` — Cancel state guard in no-token branch
* `LifecycleManagerHitlTest.java` — 3 new/fixed delta-based pause tests

### Completed (this session)

* Phase 2b: HitlConfig string→enum typing (HitlGranularity, HitlTimeoutPolicy, HitlRejectionPolicy)
* Phase 3d: Discriminating status codes — cancelDiscussion exception mapping (409/404 instead of 500)
* Phase 4b: Strict ownership + eddi-approver role for HITL endpoints (requireOwnerAdminOrApprover)
* Phase 4d: Micrometer HITL counters (eddi\_hitl\_pause/resume/timeout\_count with surface tag)
* Phase 6a: Deduplicated executeLifecycle/executeLifecycleFromIndex → shared executeTaskRange()
* Phase 7b: HitlCrashRecoveryObserver unit tests (6 tests)
* Phase 7b: OwnershipValidator approver role tests (6 tests)

***

## 🔧 HITL Review Fixes — Phases 5/6: Group Correctness + Config Surface (2026-07-02)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** Continuing 7-phase implementation plan — group API correctness, config surface, and architecture.

### Fixes Implemented

| ID  | Severity | Phase | Fix                                                                                                                                                                                              |
| --- | -------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| #10 | MAJOR    | 5b    | **Non-EXECUTE + TASK fallback**: TASK granularity only applies to EXECUTE phases (they have a SharedTaskList). Non-EXECUTE phases (OPINION, SYNTHESIS, etc.) now fall back to PHASE-style pause. |
| #5e | MAJOR    | 5e    | **Resume ordering**: Timeout schedule deleted only AFTER the CAS succeeds (both approve + reject paths). If CAS fails, schedule preserved → timeout can still fire.                              |
| #5f | MAJOR    | 5f    | **Config drift guard**: On resume, bookmarked phase name validated against loaded config. If config was edited while paused → FAILED + ERROR transcript entry.                                   |
| #10 | MAJOR    | 5d    | **Nested group HITL guard**: Sub-group returning AWAITING\_APPROVAL → SKIPPED entry instead of extracting partial answer. Nested HITL not supported in v1.                                       |
| #11 | MAJOR    | 5c    | **Timed-out task fixup**: After wave timeout, IN\_PROGRESS tasks reset to ASSIGNED (prevents permanent stranding).                                                                               |
| #5a | MAJOR    | 5a    | **Task rejection policy**: New `onTaskRejection` field in HitlConfig (FAIL/RETRY). RETRY resets rejected tasks to ASSIGNED for re-execution.                                                     |
| #15 | MAJOR    | 4c    | **Audit trail**: Audit collector added to resume path (same as say path).                                                                                                                        |
| #6c | MINOR    | 6c    | **Pause reason**: `hitlPauseReason` field on GroupConversation — human-readable reason set at commitPause.                                                                                       |
| #6d | MINOR    | 6d    | **Bookmark timeout fields**: `hitlTimeoutPolicy` + `hitlApprovalTimeout` copied from config at pause time for REST visibility.                                                                   |

### Files Changed

* `GroupConversationService.java` — HITL gate type check, drift guard, resume ordering, nested guard, timeout task fixup, rejection policy, bookmark population
* `GroupConversation.java` — 3 new bookmark fields (hitlPauseReason, hitlTimeoutPolicy, hitlApprovalTimeout)
* `AgentGroupConfiguration.java` — `onTaskRejection` field in HitlConfig
* `SharedTaskList.java` — `resetFromAnyToAssigned()` method for RETRY policy
* `ConversationService.java` — Audit collector on resume path

***

## 🔧 HITL Framework — Cancel Path Fixes: R1 + R2 MAJORs (2026-07-01)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** Final merge verdict found 2 MAJORs in cancel path — cancel-vs-pause race and CancellationException misrouting.

### Fixes

| ID | Severity | Fix                                                                                                                                                                                                                                                                                                                                                                                                                              |
| -- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| R1 | MAJOR    | **Cancel-vs-pause race**: Added `isCancelled()` guard immediately before the HITL gate. After the wave loop breaks on cancel, the HITL gate fired before the next phase-loop iteration's `shouldStop()` check, converting a cancel into a pause. Guard uses `isCancelled()` (not `shouldStop()`) so real PAUSE still routes to `commitPause`.                                                                                    |
| R2 | MAJOR    | **CANCEL\_IMMEDIATE → FAILED**: Added explicit `CancellationException` catch in the wave `allOf.get()` handler. Forward-cancels all source agent futures (since `allOf.cancel` doesn't propagate). Both generic `catch (GroupDiscussionException)` and `catch (Exception)` now check `token.isCancelled()` and route to CANCELLED instead of FAILED. Also added source-future forward-cancel in the `ExecutionException` branch. |

### Regression Test Added

* `InFlightCancel.gracefulCancelDuringExecution` — Concurrent latch-based test: launches `discuss()` on separate thread, blocks `say()` with latch, fires `cancelDiscussion(GRACEFUL)`, asserts CANCELLED state.

### Files Changed

* `GroupConversationService.java` — R1 cancel guard before HITL gate + R2 CancellationException handling + cancel-aware generic catch blocks
* `GroupConversationServiceHitlTest.java` — In-flight cancel test with proper agent mock wiring

***

## 🔧 HITL Framework — Final Ship Fix: 2 BLOCKERs + 2 MAJORs in group TASK path (2026-07-01)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** Ship/no-ship verdict found 2 BLOCKERs + 2 MAJORs, all in the group TASK surface.

### Fixes

| ID            | Severity | Fix                                                                                                                                                                                                                                                                     |
| ------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| NEW-1         | BLOCKER  | **Submit gate ≠ pause gate**: `submitForApproval` now gates on `taskLevelHitl && phase.requiresApproval()`. Without both, `completeTask` is used. Prevents TASK\_FORCE preset from stranding all tasks in AWAITING\_APPROVAL when the phase doesn't require approval.   |
| NEW-2         | BLOCKER  | **Cancel-of-paused silent no-op**: `activeTokens.remove()` is now unconditional in the `finally` block. Paused conversations have no running thread, so a lingering token caused `cancelDiscussion` to take the no-op signal branch. Resume re-registers a fresh token. |
| NEW-3         | MAJOR    | **Control token write-only**: Added `token.shouldStop()` safe-points at the top of both the phase loop and the wave loop. Registered the wave `allOf` future via `setActiveFuture()` so IMMEDIATE cancel can interrupt.                                                 |
| AUTO\_APPROVE | MAJOR    | **TASK auto-approve infinite loop**: When TASK granularity + APPROVED verdict + null taskApprovals (e.g., timeout handler), `resumeDiscussion` now synthesizes APPROVED for all AWAITING\_APPROVAL tasks. Previously caused infinite reschedule.                        |

### Regression Tests Added

* `SubmitGateAlignment` — TASK granularity + requiresApproval=false → tasks COMPLETED not stranded
* `CancelOfPaused` — Cancel of AWAITING\_APPROVAL group does DB write to CANCELLED
* `AutoApproveTaskSynthesis` — APPROVED + TASK + null taskApprovals auto-approves all tasks
* `TaskResumeCompletesDependent` — TASK resume re-enters same phase, clears hitlPauseType

### Files Changed

* `GroupConversationService.java` — All 4 fixes + `taskLevelHitl` local variable in `executeTaskExecutionPhase`
* `GroupConversationServiceHitlTest.java` — 4 new regression test classes (13 → 17 tests)

***

## 🔧 HITL Framework — Delta Code Review Fix #2: BLOCKER + MAJORs (2026-07-01)

**Repo:** EDDI (`feat/hitl-framework`) **Trigger:** Delta code review identified 1 BLOCKER + 7 MAJORs + 2 MINORs in group TASK surface.

### Fixes

| ID      | Severity | Fix                                                                                                                                                                                                                                                           |
| ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BLOCKER | BLOCKER  | **TASK resume index**: `resumeDiscussion` now reads `hitlPauseType` before clearing. TASK resumes at same phase (re-entry idempotent via findExecutableTasks); PHASE resumes at +1.                                                                           |
| MAJOR-1 | MAJOR    | **Mutual exclusion of PHASE/TASK gates**: TASK gate fires only when `phase.requiresApproval() AND taskLevelHitl AND hasAwaitingApproval()`. PHASE gate fires only when `NOT taskLevelHitl`. Eliminates double-pause.                                          |
| MAJOR-2 | MAJOR    | **Group timeout scheduling**: `commitPause` now creates a one-shot `IScheduleStore` schedule for group HITL timeouts (reads `approvalTimeout` + `timeoutPolicy` from group config).                                                                           |
| MAJOR-3 | MAJOR    | **Schedule deletion on resume/cancel**: Added `IScheduleStore.deleteSchedulesByName()` (MongoDB + PostgreSQL). Called in `ConversationService.resumeConversation`, `cancelConversation`, `GroupConversationService.resumeDiscussion`, and `cancelDiscussion`. |
| MAJOR-4 | MAJOR    | **REJECTED branch CAS**: Rejection now uses `updateIfState(gc, AWAITING_APPROVAL)` instead of plain `update(gc)` to prevent concurrent approve clobbering reject.                                                                                             |
| MAJOR-5 | MAJOR    | **activeTokens lifecycle**: `executeDiscussion` now registers control token at start (`activeTokens.put`) and removes it in `finally` block when discussion truly ends (not paused).                                                                          |
| MAJOR-6 | MAJOR    | **listPendingApprovals ownership filter**: `RestAgentEngine.listPendingApprovals()` now filters by caller identity. Admin sees all; non-admin sees only their conversations. Added `OwnershipValidator.isAdmin()`.                                            |
| MINOR-1 | MINOR    | **Metrics/events guard**: `counterGroupDiscussion.increment()` and `onGroupStart` only fire when `startPhaseIndex == 0` (fresh discussion, not resume).                                                                                                       |
| MINOR-2 | MINOR    | **Fail-closed on null owner**: Added `OwnershipValidator.requireOwnerOrAdminStrict()` for state-changing ops where null-owner resources should deny access (admin exempted).                                                                                  |

### Files Changed

* `GroupConversationService.java` — BLOCKER, MAJOR-1/2/3/4/5, MINOR-1 fixes
* `ConversationService.java` — MAJOR-3 schedule deletion on resume/cancel
* `RestAgentEngine.java` — MAJOR-6 ownership filtering
* `IScheduleStore.java` — MAJOR-3 new `deleteSchedulesByName` API
* `MongoScheduleStore.java` — MAJOR-3 MongoDB implementation
* `PostgresScheduleStore.java` — MAJOR-3 PostgreSQL implementation
* `OwnershipValidator.java` — MAJOR-6 `isAdmin()`, MINOR-2 `requireOwnerOrAdminStrict()`
* 6 test files — Constructor updated for new `IScheduleStore` param + test fixes

***

## 🐛 Fix: PostgreSQL group conversations broken — JDBC `?|` operator escape (2026-07-02)

**Repo:** EDDI (`fix/postgres-group-conversation-jdbc-escape`) **Severity:** Critical — **all** group conversations fail on PostgreSQL; MongoDB unaffected.

### Root cause

`PostgresUserMemoryStore.getVisibleEntries()` builds a dynamic SQL query that uses PostgreSQL's `?|` (array overlap) operator for group-scoped visibility filtering. The JDBC driver interpreted the `?` in `?|` as a bind parameter placeholder, inflating the expected parameter count by one. This caused `PSQLException: No value specified for parameter 5` every time `groupIds` was non-empty.

Group conversations always pass a `groupId` context when starting agent sub-conversations (`GroupConversationService.executeAgentTurn()` → `startConversation()` with `groupId` in context), so this bug was triggered on **every** group conversation turn — making group conversations completely non-functional on PostgreSQL.

### Fix

* **`PostgresUserMemoryStore.java`**: Changed `?|` to `??|` (JDBC escape syntax for a literal `?`). Single character fix.

### Regression guard

* **`PostgresUserMemoryStoreTest.java`**: Added two integration tests (Testcontainers PostgreSQL) that exercise `getVisibleEntries` with non-empty `groupIds`:
  1. `groupIdsDoNotBreakQuery` — verifies the query doesn't throw with non-empty groupIds (would have caught this bug directly)
  2. `groupScopedEntriesVisible` — verifies group-scoped entries are correctly returned when groupIds match, and excluded when they don't

### Why existing tests missed it

* The **unit test** (`PostgresUserMemoryStoreUnitTest.getVisibleEntries_withGroupIds_includesGroupClause`) tested with non-empty groupIds but mocked the JDBC PreparedStatement — never sent actual SQL to PostgreSQL, so the `?` parsing was invisible.
* The **integration test** (`PostgresUserMemoryStoreTest.selfAndGlobalVisible`) ran against real PostgreSQL but only tested with `groupIds = null` — never exercised the group visibility branch.

***