July 2026
Archived entries for July 2026, newest first. For recent work see the live changelog.
🔓 fix(ci): secret scanning could never pass on a pull request from a fork (2026-07-31)
Repo: EDDI (ci/gitleaks-cli-for-forks)
The Secret Scanning job failed on every pull request opened from a fork. On #623 it was the only failing check: twelve passed and seven were skipped. Nothing downstream was blocked: two jobs list gitleaks in needs: and neither is stopped by it. docker is gated on github.event_name == 'push' and skips on pull requests; notify-slack runs if: always() and reads needs.gitleaks.result only to pick a status icon, so every fork pull request posted a Slack card with a red Secret Scan marker. The cost is narrower than a blocked pipeline but not harmless: an outside contributor cannot get a green run, a permanently red check trains reviewers to ignore that check, and it cannot be made a required status without blocking every fork.
Two causes, stacked. gitleaks/gitleaks-action requires a paid GITLEAKS_LICENSE for repositories owned by an organization, and GitHub deliberately withholds repository secrets from pull_request runs that originate in a fork. So the license was absent by design, not by misconfiguration, and no amount of secret management on this side would have supplied it. The failure is structural rather than incidental, and the open pull requests are a natural experiment. Of the 21 open, 12 come from forks: all 9 that have a Secret Scanning result failed it, and the other 3 never ran it. Of the 9 from branches in this repository, 7 passed, 1 has no result, and 1 (#430, a Dependabot bump last updated in May) failed — its logs have since expired, so that one is unattributed rather than explained. Dependabot is not the discriminator: five other Dependabot pull requests pass.
The scanner and its wrapper have different licenses. The gitleaks CLI itself is MIT and needs no key at all — only the Action wrapper is commercially licensed. Running the binary directly restores coverage on forks, which is precisely where an unreviewed secret is most likely to arrive. The two alternatives both lose that: dropping the job to continue-on-error keeps it green while scanning nothing, and gating it on github.event.pull_request.head.repo.fork skips forks outright.
Pinned by version and checksum, following the existing convention. This repository already installs a third-party binary this way: PREFLIGHT_VERSION and PREFLIGHT_SHA256 sit in the workflow-level env: block and the install step pipes the recorded hash through sha256sum -c -. GITLEAKS_VERSION and GITLEAKS_SHA256 join them there and the install step mirrors that shape, so a retagged or substituted release fails the step rather than executing. This is stronger than the commit-pinned Action it replaces: the Action pin covers the wrapper, not the binary the wrapper downloads at runtime. Output is redacted with --redact=100, so a real finding never prints the secret into a public log.
Design decision — scan the event's commits, never the full history. A whole-repository scan on a full-depth checkout reports 61 findings across 16 commits, dated 2016 to 2026. Most are in files deleted years ago: Keycloak property files from 2016-2019, vendored licence HTML from 2022, an old keycloak-dev.json. The licence files are plainly false positives, licence prose tripping generic-api-key. Several of the old Keycloak entries are UUID-shaped, which is what a real Keycloak client secret looks like, so they are worth a maintainer's eye even though the files are long gone and the history is public either way. No values are reproduced here or in the pull request. Whatever their status, none is something a contributor could fix, so repo-wide scope would fail the build for everyone. Scoping to the event range keeps the same scope gitleaks-action used, so this is not a coverage change.
Correction to an earlier revision of this entry. It said 82 findings across 9 commits, concentrated in docs/gdpr-compliance.md and test fixtures, and called all 82 false positives. That was measured on a shallow clone: at the graft boundary a commit appears to add entire files, which both inflated the count and misattributed it. On a full-depth checkout, which is what fetch-depth: 0 gives CI, the real figure is 61 across 16 commits with a different profile. The blanket "all false positives" reading was also not supported by the measurement. The conclusion is unchanged; the numbers behind it were wrong and are corrected here.
Design decision — the scanner's own config comes from the base branch on a pull request. The checkout on a pull request is the contributor's tree, and gitleaks reads two files from it: .gitleaksignore (fingerprint allowlist) and .gitleaks.toml (auto-detected config). Those two files are the whole surface: gitleaks.toml without the dot and .gitleaks.yaml are not auto-detected, which I checked rather than assumed. Either can switch a finding off. Verified by building the attack: commit a secret, read back the fingerprint gitleaks reports, add it to .gitleaksignore in the same pull request, and the scan passes clean. A permissive [allowlist] regexes in an added .gitleaks.toml does the same, and the repository has no .gitleaks.toml today, so adding one meets nothing. --gitleaks-ignore-path does not help: the flag is honoured but the repository-root file is still read alongside it. Restoring both from base.sha before scanning does, and it keeps the base's own legitimate entries. This was not a regression — gitleaks-action read the checked-out config the same way — but fork pull requests were never scanned before, so there was nothing to bypass; making fork scanning work is what makes the boundary matter.
Design decision — a range that cannot be resolved is an error, not an empty scan. On a pull request the step requires both base.sha and head.sha to exist in the checkout and exits non-zero with a ::error:: annotation if either is missing. The tempting fallback is to narrow the range and carry on, but that converts a broken fetch into a scan that passes without having looked at anything — the failure mode that is worst here, because it is invisible. Only the push path falls back, to the tip commit, and it logs that it did. fetch-depth: 0 on the checkout is what makes both SHAs reachable and is now load-bearing rather than incidental.
Verification. Both steps were extracted from the committed YAML and executed verbatim against this repository with the real 8.30.1 binary. Nine scan cases, three install cases and two configuration-bypass cases:
Pull request, valid range, no secrets
pass
exit 0
Pull request, base.sha unresolvable
fail loudly
exit 1, ::error::, scanner never invoked
Pull request, head.sha unresolvable
fail loudly
exit 1, ::error::, scanner never invoked
Pull request, empty range
pass
0 commits scanned, exit 0
Pull request planting a Bearer token of 40 hex characters
fail
leaks found: 1, exit 1
Push, valid before
pass
exit 0
Push, before all zeros (new branch)
tip commit, logged
exit 0
Push, before absent from the checkout
tip commit, logged
exit 0
Event that is neither push nor pull request
tip commit, logged
exit 0
Install, correct version
pass
checksum OK, ELF x86-64 extracted
Install, tampered tarball
reject
sha256sum -c exit 1
Install, nonexistent version tag
fail, leave nothing behind
exit 22, no binary written
Fork allowlists its own secret via .gitleaksignore
blocked
exit 1; base's own entries kept
Fork adds a permissive .gitleaks.toml
blocked
exit 1; file removed, logged
The detection test matters because the first attempt at it used the AWS documentation example key, which gitleaks allowlists — it reported clean, which would have made a scanner that detects nothing look correct. The planted-secret case is what proves the job is not vacuous.
Files: .github/workflows/ci.yml (the gitleaks job only; no other job, and no application code, is touched).
🧨 fix(secrets): export was corrupting the configs it exported (2026-07-30)
Repo: EDDI (fix/code-review-validation)
The Integration Tests job went from 156 failures + 14 errors down to 18 failures, 0 errors after the fixture and round-trip fixes. The remaining 18 split two ways, and the second one is the serious find.
15 of them: the same outer type, hardcoded inline
ComplexRulesAgentEngineIT, HitlToolPauseResumeIT, HttpCallsAgentEngineIT, LlmAgentEngineIT and PropertySetterAgentEngineIT build their output configs as inline JSON strings rather than loading fixtures, and every one of them wrote "outputs": [{"type": "text", …}] — the key OutputConfiguration.Output never declared. The file sweep could not see them because they are Java string literals. Removed; the inner valueAlternatives[].type (a real field) is untouched.
The other 3: SecretScrubber corrupts exported agents
ImportMergeIT imports the ZIP EDDI itself exported one test earlier, and it 400'd with:
SecretScrubber runs on the export path and replaces suspected secrets with ${vault:REDACTED}. Its second heuristic — Shannon entropy > 3.5 bits/char on any string ≥ 14 chars — cannot tell a long identifier from a long key. Simulating it against the weather-agent export, it rewrites four ordinary configuration values:
dynamicvaluematcher
a behaviour condition type
3.68
currentWeatherDescription
a property name
3.57
properties.count+1
a fromObjectPath
3.61
memory.current.httpCalls.currentWeatherDescription
a fromObjectPath
3.87
So export → import was never lossless: exported agents came back with a condition that resolves to no class, and property names and memory paths replaced by a vault reference.
This is pre-existing — SecretScrubber is byte-identical to main, and main is green. It was invisible because nothing validated a ruleset on write: the corrupt condition stored happily, the rule silently never matched, and the import returned 201. #620's write-time rule validation is what turned a silent corruption into a loud 400. The validation is doing exactly its job.
The fix exempts schema-fixed field names (type, name, fromObjectPath, toObjectPath, expressions, actions, …) from the entropy heuristic only. A condition type names a Java class and a fromObjectPath names a memory path — neither can be a credential, so the exemption costs no secret coverage, and field-name detection still runs first, so a field actually called apiKey or token is redacted regardless. Mutation-checked: with the exemption removed the new test fails with all four values rewritten to ${vault:REDACTED}.
🔀 chore(merge): main (#622, caller-bound MCP) into the wave-4a branch (2026-07-30)
Repo: EDDI (fix/code-review-validation)
main moved by 16 commits while #620 was open (#622, the operator-write foundation, plus caller-bound MCP credentials and HITL endpoint patterns), which put the PR into CONFLICTING. Worth recording because a conflicting PR has no computable merge ref, so no workflow can run at all — the Integration Tests failure GitHub was still showing on #620 was a stale result from before the fixes that address it. Resolving the conflict is what lets CI produce a real answer.
Git reported exactly one textual conflict — docs/changelog.md, a pure union of entries — and silently auto-merged three code files: McpToolProviderManager, AgentOrchestrator, and McpToolProviderManagerAdditionalTest. The auto-merges were the part worth checking, and one of them was wrong.
The silent breakage
#622 made the MCP tool cache credential-scoped, not just the client cache: toolCache is now keyed on cacheKey(config) (url|<sha-256 of apiKey>, or url|anonymous) so a tool list discovered with one credential can never be served to another. Our F12 TTL tests seeded that cache under the bare URL. After the merge the seeded entry no longer matched the lookup, and the two tests failed differently:
freshEntryIsServedFromCachefailed loudly —expected: <1> but was: <0>.staleEntryIsNotServedkept passing, vacuously. Nothing was served because nothing matched the key, not because the entry was stale. It would have gone on "passing" while testing nothing.
The second is the reason this is in the changelog: a green suite after a merge is not evidence that the merge is correct.
The helper now derives the key by reflectively calling the production cacheKey rather than reconstructing it — the same idiom #622's own tests use — so the next change to the key shape carries these tests along instead of hollowing them out. Production was correct throughout; only the test was stale.
AgentOrchestrator's side of the merge was javadoc-only on our branch and merged cleanly; McpToolProviderManager's validation block interleaved coherently (validateServerUrl + our validateTransport + #622's validateCallerBoundKey in one guard, with our one-time deprecated-transport warning after it). Our URL-keyed deprecatedTransportWarned set is unaffected by credential scoping, which is right: the transport is a property of the server, not of the credential.
🧩 fix(rag): two guards disagreed about chunkStrategy and neither test could tell (2026-07-30)
Repo: EDDI (fix/code-review-validation)
chunkStrategy has no reader — ingestion always builds a DocumentSplitters.recursive splitter — so an unsupported value is inert. Rejecting it at save time so the author hears about it is right. It was implemented twice, at two layers, with opposite intentions:
RestRagStore.prepareForWrite (create/update)
normalise legacy aliases, else 400 — author-facing, correct
RestRagStore.duplicateRag
normalise only, deliberately no rejection — "a copy of an existing document must not be refused just because the rules tightened after it was stored"
RagStore.validate (store, every write)
normalise legacy aliases, else throw
The store hook runs on create and update, so it silently overrode the duplicate exemption one layer down. Two paths were broken:
duplicateRag— refused to copy a document the same store happily serves throughreadRag.ZIP import —
RestImportService.createNewRagswrites throughcreateResourceDirect, i.e. straight to the store with no REST layer in front, and catches onlyResourceStoreException. AnIllegalArgumentExceptionescaped and rolled back the entire agent import over a field that changes nothing. (UpgradeExecutorreplays documents the same way.)
Why the test suite couldn't see it
This is the interesting part. Both layers were tested, and both tests passed:
RestRagStoreWriteValidationTest.duplicateDoesNotRejectAnUnsupportedStoredStrategyasserts duplicate returns 201 — but it buildsRestRagStorewithmock(IRagStore.class), socreatewas a stub and the store's write hook never ran.RagStoreValidationTest.createRejectsUnknownChunkStrategyasserted the store rejects that exact value.
Neither test crossed the boundary, so the contradiction was invisible. Demonstrated, not assumed: with the store's content.validate() restored, the new cross-layer test fails 3 of 6 cases with the real IllegalArgumentException, while RestRagStoreWriteValidationTest stays completely green.
Fix
The store now normalises but never rejects. Legacy aliases (paragraph, sentence) are still rewritten to the recursive they always meant — a data fix that is safe on every path, and it has to live in the store because import bypasses REST. An unsupported value is left verbatim rather than rewritten, so a duplicate is a faithful copy of its original. The author-facing 400 stays at prepareForWrite, the only layer that can tell "an author typed this" from "this document already exists" — the same layering decision made for RestMcpCallsStore in #619.
New RagStoreLayeringTest wires the real RagStore behind the real RestRagStore, mocking only the storage layer, and pins the division of labour: author input 400s at the boundary; a stored document is duplicable; a direct store write (as import performs it) does not abort; legacy aliases normalise on both paths. RagStoreValidationTest's two rejection cases are rewritten to assert leniency, with a comment recording that their original assertion was the defect.
🩺 fix(configs): EDDI could not read the configs EDDI writes — 8 red ITs, 3 real breakages (2026-07-30)
Repo: EDDI (fix/code-review-validation)
The Integration Tests job went red on this branch with eight failures across AgentConfigurationIT, AgentDeploymentComponentIT and AgentEngineIT, every one of them nothing but:
No field name, no constraint, no violation message anywhere in the job log. main was green, so the branch caused it. The cause turned out to be two different mechanisms, and the second one breaks production, not just tests.
Mechanism 1 — unknown keys became fatal, and EDDI's own fixtures had three
This branch added StrictConfigurationBodyInterceptor, which re-parses inbound configuration bodies with FAIL_ON_UNKNOWN_PROPERTIES and 400s on any key the model does not declare. That is the right call for a config-driven engine — a typo'd key in behavior.json is a behavioural bug, not something to discard silently — and it immediately earned its keep by catching three keys EDDI had been ignoring for years:
language on dictionaries
The model declares lang. Every shipped dictionary fixture set language, so lang was always null.
type on outputs[] items
OutputConfiguration.Output declares only valueAlternatives. The outer type was never read.
maxOccurrence: "ever"
Obsolete pre-v6 key. The current Occurrence reads maxTimesOccurred/minTimesOccurred, so the condition had no bound at all — which this branch's new Occurrence check now (correctly) refuses.
All three are corrected in the fixtures. language is deleted, not renamed to lang — renaming looks like the fix the original author intended, but it is a behaviour change: IDictionary.appliesToLanguage is isNullOrEmpty(dictLang) || dictLang.equals(userLanguage), so a null lang applies to every language while "en" applies only when the turn's user language is exactly "en". The ITs never set one, so renaming would have silently switched off the correction dictionaries these tests assert on. Deleting preserves the asserted behaviour exactly; opting those fixtures into language scoping is a separate, deliberate change.
maxOccurrence: "ever" becomes minTimesOccurred: "1" inside its enclosing negation — NOT(occurred ≥ 1) is exactly the maxTimesOccurred: 0 that agentengine/rules.json already uses to express the same "not yet welcomed" rule.
Mechanism 2 — EDDI serialized three keys it could not read back
The setup wizard's 400 had a different cause, and finding it mattered more than the fixtures. AgentSetupService does not write to the stores directly: it builds each config as an object and posts it through EDDI's own internal typed REST clients. So every generated config crosses the same strict boundary — and LlmConfiguration.Task serialized agentMode, which nothing could deserialize. isAgentMode() is a derived getter (a pure function of tools, enableBuiltInTools, a2aAgents) with no field or setter behind it.
A sweep for the same shape found three in total, now all @JsonIgnore:
LlmConfiguration.Task.isAgentMode()— derived from tools/builtInTools/a2aAgentsLlmConfiguration.Task.getSystemMessage()— a read-through ofparameters.systemMessage, so it also duplicated the prompt into every stored taskAgentConfiguration.MemoryPolicy.isEffectivelyEnabled()— derived fromstrictWriteDiscipline
This was never only a test problem. Three paths hand EDDI-serialized configs straight back to EDDI's REST boundary: the setup wizard, EDDI-Manager's GET → edit → PUT, and export → ZIP import. All three would have 400'd on any LLM or agent configuration in production. Ignoring the derived keys also stops persisting values that are recomputed on every read.
And one silently inert config, six years old
HttpPostResponse.retryApiCallInstruction was renamed from retryHttpCallInstruction in the v6 http→api sweep with no @JsonAlias. Jackson discarded the old key, so every config written before the rename lost its retry policy without a word in the logs — including EDDI's own documented reference agent, where the Agent Father's create_agent call declares maxRetries: 3 on 502/503 and has been running with no retry at all. The alias is added: stored and exported JSON configs are the one backward-compatibility contract EDDI keeps, and without it those documents would now 400 outright.
Four guards, because none of this was reachable from the unit suite
ITs need Docker and so only fail in CI; the strict boundary and the round-trip asymmetry are both checkable in seconds without it.
StrictBoundaryShippedConfigsTest— sweepssrc/test/resources/testsanddocs/agent-configs, strict-parsing every config against its model (26 checked). Sweeps the tree rather than listing files, so a fixture added later is covered automatically, and prints what it skipped so "passed" can never quietly mean "looked at nothing".StrictBoundaryRoundTripTest— asserts no config model serializes a property it cannot deserialize.RuleSetStoreShippedRulesetsTest— runs every shipped ruleset, and the wizard's generated one, through the real save-time validation.SetupWizardConfigsPassStrictBoundaryTest— round-trips each config the wizard generates, so the wizard can never again 400 on its own output with the cause buried in"Failed to set up agent: …".
Two of these guards were wrong before they were right, and both mistakes are worth recording:
The round-trip test was first written by instantiating each model from
{}and re-parsing. It passed onLlmConfigurationwhile theagentModebug sat in its nestedTask— a default instance has an emptytaskslist, so the nested model was never reached. It now uses Jackson introspection to compare serializable against deserializable property sets and walks into property types, and carries a second test asserting the traversal actually reachesTask— otherwise the sweep could silently regress to checking only the twelve roots.The ruleset sweep initially reported that the setup wizard generates an invalid ruleset:
'inputmatcher' requires a non-empty 'expressions'. That was the test's fault —IExpressionProviderwas mocked, so everyexpressionsvalue parsed to empty and everyinputmatcherin every ruleset "failed". With a realExpressionProviderthe wizard's ruleset passes. Reported as a product bug it would have been a wild goose chase.
🧹 fix(llm): bound the streaming warn-suppression key set (2026-07-30)
Repo: EDDI (fix/code-review-validation)
StreamingLegacyChatExecutor suppressed its "configured timeout is below the default" warning by remembering keys in a static Set, with a comment claiming it was "bounded because a config edit re-keys the entry". That reasoning is backwards: re-keying adds a key and leaves the previous one behind, so config edits were precisely what made the set grow. Keyed on (taskId, timeoutMs), the key space is unbounded over the lifetime of a process.
Now bounded by an explicit MAX_WARNED_TIMEOUT_KEYS (1 000) size check that clears on overflow.
Two decisions worth recording, because the obvious implementations are both wrong here:
Not a Caffeine cache. This is read on the per-request path — once per streaming turn, again per cascade step — and putting cache machinery there measurably slowed it: two timing-sensitive tests failed against their 66 ms budget when it was tried. A plain key set plus an O(1) size check costs nothing per call. (Isolated by reverting: the same tests pass with the set restored.)
Clear on overflow, not refuse to add. A hard cap would silently stop warning about genuinely new misconfigurations — the exact failure this warning exists to prevent. The worst case after a clear is that an already-warned task warns a second time: noise, not silence.
The misleading comment is replaced with the corrected reasoning rather than deleted, so the next reader does not re-derive the same wrong conclusion.
🔍 fix(all): critical re-review of the applied code-review fixes — 61 defects closed, 11 theatre tests made real (2026-07-29)
Repo: EDDI (fix/code-review-validation)
After ~120 findings from the external review had been applied across four waves, seven reviewers re-read the whole cumulative diff adversarially — explicitly assuming the previous agents got things wrong — and a mutation runner proved, empirically, which tests would actually fail if their fix were reverted.
They found 64 defects (4 critical, 10 high) and 11 tests that pass with the fix removed. Several of the defects were regressions introduced by the fixes themselves; one was worse than the bug it replaced. 61 closed, all 11 mutations now bite.
The critical four
PostgresResourceStorage— the D1 fix broke what it repaired. The rewritten reverse-lookup emitted the jsonpath operator@?unescaped, and pgjdbc parses a bare?as a bind placeholder — so on a real PostgreSQL the queries now threw where before D1 they merely returned empty. This shipped in the wave-1 merge and was live onmain.RestWorkflowStore— consequence of the above: cascade-delete of workflow extensions was a permanent no-op on PostgreSQL, and "which agents use this workflow" returned 500.ConverseWithAgentTool— F18's delegation guard was inert. The depth context was attached tostartConversationonly, while the delegated question travelled throughsay()with an empty context — so the callee always read depth 0 and the A→B→A cycle still recursed unbounded. The fix mirrors thegroupDepthpattern groups already use: attach the context to every turn, including the branch that reuses an existing conversationId.Conversation— G6 silently dropped data. Changed-only upserts permanently lost alongTermwrite whenever the turn that set it never reached teardown (HITL pause, error, cancel), because the next turn's baseline already contained the un-persisted value.
The one that mattered most
G2's secret fail-closed was incomplete. scrubSecretInput only rewrote input:initial and the input output — but InputParserTask has already written the plaintext to input:normalized in the same step, and every IData of a step is serialized into the persisted document. Worse, the scrub was a silent no-op whenever a normalizer was configured, because {memory.current.input} resolves to the normalized text, not the raw. A secret could still persist despite the fix reporting success.
Fixes that broke stored configs or existing deployments
B7's
modelID→modelIdrename had no legacy fallback, so every storedgemini-vertexconfig using the previously-working spelling silently built a nameless model. AGENTS.md is explicit that stored MongoDB/ZIP configs must keep working. Now reads both, preferring the canonical spelling, with a deprecation warning.I3 hard-rejected the MCP
ssetransport that the config's own javadoc advertised — and in the agent path the throw was swallowed, so an existing agent silently lost every tool from that server and burned circuit-breaker budget each turn.B14's fail-closed migration aborted forever. It refused to proceed whenever the v6 collection merely existed — the normal state, since every store constructor creates its collection — and never marked itself complete, so it retried and aborted indefinitely. Now distinguishes "exists and empty" from "exists with data".
B9's fail-closed capability table disabled vision on Azure OpenAI, where the model name is an operator-chosen deployment name that rarely reproduces canonical punctuation (
gpt4o-prodfailed thegpt-4osubstring test).
Fixes that quietly negated each other
E19 and E8 collided. E19's per-invocation random nonce was embedded in the Qute template text, and E8's compiled-template cache is keyed on that text — giving the httpcall output path a 100% cache miss rate and unbounded churn through a bounded cache. E19's injection-safety is kept; the cache key is now stable.
G18 and G20 collided. An entry the ledger dropped locally had already consumed its sequence number, so a store outage or full queue permanently made verification report BROKEN — the ledger accusing the deployment of deleting records it had dropped itself.
The 11 tests that were theatre
A test that passes with its fix reverted is not coverage. The mutation runner proved these did exactly that, and all 11 now fail when the fix is removed — among them: the F18 delegation-depth wiring (the guardrail tests covered the tool and the resolver but nothing pinned the call site, so hard-coding the depth to 0 left all 11 green), the C10 map-entry removal, B3's configurable drain timeout (a hardcoded 10s passed it), the E19 nonce's unguessability — the actual security property — and three separate holes in the audit ledger's drop/dead-letter accounting.
Verification
Clean test-compile, full suite with 0 non-environmental failures, and all 11 recorded mutations re-run and confirmed biting. Not verifiable here and left to CI: anything needing Docker or a bound socket — in particular a Testcontainers test executing the corrected JSONB path against a real PostgreSQL, which is what would have caught the @? regression in the first place.
✅ fix(configs): code-review findings wave 4a — write-time config validation (E6) and request-body validation (A11) (2026-07-29)
Repo: EDDI (fix/code-review-validation)
E6 is the highest-leverage item in the whole review: it converts most of workstream E from silently wrong into loudly wrong at save time. For a config-driven engine, silent acceptance of invalid config is the worst available failure mode — the agent author gets no feedback and the agent looks healthy while behaving wrongly.
E6 — the prescribed fix was unsafe, and was not applied as written
The review said: "enable FAIL_ON_UNKNOWN_PROPERTIES on the REST mapper only (never the persistence mapper — it needs schema-evolution tolerance)". That instruction assumes a REST-only surface exists. It does not.
SerializationCustomizer.configureObjectMapper is the shared recipe behind the CDI mapper, the @PersistenceMapper mapper and the Postgres JSONB mapper. customize() is not REST-only either — PersistenceModule.buildMongoClientOptions calls new SerializationCustomizer(false).customize(objectMapper) on the MongoDB BSON mapper, so flipping the flag there would have made Mongo document reads strict and broken loading of any stored document written by a newer version. The CDI mapper is also injected into the MicroProfile REST client and ~15 services that parse third-party JSON (Slack, web search, Dream, rule deserialization), none of which control their input's shape.
So instead of the flag flip, this adds StrictConfigurationBodyInterceptor — a JAX-RS ReaderInterceptor scoped to inbound JSON bodies whose target type is a first-party configuration model. It re-parses with a strict copy of the REST mapper and translates only UnrecognizedPropertyException into a 400 naming the field, its JSON path and the known fields. Every other parse failure falls through unchanged, and an empty body keeps its original stream.
A comment in SerializationCustomizer now records why that flag must stay false, so the next person doesn't "fix" it.
E6 — the validation hook
AbstractResourceStore gained protected void validate(T), defaulting to a no-op and invoked from create()/update() before anything reaches storage. Read paths deliberately do not call it, so stored documents keep loading.
RuleSetStore.validate hoists the wave-1 condition checks to save time without duplicating them: it serializes and runs the result through the very same IRuleDeserialization the deploy path uses. That covers empty action/input matchers, unknown occurrence values, context-type mismatches, empty negations and unknown condition types — each naming the offending rule. Previously all of these surfaced only at agent-deploy time, with a message naming neither rule nor group.
RagStore.validate closes the I3 leftover: RagConfiguration.validate() already rejected unimplemented chunkStrategy values, but its only caller ran at retrieval time, so a POST with "paragraph" returned 201 and the field was silently ignored.
A11 — request-body validation
There was zero jakarta.validation usage in the repo and no validator in pom.xml. Added quarkus-hibernate-validator (unpinned, via the BOM) and constrained the bodies that actually matter: DiscussRequest.question (unbounded free text passed straight to an LLM) and AttachmentRef.data (unbounded inline base64, now bounded by a validator deriving its ceiling from configuration rather than a hardcoded magic number).
Verification
Clean test-compile green first attempt. Full suite: 12,688 tests, 0 real failures — every one of the 308 failures/errors was classified programmatically from the surefire XML against the known environmental markers (this machine cannot bind loopback sockets), not eyeballed. All six mutation checks bite.
Because E6 makes previously-accepted input fail, the upgrade path was verified beyond the suite: the bundled initial-agents Agent Father config passes the new save-time validation (22 actionmatchers, 23 inputmatchers, 2 negations, all occurrences legal), and the import/export round-trip still works — RestImportService writes rulesets through the same store path the new hook guards.
Known limits, stated plainly
The interceptor's JAX-RS provider registration cannot be verified locally (no loopback sockets); its logic is fully unit-tested but CI is the gate for the wiring.
McpCallsConfiguration.validate()andLlmConfiguration.validate()still run at conversation/execution time rather than save time — the same defect class, now trivially fixable via the new hook. Left for a follow-up rather than widened into this change.
🔑 feat(operator): the foundation for an agent that can safely write (2026-07-29)
Repo: EDDI (feat/operator-write-foundation)
Groundwork for a workspace operator agent that manages a deployment — creating and updating agents and groups — while acting as the person chatting to it. Five commits, no new capability granted: the operator's endpoint allow-list is untouched and still read-only. What changes is that widening it is now safe, where before it was neither safe nor functional.
Generated writes did not work at all. McpApiToolBuilder.buildBodyTemplate emitted Qute variables for a request body but registered none of them, and AgentOrchestrator builds the tool schema from ApiCall.getParameters() alone. So the model had no documented way to fill a body; with strict rendering off the variables rendered empty and every generated POST/PUT/PATCH went out structurally valid and semantically empty. Adding a write endpoint before this would have produced garbage requests that fail at the far end rather than at the config.
The body is now one model-written variable. A per-property template looked more helpful and was worse three ways: every variable became a required tool parameter (a Map<String,String> has nowhere to record optionality, so a PATCH of one field forced the model to restate all the others); values were substituted into the JSON unescaped, since the templating engine runs in TEXT mode, so a value containing a quote could break the body or add fields the schema never declared; and the HITL card shows tool arguments, so "the arguments are the request" only holds if the body is one of them. The shape a decomposed template implied now lives in the parameter description, which names each property with its type and marks which are required.
Approval patterns can address the endpoint a tool calls. Names come from operationId or a slug and drift when a spec changes, and ToolApprovalGate allows an unmatched call — so a renamed or newly generated write arrived ungated and silently. Method and path were available and discarded one line into registration; they now travel alongside, so a pattern may match a bare name, source:name, or source.method:path:
gates every mutation without naming a tool, while http.post:/agentstore/agents addresses exactly one. Both speak the same METHOD /path vocabulary as the endpoint allow-list, so the two can be generated from one source instead of maintained in two. Documented in docs/hitl.md.
Design decision — enumerate downward, never upward. Whether something is gated stays in requireApproval/exempt; per-endpoint tuning may only lower friction. A missed exemption costs an approval prompt; a missed requirement is an ungated write. The same reasoning restricts the method qualifier to http: it is the only source whose tools record an endpoint, so mcp.post: is rejected at save time rather than saved as a pattern nothing could ever match.
MCP tool calls can now run as the chatting user. They previously ran as whatever static credential the config named, and a ${caller:token} there passed through the global-variable and secret resolvers untouched and was sent as the literal placeholder — failing silently rather than closed. The transport supported this all along: customHeaders has three overloads and EDDI used the constant one. The per-request McpHeadersSupplier overload makes the credential per-call while the client stays cached.
Discovery must be told apart from invocation explicitly. The first version of this decided by asking whether a caller was bound to the thread — but discovery runs inside the turn, on a thread that is bound, so initialize and tools/list went out with the first caller's token. Since the client is cached, that session was then reused by everyone after them, and the tool list reflected one user's permissions while being offered to the next. langchain4j distinguishes the two — DefaultMcpClient.listTools() delegates with a null InvocationContext while McpToolExecutor always builds one — but it does not enforce anything; EDDI has to read McpCallContext.invocationContext() and act on it, which it now does. Only a tool call carries the caller; discovery goes unauthenticated on a caller-bound config and the server decides.
Fixed in passing — a privilege bug. MCP clients were cached by URL alone, so two agents naming the same server with different credentials silently shared whichever client was constructed first. The key now includes a digest of the configured credential: a digest so a literal key never becomes a map key, taken unresolved so configs sharing a vault reference still share a client. This does not multiply clients per user — a caller-bound config yields one client whose supplier reads the caller per request.
Not solved, deliberately. On an expired MCP session the transport retries initialize() on an HTTP callback thread where the caller binding does not exist. That path now sends the request unauthenticated rather than falling back to the static key under the caller's intent — a visible failure instead of the wrong authority.
Corrections to earlier analysis, recorded because they changed decisions. McpCallsConfiguration.toolsWhitelist does not fail closed — AgentOrchestrator skips filtering when the list is empty, the same shape as the approval gate. toolApprovals is not agent-only: LlmConfiguration carries a per-task override that fully replaces the agent-level block. And a PUT on a config does not reach a running agent, because HistorizedResourceStore.update creates version + 1 while agents pin a version — so self-modification takes a chain of write, re-point, redeploy, each independently gated. The thing to guard is whatever can re-point a version reference, not PUT in general.
Verification. 566 tests pass on a clean build across the affected suites; checkstyle clean. Each fix mutation-checked by reverting it. Four tests written during this work were found vacuous by that check and rewritten — three asserted against a helper or state seeded through the very wrapper under test, one seeded a thread binding that the fix then restored, so it passed either way.
Next: per-endpoint approval friction (timeoutPolicy, approvalTimeout and the pause message are still single scalars for every gated tool), agent-readable documentation (EDDI's MCP client never reads MCP resources, so eddi://docs/* is reachable from a desktop client but not from an agent), widening the allow-list, and the Manager scope picker and approval surface.
⚙️ fix(runtime): code-review findings wave 3 — concurrency, lifecycle, cancellation, graceful shutdown, Dream wiring (2026-07-28)
Repo: EDDI (fix/code-review-concurrency)
Third wave of the 124-finding external review. 16 fixed, 1 partial. This is the wave where the findings were hardest to fix correctly, because the bugs are non-deterministic and several of the obvious fixes are wrong.
Pre-merge review pass (2026-07-29)
Like #618, this PR reached "approved" without CI or any review bot having seen it — a stacked base disables CodeRabbit, and a base retarget does not fire the CI trigger. A dedicated pass over the 37-file diff produced 6 findings that survived adversarial verification (12 of 18 were refuted) plus 22 from completeness/test critics, and Copilot found four more once CI could finally run. The high refutation rate is the point: concurrency invites "this looks racy", and verifiers were required to name the interleaving or drop the claim.
A destructive primitive behind a missing ownership check (HIGH). The schedule REST surface never checked
schedule.userId, which on its own was inert. This PR'sdreamType=dream_consolidationdispatch armed it: anyeddi-editorcould create and fire a schedule that bulk-deletes another user's persistent memories.RestScheduleStorealready injectedOwnershipValidatorand simply did not use it here. Now admin-or-self on create, update (the re-point path) andfireNow— refusing with 403 rather than silently rewritinguserId, which would hand back a schedule that does something other than what was asked.system:schedulerand blank ids stay exempt so existing stored schedules and Manager round-trips keep working.Dream consolidation crossed agent boundaries.
process()readgetAllEntries(userId)— userId-only, agent-unscoped — while every knob it obeyed came from one agent's config, so agent A'spruneStaleAfterDaysdeleted agent B's memories and A's model endpoint saw B's text. Cycles are now scoped to the firing agent's ownsourceAgentIdwrites, withcrossAgentMaintenance: trueas an explicit opt-in. Newly reachable in this PR, which gaveprocess()its first scheduled caller.A transient LLM blip permanently disabled a schedule. A single failure aborted the whole cycle and marked the fire FAILED, so three consecutive 429s dead-lettered the user's dream schedule. Transient classes (429/timeout/5xx) now skip the group and continue.
The B2 interrupt fix destroyed the bookkeeping it was protecting. The restore in
fire()ran beforelogFire(), and the sync Mongo driver throwsMongoInterruptedExceptionon connection checkout while the flag is set — so on exactly the interrupt the restore existed to handle, the FAILED fire log was lost andfailCountnever incremented. The flag is now parked and re-asserted in afinallyafter the store round trip, in bothfire()and the Dream fast-path. The residual half was inSchedulePollerService, which ranmarkFailed()on the same still-interrupted thread: the schedule stayed CLAIMED withnextFirein the past, was re-claimed every lease expiry, and could never reachmaxRetries— an interrupt turned a failing schedule into an unbounded re-fire loop.A draining node answered 500 instead of "retry elsewhere".
RestAgentEngine.sayInternal's trailingcatch (Exception)swallowed theRejectedExecutionExceptionfrom the new shutdown gate and rethrew it as a generic 500 — defeating the point of the graceful-shutdown work in this same PR.Also: the parallel-phase batch deadline was sized at one member attempt, so it always fired first and made the per-member RETRY/ABORT/attributed-SKIP branches unreachable;
maxSummarizationCallssilently stopped being enforced for stored configs (now honoured as an explicit backstop, deprecated in favour ofmaxCostPerRun);BaseRuntimeswallowedonCompletefailures with no identifying context; andWorkflowStoreClientLibrarydocumented an invariant the code neither enforced nor detected — now the component key no longer depends on it at all.
Docs corrected against the code, not against intent — the third and fourth instances of that error in this stack, so every claim was re-read out of the implementation: architecture.md told operators to create Dream schedules with the create_schedule MCP tool, which has no metadata parameter and therefore cannot set the marker the dispatcher matches on, so the documented procedure produced a schedule that never consolidated (REST is the only working route today, now written out with the two gotchas that bite: a message is required for CRON triggers even though the Dream path ignores it, and an unset userId defaults to system:scheduler, which DreamService refuses). IEventBus and InMemoryConversationCoordinator both claimed the coordinator is selected at runtime via eddi.messaging.type; it is @IfBuildProfile("nats"), a build-time condition, and that property is read by no Java code at all.
One disagreement adjudicated rather than deferred to severity. The completeness critic rated the NATS C13/C10 parity gap CRITICAL; two independent verifiers refuted it because @IfBuildProfile("nats") keeps that class out of shipped artifacts. Both cannot be right. The code defect is real and was fixed, but the CRITICAL rating was not — it is unreachable unless someone builds with that profile, and the class now records why the two coordinators differ.
Two tests were relabelled rather than trusted. The critics caught that both new GracefulShutdownService interrupt tests pass identically with and without the fix — sleepQuietly restores the flag, so the old code's next sleep threw immediately and it exited just as fast. That fix buys accurate logs (an interrupt was being reported as a 30-second timeout), not changed behaviour, and the tests now say so instead of implying coverage they do not have.
Cancellation that cancelled nothing (C1)
CompletableFuture.cancel(true) does not interrupt a runAsync/supplyAsync body — the JDK documents mayInterruptIfRunning as having no effect there. Five call sites in GroupConversationService relied on it, so "cancelled" agent threads kept mutating gc.getTaskList(), gc.getTranscript() and the errors list after the orchestrator had already persisted the document. Replaced with a cooperative MemberTurnCancellation token checked at the agent turn's own await points, plus a bounded drain. This is also the root cause of C7 (resetStrandedInProgressTasks could strand the very task it exists to rescue, because a falsely-"cancelled" thread flips state between the snapshot and the mutate).
The ~100-turn scalability cliff (C4)
ConversationService submitted the inner pipeline through the same bounded pool as the outer coordinator callable and then blocked on future.get(). With no quarkus.thread-pool.* overrides that is the 200-thread default: at ~100 concurrent turns every thread is a waiter, no inner task can ever be scheduled, and every turn fails at the 60s watchdog. A cliff, not a gradual degradation.
Fixed by routing nested submissions to a virtual-thread executor via a ThreadLocal marker scoped to the callable body. Chosen over CompletableFuture composition deliberately: the coordinator's ordering contract is "the callable returns ⇒ the turn is done", so making the outer non-blocking would need an IEventBus/IConversationCoordinator SPI change and would let the next turn of the same conversation start while the previous one still ran. Virtual threads are safe here — there is not a single @RequestScoped bean in src/main, and three existing callers already drive the pipeline with no request context on virtual-thread executors. The marker is cleared before callbacks run, so submitNext still schedules on the managed executor exactly as before; watchdog and timeout semantics are unchanged.
Re-execution and lost turns (C3, C9, C10, C13)
C9 —
onCompletesat inside thetrywhosecatch (Throwable)calledonFailure, so any unchecked throw on the completion path resubmitted the already-executed callable as a retry — LLM calls, tool side effects and cost all running twice. Completion dispatch moved outside the guarded region and both callbacks gated behind a one-shotAtomicBoolean.C13 — The coordinator retried failed turns 3×. Because
onFailurecan only be raised from inside the executor task, every retry re-ran a turn that may already have called an LLM and spent money. Retry removed entirely; genuinely pre-execution failures surface as a synchronous throw and are handled by C10's rollback.C3 — A timed-out turn still persisted over a newer one, because the stale-completion guard used the interrupt flag and the pipeline cleared it via
Thread.interrupted(). Replaced with a per-submission abandonment token set before delegatingcancel(), so nothing the work itself does can clear it.C10 — A throwing
submitleft the callable queued with nothing scheduled to run it, wedging that conversation permanently and leaking the map entry for the JVM's lifetime.
No graceful shutdown existed at all (B3)
grep -rn ShutdownEvent src/main matched nothing. A rolling deploy dropped every queued and in-flight turn with no drain and no readiness flip. Added GracefulShutdownService + ShutdownReadinessHealthCheck.
/rerun destroyed output and regenerated nothing (C5)
Selective execution passes a sublist with startIndex=0, so the loop index is sublist-relative — but the component-cache key was built from that relative index while the cache stores under the absolute index. The output task then ran with component == null and no-op'd, after the prior output had already been deleted. indexOffset was already threaded in and used only for HITL bookkeeping.
This one survived because every existing LifecycleManagerTest stubs the component map empty — the exact condition that hides it. The new test populates it at absolute indices and fails if the offset is removed.
B2: the finding's premise was inverted
The review claimed "interrupt flags swallowed in 18 of 20 handlers". Auditing all 28 sites individually (27 explicit catch (InterruptedException) plus one hiding behind a broad catch (Exception)) found the opposite: 14 already restored the flag correctly and 9 rethrew; only 4 genuinely swallowed it. Fixed those 4, plus:
ScheduleFireExecutor— a broadcatch (Exception)swallowingInterruptedExceptionfromlatch.await(5, MINUTES), so the poller kept firing schedules after being interrupted for shutdown.NatsConversationCoordinator— the mirror bug, not in the finding: it calledinterrupt()unconditionally oncatch (InterruptedException | TimeoutException), so a drain timeout left the shutdown thread flagged and would abort the@PreDestroysteps after it.
One restore is deliberately placed in a finally after the store round trips rather than at the top of the catch: the sync Mongo driver aborts with MongoInterruptedException when the calling thread is flagged, so an early restore would skip the very EXECUTION_INTERRUPTED write that branch exists to perform.
Dream wired up (I1, G8)
Per the repo owner's decision, DreamService is now registered with ScheduleFireExecutor rather than deleted, with its ceiling switched from maxSummarizationCalls to the dollar-based maxCostPerRun the project's own guidance prescribes. docs/architecture.md's claim that it performs scheduled maintenance is now true.
G8 mattered much more once Dream actually runs: consolidation upgraded self visibility to global whenever a group spanned multiple agents — and with summarizeGroupBy defaulting to category and preserveAgentProvenance defaulting to false, cross-agent grouping was the default path. Two agents' private memories became one entry every agent could read.
Partial
F6 — the REST/pipeline half is done (client disconnect now sets cancelled, and cancellation is checked at more points). The in-modules/llm half — cancellation checks inside the tool loop and the cascade — is deferred, since that module is owned by another workstream.
Disconnect is detected by testing
SseEventSink.isClosed()before and around each send — not by aConnectionCallback, which RESTEasy Reactive does not invoke on this path, asRestAgentEngineStreamingdocuments at the call site. An earlier draft of this entry namedConnectionCallback: it described the approach that was tried, not the one that shipped.
Verification
Clean compile passed first attempt, with no repairs needed despite three cross-workstream signature changes. Full suite as of the original wave-3 work: 12,633 tests, 0 non-environmental failures (308 listed failures/errors all carry a loopback/selector/event-loop signature; this machine cannot bind sockets). All four mutation checks bite — C1, C5, C9 and B2 each fail a test when reverted, verified against whole test classes and with surefire reports checked to confirm the new classes actually executed rather than being silently skipped.
The pre-merge review pass above re-ran the suite after its fixes: 12,912 tests, failures confined to the same 15 known network-dependent classes. Both figures are real runs at different points — the earlier one is not superseded, it just predates ~280 added tests.
🔎 fix(llm): review follow-ups — workflow version parse, log sanitization (2026-07-29)
Repo: EDDI (fix/review-followup-workflow-version)
Two findings Copilot raised against #618 after it had already been approved, kept out of that PR so they arrive small enough for CodeRabbit to actually review (#618 reached 120 files, past CodeRabbit's 100-file limit, so it merged without ever getting a CodeRabbit pass).
WorkflowTraversalaborted tool discovery for a whole turn over one malformed URI. Every other malformed-URI branch in that loop warns, marks the traversal degraded and continues. The version parse did not.String.replaceAllreturns its input unchanged when the pattern does not match, so a workflow URI carrying?version=abcpassed thecontains("version=")guard and reachedInteger.parseIntas the literal string"version=abc". TheNumberFormatExceptionescapeddiscoverConfigsentirely — so a single bad workflow URI took out httpcall, mcpcall and RAG tool discovery for that turn, rather than skipping the one workflow that was broken. Now matched explicitly, with "present but unusable" treated exactly like "absent" (and a digit run too large for anintfolded into the same path).MemoryItemConverterlogged raw exception messages in both the prompt-snippet and global-variable catch blocks. Exception text can carry user-controlled values, so this is the same CWE-117 class CodeQL flagged five times in #618; routed throughLogSanitizer.
🐳 fix(demo): the Open WebUI seeder did nothing on a second run (2026-07-29)
Repo: EDDI (feat/openai-api-adapter)
Two defects in src/main/docker/seed-demo-agent.sh, both found by actually re-running the stack rather than reading it. Also .env.example and docs/open-webui-integration.md §1.
Adding an LLM key later silently did nothing. The seeder guarded on "is any model exposed" and exit 0-ed if so. Because the MongoDB volume persists, the most common second run is exactly the case it broke: the rule-based agent is already there, the user has now set EDDI_DEMO_LLM_API_KEY, and they re-run to get an agent that can actually answer questions — and got no new model and no explanation. The guard is now per agent, keyed on the model-id prefix each descriptor name slugifies to (eddi-demo-agent-, eddi-llm-demo-), so each run creates only what is missing and says what it skipped. Vault storage was split into its own step that runs whenever a key is supplied, so changing the key rotates it instead of leaving the old one behind an "already exists" check.
The final "Ready" listing could omit the agent just created. The poll exited on grep -q '"id"', which a pre-existing agent satisfies immediately — so a freshly deployed LLM agent, still inside the adapter's 30s model-cache TTL, was absent from the output and looked like a failure. Verified it was only a display problem: re-querying after the TTL showed all four models. The loop now waits for the specific prefixes expected on this run.
Verified live, not reasoned about. Ran the stack against a populated volume: first run created the rule-based agent, second reported already deployed — skipping and created no duplicate, third (with a key) added the LLM agent and listed all four models. The old code would have exited at step two.
Also: the closing "open this URL" line hardcoded port 3000 and was wrong whenever OPEN_WEBUI_PORT was remapped — compose now passes OPEN_WEBUI_URL. .env.example gained an Open WebUI demo section (the demo's .env is gitignored, so these variables were undiscoverable from a fresh clone). The docs gained three subsections that only exist because they bit during testing: re-running, port collisions (Bind for 0.0.0.0:7070 failed when another EDDI holds the port), and down vs down -v.
Doc accuracy pass. Cross-checked §3 against application.properties — all 11 eddi.openai-compat.* keys and all 11 defaults match; the §8 error table's 8 codes all exist in OpenAiErrorResponse; the documented endpoints match RestOpenAiAdapter's @Paths. No drift found there.
🔒 fix(openai): two CodeQL alerts — logged intent, and a regex flagged as ReDoS (2026-07-29)
Repo: EDDI (feat/openai-api-adapter)
CodeQL failed the PR with 3 new alerts. Two are addressed here; the third is deliberately left.
java/log-injection (medium, PostgresUserConversationStore:139). The delete path logged intent unsanitized. That file predates this branch, but the OpenAI adapter is what makes it carry attacker-influenced data: the intent is channel:openai:<agentId>:<chatKey> and the chat key comes from a request header, so a newline in it could forge log entries. Routed through the existing LogSanitizer.sanitize(), which the bridge already uses for its own logging. The Mongo store logs nothing, so there was no counterpart to mirror.
java/polynomial-redos (high, AgentModelResolver.slugify). The dash trim (^-+|-+$) is replaced with a character walk. This is not a fixed vulnerability, and should not be read as one. The alert is a false positive twice over:
The
NON_SLUG_CHARSpass on the line immediately above collapses every run of non-slug characters into a single-, so-+can never match more than one character.Measured directly, the regex is linear anyway — 3ms on 400k separator characters. Java's engine anchors on
$rather than backtracking, so the quadratic path CodeQL models does not exist here.
It was replaced regardless: a standing high-severity alert competes for attention with real ones, and character walking is no harder to read than the regex was.
A vacuous test was written and then removed. The first version of this change asserted slugify completed within 2 seconds on 400k separators. Measuring the old regex showed it finishes in 3ms — so that assertion would have passed against both implementations and proved nothing. It was deleted rather than shipped; a test that cannot fail is worse than no test. What remains asserts trimming behaviour across the implementation swap, and is mutation-checked: stubbing out the trim kills 3 tests.
Dismissed with justification: java/user-controlled-bypass (high, OpenAiAuthFilter:84). The filter returns early when the request path is not under /v1, and the path is user-controlled — which CodeQL reads as authentication being skippable.
It is not, because this filter is neither the only nor the first check in front of those paths. application.properties ends with quarkus.http.auth.permission.authenticated.paths=/,/* at policy authenticated, and Quarkus HTTP authorization runs before JAX-RS request filters — so every path the filter declines has already been required to authenticate. The one exception is /v1/*, which carries its own permission entry at policy permit precisely so the shared API key can be checked in the filter rather than rejected at the OIDC layer as a malformed JWT. That permit set is exactly what isGuarded returns true for. The guard therefore does not choose between authenticated and anonymous; it chooses between the adapter's key check and Quarkus' own, and declining is the safe branch.
The reasoning lives in a javadoc block on isGuarded, not only in the GitHub dismissal, so a reader of the code finds it where the suspicious-looking early return is.
The dismissal is pinned by a test. It rests entirely on two lines of configuration, and a security finding waved away on the strength of config that nobody re-checks is how a real bypass eventually ships. quarkusStillGuardsEverythingThisFilterDeclines reads src/main/resources/application.properties and asserts the catch-all paths, the catch-all policy, and the /v1/* exemption. Flipping the catch-all policy to permit fails it. Note it reads the file from source: the first version loaded /application.properties from the classpath, where src/test/resources shadows the production file and declares none of these keys — it asserted nothing and failed loudly on first run.
✨ feat(openai): render structured outputs, report token usage (2026-07-28)
Repo: EDDI (feat/openai-api-adapter)
Two gaps closed in the /v1 adapter, both from the honest support assessment of the previous session. Files: new OpenAiOutputRenderer, new TokenUsage + StreamOptions DTOs, changes to OpenAiConversationBridge, OpenAiSseWriter, RestOpenAiAdapter, ChatCompletionResponse, ChatCompletionChunk, ChatCompletionRequest. Docs: docs/open-webui-integration.md §7.1, §7.2, §10.
Structured outputs are no longer dropped. An EDDI turn carries eight output types; the OpenAI protocol carries one string, and the shared ConversationOutputExtractor keeps only the text. Through that lens a wizard agent whose whole turn is "Which provider?" plus five quick replies arrived as a question with no visible answers — indistinguishable from a broken agent. OpenAiOutputRenderer now takes the shared extractor's text verbatim (a reply's wording must not depend on which channel it left through) and appends a Markdown rendering of the rest: quick replies as backticked values, images as , application links as Markdown links, buttons as their label, input fields as a described prompt. agentFace and other are dropped deliberately — an avatar has no text equivalent, and captioning it would add a line the agent author never wrote.
The shared extractor is untouched. Its other callers (
GroupConversationService,CreateSubAgentTool,ConverseWithAgentTool) feed agent-to-agent prompts, where interaction affordances are noise. The renderer is adapter-local.Quick replies render as literal values, not a numbered list. A numbered list invites
2as an answer, which no input matcher recognises. Thevalueis what a chat UI puts on a button and therefore what a user would retype;expressionsstays internal (asserted by test — it is an internal identifier that must not be shown).Both POJO and Map item shapes are handled — a turn that just ran yields typed items, a rehydrated conversation yields Maps.
Streaming needed a split. When the model streamed the prose token by token, re-rendering the full text at
onCompletewould have sent the whole reply twice, sorenderExtras()emits the affordances alone.
usage is now reported — the previous entry's claim that EDDI does not surface token counts was wrong. LlmTask writes audit:token_usage (inputTokens/outputTokens/totalTokens) into the current step, accumulated across every model call the turn made — cascade steps and tool round-trips included.
This required
returnDetailed=trueon bothsayandsayStreaming. The filtered snapshot keeps onlyinput:initial,actions,output*andquickReplies*, dropping every audit key — which is why the counts looked unavailable. The snapshot is read in-process and never serialized to the client, so the cost is one extra step's worth of references. This flag is load-bearing and otherwise invisible, so a test pins it: flipping it back would silently removeusagefrom every response with nothing else failing.Absent, not zero, for rule-based agents. They call no model;
0 tokensreads in a client as a measurement rather than an absence.totalTokensis derived when a provider omits it but reports both parts — a usage block whose parts do not add up is worse than one that computes the sum.Streaming usage is opt-in via
stream_options.include_usage, emitted as a trailing empty-choicesframe afterfinish_reasonand before[DONE], per spec. An unrequested empty-choices frame is a protocol deviation some clients reject.
Testing: 183 adapter tests (up from 151) — new OpenAiOutputRendererTest (21) plus usage coverage in the bridge, SSE writer and wire-format suites. Two mutation checks were run rather than trusting green: stubbing renderQuickReplies to null killed 7 tests, and reverting returnDetailed to false killed the 2 flag-pinning tests.
Not done: exposing agent groups as models. Assessed rather than assumed — every piece exists (groups list via readDescriptors("ai.labs.group", …), discuss() returns a synthesizedAnswer, continueDiscussion() gives multi-turn, GroupDiscussionEventListener gives streaming), but it needs a second bridge with its own conversation mapping, streaming path and approval surface. That is a feature-sized change, not an addition to this one, so it is recorded as gap #1 in §10 instead of half-landed here.
🧠 fix(llm): code-review findings wave 2b — LLM core, persistent memory, migration, import/export (2026-07-28)
Repo: EDDI (fix/code-review-llm-memory)
Second half of wave 2, stacked on the access-control PR. Covers the LLM tool pipeline, the persistent-memory subsystem, the v5→v6 migrations, and import/export.
Pre-merge review pass — and corrections to the claims below (2026-07-29)
This PR reached "approved" having been reviewed by no CI run and no review bot: CodeRabbit reported Review skipped: reviews are disabled for this base branch because it was stacked on a disabled base, and a base-branch retarget fires edited, which is not in the default pull_request trigger set — so no workflow ever ran on its head SHA. A dedicated review pass over the 91-file diff found 16 findings that survived adversarial verification, plus 21 from completeness/test critics. All are fixed here except one, named below. Several entries further down overstated what the code delivered; those claims are corrected here rather than quietly edited away.
Fixes with user-visible consequence:
F14 broke the error path it was meant to unify. Routing rule-triggered MCP calls through
ToolExecutionService.executeToolWrappedwas right in intent, but that wrapper catches every exception and returns an error string.RetryConfiguration.executeWithRetrytherefore never saw a throwable: retry never retried,continueOnErrorbecame dead code, and a failed MCP call was stored as a successful response. The metering wrapper is kept; a real failure signal is restored on top of it.F18's delegation-depth guard was inert in production.
delegationDepthwas injected only into the callee'sstartConversationcontext, landing on step 0; the follow-upsaycarried no context, so the turn that actually decides delegation read nothing. The claim below that F18 was "mutation-checked" was true of the test, not of production — the test drove the mechanism directly and never crossed thesayboundary where the value was lost. The depth now propagates to the turn that reads it.F15 was fixed on two of three merge routes. Within-server dedupe and the
AgentOrchestratorsource merge were handled; the cross-server merge indiscoverTools()was stilladdAll/putAll, so two MCP servers advertising the same tool name still shadowed silently.G12 / G5 / G7 shipped on MongoDB only. The "a turn is never silently discarded" guarantee, the
most_accessedrecency reservation, and global-entry ownership preservation were all absent from the PostgreSQL adapter — which answered benignly rather than signalling the gap. Now ported, with tests. This is the third cross-backend gap in this stack (after scheduleuserIdand the auditsequence), which is no longer coincidence: each was a feature built against one backend, silently missing on the other, and invisible because the degraded answer looked like a normal one. The cross-backend conformance suite (D4/J3) is the real fix and remains outstanding.A summarizer could be handed another vendor's API key. When
conversationSummarynames a differentllmProviderthan the parent task, the parent's resolved parameters —apiKey,baseUrl— were inherited and passed to that other vendor's client. Not theoretical: the pre-PR POJO defaults serializedllmProvider: "anthropic"into stored configs. Credential- and endpoint-bearing keys now stop at a provider boundary; vendor-neutral tuning keys still travel.Validation moved to the boundary where rejecting is safe.
McpCallsTask.configure()and RAG retrieval were both throwing/dropping on stored configs — a workflow-load failure and a silently empty knowledge base respectively. Both are now lenient on read (stored configs stay loadable, which is the one backward-compat contract that matters) and strict on write, inRestMcpCallsStoreandRestRagStore.Also fixed: the v6 rename migration aborting permanently when a v6 collection merely exists (EDDI creates those itself via
createIndex) instead of skipping; the new pre-migration backup duplicating conversation transcripts outside the reach of GDPR erasure; export cleanup recursively deleting a sharedtmp/<agentId>it could not prove it created, reachingtmp/import/; and the streaming no-partials fallback overwriting thewarningkey thatresponseValidationdispatches on, silently skippingonTruncation/onContentFilter.
Still not fixed — I5. AgentConfiguration.maxCheckpointsPerConversation is still ignored at runtime. The test named explicitRetentionIsHonoured exercises an overload no production path calls, so it proved nothing; the misleading label is removed rather than left to imply coverage. Wiring the value through needs a session-scope slot on IConversationMemory and propagation via IAgent/Agent, which is a larger change than a review fix should smuggle in. Any statement below that I5 is complete is wrong.
One critic finding was investigated and rejected: the ExpressionFactory.setDomain removal was claimed to change and(...)/or(...) parsing, but domain splitting happens only in setExpressionName(String) and the parser builds children through constructors that assign the name verbatim — the deleted line really was a no-op.
The tool pipeline had a second door
F14 —
executeToolWrappedis genuinely well-built and has one production call site, so every one of the seven tool sources routes through it. ExceptMcpCallsTaskdoesn't — a behaviour-rule-triggered lifecycle task invoking the same external MCP tools, which resolved aToolExecutorand called it directly. It also bypassedToolApprovalGate, sohitlConfig.toolApprovalsgated LLM-initiated calls and not rule-initiated ones — a human-approval gate with a hole in it.F18 —
converse_with_agenthad no guardrails at all: it never consultedDynamicAgentConfig, andallowDelegationwas never checked anywhere. Agent A could call B, which calls A — and with noconversationIda fresh conversation starts, so the busy-guard never breaks the cycle. Prompt injection in a user message was sufficient to start it. Now enforcesallowDelegation, a target allowlist, and a delegation-depth counter propagated through conversation context (reusing the mechanism groups already use forgroupDepth).F17 —
maxCreatedAgentsPerDiscussionwas enforced per turn, not per discussion, becausesharedCreatedIdswas created fresh in everybuildToolListcall. A 5-member × 3-phase discussion with the default cap of 5 permitted up to 75 agents deployed to production.F15/F16 — Remote MCP tools silently shadowed built-in tools (specs accumulated in a
List, executors in aMap, so duplicates reached the model and last-write-won), and their descriptions entered the prompt verbatim with no length cap or sanitisation — whitelisting operates on names, so a whitelisted tool whose description changes was ungoverned.A10 — The MCP client did no URL validation at all, unlike its A2A sibling, while a discovery endpoint echoed the response body — a full SSRF read primitive.
Two real bugs found by the tests, not by the review
The review's F12 fix (cache the workflow traversal that runs 3–4× per LLM task per turn) introduced two defects that only surfaced when WorkflowTraversalTest started returning 0 instead of 1:
The cache memoized failure-derived results. A traversal whose workflow read threw still cached its empty result for the full TTL, and replayed it to the other traversals of the same turn — an agent silently losing its httpcalls/mcpcalls/RAG configuration with nothing in the logs but one WARN. Now only complete traversals are cached.
The cache key omitted the target class while the value was cast with an unchecked
(List<StepConfig<T>>). Justified by a comment asserting a 1:1 mapping that nothing enforces — any future caller asking for the same step type with a different class would get another caller's entry and aClassCastExceptionfrom a cache hit with no connection to the calling code.
This is why the triage pass asked "stale test, bad fixture, or real bug?" for every failure rather than adjusting tests until green.
F13 was fixed but unreachable
The wave-2 agent added inheritedParameters overloads to SummarizationService and tested them directly — but no caller in src/main passed them. ConversationSummarizer still called the 4-arg overload, so the rolling summary still could not authenticate and still silently never materialised. Threading the parent task's resolved parameters through LlmTask → ConversationSummarizer → SummarizationService is what actually closes it; both hops now have tests that fail if the parameters are dropped.
DreamService remains on the un-inherited path — it is a background job with no parent task, so it needs a credential source of its own. That is part of wiring Dream up (finding I1), scheduled for wave 3.
Memory & properties
G2 —
scope: "secret"failed open to plaintext. On vault failure the method returned the plaintext before the scrub block, persisting the secret twice: as a conversation property and as rawinput:initialdata. Vault-disabled is the default (eddi.vault.master-keyships empty), so this was the common path. Now fails closed, scrubbing first.G1 — User-memory search and delete crossed agent boundaries:
getVisibleEntriesbuilds a proper self/group/global filter, butfilterEntriesandgetByKeyfilter onuserIdalone — and the tool path used that unscoped pair.G13 — Token-aware windowing could emit a prompt with no user message at all: the backward fill breaks on the first message that doesn't fit, and if the anchors alone exceed the budget the code only warned. The model then answered with no idea what was asked. The final user message is now reserved first; anchors get trimmed instead.
G12 — A turn could be silently lost:
replaceOnewith no upsert whoseUpdateResultwas discarded, so a conversation deleted mid-turn by erasure or a retention sweep discarded the turn while the caller got a normal response.G5/G6/G7 —
most_accessedrecall did an N+1 write inside an open read cursor and was self-reinforcing (only already-top-N entries got incremented, so a new entry could never climb in);storePropertiesPermanentlyrefreshedupdatedAton every longTerm property every turn, somost_recentdegenerated to "everything is recent" anddeleteOlderThannever expired anything for an active user; and re-upserting a recalled entry silently flipped its owner to the reading agent.G9/G10/G11 —
ConversationPropertiesbroke theMapcontract (clear()/remove()left the template map stale, so a checkpoint rollback left post-checkpoint properties visible);scope: "step"was persisted despite the docs; one malformed field failed the entire conversation load.
Migration & import/export
B12 — Migration irrecoverably erased typed BSON values: it deleted the legacy
valuefield unconditionally but only wrote a replacement for String/Map/Integer/Float, dropping doubles, longs, booleans and arrays with no error.B13 — The template migrator rewrote any
{...+...}sequence, corrupting JSON bodies and arithmetic that merely sat in a document containing Thymeleaf syntax.B14 — The rename migration skipped when the v6 collection already existed and still marked itself complete, abandoning the v5 data.
D11/D12 — Import had no rollback (any failure mid-way left every already-created resource orphaned), and neither import nor export ever deleted their temp directories —
ZipResourceSourceisAutoCloseableand itsclose()does the cleanup, but two call sites constructed it outside try-with-resources.
Verification
Full unit suite: 12,384 tests, 0 non-environmental failures. G2 and G12 mutation-checked, plus a second sharper mutation for G2 that removes only the input-scrub call — both halves are independently covered. F13's new wiring is mutation-checked at the LlmTask hop.
🧵 fix(security): carry caller identity across the cascade and group dispatches (2026-07-28)
Repo: EDDI (feat/caller-identity-passthrough)
The caller binding is a ThreadLocal, and four more dispatch sites hand a turn to a fresh virtual thread without carrying it. A ${caller:token} apicall reached from any of them failed closed with "the conversation turn has no authenticated caller" — safe, but invisible to the agent designer and dependent on unrelated configuration.
CascadingModelExecutor:577— every cascade step runs onTIMEOUT_EXECUTOR. The same agent config worked or failed purely on whethermodelCascadewas set.GroupConversationService:282— the whole discussion is dispatched to a virtual thread, so no member agent had a caller.GroupConversationService:2366— parallel-phase speakers fan out to further threads.GroupConversationService:3772— the HITL resume path.
CallerIdentityContext gains withIdentity for Runnable/Callable, withIdentitySupplying for Supplier (needed by CompletableFuture.supplyAsync), and captureOrCurrent(), which prefers the active request and falls back to the thread's binding — the group discussion is dispatched from the REST thread, the cascade from mid-pipeline.
Design note. The Supplier variant is named apart from the withIdentity overloads on purpose: a value-returning lambda satisfies both Callable and Supplier, so same-named overloads are ambiguous at every call site.
Review round two found a fifth site and two flaws in the wrapper itself:
GroupConversationService:1788— the task-force EXECUTE phase fans out again throughCompletableFuture.runAsync, so every task wave lost the caller. Missed first time because the search pattern coveredsubmitandsupplyAsyncbut notrunAsync.The parallel-phase fan-out captured with
current(), which is null whendiscuss()is called synchronously on the REST thread — onlycaptureOrCurrent()sees the request there.A null identity was a no-op, so work dispatched without a caller inherited whatever binding the pooled thread still carried from the turn before. It now binds null, masking it.
Nested wrappers cleared instead of restoring, so an inner wrapper wiped the outer caller and the rest of that turn ran unauthenticated. The previous binding is now saved and restored.
Note on scope. A group member agent now acts as the person who started the discussion. That follows the feature's model — the operator acts as the chatting user — but it is worth stating, because a member agent's reads are now attributed to that user in the audit trail.
1204 tests pass across the affected suites. Neutering the Runnable wrapper fails the propagation test, so the binding is pinned rather than assumed.
🔓 fix(security): ${caller:token} never reached its resolver (2026-07-28)
Repo: EDDI (feat/caller-identity-passthrough)
A critical review of the caller-identity PR found the feature was dead on arrival, and the same defect silently disables two older documented features.
ApiCallExecutor.buildRequest runs every header value through prePostUtils.templateValues before the caller-identity, global-variable and vault resolvers. TemplatingEngine's trigger regex \{[a-zA-Z#/!] matches {c, so Bearer ${caller:token} is handed to Qute, which parses {caller:token} as a namespaced expression. An unresolvable namespace is a hard failure regardless of strictRendering, and no caller resolver existed. Proven against the project's own build:
So ${vault:...} and ${vars:...} in apicall headers have never worked either.
Fix. CallerNamespaceResolver returns the caller placeholder verbatim so it round-trips through templating to the real resolver. It resolves nothing itself.
Deliberately caller only. vault/eddivault keep failing loudly in templated positions. Letting them through would widen where a secret is substituted, and the resolved request body is written to conversation memory unscrubbed — a vault reference in a body template would put a plaintext API key into MongoDB. Docs claiming vault works in apicall headers were the bug, not the behaviour.
Why no test caught it: all three ApiCallExecutor suites stub templating as a pass-through, and CallerIdentityResolverTest calls the resolver directly. Nothing exercised header -> Qute -> resolver. CallerNamespaceResolverTest now does, including a guard-rail test that fails without the resolver and one asserting vault still refuses.
Also fixed
setup-apihardcodednullfor the LLM base URL while the manager sent Ollama's URL asapiBaseUrl— the tool target — pointing every generated tool at the model server.CreateApiAgentRequestgains anllmBaseUrlfield, passed tocreateLlmConfig.The by-value token redaction added last round was untested through
ApiCallExecutor; deleting the call kept every test green. Now covered with a real resolver and mutation-checked.Docs overclaimed:
${caller:userId}is resolved in headers and query parameters, not "anywhere";rejectTokenReference's Javadoc claimed request bodies are checked.
🔐 feat(security): forward the caller's identity to apicall headers (2026-07-28)
Repo: EDDI (feat/caller-identity-passthrough)
An agent could only call an API with a static credential baked into its apicall config. That is the wrong shape whenever the API being called is EDDI's own: an OIDC token expires within the hour, cannot be least-privilege, and collapses every action to one synthetic principal in the audit trail. The EDDI-Manager "Platform Operator" needs exactly this — an agent that reads the platform on behalf of whoever is chatting with it — and could only be built by smuggling the user's bearer through the per-turn conversation context, which persists the token to MongoDB.
What changed. Apicall headers may now reference the authenticated caller:
${caller:token}— the caller's raw bearer token${caller:userId}— the caller's principal name (not a secret)
ApiCallExecutor resolves these last in the header chain (after global variables and vault refs), because resolution needs the target URI.
Files
engine/security/CallerIdentity.java— record (token, userId, origin); deliberately not part ofIConversationMemory.engine/security/CallerIdentityContext.java— captures the identity on the REST request thread and binds it to whichever pool thread runs the turn.engine/security/CallerIdentityResolver.java— the${caller:...}resolver, mirroringSecretResolver/GlobalVariableResolver.engine/security/OriginMatcher.java— scheme/host/port comparison with default-port normalization.engine/internal/ConversationService.java— captures inprocessConversationStepand decorates the callable.modules/apicalls/impl/ApiCallExecutor.java— resolves in headers; rejects a token reference in a query parameter.
Threading — the non-obvious part. A turn is built on the request thread but executed twice removed from it: submitInOrder hands it to a coordinator thread, which hands the pipeline to another thread via runtime.submitCallable. Request-scoped beans (SecurityIdentity) resolve at none of those points. So the identity is captured while the request context is still live and travels with the callable (withCallerIdentity), not with a thread. Binding the coordinator thread would have missed the pipeline entirely — an easy and silent mistake.
Design decisions
Same-origin only. The token is released only when the outbound call targets the exact
scheme://host:portthe caller addressed, taken from the inbound request rather than configuration. An agent config naming a third-party host therefore cannot exfiltrate a user's token, and the feature needs no allow-list to be safe out of the box.Headers only.
${caller:token}in a query parameter or request body is rejected — tokens in URLs leak through access logs, proxies and browser history, and outside a header the reference is never substituted.${caller:userId}is resolved in headers and query parameters.Fails closed. An unsatisfiable reference throws rather than resolving to
"", which would silently sendBearerand surface far away as a puzzling 401.Never stored. Resolution happens while building the request, and
scrubSensitiveHeadersredacts it before the request is written to conversation memory. Header-name matching alone was not enough — a token placed in an unconventionally named header would have slipped through — so the resolved token is additionally matched by value.Opt-out.
eddi.caller-identity.enabled(defaulttrue) forbids the feature outright.
Async boundaries. Two further hand-offs lose a ThreadLocal binding and had to be covered explicitly, or ${caller:token} would fail closed for no reason the config author could see: the HITL resume path (runtime.submitCallable(resumeCallable, ...)) and fire-and-forget batch calls in ApiCallExecutor. CallerIdentityContext.propagate() carries the binding across the latter; the resume path captures its own caller, since a resume is itself an authenticated request.
Tests. CallerIdentityResolverTest (24) and CallerIdentityContextTest (10) — same-origin refusals (host, port, scheme downgrade), fail-closed paths, regex-escaping of tokens containing $/\, thread isolation, and clearing on pooled threads. Disabling the same-origin guard fails 4 tests (mutation-checked). All 227 ConversationService*Test tests still pass.
Note for the manager: the operator ships this as its caller-identity auth mode, provisioning apiAuth as Bearer ${caller:token}; the conversation-context workaround and its token-at-rest warning are gone.
🔐 fix(security): code-review findings wave 2a — access control, A2A ownership, GDPR & audit ledger (2026-07-28)
Repo: EDDI (fix/code-review-access-control)
Second wave of the 124-finding external code review, split into two PRs so each stays under CodeRabbit's 100-file review limit (wave 1 at 151 files was skipped by it entirely). This half is the access-control and compliance surface.
The guard triad existed — it just wasn't called everywhere
EDDI already has OwnershipValidator, ConversationAccessGuard and HitlAccessGuard, used correctly in engine/internal, engine/hitl and the MCP surface. Every finding here is a place that never called them.
A1 (critical) — The SSE turn endpoint had no ownership check while its non-streaming twin did. The turn then executed under the target conversation's
userId, loading that user's long-term memories into the prompt and running tool calls in their context. Fixed in two layers: the guard moved down intoConversationService.say/sayStreamingso no future REST adapter can omit it, before the memory snapshot loads — plus a REST-layer check so denial is a plain 403 rather than an error event on an already-200 SSE stream.A2 (critical) — Attachment endpoints authorised the path parameter, not the caller.
IAttachmentStore.load(ref, requestingConversationId)checks that the named conversation owns the blob — and the caller supplies that name, so the check was self-satisfying. All five methods now require caller ownership, checked on the request thread before the async hop (SecurityIdentityis request-scoped).A3 —
?deleteOlderThanDays=0permanently deleted every ended conversation in the deployment, from an endpoint with no role at all. Now admin-only with a minimum of 1 day.A4 — The tool control plane carried no
@RolesAllowedat all: rate-limiter reset, cost-budget reset, and a history endpoint dumping raw tool arguments and results for any conversation.A5 —
/propertiesstore/properties/{userId}had neither role nor ownership check over the sameIUserMemoryStorethatRestUserMemoryStoreguards on all nine of its methods. Writes there land in the victim's next system prompt.A6 — A conversation-id oracle: it returned another user's live
conversationId, which is the discovery half of A1 and A3.A7 — Template preview read any conversation's memory and returned a flattened dump of properties/context/memory — and since the caller supplies the template, it was effectively a query language over someone else's conversation.
A8 — Config stores with full CRUD and no role. The review named one; the actual inventory was six, including two it missed (
IRestCapabilityRegistry,IRestWorkflowStepStore).IRestVersionInfowas deliberately left alone — it has no@Pathand is a mixin, reasoning recorded in the code.A9 — A2A sat entirely outside the ownership model: caches keyed on a caller-supplied id,
contextIdas an unauthenticated read+write conversation handle, conversations created withuserId = null(whichOwnershipValidatortreats as "legacy — allow", so permanently unowned), and rawe.getMessage()returned to arbitrary peers.A12 — The global exception mapper returned the raw driver message as the 500 body — collection names, hostnames, replica-set topology. Now logged at ERROR with a correlation id.
GDPR & audit (G14–G20)
G14 — Erasure was served stale indefinitely from a Caffeine cache with no TTL that erasure never invalidated.
G15 — The erasure cascade missed three stores while
docs/gdpr-compliance.mdasserted it "covers all data stores": conversation checkpoints (which carrypropertiesCopyincluding PII, behind a javadoc claiming it was "used during GDPR erasure" when its only caller was unreachable), group transcripts, and schedules that kept firing under an erased user's id.G16 —
AuditHmac.verifyHmachad zero production callers; the docs told operators to "recompute the HMAC and compare it" and the product shipped no way to do so. Added admin verification endpoints.G17 — GDPR pseudonymisation did
updateMany($set userId)with no HMAC recompute, anduserIdis a signed field — so every routine erasure produced rows cryptographically indistinguishable from tampered ones. The class javadoc claiming a write-once contract was literally true and substantively false.G18 — No hash chain: entries were independently signed, so deletion and reordering were undetectable. Added a per-conversation sequence inside the signed payload (chosen over a global chain, which would serialise all audit writes).
Follow-up (review): the first cut shipped this on MongoDB only.
PostgresAuditStorepersisted no sequence and leftsupportsSequence()at itsfalsedefault, soAuditLedgerServiceskipped assigning one entirely — every PostgreSQL deployment silently had no deletion detection, reported asUNAVAILABLE, which reads like "not applicable" rather than "unprotected". Added the column, an idempotentALTER TABLEdefaulting old rows to theUNSEQUENCEDsentinel, the(conversation_id, sequence)index verification reads, andsupportsSequence() = true. This is the second cross-backend gap in this PR (after scheduleuserId), both in compliance code, both invisible because the degraded answer looked benign — the case for the deferred D4/J3 conformance suite.
G19 —
eddi.vault.master-keyships empty, so entries were written unsigned by default while the docs present the ledger as evidence-grade.ComplianceStartupCheckshad zero references to vault/HMAC/audit.G20 — Unbounded audit queue that re-offered failed batches into itself — an OOM loop under a slow store.
Documentation corrected (I7–I13)
Several docs described behaviour that did not exist. docs/semantic-parser.md documented a stemming extension that does not exist, four times, including inside the flagship copy-paste config — copying it throws UnrecognizedExtensionException and the agent will not start. The same doc's expression table claimed number(42) and time(15:00) where the code emits integer(42) and epoch millis, so rules written from it never fired. docs/conversation-memory.md and docs/properties.md both contradicted AGENTS.md §5.1 on the template model ({properties.X.valueString} fails at runtime — MemoryItemConverter puts raw values).
Verification
Full unit suite: 12,384 tests, 0 non-environmental failures (308 listed failures/errors all carry a loopback/selector/event-loop signature — this machine cannot bind sockets; CI is the gate for those). Five high-stakes fixes were mutation-checked — revert the fix, confirm a test actually fails, restore: G2, G12, A1, A2 and F18 all bite, verified against whole test classes (a -Dtest=Class#method filter silently runs 0 tests and exits 0 when the method is in a @Nested class, which reads exactly like a pass).
🐞 fix(engine): code-review findings wave 1 — parser, rules, templating, apicalls, datastore, LLM infra, deployment (2026-07-28)
Repo: EDDI (fix/code-review-findings)
First of four waves applying a 124-finding external code review. Wave 1 covers the findings whose file sets are disjoint, so they could be worked in parallel without conflicting. 53 findings: 50 fixed, 3 partial. Every finding was verified against source before being acted on — none turned out to be a false positive.
Crash / correctness
B1
readActions(..., limit)didactions.subList(0, limit)with every caller defaultinglimit=20, so any config with fewer than 20 actions threwIndexOutOfBoundsException→ 500. Fixed inOutputStore,RuleSetStore,ApiCallsStorewithMath.min(limit, actions.size()). Invisible to tests because they all mock the store.B4
Permutationaccumulatedn!into anint. From n=13 the product wraps; at n=17 it goes negative, so the iterator yielded a single permutation and parse quality silently got worse on longer sentences. The counter was redundant —calculateNext()already terminates — so it was deleted, with an explicitlength < 2guard replacing the one case it was load-bearing for.B5
containsPunctuationasked whether a string contains its own characters — always true. Now consults the configuredpunctuationRegexPattern(not thePUNCTUATIONconstant, which is only the default and would disagree with the replacement regex on a customised deployment).B6
isOrdinalNumberreturnedStringfrom anis*predicate;indexOf("")is always 0, so"5."returned""and the three callers disagreed on what that meant. Renamed toextractOrdinalValue→Optional<Integer>.B7
KEY_MODEL_ID = "modelID"(capital D) blanked the model name for Vertex Gemini, losing capability lookup, token estimation and audit model naming.B8 The Ollama builder — the default local provider — silently dropped
temperature,maxTokens,topP/topK. Added, plus a shared unrecognised-key warning across all 11 builders.B9
ModelCapabilityServicewas the inverse of its own javadoc: unknown models resolved to supported, so images were forwarded and 400'd at the provider. Now fails closed, with a per-task override so an unlisted-but-capable model can still be asserted.
Behaviour-rule engine (E1–E5, E11)
The systemic defect was silent acceptance of invalid config — for a config-driven engine, the worst failure mode, because the agent designer gets no feedback and the agent looks healthy.
E1
NOT_EXECUTEDwas folded intoSUCCESS, so every path yielding it made a guard rule fire unconditionally. Now treated asFAIL— this is the amplifier that made E2–E5 dangerous rather than merely wrong.E2 Multi-child negation (the form the docs demonstrate) was ignored; now AND-combines children.
E3 Empty/misspelled matcher configs matched everything via
indexOfSubList(x, []) == 0; now rejected atconfigure().E4 A typo'd
occurrencesilently becamecurrentStep, converting the exact guard AGENTS.md §5.3 mandates into the globally-firing rule it warns against. Now throws, naming the bad value and the legal set.E5
sizematchercould never match a collection (parseInt("[a, b]")); the documented example was unreachable.E11
ContextMatcherNPE'd on a runtime/config context-type mismatch, killing the turn.
Templating & output (E7–E9, E13)
E7
strict-renderingdiffered between dev (false) and prod (true), so a missing property rendered empty in dev and in prod shipped the raw template literal to the end user, while quick replies putnullin the list. Aligned on one lenient, safe behaviour across profiles.E8 Templates were re-parsed on every output, httpcall body/header and property instruction, every turn. Now a bounded Caffeine cache.
E9
TemplateModewas accepted and ignored (no escaping), and the HTML branch was unreachable because"output:html".startsWith("output")is always true.E13 Only 3 of 8 output types were templated —
inputField,button,applicationLink,agentFace,othershipped{properties.x}raw. Now an abstracttemplatedCopyonOutputItem, so a new type cannot silently miss templating.
Parser (E14–E18)
E14
appendExpressions=falsedisabled the whole parser rather than just the merge — the entire storage block sat inside the flag, so no expressions and no intents reached any rule. Store/merge decisions are now separate.E15 Dictionary
langwas dead: no implementation overrodegetLanguageCode(), so an agent withenanddedictionaries matched both against every turn.E16
PhoneticCorrectionkept only the last word per phonetic code — and the codes are lossy by construction, so night/knight/nite collapsed to one entry.E17 No caps on solution enumeration (13.8s at 15 tokens with a dense dictionary). Added config-driven
maxInputTokens/maxSuggestions/maxSolutions, aHashSetfor the O(k²) scan, and cachedvalues().E18 Damerau-Levenshtein returned the entire dictionary as candidates and computed
accuracy = 1.0 - distance, yielding 0.0 and −1.0 against a documented 0..1 contract.
API calls (E10, E19, E20)
E10 A
"*"httpcall fired once per action, duplicating non-idempotent POSTs on a multi-action turn.E19 Output JSON was built by string concatenation from upstream API response bodies — a
"broke the JSON and a crafted value injected arbitrary output items into the agent's reply. Now built with Jackson.E20 httpcalls had no timeout and no response-size cap; success bodies landed unbounded in conversation memory against Mongo's 16MB limit. Added per-call
timeoutInMillis/maxResponseSizeInByteswith defaults.
Datastore (D1–D3, D5–D10)
The two backends had silently diverged, because nothing tests them against each other.
D1 Postgres
data->'a.b.c'looks up a literal key of that name — it does not traverse. Dotted paths returned zero rows forever while Mongo traversed correctly. Now a real JSONB path, keeping the existing injection-safety.D2 The Postgres factory accepted an
indexesargument and dropped it; real callers passed real hints and got sequential scans.D3 Mongo used
$set(merge), Postgres usedEXCLUDED.data(replace) — so clearing a config field was a no-op on one backend and took effect on the other, both returning 200. Unified on full-document replace.D5 The cascade-delete reference guard never worked: the query said
WorkflowSteps...(capital W) against a field persisted asworkflowSteps, and Mongo paths are case-sensitive — so shared configs were cascade-deleted while other workflows still referenced them. The guard now also fails closed on a query error.D6
PRIMARY KEY (id, collection_name)putidfirst, soWHERE collection_name = ?could not use it, and nothing indexeddata— every listing was a sequential scan over the single shared table. Reordered for new tables; existing tables get the equivalent index (column order can't be changed byCREATE TABLE IF NOT EXISTS).D7 (partial) Restored the seven index declarations lost when the DB-agnostic
DescriptorStorereplaced the legacy one, and madeoriginIda real indexed constant. Not done: splitting config descriptors from per-conversation descriptors into separate collections — that needs a data migration and two callers outside this workstream.D8 Descriptor listings did one
read()per id after fetching up to 10,000 ids. AddedreadMany.D9 (partial) Fixed the
intoverflow inResultManipulator.limitEntities(index * limitwrapped negative on deep pages). Not done: keyset pagination — it changes theindex/limitpaging contract across every REST store and the Manager UI. Deep offsets are at least index-served now.D10 History and current-row writes were non-atomic; a crash between them left an archived-as-deleted row with the live row still present. Added
storeHistoryAndUpdate/storeHistoryAndRemove.
LLM infrastructure (F1–F5, F20)
F1 Rate limiting was a single global bucket per tool name, so one conversation starved every other user of that tool. Now keyed on
(conversationId, toolName), with an optional global bucket for provider-quota protection, and the gauge is finally tagged.F2
ChatModelRegistrycaches were unbounded and keyed on Qute-resolved parameters — so"modelName": "{properties.preferredModel}"minted a retainedChatModel+ HTTP client per conversation-derived value. Now bounded, matching the sibling factories.F3 Embedding model/store caches never evicted on credential rotation and used
expireAfterAccess, which resets on read.F4
mongoClientCachewas unbounded, keyed on a raw connection string (a credential), and leakedMongoClients on eviction.F5 Cost-budget eviction claimed "oldest" but iterated
ConcurrentHashMap.keySet()— arbitrary order — so an in-flight conversation's spend could be dropped, resetting its budget to $0.F11 (part)
ObservableChatModelleaked a platform thread per timeout from an unbounded cached pool.F20 Every wizard-created agent hardcoded
logRequests/logResponsestotrue, writing all conversation content to application logs with no opt-out — a Pillar 1 violation. Now config, defaulting tofalse.
Deployment & CI (H1–H3, H5–H10, H12–H15)
H1 Both k8s and Helm scaled to 2–10 replicas against a coordinator whose per-conversation serialisation is a JVM-local
synchronized(queue)— so two turns of one conversation ran concurrently on different pods, both replaced the whole document, and one was silently lost. Pinned to 1 with the reasoning inline; autoscaling is now gated behind a genuinely distributed coordinator.H2 (CI half)
always()cancelled the implicit needs-gate and the tag branch short-circuited it, so a tag push published, signed and attested regardless of test result. Also addedintegration-testtodocker's needs, since it is the only job running the JaCoCo gate.H3 The image scan ran with
exit-code: 0, so no vulnerability could block a release.H5 No container-level
securityContextin any manifest — both deployments were rejected by a restricted Pod Security Standards namespace.H6 (partial) Replaced mutable tag
labsai/eddi:6with an immutable patch tag and added a digest value +cosign verifyguidance. Pinning the literal digest remains a release-time step — inventing one would break everykubectl apply.H7
networkPolicy.enabledwas a dead Helm value with no template behind it; added one, plus link-local to the egress deny list so the network layer mirrors the app's own SSRF protections.H8 Secrets were injected as env vars (readable via
/proc/<pid>/environ, leaks into crash dumps). Verified Quarkus can read them from files, then switched to a mounted projected volume.H9 The quick-start compose file hardcoded auth off with no override.
H10 Helm shipped default PostgreSQL/Keycloak credentials and rendered an empty vault key without failing; now
required.H12 The production image copied the entire docs tree — changelog, incident-response playbook, internal review standards — and served it to MCP clients.
H13 ZAP ran after publish, against an instance with auth deliberately disabled, passive-only, unable to fail. Dropped rather than cited as coverage it never provided.
H14 Fuzzing targeted vendored copies of the security-critical parsers with no drift check, so PRs touching the real sources fuzzed a stale duplicate.
Verification
./mvnw clean test-compile green from scratch (deliberately clean, not incremental — several signature changes crossed workstream boundaries, and incremental builds reuse stale .class files for unedited callers). 158 targeted tests pass. E1's fix was mutation-checked: reverting it fails 2 tests in RuleTest.
Deliberately deferred to later waves: E6 (write-time config validation), E12, D4/J3 (cross-backend conformance suite — needs Testcontainers), the pom.xml batch (H4/H11/H15 + J1/J8/J9), and the doc corrections these fixes imply (I7, I8, I12, semantic-parser.md, httpcalls.md).
✨ feat(openai): OpenAI-compatible API adapter for Open WebUI (2026-07-27)
Repo: EDDI (feat/openai-api-adapter)
New /v1 surface presenting deployed agents as OpenAI "models", so Open WebUI, the Python openai SDK, LangChain and LiteLLM can drive EDDI conversations. New package integrations/openai/, parallel to integrations/slack/. Disabled by default. Full guide: docs/open-webui-integration.md; design rationale in planning/openai-api-adapter-plan.md.
Four earlier drafts of this plan were built on premises that turned out to be false. Each was verified against source before implementing; the corrections shaped the design:
Content negotiation cannot dispatch sync vs streaming. The plan routed on
Acceptvia two@Producesmethods. Butopenai-pythonhardcodesAccept: application/jsonand never varies it bystream, and Open WebUI sends noAcceptheader at all (it sniffs the response content-type). Every streaming request would have landed on the JSON method. → one method, dispatching on thestreamfield of the body, which is what the OpenAI spec says and what vLLM/llama.cpp/LiteLLM/Ollama all do.Open WebUI does not map
chat_idto theuserfield. #27174 is an issue, closed as not planned;payload['user']is set only for pipeline models and is an object, not a chat id. The plan's entire per-chat isolation mechanism did not exist. → key onX-OpenWebUI-Chat-Id(#15813, gated byENABLE_FORWARD_USER_INFO_HEADERS), withuseras a defensively-typed fallback.IRestAgentAdministrationis@RolesAllowed({"eddi-admin","eddi-editor"})at type level, and Quarkus enforces it via a CDI interceptor. Injecting it for/v1/models, as planned, would have 403'd for every ordinary caller. → readIAgentFactory+IDocumentDescriptorStoredirectly. REST facades are the authorization boundary; a public surface must not reach around one.The auth story did not exist.
quarkus.http.auth.permission.authenticated.paths=/,/*captures/v1/*, so ansk-…bearer would be rejected by OIDC before the adapter ran. → explicit/v1/*permission entry plus two modes (permit+ shared key, orauthenticated+ OIDC), constant-time key compare, and a startup guard.UtilityAgentProvisionerwas cut. It had an unauthenticated caller creating and deploying an agent, cloning another agent's credential reference and consumingmaxAgentsPerTenantquota — blocked by the role checks above anyway, and in tension with Pillar 1 (the engine authoring agent config at runtime). → replaced by<model>:statelessvariants: start, say, end. No writes, no roles, and useful beyond title generation.The new-chat heuristic was cut.
userMessageCount == 1 && hasTurns → endConversationis unreliable (regenerate and edit-and-resend look identical to a first message), destructive (memory loss is irrecoverable), and directly contradicted the plan's own recommended Open WebUI Filter, which strips history to the last user message — under which every turn would have destroyed the conversation. A new chat is a new chat key, which is a new intent. No inference needed.HITL was unaddressed.
onSkippedfires for bothAWAITING_HUMANandIN_PROGRESS; the plan mapped both to "conversation busy", which would make any agent usingPAUSE_CONVERSATIONpermanently unusable with a misleading message. → sentinel-snapshot discrimination, mirroringSlackEventHandler, and pauses surface as chat text with200, never as an HTTP error — a 4xx makes clients discard the user's message.
Components (src/main/java/ai/labs/eddi/integrations/openai/): RestOpenAiAdapter (/v1), OpenAiConversationBridge, AgentModelResolver, OpenAiMessageMapper, OpenAiSseWriter, OpenAiAuthFilter, OpenAiStartupGuard, OpenAiCompatConfig, OpenAiApiException + OpenAiExceptionMapper, and 11 wire DTOs under model/.
Other design decisions:
Model ids are
<slug>-<last 6 of agentId>. Agent names are not unique, so a bare slug would be non-deterministic with two agents called "Support". Name and slug lookups are accepted but only when unique; an ambiguous match returns 400 listing candidates rather than guessing.Slugging folds accents via NFD rather than dropping them —
Übersichtwas slugging tobersicht, mangling every non-ASCII agent name. Caught by its own test.Images map to
attachment_Ncontext entries, so they flow through the existingAttachmentForwarderwith its vision gating, byte caps and SSRF-guarded fetching. Zero core changes. Two parsing fixes:data:image/png,payloadis legal and has no;(scanning to;threw), and remote URLs get a concrete MIME from the extension —image/*passes the forwarder'sstartsWith("image/")gate but is rejected by providers when handed toImageContent.from.usageis omitted rather than zero-filled when the agent called no model. (This originally read "EDDI does not surface per-request token counts here" — that was wrong, and is corrected in the 2026-07-28 entry below.)In-flight completions are semaphore-bounded — each holds a worker thread, since the bridge blocks on the turn as the Slack handler does.
Reused, not duplicated:
ConversationOutputExtractor.extractResponse()already existed inengine/memory(added upstream) and handles more output formats than the Slack-local copy the plan proposed extracting. Phase 2 of the plan became unnecessary. Noted separately:SlackHitlSupport.extractSlackResponseTextstill duplicates it and should delegate.
Self-review pass (after the feature was complete) found and fixed six defects, each now covered by a test:
Semaphore permit leak. The streaming path handed its permit to the
StreamingOutputbody to release. If that body never runs — a client disconnecting before serialization starts, say — the permit is never reclaimed, and aftermax-concurrent-requestssuch events the adapter returns 429 permanently, until restart. The resource now releases unconditionally and the stream body takes its own permit inside one try/finally.GET /v1/models/{id}echoed the caller's string, not the canonical id — a lookup by agent name returned{"id":"Customer Support"}, which is absent fromGET /v1/models, so a client round-tripping the answer would ask for a model that does not exist.ResolvedModelnow carries requested and canonical ids separately (plus the descriptor timestamp, which was hardcoded to 0).InterruptedExceptionwas swallowed in the turn wait, leaving the worker thread looking healthy with its shutdown signal gone.A null exception message rendered to the user as the literal text "null" in the stream error path (NPEs carry no message).
hasSentContent()lied — it returned "the stream has started", true even after a content-freefinish(). RenamedhasStarted().The
eddi.openai.requestscounter documented in the plan was never implemented. Added withmodeandoutcometags, sopaused(reviewers behind) andbusy(clients racing) are distinguishable from real errors.
Also verified rather than assumed: quarkus.rest.jackson.optimization.enable-reflection-free-serializers=false in this project, so @JsonProperty on record components works — finish_reason and owned_by serialize correctly. Added OpenAiWireFormatTest to pin that, since the non-streaming response shape had no coverage at all.
Corrected a documentation claim rather than the code: unimplemented /v1 paths (/v1/embeddings etc.) return Quarkus' plain 404, not an OpenAI error envelope. A catch-all route would risk shadowing the real endpoints for a cosmetic gain.
Tests: 148 unit tests plus 16 integration tests, all green. Mutation-checked — reverting the model-ambiguity guard, the identity refusal, and the HITL sentinel distinction each makes the relevant tests fail. RestOpenAiAdapterTest (@QuarkusTest, binds a socket) is deferred to CI per the local-environment constraint.
Attachment support covers all three binary content-part types: image_url, file (inline PDFs and documents) and input_audio. All map to attachment_N context entries, so EDDI's existing forwarder does the real work — capability gating, byte caps, PDF text extraction, SSRF-guarded fetching. Three details the wire formats make easy to get wrong, each pinned by a test:
input_audio.datais raw base64 with nodata:prefix, unlike every other binary payload in the protocol, with the type in a separateformatfield.mp3maps toaudio/mpeg—"audio/" + formatwould produceaudio/mp3, which is not a real media type.file.file_datais a full data URI, and the declaredfilenamebeats a genericapplication/octet-streamtype, since clients that base64 a file without sniffing it send exactly that.file.file_idreferences the OpenAI Files API, which EDDI does not implement; those parts are skipped with an actionable warning rather than becoming empty attachments.
Stateless requests have two routes to the same behaviour. The :stateless model suffix exists because a model name is the only per-request dimension a UI like Open WebUI can express — its title-generation setting is a dropdown, so a query param, header or body field could not be selected there at all. It follows the ecosystem convention for behavioural model variants (OpenRouter's :nitro/:floor/:free, Ollama's llama3:8b) and has the side benefit of appearing in GET /v1/models, making the capability discoverable. Alongside it, a stateless body field gives programmatic callers the explicit parameter (extra_body={"stateless": True} in the Python SDK). The two are OR-ed rather than letting either win: model:"x:stateless" plus stateless:false is self-contradictory, and running stateless only loses continuity while running stateful would persist a conversation the caller may not have wanted. expose-stateless-variants=false blocks both routes, so the switch cannot be circumvented by moving the request into the body.
Gateway-agent recipe documented in §11 of the integration guide, rather than building a passthrough proxy mode. A thin LLM-only agent behind this adapter already provides vaulted API keys, audit, tenant quotas and cost tracking — which is what the passthrough idea was actually after. Building a real proxy would mean entering the LLM-gateway market (LiteLLM, Portkey, Cloudflare AI Gateway) with a worse product, inheriting per-provider streaming/tool/vision passthrough maintenance for zero agent value, and shipping a feature with neither logic nor configuration — which is the opposite of Pillar 1. The recipe's limits are stated up front: single-turn only, one agent per model, no caching/fallbacks/virtual keys.
OpenAiCompatIT closes the one gap the unit tests structurally could not. The adapter serves sync and streaming from a single JAX-RS method, dispatching on the stream body field rather than the Accept header — a design forced by real client behaviour. A refactor toward content negotiation would look correct in every unit test and silently return JSON to every streaming client, so the guard has to live at the HTTP layer: stream:true with Accept: application/json (what openai-python sends) and with no Accept header at all (what Open WebUI sends) must both yield text/event-stream. 16 tests also cover the response wire shape, per-chat conversation isolation end to end, the error envelopes, and tolerance of the unknown request fields every client sends.
It deploys the shared minimal agent (parser/rules/output/templating), so it needs no LLM credentials and is deterministic. Runs in CI only — Quarkus cannot boot in the dev sandbox (Unable to establish loopback connection), the same environmental limit that already fails SlackWebApiClientTest. The test compiles and is discovered by failsafe locally; whether it passes is for CI to say.
A real defect the integration test caught immediately. OpenAiCompatIT.streamingBodyIsWellFormed failed on its first CI run (passing only on failsafe's retry) with content arriving after the [DONE] sentinel. Root cause: ConversationService.sayStreaming ends with conversationCoordinator.submitInOrder(...) and returns immediately — every handler callback fires later on another thread. The bridge wrote the terminator in its finally as soon as sayStreaming returned, so the response closed mid-turn and late tokens raced a closing stream. Fixed by awaiting a CompletableFuture completed by whichever terminal callback fires (onComplete/onSkipped/onError), bounded by request-timeout-seconds.
The unit tests had all passed because their Mockito stubs invoked the handler synchronously, hiding the asynchronous contract entirely — a mock that was more convenient than the real collaborator. A regression test now drives the handler from a background thread, and mutation-checking confirms removing the await makes it fail. This is precisely the class of bug an HTTP-level test exists to find, and it was found on the first run.
A runnable demo — and two more defects it caught. docker-compose.openwebui.yml brings up MongoDB, EDDI (built from the working tree, since no published image has the adapter), Open WebUI and a seeder that deploys a small rule-based agent so the model list is not empty. Standing it up and actually talking to an agent found two bugs that 146 unit tests and 16 integration tests had not:
plainTextleaked into every assistant message. Jackson treatsChatMessage.isPlainText()as a bean property, so responses carried a field absent from the OpenAI schema. Harmless to clients, wrong on the wire — and invisible toOpenAiWireFormatTest, which asserted expected fields were present but never that there were no extras. Fixed with@JsonIgnore; the tests now assert the exact field set.The create-conversation race returned a bare 500. Open WebUI issues the completion and its title request concurrently with the same chat id, so both find no mapping and both insert. The handler caught
ResourceAlreadyExistsException— but the Mongo store lets a rawMongoWriteException(E11000) out, so the catch was dead code against the real store. Now caught broadly and resolved by re-reading the mapping, which is datastore-agnostic (the Postgres store reports it differently again). Verified with four concurrent requests to one chat: all 200.
Also noted while building the demo, and not fixed here because it is unrelated to this PR: POST /backup/import/initialAgents cannot import the bundled Agent Father on Linux. The ZIP's entries are correctly forward-slashed, but extraction writes them as single files with literal backslashes, so the workflow directory never exists and the import 500s. The demo seeds through the ordinary REST API instead.
A documentation error found by using the demo, and two settings verified from Open WebUI v0.11.0's notes.
The guide claimed Open WebUI injects RAG context into the system message. It does not, by default. Verified in the running container: RAG_SYSTEM_CONTEXT defaults to False (open_webui/env.py), and middleware.py then calls add_or_update_user_message(...) rather than add_or_update_system_message(...). So dropping a PDF into a chat delivers several thousand tokens of ### Task: …, <context><source id="1">…</source></context> and <attached_files> markup as {memory.current.input}, with the user's real question buried at the end — which breaks input matchers, makes property setters capture the whole blob, and stops quick replies firing. Corrected in the guide, added to the must-configure table, and RAG_SYSTEM_CONTEXT=true set in the demo compose. This was found by actually uploading a file, not by review.
Also verified and documented: AIOHTTP_CLIENT_STREAM_IDLE_TIMEOUT (new in v0.11.0) ends a stream when the upstream goes quiet — sized for an LLM's time-to-first-token it will cut EDDI agents short, since a rule-based agent emits nothing at all until its turn completes; and passthrough_params on a connection forwards non-standard body fields verbatim, which makes the stateless field settable from the Open WebUI UI rather than only via extra_body.
The demo agent's prompt was also reworded. A property setter bound to {memory.current.input} stores the whole turn, so answering "Gregor. What are you capable of?" stored all of it as the name and looked like a parsing bug. It now says it stores the next message verbatim — which is what a naive slot-filler does, and a fair thing for a demo to teach.
A second injection path, and an LLM-backed demo agent. Setting RAG_SYSTEM_CONTEXT=true moved the retrieved chunks to the system message as intended, but an attachment-carrying turn still arrived polluted — this time with an <attached_files> block. Different mechanism: Open WebUI's built-in Files tool, gated by use_builtin_tools, which depends on a per-model builtin_tools capability with no environment override. On an EDDI model that capability is pure cost — the adapter never returns tool_calls, so the tools can never fire, yet enabling them rewrites the user's message. Documented as a per-model toggle in the model editor, and added to the must-configure table alongside RAG_SYSTEM_CONTEXT.
The demo also gained an optional LLM agent, with its provider key in the Secrets Vault. The rule-based one demonstrates the transport and the state bridge without credentials, but it has no model, so it cannot answer questions about anything — which made "what is this pdf about?" look like an adapter failure when it was simply an agent with nothing to think with. Set EDDI_DEMO_LLM_API_KEY (plus optional _TYPE/_MODEL) and a second agent is deployed whose system prompt references {context.openai_system_message}, so it can answer about uploaded files. The key is stored via PUT /secretstore/secrets/default/demo-llm-api-key and the config holds only ${vault:demo-llm-api-key} — an agent config is exported, diffed, rendered in the Manager UI and logged, so a literal key would travel with all of it.
Verified on a clean run: both agents reach READY, all four models are exposed, the stored config reads "apiKey": "${vault:demo-llm-api-key}", and a scan of every collection in the database finds the plaintext key in none of them. The LLM call itself is still unverified — the run used a deliberately fake key.
Getting there also surfaced a documentation error in AGENTS.md §5.5: the workflow step type for LLM interaction is listed as eddi://ai.labs.langchain, but LlmTask.ID is ai.labs.llm, so a workflow built from the table fails to deploy with Extension 'ai.labs.langchain' not found. The config-store URI in that row (eddi://ai.labs.llm/...) was already right; only the step type was stale. Corrected.
Not implemented (v1): /v1/embeddings, tool_calls passthrough (would cause double-execution — EDDI's tools do not exist in Open WebUI), n > 1, logprobs. Quick replies and inputField outputs are dropped; only text reaches OpenAI clients. (Superseded by the 2026-07-28 entry above — they are rendered as Markdown now.)
🎨 Keycloak login theme matching the EDDI corporate identity (2026-07-27)
Repo: EDDI (feat/keycloak-eddi-theme)
Users redirected from the Manager or Workforce UI to Keycloak went from EDDI's amber-on-near-black design to stock Keycloak blue-on-white. This adds an eddi login theme — no FreeMarker overrides — plus the wiring to activate and ship it. As finally shipped the theme is eleven resources: theme.properties, one stylesheet, two enhancement scripts, four Noto Sans subsets, the wordmark and the favicon. The sections below are in the order the work happened, so earlier ones describe smaller inventories.
What changed.
keycloak/themes/eddi/login/theme.properties—parent=keycloak.v2,styles=css/styles.css css/eddi-login.css,kcHtmlClass=login-pf pf-v5-theme-dark.keycloak/themes/eddi/login/resources/css/eddi-login.css— the palette, a block of PatternFly global-token overrides, and a short list of targeted rules (logo, page background, card border, button label, autofill)..../resources/img/{logo_eddi.png,favicon.ico}— byte-identical copies of the app's own assets.keycloak/eddi-realm.json—loginTheme,displayName,displayNameHtml.docker-compose.auth.yml— bind-mounts./keycloak/themes/eddiinto/opt/keycloak/themes/eddi.install.sh— fetches the theme resources alongside the realm JSON (only the realm JSON is fatal; see the atomicity note below), and setsloginThemethrough the Admin API after startup.planning/keycloak-eddi-theme.md— the full design, the verified-facts table, and the corrections below.
How it works. Three layers. kcHtmlClass adds PatternFly 5's built-in .pf-v5-theme-dark to <html>, which flips every PF component the login pages use — including ones we would never enumerate (tiles, data lists, helper text, panels). A :root block of --pf-v5-global--* overrides then recolours PF's dark defaults to EDDI's stone/amber palette. A handful of targeted rules cover what tokens cannot express. We own zero FreeMarker, so a Keycloak upgrade can only break this loudly.
Design decisions.
Pure CSS over template overrides. Every
.ftlwe do not ship is a file we do not re-diff on every upgrade. The logo is applied to#kc-header-wrapperwith accessible image replacement (text-indent: -9999px), keepingdisplayNameHtmlin the accessibility tree so screen readers still announce "EDDI".stylesmust name the parent stylesheet. Theme properties merge key-by-key with the child winning, sostylesreplaces rather than appends; omittingcss/styles.csssilently drops all base styling. It resolves up the inheritance chain, so we ship no copy.Prefer global-token overrides to component rules. Component rules are what rot on upgrade.
Login theme only. The account console, admin console and email templates remain stock; adding them later means sibling directories reusing the same palette block.
Three things the live run corrected — none were visible from source reading alone.
:where()specificity 0 does not mean:rootalways wins. PF's dark theme also sets component variables on component elements (:where(.pf-v5-theme-dark) .pf-v5-c-login { --pf-v5-c-login__main--BackgroundColor: … }). Custom properties resolve from the nearest declaring element, so that beats:rootregardless of specificity.The dark theme re-points components at a different tier of globals. The primary button reads
primary-color--300/--400, the cardBackgroundColor--300, form controlsBackgroundColor--400, the underlineBorderColor--400. Overriding only the--100tier left the Sign In button Keycloak blue (#06c) and the card the wrong grey. Both tiers are now set.Field-level error text reads
danger-color--200. Set to a red-700 (#b91c1c) it rendered at ~2.4:1 on the card — a WCAG failure on the most important message on a failed login. On a dark surface the higher tiers must get lighter, not darker; it is now red-400 at 6.40:1.
Verification. Against a running quay.io/keycloak/keycloak:26.0 on a fresh volume, reading computed styles rather than eyeballing. Realm import log clean. All assets 200, including the inherited css/styles.css and our favicon. Flows exercised: sign-in, failed login, update password (forced action), re-authenticate, logout confirmation, forgot password, error page. Contrast: button label 9.20:1, links 10.61:1, labels 6.91:1, error text 6.40:1 — all past AA. Mobile 375×812: no horizontal overflow. Full measurements in planning/keycloak-eddi-theme.md §6.1.
Operational note — existing installations. Realm import is one-shot: Keycloak skips realms that already exist, so editing eddi-realm.json does not reach an existing deployment. install.sh now applies loginTheme through the Admin API so upgrades pick it up; for local development the equivalent is docker compose -f docker-compose.yml -f docker-compose.auth.yml down -v (⚠️ destroys keycloak-data — all users and sessions).
Incidental finding, now fixed. "temporary": true on an imported credential does not create an UPDATE_PASSWORD required action in Keycloak 26 — the seeded users log straight through (requiredActions is empty after import). That made the comment at the top of docker-compose.auth.yml ("password change required on first login") wrong. The comment now says so explicitly, including why the "temporary": true flag is misleading, so the next reader does not "fix" the realm JSON in the wrong direction. The same block listed only two of the three seeded users and labelled them loosely; it now lists all three with their actual realm roles. Recorded as F19 in the plan.
A colour-scheme control, and the locale picker no longer looks like a stray field. Both came from looking at the rendered page.
The switcher exists now. The earlier decision to omit it was wrong for a reason I had not checked: the Manager and Workforce show a theme switcher on every page, so offering one here is consistent rather than duplicative. It is a quiet icon button beside the locale picker with the same three states as the rest of the product — system (default) / light / dark — persisted per origin. It deliberately does not sync with the Manager's setting: Keycloak is a different origin, so that
localStorageis unreadable from here. The control is a genuine enhancement — "system" removes the attribute so the CSS media query governs, which means the page follows the OS with no JavaScript at all, and the button only adds the ability to override.The locale picker was my fault. Keycloak renders it as a native
<select>wrapped in a.pf-v5-c-form-control, so the field styling hit it too and gave it the recess, border, radius and 38px height of the password input — sitting next to the H1, it read as a stray form field. It is now quiet at rest and reveals its affordance on hover or focus.A 16px misalignment put the picker past the right edge of the fields, close enough to look like a mistake. The cause is the header grid's tracks overflowing its padding box by exactly the column gap. Worth recording because two plausible fixes do not work —
minmax(0, 1fr)on the title track andmin-width: 0on the title were both verified against the live page and neither moved it. Zeroing the gap and moving the spacing onto the utilities does.
Also corrected: the claim that start-dev randomises the resource-path hash per boot is false — it survived a container restart. That matters beyond testing, because it means a theme edit does not change the CSS URL, so returning browsers can hold a stale stylesheet. The response does send Cache-Control: no-cache, so revalidation happens, but the "bump the version to bust the cache" assumption in the plan was wrong.
Light mode, all 30 locales, and two palette corrections. Three things came out of questioning earlier decisions.
Locales — the single-locale choice was wrong. Enabling internationalisation with
supportedLocales: ["en"](done purely to getlangon<html>without a switcher) silently discarded 29 translations the theme already had. Keycloak's server-info endpoint confirms theedditheme inherits 30 locales — ar, ca, cs, da, de, el, en, es, fa, fi, fr, hu, it, ja, ko, lt, lv, nl, no, pl, pt, pt-BR, ru, sk, sv, th, tr, uk, zh-CN, zh-TW — for free. All 30 are now offered; German renders aslang="de"with "Passwort vergessen?" / "Anmelden". The switcher Keycloak renders is a native<select aria-label="languages">wrapped in.pf-v5-c-form-control, so it inherits the field styling and is keyboard-accessible without extra work.Light mode via
prefers-color-scheme. The Manager's own default isdefaultTheme: "system", so a user on a light OS already gets a light Manager while the login page stayed hard dark. Following the system preference matches that default exactly. (At this point no toggle was added, on the reasoning that the Manager owned the explicit choice. That was revisited — see the colour-scheme control above — andprefers-color-schemeis now what the control's "system" state resolves to.) Values are the Manager's own light token block. Because every PatternFly token is declared asvar(--eddi-…), the light block is a palette swap plus the treatments that are inherently directional — a highlight lit from above, a recess, a shadow, and the bloom, which on a light page reads as a stain rather than as light and so drops to a faint wash. The white wordmark is handled withfilter: invert(1);invert()leaves alpha alone, so no second asset is needed.Two palette errors, found by reading the Manager's tokens properly.
#0c0a09is its--color-primary-foreground— the label on an amber fill — not its background; the dark background is#09090b. The page colour was wrong from the first commit. (The button label was coincidentally correct.) Destructive is#dc2626, not#ef4444; that is now used in light mode, while dark keeps a lighter red because#dc2626is only ~3.2:1 on our dark card.
Two contrast defects that only light mode exposed: amber-400 links and an amber-500 focus ring are 1.7:1 and 2.15:1 on white. Links and the ring now step down to amber-700 in light (5.02:1, clearing 4.5:1 for text and 3:1 for a focus indicator). Verified in both schemes: light gives title 21:1, field text 16:1, links 5.02:1, labels 4.8:1, button label 9.2:1; dark is unchanged.
Fonts gained the Latin-ext, Cyrillic and Greek subsets, because Latin-only would put fallback glyphs inside otherwise-branded words for Czech, Polish and Turkish. Measured rather than assumed: unicode-range normally defers those files, but the switcher lists all 30 languages under native names, so every range is in the DOM and all four are fetched (~245 KB). Accepted — font-display: swap keeps it off the paint path, it caches for the session, and the page already loads a ~1.5 MB PatternFly stylesheet.
Least-privilege default role — a privilege-escalation trap closed. eddi-realm.json composited default-roles-eddi to eddi-admin, eddi-editor and eddi-user, so any user created without explicit realm roles became an admin — which is exactly what self-registration produces. Nothing was exposed, because registrationAllowed: false, but enabling registration (a one-flag change that looks innocuous) would have silently granted admin to everyone who signed up. Found while verifying the seeded users' roles for the compose comment, and independently flagged in review afterwards.
The composite is now eddi-user alone, in both places it is declared. Verified on a fresh import, since editing the file does not touch a running realm:
default-roles-eddi→eddi-user, manage-account, offline_access, uma_authorization, view-profile— Keycloak's own built-ins survive, so refresh tokens and account access are unaffected.A user created with no roles →
eddi-userand the built-ins. No admin.The three seeded users are untouched:
eddikeeps admin+editor,viewerkeeps viewer,userkeeps user — they declarerealmRolesexplicitly, and Keycloak's import does not add the default role on top.
Existing deployments are detected, not migrated. Realm import is one-shot, so a realm created before this keeps the old composite. install.sh now checks it on every run and warns when the default role still grants eddi-admin/eddi-editor, printing the exact fix — but changes nothing. That asymmetry with the branding fields (which are forced) is deliberate: branding is cosmetic, whereas silently stripping elevated defaults during an upgrade could break a deployment that granted them on purpose. To apply it manually: Admin console → Realm roles → default-roles-eddi → Associated roles → remove them, or DELETE /admin/realms/eddi/roles-by-id/{id}/composites. Both branches of the check were verified against the running realm — silent once fixed, firing when the elevated roles are put back.
A duplicated token, and the bug hiding behind it. CodeRabbit flagged --eddi-chevron as declared twice in the light media query (a Stylelint error). It was worth looking at why rather than just deleting the second line: the patch that introduced the chevron had matched the same anchor twice, so it inserted both copies into the media query and none into :root[data-eddi-theme="light"]. The duplicate was the harmless symptom; the real defect was that choosing "light" explicitly from the toggle while the OS is dark left the chevron at the dark grey. Duplicate removed, missing declaration added, and all three paths checked: system follows the OS, explicit light #78716c, explicit dark #a1a1aa.
The header utilities looked accidental on narrow screens. Below PatternFly's header breakpoint the grid collapses to a single column and the utilities wrap onto their own row — which is fine — but the locale picker stretched to fill it: 268px at a 560px viewport against 82px on desktop, so the same control looked like two different things depending on width.
The fix is to keep both utilities content-sized. Worth recording why it needed a newer property: width: auto does not size a <select> to its selected option (it fills), and max-content sizes it to the widest option in the list — which here is a 30-language menu, so that is worse. field-sizing: content is the one that does the right thing. It is Chrome/Edge-only at present, so a max-width cap is the fallback: capped rather than full-bleed everywhere else.
Measured at 375, 560 and 1280: the picker is 76px at every width, the row stays right-aligned with the form fields, no horizontal overflow, and no dead selectors.
Locale picker restored on the admin console; both realms offer the full set.
The admin console had no language picker because I had given master a single locale — a deliberate trade to get lang on <html> without adding a switcher, but the wrong one once that console is EDDI-branded. Both realms now show the picker.
On which languages: they were briefly narrowed to the Manager's 14 (its bundle declares supportedLngs as cs, de, es, fr, it, ja, ko, pl, pt-br, ru, tr, zh-cn, zh-tw with fallbackLng: "en"), on a parity argument — a user who picks a language at login lands somewhere that speaks it. That was reverted on review: both realms offer all 30 locales Keycloak ships, deliberately a superset. The reasoning that won is that a localised login page is worth having even when the app behind it falls back to English; the alternative denied 16 free translations to speakers of ar, ca, da, el, fa, fi, hu, lt, lv, nl, no, pt, sk, sv, th and uk in order to avoid an inconsistency they would only notice after signing in.
All four font subsets are therefore shipped (~245 KB), Greek included.
RTL now verified, having previously only been assumed. The superset makes Arabic and Farsi reachable, and the wordmark uses text-indent: -9999px, which in a right-to-left document pushes text the other way. Checked in Arabic: <html lang="ar" dir="rtl">, no horizontal overflow — overflow: hidden on the header contains it — with the logo, the 30-option picker and the theme toggle all intact.
Admin console follow-ups — a logo collision, a missing toggle, and a shipped bug.
Two logos on top of each other. Keycloak's
masterrealm shipsdisplayNameHtmlas<div class="kc-logo-text"><span>Keycloak</span></div>, and that markup lands inside#kc-header-wrapperwhere we paint the EDDI wordmark — withkeycloak.v2's stylesheet still carrying the rule that gives that div the Keycloak logo. This corrects F5 in the plan:div.kc-logo-textis not dead CSS in general, only for realms whosedisplayNameHtmldoes not contain that markup. Fixed both ways — the theme now suppresses any logo box inside the header, andinstall.shsets master'sdisplayNameHtmlto plainEDDIso the header also has a correct accessible name.displayNameis deliberately left as "Keycloak", because it labels the realm in the admin console's realm selector.The theme toggle was missing there. Keycloak only renders the header-utilities container when it has a locale switcher to put in it, and master has a single locale — so the control had nowhere to attach and was silently absent.
eddi-theme.jsnow creates the container when it is not there.A bug this branch shipped in
97c7a0d0: the patch that added the master-realm block wrote a literal instead of a line continuation, so the jq path readjq '...'—bash -naccepts it, but at runtime jq would takenas its filter and the branch would fail on any machine that has jq. Repaired, and the whole file checked for further damage.
Keycloak manages pf-v5-theme-dark itself. The template ships an inline script that adds and removes that class from prefers-color-scheme, so kcHtmlClass only sets the initial state and the class is stripped on a light OS. Nothing depends on it: because the token block assigns both the --100 and the --300/--400 tiers, components render correctly either way — verified by forcing dark on a light OS and confirming page #09090b, card #18181b, amber button with near-black label.
Admin console login themed, and two picker defects.
The admin console is now branded too. It authenticates against Keycloak's
masterrealm, which is not part ofeddi-realm.json, so it kept the stock polygon-and-Keycloak-logo page while the EDDI realm was branded.install.shnow setsloginThemeonmasterthrough the Admin API — but only when master has no login theme of its own, so on a Keycloak shared with other products an operator's existing admin branding wins. It also enables internationalisation there with a single locale: master ships with i18n off, which meant nolang/diron a page we now own the appearance of (WCAG 3.1.1). One locale adds the attributes without adding a switcher, and loses nothing, since i18n-off was English-only anyway. Verified:<html class="login-pf pf-v5-theme-dark" lang="en" dir="ltr">with no locale picker.The locale picker had a stray underline. PatternFly draws the field underline with
::after; quietening the picker removed its background and border but left that line hanging under the label with no box around it.And its label overlapped the caret. PF's caret is not part of the
<select>— it is a 32px sibling grid item overlaying the field, and the select's width comes from the wrapper's grid rather than its content, so it does not grow for a longer label. "Deutsch" therefore ran 17px underneath the caret, and more right padding could not help because the caret is not in the select's box. Fixed by hiding PF's caret and drawing our own as a background inside the select's own padding: both are now in one box, so overflowing text clips at the content edge instead of painting over the chevron. Measured 5px clearance for "Deutsch"; the chevron colour follows the colour scheme via--eddi-chevron.
Accessibility and keyboard pass. Audited against the running page with real key events rather than assumptions.
Already correct, and left alone — Keycloak gets these right: tab order (username → password → show-password → forgot → Sign In, no tabindex anywhere), <label for> association on both fields, autocomplete="username"/current-password, autofocus on username (restored after a failed attempt), a single <h1>, a submit button inside the form so Enter submits from any field (verified with a real Return keypress), and an aria-live="polite" region that announces the error.
Three real defects fixed:
A focus-ring regression I had introduced.
overflow: hiddenon the input-group — added to clip the password field and its toggle into one control — also clipped the toggle's focus ring, because that ring uses a positiveoutline-offsetand the button sits flush against the group's edge. A keyboard user tabbing to "Show password" got a partially invisible indicator (WCAG 2.4.7). Rings inside the group now draw inward (outline-offset: -2px), verified fully visible.No
langon<html>— a WCAG 3.1.1 Level A failure. The template only emitslangwhen the realm has internationalisation enabled, which it did not. Fixed in realm config, not FreeMarker: enablinginternationalizationEnabledemitslanganddiron<html>. (This was first done with a single supported locale to avoid adding a switcher; that was the wrong trade and the realm now offers all 30 — see above.)Errors were not programmatically associated with their field. Keycloak renders
<span id="input-error-password">and setsaria-invalid="true", but neveraria-describedby, so a screen reader reaching the field announced "invalid entry" with no reason; the message was only spoken once, via the live region, at page load. Also, untouched fields carriedaria-invalid="", which is not valid ARIA and reads as true to some assistive tech.
That last fix required the theme's first JavaScript (resources/js/eddi-a11y.js, the accessibility helper, wired via scripts= in theme.properties; eddi-theme.js was added later for the colour-scheme control): CSS cannot set ARIA attributes. It is ~20 lines, adds no event handlers, reads no user input and makes no network calls. This is a deliberate departure from the "pure CSS, no FreeMarker" principle — the alternative was overriding templates, which is worse, or leaving the gap. keycloak.v2 sets no scripts of its own, so nothing is replaced; that needs re-checking on upgrade, since scripts replaces like styles does.
Not done, deliberately: arrow-key navigation between fields and an Esc handler. Neither is standard for a login form, both would fight the platform conventions assistive technology relies on, and adding them would mean shipping keyboard handlers to an auth page for no real gain.
Second design pass — after actually looking at the rendered page. Everything up to here was measured, never seen. Seeing it made the problem obvious: the page ignored its own logo's design language. The wordmark is hairline, widely tracked, geometric — an instrument face — while the form under it was heavy 700 labels, pure-black input voids and a flat slab button. The direction taken is a machined panel lit from a single thin source:
Signature — a hairline amber light-line across the card's top edge, brightest at centre and fading to nothing at both ends, with a matching ambient bloom behind the wordmark so the logo reads as the light source. One gesture expressed twice rather than several competing effects. It animates in once on load (620ms) and is suppressed under
prefers-reduced-motion.Depth. The flat page fill became a layered radial wash; the card gained a 1px inset top highlight, a falloff gradient and a real shadow, so it sits off the page instead of on it.
The password field seam. The field and its visibility toggle are siblings in a
pf-v5-c-input-group, each with its own box — they met in a hard seam with mismatched corners, the most obviously unfinished detail on the page. The group is now the field; its children go flat, so the pair reads as one control.Inputs moved from pure black (which pulled more attention than the button) to a translucent recess with an inset shadow, plus an amber focus glow.
Type now echoes the wordmark: title dropped to weight 300 with tracking, labels to 12px tracked caps. The title had to be set through
--pf-v5-c-title--m-3xl--FontWeight, since.pf-v5-c-title.pf-m-3xlcarries the weight at two-class specificity and beats a plain declaration.Deliberately not done: no gradient on the button. It is already the heaviest element; a gradient there would compete with the signature. The primary button did gain vertical padding, which incidentally takes it to a 44px touch target on mobile.
Design pass — the page was correct but not designed. Three fixes after looking at what the measurements implied rather than only whether they matched the palette:
Typography. The page rendered in PatternFly's Red Hat Text/Display — the one EDDI surface in a different typeface, and matching neither the Manager (Noto Sans) nor this plan's own stated
system-uiintent, because nofont-familywas ever set. Now ships the same variable font the Manager loads. At this point that was the Latin subset only — 35 KB for weights 100–900 — with other scripts falling through to the system stack; the Latin-ext, Cyrillic and Greek subsets were added later when the realm began offering all 30 locales (see above). Overriding--pf-v5-global--FontFamily--textalone is not enough — PF re-declares it asvar(--…--text--vf)when variable fonts are present, so the--vfpair must be set too.Inputs were invisible as inputs. Fixing the card background had left fields at the card's own
#18181b, separated from their container by a 1px hairline alone. Fields are now recessed to the page colour (#0c0a09), so they read as cut into the card rather than laid on it; the password-visibility toggle follows so the input group stays one control.Three corner radii on one card (8px card, 4px input, 3px button). Replaced with a deliberate two-step scale — 6px controls, 12px card — driven through
--pf-v5-global--BorderRadius--smso every PF control inherits it.
WCAG 1.4.11 — since closed. This was recorded as a known gap at the time: the resting input border (#27272a on #18181b) was 1.19:1 against a 3:1 requirement for UI component boundaries, and reaching it meant departing from the border colour the Manager uses throughout. It is now fixed with a dedicated --eddi-field-border token — zinc-500 on dark (3.67:1), stone-500 on light (4.8:1) — while the card keeps --eddi-border, because a panel edge is decoration and 1.4.11 does not apply to it. Fields consequently read more strongly than the Manager's do; that is the cost of the criterion.
Review pass — four things the first commit had not established.
Graceful degradation confirmed. Emptying the theme directory while
loginTheme: eddistays set — exactly what a failedinstall.shasset download produces — yields HTTP 200 with the built-in theme andERROR ... Failed to find LOGIN theme eddi, using built-in themesin the log. So the decision to make theme downloads non-fatal is safe provided the fallback is all-or-nothing: a missing theme cannot take authentication down. A partial one is a different matter —theme.propertieswould still resolve and Keycloak would render the eddi theme with its stylesheet or fonts 404ing — soinstall.shdiscards the whole theme directory if any resource fails rather than leaving a half-built one mounted.The verification protocol is not vacuous. Run against that deliberately broken state, both §6 step 2 (log grep) and step 3 (html-class and stylesheet assertions) fail as designed. Checks that cannot fail are worthless; these can.
The
install.shrealm update was executed, not just written. Against a realm reset tologinTheme: "", the GET-modify-PUT returns 204, sets all three fields, and is idempotent on a second run.A dead CSS selector removed.
.pf-v5-c-login__main-header-descmatches nothing and appears in nokeycloak.v2template — the same mistake (styling an element the theme never emits) that killed the originaldiv.kc-logo-textapproach, reproduced at low stakes. Found by auditing every selector in our stylesheet against the live DOM..pf-v5-c-login__main-footer-bandwas checked the same way and kept: it is emitted bylogin.ftland only hidden becauseregistrationAllowed: false. Also verified the keyboard focus ring —:focus-visibleneeds a real key event to match, and does then givesolid 2px #f59e0bat 8.25:1 on the card, past WCAG 2.2's 3:1 for non-text indicators.
🧹 fix(release): retire stale deployment records, quiet the Postgres health line, revive two dead checks (2026-07-27)
Repo: EDDI (fix/release-6.2-polish) + eddi-chat-ui (fix/release-6.2-polish)
Findings from a full smoke test of labsai/eddi:6.2.0-b638 against both datastore backends (MongoDB and PostgreSQL). Agent CRUD, multi-turn conversations, PropertySetter + Qute templating, undo/redo, group conversations (ROUND_TABLE, followup/continue/close), MCP handshake and SSE streaming all behaved identically on both — no engine defects found. The items below are the rough edges that surfaced.
1. Deleting an Agent orphaned its deployment record (both datastores)
RestAgentStore.deleteAgent cascaded to schedules, workflows and the capability registry but never to deployments, and IDeploymentStore had no delete method to call. Every deleted-but-once-deployed Agent therefore left a row at status deployed; on each startup the runtime retried the redeploy, failed with ResourceNotFoundException, and logged a full ERROR stack trace — forever, accumulating with each deletion. Reproduced on current code; this instance had an orphan dating to 2026-04-01.
AgentDeploymentManagement.checkDeployments already carried self-heal logic for exactly this case (isCausedByResourceNotFound → mark undeployed), but it was unreachable: AgentFactory.deployAgent returns void and swallows the ServiceException internally, so the catch blocks never fired. That is why the April orphan survived for months despite the code being there.
IDeploymentStorage/IDeploymentStore: addeddeleteDeploymentInfos(agentId), implemented inMongoDeploymentStorage(deleteMany) andPostgresDeploymentStorage(DELETE … WHERE AGENT_ID = ?)RestAgentStore.deleteAgent: clears the Agent's deployment records; a failure here logs a warning and still deletes the AgentAgentDeploymentManagement.checkDeployments: checks up front whether the Agent config still exists and retires the record instead of attempting a doomed deploy.isAgentConfigMissingtreats onlyResourceNotFoundExceptionas proof of absence — a store outage leaves the record untouched, so a transient DB failure can never mass-retire live deployments.
Three call sites delete Agents, not one. McpAdminTools.deleteAgent delegates to RestAgentStore and inherits the cascade, but GroupConversationService's ephemeral cleanup and TeardownAgentTool call agentStore.deleteAllPermanently directly — and dynamic sub-agents do get deployment records, because AgentSetupService.deployAndWait goes through RestAgentAdministration. Every ephemeral group agent was therefore orphaning a record. Both now retire their records too (null-tolerant, non-fatal), following the existing field-injection pattern in those classes so the directly-constructed unit tests keep working.
Ordering: the cascade runs after restVersionInfo.delete, not before. That method validates its arguments internally and throws on a stale or unknown version — clearing the records first would strip a still-live Agent of what it needs to come back up. Pinned by a test.
Design note: the pre-check was chosen over making deployAgent rethrow — that method's dummy-agent-on-failure contract is relied on by the on-demand deploy path, and widening it for this would have been a far riskier change than a cheap existence check.
Verified against the real Postgres instance: the April orphan was retired, the ERROR + stack trace was replaced by a single WARN … retiring stale deployment record, and deleting a fresh Agent logged Cascade-deleted 1 deployment record(s) with the row gone.
That run was observed against the first implementation, which retired a record by marking it
undeployed— so the orphan was seen flippingdeployed→undeployed. The review pass below replaced that with a scoped delete, so the record is now simply absent. The ERROR-to-WARN result and the cascade behaviour are unchanged.
2. MongoDB health check reported UP with no MongoDB (postgres profile)
Under QUARKUS_PROFILE=postgres, with the Mongo container stopped and its hostname unresolvable, /q/health still listed "MongoDB connection health check": "UP" — noise that would equally mask a genuine outage in Mongo mode.
The obvious %postgres.quarkus.mongodb.health.enabled=false does not work: that property is fixed at build time, so it cannot apply to one image pointed at either datastore (confirmed — even passing it as a runtime env var leaves the check in place). Disabled through smallrye-health instead, which is runtime-scoped:
3. Two checks that could never fail
InfrastructureIT.openApiSpecassertedanyOf(200, 404)against/q/openapi— a path that never serves the spec (quarkus.smallrye-openapi.path=/openapi). It passed whether or not OpenAPI worked. Now asserts200on/openapiplus real body content.AGENTS.mddocumented EDDI-Manager as served at/chat/production. It is served at/manage;/redirects to the/welcomechooser and/workforceis the group workspace./chat/productionis the standalone chat widget. The wrong path in this file is what sent an earlier review to the wrong URL.
4. eddi-chat-ui: dead-end landing and an agent name that never loaded
main.tsxrouted its catch-all to a hard-coded/chat/production/default. No instance has an agent literally nameddefault, so this 404'd and left the visitor on a spinner that never resolved. Replaced with a newAgentPickerthat lists the instance's actual agents, with explicit empty and error states. Reached only by deep-linking/chat/productionwithout an agent id — the Manager's own chat (/manage/chat) was never affected.fetchAgentDescriptorcalledGET /agentstore/agents/{id}with noversion, which answers400. Even fixed, that endpoint returns the Agent config and carries no name at all — so the agent name had never rendered. Now resolves the current version and reads/descriptorstore/descriptors/{id}?version=N.
Note: eddi-chat-ui builds its bundle directly into EDDI/src/main/resources/META-INF/resources/, so the two repos ship together. The superseded chat-ui.p4wYUapg.js / chat-ui.D213XXZR.css were removed; chat.html now points at the new hashes.
Review pass on PR #611
CodeQL log injection (RestAgentStore).
idis a raw REST path parameter and reached twolog.infof/warnfcalls unsanitized. Now routed throughLogSanitizer.sanitize, the pattern already used across the config REST stores. The equivalent logs inTeardownAgentToolandGroupConversationServicewere left raw on purpose: both only reach them after the id has passed acreatedAgentIds.contains(...)check, so the value is a server-generated agent id, and every neighbouring log line in those classes prints it the same way.Retire scoped, not agent-wide (CodeRabbit, Major). The sweep proves only that one
(environment, agentId, version)is missing, so retiring with an agent-widedeleteDeploymentInfos(agentId)acted far beyond its evidence — it would take out sibling records for versions nobody checked. AddeddeleteDeploymentInfo(environment, agentId, version)and use it here; the agent-wide variant stays for the delete cascade, where the whole Agent really is gone. In practice the harmful case is hard to construct —HistorizedResourceStore.readfalls back to history, so an older version normally still resolves — but the mismatch between what is checked and what is acted on is the kind that bites once versioning semantics shift. Covered by a mixed-version regression test.Pin the sibling in the mixed-version test. The regression test asserted the missing version's record was deleted and that no agent-wide delete happened — but it would still have passed if the code had also removed the live sibling through the scoped API. Added the explicit negative (
never()on version 2, in any environment) andtimes(1)on the deletion that should happen. Mutation-checked: injecting a sibling delete now fails the test, where before it slipped through. While there, thenever()verifications in these classes moved fromanyString()/anyInt()toany(), which also matches null —anyString()does not, so a null-argument regression would have verified vacuously.Retire by delete, not by upsert.
setDeploymentInfoupserts. If an Agent were deleted between the sweep reading the deployment list and reaching the retire branch, marking itundeployedwould resurrect the row the delete cascade had just removed — an inert but permanent tombstone. The sweep now callsdeleteDeploymentInfos, which is idempotent, and an Agent that no longer exists has nothing to undeploy.TOCTOU between the pre-check and the deploy (CodeRabbit, Major) — acknowledged, not fixed. If an Agent is deleted in the window between
isAgentConfigMissinganddeployAgent, the record can staydeployedand be cached indeploymentInfosuntil the next restart. The suggested remedy is to makedeployAgentsurface a definitive missing-result — the contract change deliberately avoided above, since the dummy-agent-on-failure behaviour is relied on by the on-demand deploy path. The window is milliseconds, the consequence is one stale row, and it self-corrects on restart, so it is not worth widening a core contract on a release-polish branch.CodeRabbit nitpicks, all taken:
.gitattributesmarks the chat-ui bundles and Vite assetslinguist-generatedso reviewers stop being handed minified output;MongoDeploymentStoragegains anagentId-leading index, since the existing compound index starts withdeploymentStatusand cannot serve the new agentId-only delete; and two coverage gaps closed — deployment cleanup on the cascade-success path, and the guarantee that a failed permanent delete leaves the records alone.Style nits from Copilot: imports instead of inline
java.util.concurrent.*FQNs inTeardownAgentTool(per §4.7), and four comments the Eclipse formatter had reflowed into orphaned fragments (// throws,// path,// it,// up.). Comment lines are now short enough to surviveformatter:format.
Not fixed here (deployment config, not the release artifact)
The stored Anthropic API key is invalid — every LLM-backed agent fails with
invalid x-api-key. EDDI handles it correctly (non-retryable, conversationERROR, no stack trace to the client), but any demo needs a working key.VaultSaltManagerreports existing DEKs with no per-deployment salt and is falling back to the legacy fixed salt; the KEK migration should be run.
🔒 fix(ci): remove accidentally-committed langchain4j-mcp decompiled sources (2026-07-27)
Repo: EDDI (feat/v6.2.0-prep)
Both the Dependency Review and Trivy Filesystem Scan CI checks were failing on this branch with 3 HIGH-severity Jackson CVEs (GHSA-r7wm-3cxj-wff9, CVE-2026-54512, CVE-2026-54513).
Root cause: commit 702e2f79b (fix: serialize SSE log events as JSON instead of toString(), 2026-07-24) accidentally committed a decompiled copy of the langchain4j-mcp jar into a new top-level lc4jmcp/ directory (123 files) — leftover from decompiling the jar to debug the toString() logging issue, swept up by a broad git add. The directory sat outside src/main/java, was never referenced by pom.xml or any source file, and Maven never compiled it — but its embedded META-INF/maven/dev.langchain4j/langchain4j-mcp/pom.xml (the jar's own build-time manifest) declared jackson-databind:2.21.3 / jackson-core:2.21.3, which both scanners picked up as if it were a real dependency manifest.
Verification: ./mvnw dependency:tree -Dincludes=com.fasterxml.jackson.core confirms the actual resolved build uses jackson-databind:2.22.0 / jackson-core:2.22.0 (patched, via quarkus-jackson:3.37.4 and langchain4j:1.18.0) — this was never a real vulnerability in the shipped app, just dead decompiled code confusing the scanners.
Fix: git rm -r lc4jmcp — no source or config referenced it, so removal is inert.
🧪 test(ui): close entity streams and assert real content in welcome/workforce tests (2026-07-26)
Repo: EDDI (feat/v6.2.0-prep)
Copilot review feedback on RestWelcomeResourceTest / RestWorkforceResourceTest: the tests left the returned InputStream entity open and never read it, despite a display name promising a "readable" entity.
Changes (test-only, no production code touched):
Added a private
readEntity(Response)helper to both test classes — asserts a non-nullInputStreamentity, reads it fully inside try-with-resources (closing the classpath stream), returns UTF-8 contentviewHtmlReturnsOkWithEntitynow asserts the body contains<html, matching its display name instead of only checking the entity typeviewDefaultAndViewHtmlServeSameShellnow compares the two bodies for equality — previously it only asserted both entities were non-null, which never actually proved both endpoints serve the same shell
Design decision: assert on <html> rather than any Vite-generated markup. welcome.html / workforce.html are SPA shells regenerated on every Manager/chat-UI asset update, so hashed asset filenames or inline styles would make content-specific assertions stale on the next chore: update Manager UI assets commit.
Files: RestWelcomeResourceTest.java, RestWorkforceResourceTest.java (6 tests, all green)
Noted, not fixed: RestManagerResourceTest wraps several calls in try { … } catch (Exception e) { /* expected */ }, which passes regardless of whether the code under test behaves correctly. Out of scope for this review fixup — worth tightening separately.
🐛 fix(memory): orphaned ConversationDescriptors on conversation deletion (2026-07-24)
Repo: EDDI (feat/v6.2.0-prep)
ConversationDescriptors (created at conversation start by ConversationSetup) were never cleaned up when a conversation was permanently deleted, causing "Memory snapshot not found — Descriptor is orphaned" warnings on every conversation listing.
Root cause: ConversationDescriptorStore.deleteAllDescriptor() existed but was never called in any deletion path. Three separate hard-delete paths all deleted the memory snapshot but left the descriptor behind.
Fixes (all in production code):
RestConversationStore.deleteConversationLog — added
conversationDescriptorStore.deleteAllDescriptor()after permanent snapshot deletionRestConversationStore.permanentlyDeleteEndedConversationLogs — added descriptor cleanup in both the happy path (old enough to delete) and the catch path (orphaned snapshot without DocumentDescriptor)
GdprComplianceService.deleteUserData — added
IConversationDescriptorStoredependency, hoisted conversation ID resolution before step 2 (eliminating a duplicate DB query), added step 4a to delete descriptors per-conversation before the bulk snapshot delete
Files:
RestConversationStore.java— 3deleteAllDescriptorcalls addedGdprComplianceService.java— new dependency, hoisted ID resolution, new step 4a, updated JavadocRestConversationStoreTest.java— 9 verify assertions added (positive + negative)GdprComplianceServiceTest.java— mock added, constructors updated,inOrderverification for deletion ordering
Design decision: soft-delete (deletePermanently=false) does NOT touch the descriptor — the DocumentDescriptorFilter JAX-RS interceptor still handles soft-deletion via its existing mechanism. Only hard-deletes clean up the ConversationDescriptor.
🐛 fix(llm,group): empty task results, maxTokens defaults, verification display (2026-07-24)
Repo: EDDI (feat/v6.2.0-prep)
Group conversations with Anthropic thinking models (e.g. claude-sonnet-5) produced empty task results and raw JSON in the UI. Root cause: langchain4j's default maxTokens=1024 was consumed entirely by thinking tokens, leaving nothing for the response text.
Bug fixes:
AnthropicLanguageModelBuilder — added
maxTokens,topP,topKsupport; defaultmaxTokens=16384OpenAILanguageModelBuilder — added
maxTokenssupport (was missing)LlmTask — null/blank response skips output creation (no empty bubbles)
GroupConversationService — verification JSON → human-readable ✅/❌ summaries
ConversationLogGenerator — null-safety on
KEY_TEXTentries
Documentation: docs/langchain.md — Output Token Limits section, provider matrix, updated examples
Tests: Anthropic maxTokens, blank output guard, verification formatting, null text filtering
📝 README: add HITL section, JSON mode, tool context ceiling (2026-07-23)
Repo: EDDI (feat/v6.2.0-prep)
The README's Features section was missing Human-in-the-Loop Governance entirely — despite HITL being a Phase 9b completed feature with a 479-line dedicated doc (docs/hitl.md). This was the biggest gap found after a systematic comparison of the README against AGENTS.md completed features, docs/changelog.md, and all documentation files.
Changes to README.md:
Added
### 🛑 Human-in-the-Loop Governancesection (8 bullets) between Smart Model Cascading and Enterprise Security — covers turn-level approval, per-tool-call gating, group phase approval, timeout policies, no-progress guard, Slack/MCP approval surfaces, and crash recoveryAdded HITL link to Documentation table —
docs/hitl.mdwas not linkedAdded JSON Response Mode to LLM Provider Support —
jsonResponseFormatpolicy (auto/on/off) with provider-aware negotiationAdded Tool Context Ceiling to Memory & Context Management —
maxToolContextTokens(default 60k) prevents provider context-window errors from tool loopsAdded Tool Calling and Multi-Model Cascading cross-reference under LLM Provider Support
What was deliberately NOT added (internal hardening, not README material):
Constant-time HMAC, versioned audit HMAC v2, tool-cache scope fail-safe, HTTP logging guard, SSE secret redaction, budget enforcement warnings, stream resilience — these belong in release notes
Provider count stays at 12 — verified against
docs/langchain.md(the authoritative source per §2 rule 7); Vertex AI is a sub-type of Gemini, not a separate providerdocs/template-preview.mdlink not added — file doesn't exist yet
🧵 Model registry: a rotation landing mid-build could re-cache a stale model (2026-07-23)
Repo: EDDI (fix/chatmodel-stale-rebuild-race)
CodeRabbit flagged this on the fix/backlog-defect-remediation PR (Major / "Heavy lift") and it was deliberately deferred from that branch. It is separate from the C1 check-then-act race fixed there (commit 96eacacde, which stopped a concurrent clear() turning a cache hit into a null return).
The defect. getOrCreate/getOrCreateStreaming resolve global-variable and vault-secret values, build a model from them, and then put it into the cache — all unsynchronised. A secret rotation (invalidateForSecret → clear()/evictMatching) or a global-variable edit (the invalidation listener → clear()) can land between the resolve and the put. The clear() finds nothing (the entry does not exist yet), and the in-flight build then puts a model constructed from the now-stale secret. Because the entry is present and valid-looking, every later turn keeps reusing that pre-rotation model until some unrelated invalidation happens to evict it — i.e. a rotation could silently fail to take effect.
The fix — an invalidation generation. A process-wide AtomicLong invalidationGeneration is bumped by every invalidation (invalidateForSecret for both the bulk-clear and targeted-evictMatching paths, and the global-variable listener). A build snapshots the generation before it resolves any value, and publishes through publishIfCurrent, which caches the model only if the generation is unchanged; otherwise it discards it (the racing caller still receives the instance for its own turn — a one-turn window inherent to lock-free building — but the next lookup misses and rebuilds from current values). The re-check and the put are held under a new publishLock that the invalidation paths also hold around their bump-and-clear, so "re-check then put" and "bump then clear" cannot interleave — without that lock the re-check would itself be a check-then-act with the same hole.
Design decisions:
Do-not-cache rather than rebuild-and-retry. CodeRabbit suggested "discard and rebuild (or at least do not cache)". Not caching fully closes the reported bug (persistent staleness) with no unbounded-retry surface under an invalidation storm; the residual one-turn exposure of the racing caller is fundamental to building without holding a lock across the whole (potentially slow) provider build.
The generation is a coarse, registry-wide signal. An invalidation for an unrelated secret still bumps it, so a concurrent build may be rebuilt needlessly — wasteful but always correct, and far simpler than trying to match the in-flight build against the specific rotated reference. Invalidations are rare.
publishLockscope is minimal. Builds (the expensive part) stay fully concurrent; the lock covers only the cheap re-check+putand the clear/evict. Both resolvers fire their listeners outside any lock, so takingpublishLockin the callbacks introduces no lock-ordering/deadlock risk, and no path holdspublishLockwhile calling back into a resolver.The counter is registry state, not conversation state. The class stays
@ApplicationScopedand per-conversation-stateless; the generation and lock are process-global memoization-cache concerns, which is the correct home for them.
Tests: ChatModelRegistryTest gains a nested StaleRebuildDuringInvalidationTests (C5) with a sync and a streaming case. Each installs a builder that fires invalidateForSecret(null) from inside build()/buildStreaming() — deterministically the interleaving "clear lands between build-start and publish" — then asserts the racing model is not served to the next caller (it rebuilds) and that caching resumes normally afterwards, with an exact build-count assertion. Mutation-checked twice: against the pre-fix code both tests fail with expected: not same but was: <same instance>, and reverting only the generation guard in publishIfCurrent (unconditional put) fails exactly those two tests and no others.
🧪 De-vacuum three tests and fix a doc/fixture drift (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
CodeRabbit and Copilot flagged a cluster of tests that passed for the wrong reason, plus two documentation/fixture mismatches. Each behavioural change is mutation-checked.
MongoTenantQuotaStoreTest.everyUpsertIsKeyedOnTenantIdOnlywas vacuous. It verifiedfindOneAndUpdate/updateOnewithatLeast(0), which never fails, and its filter loop asserted nothing when nothing was captured. NowatLeastOnce()plus an explicitupsertsChecked > 0guard. Mutation: removing the store exercise fails withWanted but not invoked; the old form passed.CacheFactoryTest.negativeLifespanIsUnlimitedread immediately — a negative-lifespan-mapped-to-short-positive-TTL bug would slip through. Now sleeps pastPAST_TTL_MILLISand asserts survival. Mutation: mapping negative → 1ms fails withexpected: <value1> but was: <null>.CacheFactoryTest.nonceCacheCapacityCoversItsTtlused whole seconds, so a capacity derived fromttl.toSeconds()(truncating) instead oftoMillis()would pass. Now uses a fractional390_500msTTL and millisecond-precision occupancy. Mutation: revertingmaximumSizeFortotoSeconds()fails withexpected: <234300> but was: <234000>.LifecycleManagerTestaudit fixture used{input, output}forllmDetail.tokenUsage, a shape production never emits —buildAuditEntrycopiesaudit:token_usagestraight through, and that key's contract is{inputTokens, outputTokens, totalTokens}. Fixture and assertion aligned to the real contract, and the inlinejava.util.Map/ConsumerFQNs replaced with imports (AGENTS.md §4.7).This changelog's D5 cache-key example still showed
arguments.length()directly afterbuildKeygained a null-coalescing guard; the formula now showsargs = arguments == null ? "" : argumentsfirst.
🐛 ToolNameResolver.canonical could return null, contradicting its own contract (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Copilot flagged that canonical(dispatchName, canonicalNames) used getOrDefault, which returns a stored null when a key maps to null — so the method could hand back null even though its javadoc promises "callers never need a null check". A null slug would then flow into the price and TTL lookups. The tell was in the test itself: nullValuedMappingDegrades was named for degrading to the dispatch name but asserted assertNull, documenting the bug as the contract and leaning on ToolInvocation to normalise it downstream.
Fixed to treat a null-valued key the same as an absent one (get + null fallback), and the test now asserts what its name always said — the result is the dispatch name, never null. Mutation-checked: reverting to getOrDefault fails the test with expected: not <null>.
The map is built by AgentOrchestrator.buildToolSetup, which substitutes the dispatch name and never stores nulls, so this was defence-in-depth rather than a live crash — but a method whose javadoc, code and test display-name all disagreed is exactly the silent inconsistency this branch set out to remove.
🔒 Redact secrets from the tool trace before it reaches the SSE stream (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
The D3 fix earlier on this branch made the tool trace reach the task_complete SSE frame — the first time tool arguments and results leave the process on a channel with no redaction of its own. Those payloads are LLM- and user-controlled, so they can carry API keys or bearer tokens. Copilot flagged it HIGH on the PR; the tradeoff had been disclosed in the PR description rather than fixed.
Now fixed at the producer. LifecycleManager.buildTaskSummary deep-redacts the collected trace through SecretRedactionFilter — the same filter the audit ledger already applies via AuditLedgerService.scrubSecrets — before putting it on the summary, bringing the live-display path to parity with the audit path. The redaction walks maps and lists recursively and scrubs every string leaf, and it operates on a fresh copy: the trace stored in conversation memory is left intact for the owner-scoped RestToolHistory endpoint. Only the summary's toolTrace is touched; buildAuditEntry reads its own audit:tool_calls key (separately scrubbed on submit), so the audit path is unchanged and not double-scrubbed.
Regression test LifecycleManagerTest.summaryRedactsSecretsInToolTrace feeds an sk-… key and a bearer token through the trace and asserts neither reaches the summary while the non-secret structure (type, tool) survives. Mutation-checked: bypassing the redaction call fails with the raw key visible in the captured payload.
📝 Budget-enforcement docs corrected to match the shipped opt-in default (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
CodeRabbit caught that three places still described enforceBudget as opt-out, default true after the flag was reverted to opt-in (false) earlier on this branch — a config knob whose documentation stated the opposite of its behaviour, which is precisely the defect class this branch exists to remove, and it contradicted the langchain.md table row in the same file. Corrected:
LlmConfiguration.Task#enforceBudgetjavadoc — "opt-out … itselftrue" → opt-in, defaultfalse, with the WARN rationale.docs/security.md(Cost Tracking config line and Behaviour bullet) — "defaulttrue" / "enforced by default" → opt-in.docs/langchain.md(Budgets prose) — "Enforcement is opt-out … defaulttrue" → opt-in.
The code was already correct (BUDGET_ENFORCE_DEFAULT resolves false); only the prose lied. Also documented that toolPricing accepts a dispatch name as well as a slug (dispatch name wins), and fixed two markdownlint nits (MD028 blockquote continuation in langchain.md, an unlabelled fence in this file).
🧹 A dead resumeToolLoop parameter, and a CodeQL note on a deliberately unguarded test parse (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Cluster dead-param-and-test-nit: one github-code-quality dead-parameter finding and one CodeQL note on a test helper. Both traced to source before touching anything — the first is real and removed, the second is a false positive left unchanged (with one clarifying javadoc line).
A — resumeToolLoop's templateDataObjects parameter was dead (real, removed)
Both AgentOrchestrator#resumeToolLoop overloads (the 7-arg delegator and the 8-arg policy-carrying body) declared Map<String, Object> templateDataObjects, and the javadoc claimed it fed "post-response and fallback rebuild". Reading both bodies end to end, it is referenced nowhere: the 7-arg overload only threads it into the 8-arg, the 8-arg never touches it, and the fallback path it named (fallbackRebuildMessages(task, memory, batch)) does not take it. LlmTask.executeResume still uses its own local templateDataObjects (template render + response-metadata put) — only the pass-through into resumeToolLoop was inert. Exactly the branch's theme: a parameter that looks load-bearing and is not.
Fix: dropped the parameter from both overloads, the delegating call, the @param/@link javadoc, the sole production caller (LlmTask), and every test call site. Because dropping an argument silently invalidates Mockito matcher lists (an unmatched stub returns null; a verify(never()) goes vacuous — both stay green, both would be wrong), every stub/verify was re-checked: the real 7-arg calls in AgentOrchestratorResumeToolLoopTest/AgentOrchestratorCoverageTest shed their Map.of(); the 8-matcher any() lists and the anyMap() verifies in the LlmTask* tests each lost exactly the sixth matcher, keeping arity aligned to the new signature. For a behaviour-preserving removal the verification is the compile plus the suites: test-compile clean, and 178 tests across the 8 named classes green (AgentOrchestratorResumeToolLoopTest 11, AgentOrchestratorToolPauseTest 8, AgentOrchestratorCoverageTest 59, LlmTaskResumeModeTest 9, LlmTaskCoverageTest 43, LlmTaskAgentModeMetadataTest 7, LlmTaskCoverage2Test 29, LlmTaskAuditLedgerTest 12). The two verify(never()) sites stay non-vacuous — in LlmTaskResumeModeTest the sibling positive verify binds the same 7-arg overload and passes.
B — CodeQL "missing catch of NumberFormatException" in a test helper (false positive, one clarifying javadoc line)
ChatModelRegistryTest.parseTimeoutLikeARealProvider does Duration.ofMillis(Long.parseLong(parameters.get("timeout"))) under an un-trimming isNullOrEmpty guard, and CodeQL wants a try/catch rethrowing IllegalArgumentException. The helper's whole job is to reproduce, verbatim, what the shipped provider builders do — every one of OpenAI/Azure/Bedrock/Ollama/HuggingFace/Gemini/Anthropic/Mistral/…LanguageModelBuilder runs builder.timeout(Duration.ofMillis(Long.parseLong(parameters.get(KEY_TIMEOUT)))) with no guard. The C3 TimeoutNormalisationTests depend on that fidelity: they prove the registry's normalizeTimeout drops blank/non-numeric/zero values before a builder ever sees them. Wrapping the helper's parse in a try/catch would make it tolerate values the real builders reject, so a normalization regression would pass silently instead of surfacing as the build()-time crash the tests guard.
Demonstrated: neutralize normalizeTimeout (early return) and the C3 tests fail with NumberFormatException thrown straight from the helper — "30s", " ", "not-a-number", " 5000 " (2 failures / 4 errors); restore and ChatModelRegistryTest is 41 green. So the finding is a false positive and production/test behaviour is untouched. Rather than a bare suppression, the helper's existing javadoc gained one sentence stating the unguarded parse is deliberate and must stay so, naming the try/catch note as a false positive — making the intent unmistakable to the next reader without weakening the test.
🔐 Constant-time audit HMAC comparison, and a retry-cost finding that wasn't (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Cluster retry-cost-and-hmac: one CodeRabbit correctness claim and one CodeRabbit security nitpick. Both were traced end to end before touching anything — one is real and fixed, the other is a false positive left unchanged with the evidence recorded here.
B — AuditHmac.verifyHmac compared digests with String.equals (real, fixed)
The compliance ledger's HMAC verification did computeHmac(entry).equals(stored) (and the v1 branch likewise). String.equals returns on the first differing character, so its running time leaks how long a prefix of a forged HMAC is correct — the classic side channel that lets an attacker reconstruct a valid tag one hex digit at a time against an endpoint that reveals verify latency.
Fix: decode both the recomputed and the stored digest from hex and compare the raw bytes with java.security.MessageDigest.isEqual, which is data-independent. The version selection is unchanged and deliberately kept non-constant-time: the v2: prefix only picks the canonicalizer and is not secret (per the frozen-v1 design, a v2-tagged value is never retried against v1). A stored value that is not valid hex — truncated, mangled, never a digest — now decodes to null and is rejected rather than throwing out of a whole-ledger verification sweep.
Because a constant-time swap is behaviour-preserving, no unit test can observe the timing difference; the added tests instead guard that the refactor did not break verification. AuditHmacTest:
bothVersionsVerifyAndBothRejectTampering— a bare-hex v1 row and av2:row both verify, and either one tampered is rejected.malformedStoredHmacIsRejected— empty / non-hex / odd-length / truncated /v2:-only stored values are rejected without throwing.v1DigestUnderV2TagDoesNotVerify— a v1 digest mislabelled with thev2:tag fails, proving the version tag is never retried against the other canonicalizer.
Mutation-checked two ways. (1) Revert the comparison to String.equals: all 29 tests still pass — expected, and the point, since the fix must not change verification behaviour. (2) Point the v1 legacy branch at the v2 canonicalizer: bothVersionsVerifyAndBothRejectTampering and the pre-existing legacyV1EntryStillVerifies both fail (TooManyActualInvocations-free, a plain assertion failure that a pre-v2 ledger row no longer verifies), proving the regression tests guard the version selection the fix preserves. Restored: AuditHmacTest 28, and the rest of engine.audit (AuditLedgerServiceTest, …BranchTest, …ExtendedTest, AuditRetentionConfigTest, AuditStoreTest) 61 — all green.
A — "tool charges are not rolled back when a retry replays" (false positive, unchanged)
CodeRabbit (Major): the tool loop runs inside AgentExecutionHelper.executeWithRetry(() -> {…}); the lambda already resets tokenHolder[0] = null because a retry replays it and would double-count tokens; the same replay re-runs the tool calls, each of which charges an @ApplicationScoped per-conversation counter with no per-attempt undo, so an abandoned attempt's spend allegedly stays counted and inflates toolCostUsd, the audit cost and isWithinBudget.
Traced against source, the premise does not hold:
Default config (
enableToolCachingtrue): the replay is free.ToolExecutionService.executeToolWrappedreturns at the cache-hit branch (step 2) beforecostTracker.trackToolCall(step 5). A replayed attempt re-issues the identical(name, arguments)call, hits the cache the first attempt populated, and is charged nothing. So the very configuration CodeRabbit assumes produces no double-count at all.enableToolCachingfalse: the replay genuinely re-executes — and the re-charge is correct. With no cache, the abandoned attempt's tool really ran (a real search-provider API call, a real HTTP POST), really cost money, and the successful attempt's replay runs it again, a second real call.ToolCostTrackeris the authoritative real-spend ledger behindmaxBudgetPerConversationand the audit dollar figure; billing two real executions is right, not "inflation". Rolling it back would under-count real spend and let an agent evade its budget by inducing retries.The token reset is not analogous.
tokenHolderis a per-ExecutionResultreporting accumulator whose number should match the final coherent attempt;ToolCostTrackeris a cumulative expenditure ledger. Different jobs, correctly treated differently.
So there is no phantom charge to roll back: every charge corresponds to a tool execution that really happened and really cost money. Production code left untouched. A characterization test pins the property that makes the finding moot — AgentOrchestratorToolCostTest.retriedLoopChargesReplayedToolCallOnce: attempt 1 executes and is charged for searchWeb, the next model call fails retryably, the whole lambda replays, and with a real ToolCacheService the replayed identical call is a cache hit — searchWeb runs once, the conversation is charged SEARCH_PRICE once, and toolCostUsd reports exactly that one charge. Mutation-check: swap the real cache for the always-miss stub (the enableToolCaching: false model) and the tool runs twice (verify(times(1)) fails with TooManyActualInvocations … But was 2 times), demonstrating that off-cache it is real re-execution, not double-accounting. Restored: AgentOrchestratorToolCostTest 10, green.
🗄️ Two cache keys that threw away information (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Two Copilot findings on the caching layer, both verified against source before touching anything. Both are the branch's theme one level down: not a control that does nothing, but a key that silently loses the distinction it was built to preserve.
A — ToolCacheService.buildKey dereferenced tool arguments that can legitimately be null
buildKey(scopeTag, toolName, arguments) called arguments.length() with no guard. Copilot claimed ToolExecutionRequest can be built without arguments; that is only half the question, so the live path was traced end to end:
dev.langchain4j.agent.tool.ToolExecutionRequest(1.18.0) declaresprivate final String argumentswith no default and no validation —arguments()returns null whenever the builder never set it.OpenAiUtils.toolExecutionRequestpasses the wire value straight through:.arguments(functionCall.arguments()), andFunctionCall.argumentsis a plain Jackson-deserialised field. A provider that omitsfunction.argumentsfor a zero-argument call — routine on OpenAI-compatible endpoints — yields a null. (The newer Responses API path defaults it to"{}"viaasText("{}"); the classic chat-completions path does not.)AgentOrchestratorline 1614 forwardstoolRequest.arguments()intoexecuteToolWrappedwith no guard. The only two coalescing sites on the branch (normalizeToolCallIds,rebuiltRequest) are on the HITL pause/resume path and never run for a live call.
The NPE was raised inside executeToolWrapped's try block, so it never surfaced as a crash — it came back to the model as Error executing tool: Cannot invoke "String.length()" because "arguments" is null, before the tool ran. Net effect: a zero-argument tool call failed with caching on (the default) and succeeded with caching off.
Fix: buildKey coalesces null to "". A null and an empty argument string both mean "no arguments" and deliberately share one entry.
B — the TTL cache key truncated to whole seconds
CacheFactory.getCache(name, ttl) keyed instances on name + ":ttl=" + ttl.toSeconds(). toSeconds() truncates, so every sub-second TTL collapsed onto ":ttl=0" and any two TTLs sharing a whole-second part collapsed together. Two callers asking one cache name for two TTLs then shared one Caffeine instance whose expiry policy was whichever of them built it first; the other TTL was accepted and silently ignored. The same method already derived its capacity from ttl.toMillis() in maximumSizeFor, so the key was strictly coarser than the sizing it keyed.
Reachability, stated honestly: no production caller triggers it today. Each of the four TTL call sites (NonceCacheService, ChannelTargetRouter, and two in SlackEventHandler) requests its cache name exactly once, from a single @PostConstruct. This is a latent defect in a shared factory, not a live incident — but it is one config change away (NonceCacheService's TTL is Duration.ofMillis(maxAgeMs + clockSkewMs + buffer) over two operator-settable millisecond properties) and the fix is a single token.
Fix: the key renders ttl.toString(). It is injective over distinct Durations, identical for equal ones — ofSeconds(60) and ofMinutes(1) still share, as they should — and strictly finer-grained than the toMillis() the capacity is derived from, so one key can never span two capacities.
Tests
ToolExecutionServiceTest.NullArgumentsTests(3 tests) wires the realToolCacheServiceover a realCacheFactory— a mocked cache service cannot reachbuildKey— and drivesexecuteToolWrappedwithToolExecutionRequest.builder().id(…).name(…).build().arguments(), i.e. the null taken from langchain4j's own type rather than a synthetic one. Asserts the tool runs, the result comes back verbatim, and a second call is served from the cache (proving the write survived too).CacheFactoryTest: two sub-second TTLs (1 ms / 999 ms) get distinct instances with distinct expiry; two TTLs sharing a whole-second part (10 001 ms / 10 900 ms) get distinct instances; and equal TTLs expressed differently still share one, so the fix does not leak an instance per caller.
Mutation-checked by reverting each fix: expected: <2026-07-23T09:00:00Z> but was: <Error executing tool: Cannot invoke "String.length()" because "arguments" is null>, expected: <long> but was: <null>, and 10001ms and 10900ms must not truncate onto one instance ==> expected: <a> but was: <b>. Restored: CacheFactoryTest 19, CacheImplTest 51, ToolCacheServiceTest 79, PaginatedResponseStoreTest 13, NonceCacheServiceTest 10, ToolExecutionServiceTest 32, ToolExecutionServiceExtendedTest 8, ToolExecutionServiceBranchTest 2 — 214 tests, 0 failures.
🔐 Ledger integrity, replay depth, and a verification that could not fail (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Four review findings on this branch's own remediation work — the common thread is the one the branch set out to remove: a control that looks present and does nothing.
D1 — three verify(never()) assertions that matched zero invocations either way
ToolExecutionServiceTest.nullConversationIdSkipsCostTracking and ToolExecutionServiceExtendedTest.skipsCostWhenNoConversation both claimed to cover if (enableCostTracking && conversationId != null). Both pass conversationId = null, so the only invocation a regression could produce is trackToolCall(invocation, null) — and Mockito's anyString() does not match null. The verification matched zero invocations whether the guard existed or not. Removing && conversationId != null and running all three ToolExecutionService*Test classes gave Tests run: 39, Failures: 0 — neither test noticed, while production would have hit computeIfAbsent(null, …) on a ConcurrentHashMap inside ToolCostTracker and returned Error executing tool: null to the model for every tool call.
Both now use nullable(ToolInvocation.class)/nullable(String.class).
Audit of the rest of the branch's tests for the same trap — grep of every never() verification in the touched test files, asking for each "can production pass null at that position?" — found one more: LlmTaskCoverage2Test.nullUserInput_ragSkipped pins if (userInput != null) around ragContextProvider.retrieveContext(memory, task, userInput) with anyString() in the userInput position. Same defect, same fix. The remaining anyString()/never() pairs sit on positions production fills from literals or non-null config and are sound.
All three mutation-checked by removing the corresponding guard: costTracker.trackToolCall(… or(isNull(), isA(String))) / ragContextProvider.retrieveContext(<any>, <any>, or(isNull(), isA(String))) → Never wanted here … But invoked here … with arguments: [ToolInvocation[…], null].
D2 — the nonce replay cache was under-sized against its own TTL
CacheFactory pinned nonce-replay-protection at 100 000 with a comment computing 330s × 300rps, but NonceCacheService.init() asks for maxAge + clockSkew + TTL_BUFFER_MS = 390s, so steady-state occupancy at 300rps is 117 000. Worse, Caffeine's W-TinyLFU admission is frequency-based, not LRU: a nonce is written once via putIfAbsent and never read, so all frequency estimates tie and the filter rejects the candidate — the newly inserted nonce is dropped rather than the oldest. Retention collapses precisely on the most recent (most replayable) nonces; measured against real Caffeine on a fake ticker, 17.1% of nonces still inside the 330s replay window were already forgotten. A forgotten nonce is a captured signed A2A envelope that replays inside its own freshness window.
Fix: capacity is now derived from the TTL the cache is asked for, not hard-coded. CacheFactory.RATE_SIZED_CACHES maps a cache name to the peak write rate it must absorb (nonce-replay-protection → 300/s) and maximumSizeFor(name, ttl) returns peakRps × ttlSeconds × 2.0, never below the configured floor. The ×2 head-room is because eviction is not LRU: sizing at exactly the steady-state occupancy still drops the newest entries under any burst. A future change to the replay window now moves the capacity with it.
Kept TTL_BUFFER_MS rather than dropping it — it is a real margin against a nonce being forgotten while still replayable, and it is no longer free-floating now that the capacity tracks it.
D3 — the audit canonical string was not injective, so a tampered entry could verify
AuditHmac.buildCanonicalString joins keys and values with =, ,, {}, [], | and escapes none of them. The map-to-string mapping is therefore not one-to-one: {"a": "x", "b": "y"} and {"a": "x,b=y"} canonicalize to the same bytes and share one valid HMAC, and {"calls": [{"tool": "calculator"}]} collides with the literal string "[{tool=calculator}]". String.join(",", actions) collapses ["a","b"] onto ["a,b"] the same way. For an append-only ledger that is a tamper-detection hole, and this branch made it reachable: AuditEntry.toolCalls now carries tool-trace arguments/result strings, which the LLM and the user write, and the new recursion walks into them with the same unescaped delimiters.
Fix — versioned canonical form. New entries are signed over a v2 string that escapes every delimiter inside keys and scalars and type-tags every value (s: scalar, m map, l list, n null), and the stored value carries the tag: v2:<64 hex chars>.
The back-compat constraint is absolute — any change to the canonical bytes invalidates the HMAC of every already-stored entry. So verifyHmac picks the canonicalizer from the stored value's prefix: tagged → v2, untagged (a bare hex digest) → the v1 canonicalizer, which is now frozen and documented as such. It deliberately never falls back from v2 to v1; trying both would hand the collision straight back to the attacker. AuditHmacTest.flatMapCanonicalStringUnchangedByRecursion, which pins the v1 string to a literal, still passes untouched.
Note that verifyHmac currently has no production caller — the ledger signs on write and verification is an operator/forensic operation. The fix is about what a future verifier can conclude from a stored row.
D4 — maxBudgetPerConversation stopped enforcing on upgrade
The previous commit made the ceiling conditional on a new enforceBudget defaulting to false, justified as "every built-in priced at $0.00, so no stored config was ever refused". That holds only for built-ins. For http, MCP, A2A and dynamic tools the dispatch name is the configured name, so an agent with a tool called websearch, webscraper or pdfreader was priced from DEFAULT_TOOL_PRICES and was being refused on main. Those operators' cost ceiling silently ceased to exist.
Decision — enforceBudget stays opt-IN, and the silence is fixed instead. eddi.tools.budget.enforce-by-default remains false; a new warnAboutUnenforcedBudgets names every configured task that carries a ceiling without the flag, once per task id.
Both defaults break someone, so the choice is which failure is acceptable:
Enforcing by default would make the ceilings on built-in-only agents bind for the first time — this release is what repaired built-in pricing — and start aborting tool calls mid-conversation on upgrade, with no warning and no way to have anticipated it.
Not enforcing costs the operator whose ceiling was live (http/MCP/A2A/dynamic tools dispatch under their configured name, so a tool called
websearch/webscraper/pdfreaderwas priced and refused onmain) — but that loss is detectable and announced, and re-enabling it is one field.
An unannounced new refusal is worse than an announced lapse: the first is a production incident an operator cannot predict, the second is a log line they act on. The unacceptable part was never the default — it was that a ceiling could record without refusing and say nothing, which is precisely the "config that silently does nothing" pattern this whole release set out to remove. The WARN is what makes the opt-in honest.
An earlier revision of this branch flipped the default to true; that was reverted in favour of the warning. eddi.tools.budget.enforce-by-default=true re-enables enforcement deployment-wide for operators who want the old behaviour back in one line.
Files
engine/caching/CacheFactory.java—NONCE_PEAK_SIGNED_RPS,RATE_SIZED_EVICTION_HEADROOM,RATE_SIZED_CACHES,maximumSizeFor(name, ttl); bothgetCacheoverloads size through itengine/caching/CacheImpl.java— package-privatebackingCache()so the eviction policy actually built can be assertedengine/audit/AuditHmac.java—V2_PREFIX,buildCanonicalStringV2,canonicalValueV2,escape;verifyHmacdispatches on the stored prefix; v1 canonicalizer frozenmodules/llm/impl/AgentOrchestrator.java—BUDGET_ENFORCE_DEFAULTstaysfalse; newwarnAboutUnenforcedBudgets+UNENFORCED_BUDGET_WARNEDso an unenforced ceiling is named once per task instead of passing silentlymodules/llm/model/LlmConfiguration.java—enforceBudget/maxBudgetPerConversationjavadocDocs:
audit-ledger.md(canonical-form versioning table),langchain.md,security.md,agent-father-langchain-tools-guide.md
Tests
Every behavioural test was mutation-checked by reverting the production change and confirming failure:
&& conversationId != null
costTracker.trackToolCall(or(isNull(), isA(ToolInvocation)), or(isNull(), isA(String))); Never wanted here … But invoked here … with arguments: [ToolInvocation[dispatchName=testTool, …], null] (both classes)
if (userInput != null)
ragContextProvider.retrieveContext(<any>, <any>, or(isNull(), isA(String))); Never wanted here … with arguments: [memory, …Task@…, null]
maximumSizeFor → fixed CACHE_SIZES at the builder
CacheFactoryTest.nonceCacheCapacityCoversItsTtl: a capacity of 100000 cannot hold the 117000 nonces written during a 390s replay window ==> expected: <true> but was: <false>; nonceCacheCapacityTracksTheTtl: expected: <200000> but was: <100000>
the rate-sizing rule inside maximumSizeFor
NonceCacheServiceTest.cacheCapacityCoversTheRequestedTtl: a 390s replay window at 300 signed requests/s holds 117000 nonces, but the cache is capped at 100000
canonicalValueV2 → the v1 renderer
4 failures: differentStructuresDoNotCollide, scalarMimickingNestedStructureDoesNotCollide, actionListSeparatorIsNotForgeable (expected: not equal but was: <v2:…>), tamperingIntoAColludingTwinIsRejected (expected: <false> but was: <true>)
the v1 branch of verifyHmac
legacyV1EntryStillVerifies: pre-v2 ledger rows must keep verifying against the v1 canonicalizer ==> expected: <true> but was: <false>
BUDGET_ENFORCE_DEFAULT → false
AgentOrchestratorCoverageTest.toolCall_budgetSetWithoutEnforceFlag_isStillEnforced: calculatorTool.calculate(<any string>); Never wanted here … But invoked here; AgentOrchestratorToolCostTest.ceilingWithoutFlagIsEnforced: webSearchTool.searchWeb("eddi", 3); Wanted 2 times … But was 5 times
New/changed cases: AuditHmacTest +7 (a v1CanonicalStringCollides precondition test asserts the v1 form does collide, so a future edit to the frozen canonicalizer is caught rather than silently invalidating the ledger), CacheFactoryTest +3, NonceCacheServiceTest +1, AgentOrchestratorCoverageTest +1, AgentOrchestratorToolCostTest +1 (and unenforcedBudgetRefusesNothing → explicitlyUnenforcedBudgetRefusesNothing, now setting the flag it names).
./mvnw clean compile clean; 857 tests across the audit / caching / crypto / llm-tools / orchestrator / LlmTask classes green, plus 87 in ai.labs.eddi.engine.audit.
🧵 Model registry and streaming executor — races and un-stripped parameters (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Four defects around ChatModelRegistry and StreamingLegacyChatExecutor, three of them introduced by D11 (the commit that stopped stripping timeout/logRequests/logResponses so they could join the model cache key).
C1 — a concurrent cache clear could hand null to chat(...). getOrCreate/getOrCreateStreaming did containsKey(key) and then get(key). Both caches are cleared from other threads — invalidateForSecret(null) on DEK/KEK rotation, and the global-variable invalidation listener on any variable edit. A clear landing between the two calls made the method return null; no caller null-checks it (LlmTask and CascadingModelExecutor pass the result straight to chat(...)), so the turn died with an NPE instead of simply rebuilding the model. On the streaming path a null additionally means "this provider cannot stream", so the race silently downgraded a streaming task to sync.
C2 — the streaming buffer was read without a happens-before edge. fullResponse is appended only from the provider's callback thread and read only from the executor thread. On the normal path latch.countDown()/await() publishes those writes; on the timeout/interrupt path there is no such edge — abandoned.set(true) is a volatile write ordering executor→callback, not callback→executor. The executor's toString() was therefore an unsynchronized read of a concurrently mutated StringBuilder and could observe a torn buffer (count advanced ahead of the char data, or a stale value array after a grow). The abandoned gate was also check-then-act: a callback already past the check could still push a token into the shared event sink after the executor believed the attempt silenced — breaking the documented "memory text matches streamed text" guarantee that D11 introduced the flag to provide.
C3 — un-stripping timeout made previously tolerated values fatal. Every provider builder parses it unguarded: if (!isNullOrEmpty(v)) builder.timeout(Duration.ofMillis(Long.parseLong(v))) — and isNullOrEmpty does not trim. Before D11 the value's only consumer was ObservableChatModel.wrapIfNeeded, which is deliberately lenient (guards blank, swallows NumberFormatException, drops zero/negative). So a stored " ", "30s" or "0" (historically "unlimited") went from tolerated to throwing out of build() on every turn of that agent, with no migration and no warning.
C4 — un-stripping the logging flags turned on untruncated body logging. Reaching the provider builders, logRequests/logResponses install langchain4j's LoggingHttpClient, which writes the entire request and response body at INFO with no truncation. EDDI's own ObservableChatModel/ObservableStreamingChatModel already honour both flags and deliberately truncate to 200/500 chars. Enabling the flags therefore started writing full prompts, full conversation history and full model responses to the application log.
What changed:
ChatModelRegistry
Both lookups are a single get — a miss from a concurrent clear falls through to construction, which is correct for a pure memoization cache
ChatModelRegistry.normalizeTimeout
timeout is normalised once at the boundary that feeds both the cache key and the builders: trimmed, and dropped when blank, non-numeric or non-positive, mirroring the tolerance it used to enjoy
ChatModelRegistry.warnAboutRejectedTimeout
WARN naming the model type and the offending value, emitted on the build path only so it does not repeat on every cache hit
ChatModelRegistry.builderParams
logRequests/logResponses are removed from the map handed to a builder — they stay in the cache key, so D11's correctness fix survives
StreamingLegacyChatExecutor
A per-attempt lock makes check + append + emit and abandon + toString() mutually exclusive; the pre-retry abandoned.set(true) takes the same lock
docs/langchain.md
Documents that logging is EDDI's truncating path (not the provider's) and that an unusable timeout is ignored rather than fatal
Design decisions:
C4 chose option (a): keep the flags in the cache key, strip them from the builder input. EDDI's decorators already honour both flags on both the sync and streaming path, including for providers whose builder has no logging switch, so nothing is lost — while provider-level logging is unbounded PII exposure that no config field advertises.
logRequestsAndResponses(Azure OpenAI and Gemini only) is left forwarded: it predates this branch, is provider-specific, and stripping it would be an unrelated behaviour change; it is now documented as the deliberate escape hatch.Normalising
timeoutbefore the cache key, not after. Keeping the normalised value in the key preserves D11's fix (two genuinely different timeouts remain two models) and additionally collapses" 5000 "and"5000"into one.The C2 lock is held across
eventSink.onToken. That is a bounded call — the SSE sink builds an event and calls a non-blockingsend()whoseCompletionStageis never awaited — and holding it is what actually makes "no token after abandonment" true rather than merely likely.The C1 regression test replaces the cache with a map that clears itself inside
containsKey, reproducing the exact interleaving deterministically rather than relying on a stress loop, and assertscontainsKeyis never called at all.The provider
timeoutcontract is mirrored verbatim in the test builder. Real builders open a loopback socket and cannot be constructed in the unit-test sandbox, so theisNullOrEmpty+Long.parseLongexpression is reproduced in the fake — without it the C3 tests would pass whether or not the registry normalises.
Tests: ChatModelRegistryTest gains three nested classes (C1 concurrent-invalidation, C3 timeout normalisation, C4 provider logging) and its getOrCreate_observabilityParamsReachBuilder now asserts the flags do not reach the builder; StreamingLegacyChatExecutorTimeoutTest gains abandonment_isMutuallyExclusiveWithTokenEmission. All four fixes were mutation-checked: reverting C1 fails both new tests with expected: not <null>, C3 with NumberFormat For input string: " ", C4 with logRequests must NOT reach the provider builder, C2 with The executor must not set 'abandoned' and read the buffer while a token emission is in flight.
🔐 Tool-cache scope resolution honours its own invariant (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Three defects in the per-tool cache-scope surface this branch introduced with D5/D2. Two of them could put a tool on a cross-user partition — the exact leak D5 existed to close.
B1 — a typo in a per-tool scope widened the tool instead of narrowing it. ToolCacheScope's javadoc (and the D5 changelog entry) promise that "a typo must never silently widen a tool's audience". The code did the opposite: fromConfig returns null for an unrecognized token, and resolve's if (scope == null) branch could not tell "this tool has no entry" from "this tool has an entry I could not parse", so it fell through to the task-level default. With defaultToolCacheScope: "global" plus toolCacheScopes: {"getUserProfile": "usr"} — a typo for "user" in an override written to narrow that one tool — the tool resolved to GLOBAL and got the shared "g" tag: one partition for every authenticated user. Alice's result served to Bob.
B2 — toolCacheScopes was the only per-tool map that did not speak the slug. D2 established the canonical slug as the configuration vocabulary: toolRateLimits and toolPricing both accept the dispatch name or the slug. toolCacheScopes was looked up by dispatch name alone, so {"websearch": "user"} — written in the same object as the toolRateLimits: {"websearch": 30} next to it, in the same vocabulary as builtInToolsWhitelist — was silently ignored and the tool stayed on whatever the task default was, possibly global. Every built-in was affected; none declares @Tool(name = ...).
B3 — the dedicated news TTL became unreachable. getSmartTTL received only the canonical slug after D2, so searchNews canonicalised to websearch and exact-matched its 1800s entry before the substring loop could reach Map.entry("news", 600L). No slug contains "news", so that entry was dead for every built-in and news results were cached 3× longer than the table declares. Inert until D5b made per-entry TTLs actually take effect.
What changed:
ToolCacheScope
resolve takes both names and gains resolvePerTool, which returns null only when the map has no entry for a name. A present-but-unparseable entry returns DEFAULT (USER) and logs a WARN naming the tool and the bad token. An unrecognized defaultToolCacheScope also WARNs
ToolCacheService
resolveScopeTag(dispatchName, canonicalName, …); getSmartTTL split into a lookupTTL that can say "no match" plus resolveSmartTTL(dispatch, canonical) which tries the dispatch name first
AgentOrchestrator
Passes the already-computed canonicalName into resolveScopeTag — the three per-tool maps on a Task now agree on key vocabulary
docs/langchain.md, docs/security.md, docs/agent-father-langchain-tools-guide.md
Scope keys documented as slug-or-dispatch-name; fail-safe parsing and dispatch-first TTL documented
Design decisions:
Fail safe, don't fail the load. Leniency was never the bug — the direction of the fallback was. Unknown tokens still never abort an agent load; they now land on
USER, and the WARN makes the typo discoverable instead of merely inert.A bad dispatch-name entry does not fall through to the slug entry either.
{"searchNews": "usr", "websearch": "global"}resolvessearchNewstoUSER. Any fall-through from a broken narrowing override risks landing somewhere wider, which is the whole defect.Dispatch name before slug, everywhere. Same precedence as
resolveRateLimit: an entry naming one operation is more specific than one naming the whole tool. That precedence is also what makes thenewsTTL reachable again, without deleting a table entry that is genuinely correct for news.The cache KEY stays on the dispatch name. Unchanged, and load-bearing:
searchWeb/searchNews/searchWikipediashare thewebsearchslug and must not share an entry.resolveScopeTag's old 5-arg form was replaced, not overloaded. A lingering dispatch-name-only overload would let future code reintroduce B2 silently.
Operator-visible behaviour changes:
A
toolCacheScopesvalue that does not parse now yieldsuserscope instead of inheritingdefaultToolCacheScope. Deployments relying on a typo'd key to reach aglobaldefault lose that (unintended) cross-user reuse — a cache-hit-rate change, never a correctness one.Slug-keyed
toolCacheScopesentries now bind. An entry like{"websearch": "global"}that was previously inert becomes live — audittoolCacheScopesfor slug keys that were written as documentation rather than intent.searchNewsresults are now cached for 10 minutes instead of 30.
Tests: ToolCacheServiceTest (79) and AgentOrchestratorCoverageTest (57) green. New nested suites UnparseablePerToolEntryTests and DualNameScopeKeyTests, plus TTL cases in CanonicalPutTests and a ticker-driven newsTtlIsLoadBearing against a real expiring cache. All mutation-checked: reverting each production change fails 6, 4 and 2 tests respectively.
🧾 The audit ledger stops triple-billing the turn, and starts seeing the tool spend (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Three defects in the audit / cost path, all introduced or made live by this branch's own remediation work. They share one root cause: dollar figures that are read, copied or baselined at the wrong moment.
A1 — every task after the LLM task re-reported the LLM's cost and tool calls. LifecycleManager.buildAuditEntry runs once per lifecycle task, and ConversationStep's data is never cleared between tasks. It read audit:cost, audit:tool_calls, audit:token_usage and the whole llmDetail block with no task-type gate, so a workflow [parser, behavior, langchain, output, templating] on a turn costing $0.0042 appended three ledger rows at $0.0042, and recorded that the output and templating tasks made tool calls they never touched. The ledger is append-only (EU AI Act), so an auditor summing cost reads a multiple of the truth with no way to correct it. The same file already gated the SSE tool trace on the task type ~60 lines earlier, with a comment explaining exactly this hazard — the audit reader never got the same gate. Before this branch cost was a literal 0.0 and toolCalls a literal null, which is why the lingering-key mechanism was previously harmless.
A2 — the cascade dropped the entire tool spend. LlmTask's cascade branch builds a fresh metadata map and hand-copies warning, finishReason and streamingTimeout out of the winning step's metadata. It never copied toolCostUsd — yet accumulateAuditEvidence computes the ledger cost as cascadeCostUsd + toolCostUsd. Since ModelCascadeConfig.enableInAgentMode defaults to true, this is the default path for any agent-mode task with a cascade: the ledger (and responseMetadataObjectName template data) reported token cost only. The non-cascade branches assign the agent's whole map and were always correct, so the two branches of one feature disagreed.
A3 — HITL-approved tool calls were excluded from their own turn's cost. AgentOrchestrator.resumeToolLoop snapshotted its tool-cost baseline after the verdict-application loop, which has already run every human-approved gated call through executeToolWrapped → costTracker.trackToolCall. Those charges were inside the baseline, so toolCostDelta subtracted them right back out: the cost of precisely the calls a human signed off on never reached toolCostUsd. The live path (executeWithTools) always baselined before any tool ran.
What changed:
LifecycleManager
New shared predicate isLlmTask(ILifecycleTask) used by both buildTaskSummary and buildAuditEntry, so the two readers of the same lingering keys cannot drift apart again. llmDetail, toolCalls and cost are gated on it; so is the summary's confidence (an LLM-only signal that lingered the same way)
CascadingModelExecutor
CascadeResult gains runToolCostUsd; the step loop's runCostUsd/runTokenUsage locals become one RunTotals accumulator that also sums stepToolCost(...) per step
LlmTask
The cascade branch now puts toolCostUsd (the run total) into the response metadata, matching what the non-cascade branches always carried
AgentOrchestrator
resumeToolLoop takes the tool-cost baseline at method entry, before the verdict loop; the redundant costConversationId local is gone
Design decisions:
Gate, don't clear. Clearing
audit:*between tasks would break accumulation across an LLM config's sub-tasks (accumulateAuditEvidenceis deliberately read-modify-write). Gating the reader is the narrow fix.actions,inputandoutputstay ungated.actionsis genuinely per-task, and the input/output pair describes the step context every task ran in — that is pre-existing, intended behaviour. Only the LLM-written evidence is gated.Run total, not the winning step's slice (A2). An escalating cascade re-enters
executeAgentModeStepfor every step it tries, and each of those steps really does execute (and get charged for) its tools. Copying the accepted step'stoolCostUsdwould under-bill exactly as much as the old code dropped, andreturnBestAcrossSteps/finalizeBestreturn a different step's metadata anyway.AgentOrchestratorreports its own per-call delta of the conversation total, so the per-step values partition the run's spend and are safe to sum — the same argument that already justifiesrunCostUsdandrunTokenUsagebeing run totals.RunTotalsover a fifth positionaldouble.withRun/finalizeBestwould otherwise have carried two adjacent, silently swappabledoubleparameters — in a cost-accounting path, on a branch whose whole theme is cost figures drifting apart. It is a plain local accumulator; the executor stays stateless.
Operator-visible behaviour changes:
Ledger rows for non-LLM tasks now carry
cost = 0.0,toolCalls = nullandllmDetail = nullinstead of a copy of the LLM task's figures. Summingcostover a turn now yields the turn's actual cost; historical rows written before this fix remain inflated and cannot be amended (append-only).SSE
task_completeno longer reportsconfidencefor tasks that follow the LLM task.Agent-mode cascade turns now report tool spend in
audit:costand inresponseMetadataObjectName— figures go up relative to before, because they were previously missing, not double-counted.A HITL-resumed turn's
toolCostUsdnow includes the approved gated calls. This also meansmaxBudgetPerConversationaccounting sees them.
Tests: LifecycleManagerTest.BuildAuditEntryTaskTypeGateTests (3, driving a real ConversationMemory/ConversationStep so the lingering is the production mechanism rather than a per-key stub), two cascade tool-cost tests in LlmTaskAuditLedgerTest, one resume-path cost test in AgentOrchestratorResumeToolLoopTest. Every one was mutation-checked: ungating the audit reader fails with task 'output' spent nothing … expected: <0.0> but was: <0.0042>; ungating the summary reader fails the pre-existing trace test too; dropping the cascade's toolCostUsd gives expected: <0.0095> but was: <0.002>; turning the run total into last-wins gives expected: <0.007> but was: <0.004>; restoring the old baseline position gives expected: <0.002> but was: <0.0>.
Files: LifecycleManager, CascadingModelExecutor, LlmTask, AgentOrchestrator, plus their three test classes and docs/changelog.md.
🧮 convertToObject finally reaches agent mode and streaming (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D7. jsonMode was computed once in LlmTask from convertToObject and handed to exactly one collaborator — LegacyChatExecutor, i.e. the three no-tools, non-streaming call sites. AgentOrchestrator.executeIfToolsEnabled, StreamingLegacyChatExecutor.execute and the cascade's agent-mode / live-streaming steps all built their ChatRequest without ever looking at it.
The visible consequence: AgentSetupService sets enableBuiltInTools(true) in the same method that sets convertToObject=true, so a tool-enabled agent took the agent path and got no API-level JSON at all. With addToOutput=false the quickReplies/sentiment postResponse then degraded silently — the model returned prose, startsWith("{") failed, and the raw text was stored as a string.
Why only mistral and azure-openai were bitten: AgentSetupService.supportsResponseFormat approved openai, mistral and azure-openai and injected a builder-level responseFormat=json model parameter — but OpenAILanguageModelBuilder is the only builder that reads that key. For mistral and azure-openai the parameter was dead. openai got JSON in every mode purely because the parameter was baked into its cached ChatModel.
What changed:
JsonResponseFormatPolicy (new, modules/llm/capability)
Decides per request whether ResponseFormat.JSON may be set. Carries (requested, provider, override) and resolves against whether that request carries tool specifications
AgentOrchestrator
executeIfToolsEnabled / resumeToolLoop / runToolCallLoop take the policy; the tool loop sets responseFormat on each ChatRequest, gated on !activeSpecs.isEmpty(). Previous arities kept as delegating overloads
StreamingLegacyChatExecutor
execute / executeCapturing take the policy; the streamed ChatRequest now carries the format
LegacyChatExecutor
The boolean jsonMode parameter becomes the policy, so an unsupported provider is no longer sent a format it will reject
CascadingModelExecutor
Builds a policy per step from that step's resolved modelType — a cascade routinely escalates across providers
LlmTask
Builds one policy from convertToObject + the resolved provider + the task override and passes it to all three modes, plus the HITL resume path
LlmConfiguration.Task.jsonResponseFormat
New config field: auto (default) / on / off
AgentSetupService
Stopped injecting responseFormat=json; supportsResponseFormat deleted (its only remaining caller was the test-compat delegate McpSetupTools.supportsResponseFormat, also deleted)
The provider matrix, read off the langchain4j 1.18.0 bindings rather than assumed:
openai
yes
yes
OpenAiUtils#toOpenAiResponseFormat → json_object
azure-openai
yes
yes
InternalAzureOpenAiHelper#toAzureOpenAiResponseFormat → ChatCompletionsJsonResponseFormat
mistral
yes
yes
MistralAiMapper#toMistralAiResponseFormat → JSON_OBJECT
gemini, gemini-vertex
yes
no
maps to responseMimeType=application/json, which the API rejects alongside tools
anthropic, bedrock
no
no
both throw UnsupportedFeatureException for JSON without a schema
ollama, jlama, huggingface, oracle-genai
no
no
unverified against the provider API — opt in per task with jsonResponseFormat: "on"
Design decisions:
Request level, never builder level. The changelog entry of 2026-04-02 records a Gemini
400 Function calling with a response mime type: 'application/json' is unsupportedcaused by a builder-levelresponseFormatbaked into a cached model that was later reused with tools. A cached model is shared across turns, tasks and execution modes, so anything mode-specific must live on the request. D11 has since made the model cache key finer-grained, which does not change this: identity is not the issue, reuse across modes is.The matrix is tools-aware, not provider-aware only. Gemini's answer differs depending on whether the same request also carries
toolSpecifications, so the decision is taken inside the tool loop where that is known, against!activeSpecs.isEmpty()rather than against "is this agent mode".supportsResponseFormatwas retired, not narrowed. After request-level threading the builder parameter is dead for mistral/azure-openai and redundant-plus-hazardous for openai, which leaves no provider for which the matrix entry does anything.OpenAILanguageModelBuilderstill honours a hand-writtenresponseFormat=json, so existing stored configs are unaffected.onbypasses the tools guard. An operator forcing the format onto an unlisted provider or an OpenAI-compatible gateway is making a deliberate choice; documented as such.
Operator-visible behaviour changes:
Tool-enabled and streaming agents on openai, azure-openai and mistral now send
response_format: json_object(or the Azure/Mistral equivalent) wheneverconvertToObject=true. Expect more reliable JSON, and for azure/mistral a real behaviour change on the wire.anthropic and bedrock no longer receive a schemaless JSON format on the no-tools path. Previously the request was sent, rejected, and silently retried through
LegacyChatExecutor's fallback — one wasted round trip per turn, now avoided. Enforcement there was and remains prompt-only.gemini keeps API-level JSON when no tools are present and is never sent it together with tools.
Newly created agents (
setup_agent,create_api_agent,POST /administration/agents/setup*) no longer carry aresponseFormatmodel parameter. Existing stored configs keep theirs and keep working.New optional task field
jsonResponseFormat(auto|on|off).
Tests: JsonResponseFormatPolicyTest (20) and JsonResponseFormatThreadingTest (12) are new; three wiring tests added to LlmTaskCoverageTest. Every behavioural assertion was mutation-checked (strip the agent-mode format; strip the streaming format; make the matrix tools-blind; approve every provider; drop openai from the matrix; make LlmTask pass DISABLED) — each mutation fails the tests that cover it.
Files: 8 modified in src/main, 1 new (JsonResponseFormatPolicy), 8 test files touched (incl. 52 Mockito matcher lists widened for the new parameter — an unmatched stub returns null instead of failing, so verify(never(...)) sites had to move too), docs/langchain.md, docs/changelog.md.
⏱️ timeout, logRequests and logResponses become part of a model's identity (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D11. docs/langchain.md documents timeout, logRequests and logResponses as configuration parameters. ChatModelRegistry.filterParams removed all three from the map it passed to the provider builders — so every provider's builder.timeout(...) / builder.logRequests(...) / builder.logResponses(...) branch, on both the sync and the streaming builder, was unreachable dead code.
The serious half is what that did to the cache. filterParams produces both the builder input and the ModelCacheKey, so stripping those three keys also erased them from the model's identity:
A task configured with timeout: "5000" silently received an unwrapped, timeout-free model whenever a task with otherwise identical parameters happened to be constructed first — and vice versa. Which model a task got depended on construction order. On the sync path ObservableChatModel.wrapIfNeeded was applied only to the first build, so the wrap was attached to whichever task won the race and then served to every other task. getOrCreateStreaming never called wrapIfNeeded at all.
What changed
ChatModelRegistry.filterParamsno longer strips the three keys. Cache key and builder input are the same map again, which is the invariant that was broken: anything that shapes the constructed model now necessarily shapes its identity. The keys still filtered (systemMessage,prompt,logSizeLimit,addToOutput,convertToObject,includeFirstAgentMessage) are ones no builder reads.getOrCreateStreaminghonours the three settings.timeoutreaches the streaming builder;logRequests/logResponsesadditionally wrap the model in the newObservableStreamingChatModel.New
ObservableStreamingChatModel— logging-only streaming counterpart toObservableChatModel, so the flags behave identically across providers, including those whose streaming builder has no logging switch.StreamingLegacyChatExecutorresolves one backstop from both timeout fields (resolveTimeoutSeconds), and stops an abandoned attempt from writing to the shared event sink.docs/langchain.mdgains a "Timeouts and Streaming" section and drops the implication thattimeoutalone bounds a streaming turn.
Design decisions
No
Future.getbound on the streaming path.ObservableChatModelbounds a sync call by submitting it to an executor and callingfuture.get(timeout). That shape is wrong for a stream: an overall wall-clock bound truncates a healthy long answer, and cancelling the awaiting thread does not stop the provider's callback thread. The streaming-appropriate bound already exists — every streaming builder passestimeoutto its HTTP client, and for the JDK client (JdkHttpClientsets it asHttpRequest.timeout()on an asyncofInputStreamsend) that bounds the time to the provider's first response, not the duration of the stream. So it detects a provider that never answers without cutting off one that answers slowly.ObservableStreamingChatModelis therefore observability-only.The two timeout fields stay two fields, with a derived default.
timeout(ms, model parameter, provider-level) andstreamingTimeoutSeconds(s, task-level, EDDI's overall backstop) bound genuinely different things; collapsing them would lose a capability. What was wrong was that they were unrelated: a task withtimeout: "300000"was still cut off by the undocumented 120s backstop. Resolution order is now (1) an explicit positivestreamingTimeoutSecondswins, (2) otherwise 120s raised, never lowered, to cover a longer configuredtimeout, (3) otherwise 120s. Both stored shapes keep their existing behaviour; only the previously broken combination changes.timeoutis read from the task's raw parameters when deriving the backstop. A Qute-templated value cannot be resolved at that point and leaves the 120s default in place — the pre-existing behaviour, and never a shorter bound. The alternative (threading the processed parameter map throughexecute/executeCapturing) buys correctness only for templated timeouts and costs two new overloads.A decorator must forward to the overload it was called on.
ObservableStreamingChatModeloverrides bothchat(ChatRequest, handler)andchat(ChatRequest, ChatRequestOptions, handler)and forwards each to the same overload on the delegate. Funnelling both through the three-argument form hitsStreamingChatModel's defaultdoChat—throw new RuntimeException("Not implemented")— for any model that implements the two-argumentchatdirectly. Caught byObservableStreamingChatModelTest, which is why it exists.R5 (
ChatModelListenerobservability SPI) deliberately not implemented. D11 is the bug; R5 is a feature and stays a separate backlog item.
Abandoned streams no longer write to the shared event sink
StreamingLegacyChatExecutor retries a timed-out attempt that produced no tokens, but nothing stopped the abandoned attempt's handler: its provider callback thread stayed alive and kept calling eventSink.onToken(...) while the retry streamed into the same sink. A slow first attempt whose first token landed after the retry began therefore interleaved two answers on the SSE stream, and neither matched the text stored in memory. Each attempt now carries an abandoned flag its handler checks before forwarding — set on timeout, on interrupt, and before an error retry. The flag is set before the partial text is read, so the text returned to memory is exactly the text the client was sent. The executor still cannot cancel the provider's stream (langchain4j exposes no cancellation on this path); it can and now does make the abandoned stream silent.
The cascade's live-stream backstop stopped assuming 120s
CascadingModelExecutor bounded a live-streamed step's future with a hardcoded STREAMING_STEP_TIMEOUT_MS = 125_000L, whose comment states the invariant it exists to hold: it must exceed the streaming executor's own bound, because cancelling the awaiting thread does not stop the provider's callback thread — the SSE client would keep receiving tokens for a step the cascade has already moved past. That invariant was already violable (streamingTimeoutSeconds: 300 broke it), and deriving the bound from timeout adds another way in. It is now resolveStreamingStepTimeoutMs(task) = the executor's resolved bound + a 5s margin, so the default case is still exactly 125s and a longer configured bound raises the backstop with it.
Operator-visible behaviour changes
timeout on any task
Never reached the provider builder; on the sync path only the ObservableChatModel wrapper applied, and only if that task built the model first
Reaches the provider builder on both paths, and is part of the model cache key
logRequests/logResponses on a streaming task
Silently discarded
Honoured — emitted at INFO by ObservableStreamingChatModel, and passed to builders that support them
Two tasks differing only in timeout/logRequests/logResponses
Shared one cached model; whichever built first won
Two separate cached model instances
Streaming task with timeout > 120s and no streamingTimeoutSeconds
Cut off at the undocumented 120s backstop
Backstop follows the configured timeout
Streaming task with only streamingTimeoutSeconds
—
Unchanged
Retry after an empty timed-out streaming attempt
Late tokens from the abandoned attempt could interleave with the retry's output
Abandoned attempt is silent
Live-streamed cascade step with a bound above 125s
The cascade cancelled the future while the provider was still emitting
Cascade backstop tracks the executor bound (+5s)
Cached model instances are keyed more finely now, so a deployment whose tasks differ in these settings holds a few more model objects than before. That is the point — they were sharing an instance only one of them had configured.
Tests
ChatModelRegistryTest grew an ObservabilityCacheKeyTests nest (different timeout/logRequests/logResponses ⇒ different instances, sync and streaming; construction order no longer decides which model a task gets; the settings reach the builder). Its old getOrCreate_observabilityParamsDontAffectCacheKey test asserted the defect and was replaced. New StreamingLegacyChatExecutorTimeoutTest pins the backstop resolution table including both back-compat shapes, that a short timeout cannot truncate a healthy stream end to end, and that a late token from an abandoned attempt never reaches the sink. New ObservableStreamingChatModelTest pins decorator transparency. Every behavioural test was mutation-checked by reverting the corresponding production change: 9 of the 29 ChatModelRegistryTest cases fail with filterParams restored (expected: <5000> but was: <null>, expected: not same but was: <…>), the backstop cases fail with the derivation neutralised (expected: <300> but was: <120>), and the abandoned-stream case fails without the flag (NeverWantedButInvoked: conversationEventSink.onToken("LATE")). The cascade-backstop cases fail with the hardcoded 125s restored (Cascade backstop (125000ms) must exceed the executor bound (300000ms)).
🧾 The audit ledger stops recording zeros and nulls (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D1. AuditEntry documents llmDetail as carrying token usage, toolCalls as tool execution data and cost as the monetary cost of the step. Every entry the engine has ever produced carried toolCalls = null and cost = 0.0 — passed as literals at the new AuditEntry(...) call site, with comments claiming an integration that did not exist:
Neither was true. LlmTask never wrote audit:tool_calls or audit:cost, and audit:token_usage — the key buildAuditEntry reads into llmDetail.tokenUsage — had zero writers in src/main/java. This matters beyond tidiness: the ledger is the EU AI Act Art. 17/19 traceability record, and it has been attesting that every LLM decision cost nothing and used no tools.
Same class of defect one line up: buildTaskSummary reads audit:confidence as IData<Double>, while LlmTask wrote audit:cascade_confidence as String.valueOf(...) — wrong key and wrong type. All four audit:cascade_* keys were write-only.
What was wired
audit:token_usage
LlmTask, accumulated per turn
llmDetail.tokenUsage
audit:tool_calls
LlmTask, accumulated per turn
AuditEntry.toolCalls
audit:cost
LlmTask, accumulated per turn
AuditEntry.cost
audit:confidence
LlmTask cascade branch, as a Double
llmDetail.confidence + the task_complete SSE summary
audit:cascade_model
LlmTask cascade branch
llmDetail.cascadeModel
All six audit keys moved into MemoryKeys with javadoc naming their producer and consumer, so the next reader/writer split is a compile-time concern rather than a silent one. audit:cascade_confidence, audit:cascade_cost and audit:cascade_token_usage are gone — they had no readers anywhere and never left the in-flight ConversationStep, so there is nothing to migrate.
Design decisions
Which cost signal. The two dollar figures that actually exist are the cascade's
runCostUsd(frominputPricePer1M/outputPricePer1MonModelCascadeConfig/CascadeStep) andToolCostTracker's per-conversation tool cost, which became real in D2.costis the sum of both. No token price table was invented for non-cascade tasks — those report tool cost only, and a non-cascade turn with free tools still audits at0.0. Hoisting cascade pricing to task level is a config change and stays a follow-up.Accumulate, never overwrite. A turn can drive many LLM calls: one per matching config sub-task, plus every escalated cascade step and every tool-loop iteration.
getLatestDatais last-write-wins, so each contributor read-modify-writes. Only counts a provider actually reported are touched — a provider that omitstotalTokensmust not zero what earlier calls contributed.toolCallsshape is{"calls": [...]}, each entry the tool-trace record plus thellmTaskIdthat issued it; without that tag a merged list from several sub-tasks is unattributable.Nested maps, and the HMAC time bomb they would have armed.
llmDetail.tokenUsageis a nested map andtoolCalls.callsa nested list, butAuditHmac.sortedMapStringflattened values withtoString()whileAuditStore.fromDocumentcopies only the top level — so an entry signed in memory with aLinkedHashMapwould read back with anorg.bson.Document(whosetoString()is prefixedDocument{) and fail to verify against its own HMAC. Latent only becauseAuditHmac.verifyHmachas no production callers yet. The canonicalizer now recurses through maps and lists; scalars still usetoString(), so flat maps produce a byte-identical canonical string and historical entries keep verifying — pinned by a literal-string test plus a real BSON encode/decode round-trip test.Failure-path entries deliberately unchanged. The
AuditEntrybuilt when a task throws keepscost = 0.0/toolCalls = null: the accumulators are partial at that point and the task may never have reached the model.GdprComplianceServiceandCapabilityMatchConditionalso keep their zeros — those are administrative/rule entries where zero is correct, not defective.
Three upstream signal defects fixed on the way
The ledger is only as honest as what feeds it:
LegacyChatExecutorbuilt the token map withMap.ofover three boxedIntegers. Providers legitimately report only some of the three (Bedrock and Ollama commonly omit the total), so a partial report was an NPE that killed the whole turn over telemetry. Now sharesAgentOrchestrator.tokenUsageMap, which 0-defaults.The agent tool loop double-counted tokens on retry. The accumulator was declared outside the retry lambda and
RetryConfiguration.executeWithRetryreplays that lambda, so a retried turn counted the abandoned attempt too. Reset on lambda entry.The cascade under-reported tokens.
runCostUsdwas a run total across every attempted step whiletokenUsagereported only the accepted step, so an escalating cascade produced token counts that contradicted its own dollar figure.CascadeResult.tokenUsageis now the run total; per-step usage stays in the trace.
Operator-visible behaviour changes
LLM-task audit entries gain
llmDetail.tokenUsage, andllmDetail.cascadeModel/llmDetail.confidenceon cascade turns. Entries are larger.AuditEntry.toolCallsis non-null on any turn where a tool ran;AuditEntry.costis non-zero wherever a cascade with configured prices or a priced tool ran. Cost dashboards that assumed a constant zero will start moving.HITL-resumed turns gain a full
llmDetailblock.executeResumenever wroteaudit:compiled_prompt, andLifecycleManagergates the entire block on that key — so every turn a human intervened in audited with no LLM evidence at all. Fixed.ExecutionResult.responseMetadatanow carriestoolCostUsd(the delta this model call added, not the conversation running total). Agents that surfaceresponseMetadataObjectNamein templates will see the extra key.task_completeSSE frames now really carryconfidenceon cascade turns.
Known gap — kept documented, not silently swallowed
A turn that pauses for tool approval still loses its pre-pause token usage: ToolApprovalRequiredException escapes executeWithTools before the metadata is assembled, and resumeToolLoop starts a fresh accumulator. Closing it needs the usage to survive the pause, i.e. a new field on the persisted PendingToolCallBatch — a snapshot-format change deliberately out of scope here. The under-report is bounded to the pre-pause segment of paused turns and is called out at both code sites.
Tests
New LlmTaskAuditLedgerTest backs the conversation step with a real map rather than a mock that always answers null — every assertion here is about accumulation, and a null-answering step makes a broken accumulator look correct. It covers agent-mode token usage, multi-sub-task summation, partial provider reports, tool-call merging with llmTaskId, cascade cost from configured pricing, the Double confidence key, the resumed-turn llmDetail block, and the audit-collector gate. AuditHmacTest gains the flat-map back-compat literal, nested-map/list determinism, Document-vs-LinkedHashMap equivalence and the BSON round-trip. LifecycleManagerTest gains toolCalls/cost/cascadeModel/confidence assertions.
Four pre-existing tests were repaired rather than extended — they passed for the wrong reason: LifecycleManagerTest.auditEntryWithLlmDetails asserted only containsKey("tokenUsage") on a stub for a key nothing wrote; LlmTaskCoverage2Test pinned the dead audit:cascade_confidence and audit:cascade_token_usage keys with any(); LlmTaskDeepBranchTest.auditCollectorStoresData verified atLeast(3).createData(anyString(), any()), which passes for any three keys at all.
🔌 The live tool trace finally reaches the SSE stream — and takes an unredacted payload with it (2026-07-23)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D3. The task_complete SSE frame has advertised a toolTrace field since the streaming API shipped, and has never emitted one — not once, on any deployment. The writer and the reader never agreed on a key:
Writer
langchain:trace:<modelType>:<configTaskId> (e.g. langchain:trace:openai:taskA)
LlmTask.executeTask / executeResume
Reader
langchain:trace: + task.getId().name() = langchain:trace:ai.labs.llm
LifecycleManager.buildTaskSummary
getLatestData is a startsWith scan, so those two prefixes can never overlap — summary never got a toolTrace entry and the if (summary.containsKey("toolTrace")) branch in RestAgentEngineStreaming.onTaskComplete was dead code. The "live tool call display" in the UI has never received a byte.
The fix — reader side only
buildTaskSummary now aggregates over getAllElements() instead of computing a key:
The writer key is untouched.
RestToolHistoryand everyConversationMemorySnapshotalready persisted in MongoDB scan the samelangchain:trace:prefix with an arbitrary suffix. Changing the writer would have silently broken historical tool-history replay. The literal moved intoMemoryKeys.LANGCHAIN_TRACE_PREFIXand all three call sites (writer ×2,RestToolHistory) now share it — byte-identical output, de-duplication only.Aggregate, don't take the latest.
LlmTaskwrites one trace key per LLM config task, andgetLatestDatareverses the element list and returns only the newest match. A naivegetLatestData(LANGCHAIN_TRACE_PREFIX)would have shipped a subtly wrong trace — the last task's calls only — which is worse than shipping nothing.getAllElements()is an insertion-ordered defensive copy, so write order is preserved for free.The task-type gate is load-bearing. Step data survives across tasks within a
ConversationStep, so an ungated prefix scan would make every task executed after the LLM task report the LLM's trace as its own. Reads are gated onTASK_TYPE_LANGCHAIN.Siblings are deliberately not swept in.
langchain:cascade:trace:,rag:trace:andrag:httpcall:trace:do not match the prefix and stay out of the frame.
⚠️ Security — tool arguments and results now leave the process unredacted
Operators and downstream integrators must read this. Until now a tool call's arguments and its result were reachable only through the owner-scoped RestToolHistory endpoint and the audit ledger, which runs AuditLedgerService.scrubSecrets. Making the trace reach the stream puts that same payload on the SSE channel, and nothing on the buildTaskSummary → onTaskComplete path redacts anything. Whatever a tool was called with — an API key passed as a tool argument, a token echoed back in a tool result — now streams verbatim to every client subscribed to that conversation's sayStreaming.
The correct place to fix that is the producer (the tool_call / tool_result maps built in AgentOrchestrator), so that RestToolHistory, the audit ledger and the stream all inherit one redaction rule. That is tracked separately (D12) and is explicitly out of scope here — this entry exists so the exposure is not discovered in production. Deployments that stream to untrusted clients and pass secrets through tool arguments should weigh that before taking this build.
Other operator-visible behaviour changes
The
task_completeSSE frame gains atoolTracearray on LLM tasks that executed at least one tool (including cascade turns). Purely additive — no existing field changes shape — but a strict unknown-field-rejecting SSE parser in eddi-chat-ui or EDDI-Manager would now break on it. [UNVERIFIED — requires a check of both frontends'task_completeparsers.]The frame can get large. One
tool_call+ onetool_resultentry per tool call per iteration, up tomaxToolIterations;tool_resultcarries the (truncated) tool output. There is no cap on the streamed trace — adding one would be a config field (Golden Rule 1) and is out of scope.Non-streaming
sayis unchanged.eventSinkis null there, and the only other consumer ofsummary,buildAuditEntry, reads"actions"and nothing else — audit output is byte-identical.
Tests
LifecycleManagerTest.summaryWithToolTrace was fully vacuous: it hand-stubbed getLatestData("langchain:trace:llm") for a mock task whose id was TaskId("llm") — a name no real task has — so it asserted the reader against its own stub and stayed green through the entire life of the defect. Replaced with six cases that stub getAllElements() and never getLatestData: trace reaches the summary for a langchain task; multiple trace keys aggregate in write order; a non-langchain task omits it; langchain:cascade:trace: is ignored; a non-List result is ignored without a ClassCastException; no trace keys means no field. Each was mutation-checked against a reverted fix.
Two weak neighbours tightened: LlmTaskCoverageTest.resume_nonEmptyTrace_stored from a startsWith matcher to the exact key langchain:trace:openai:taskA (the writer half of the contract the reader tests now assert), and RestAgentEngineStreamingExtendedTest.onTaskCompleteIncludesToolTrace from assertTrue(data.contains("toolTrace")) — which also passes on a stringified or malformed payload — to a parsed-JSON shape assertion.
🧱 The in-turn tool context finally has a ceiling — maxToolContextTokens (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D6b (the bug half of D6; the cross-turn history-lossiness half is an improvement and stays out of scope). AgentOrchestrator.runToolCallLoop grew ONE currentMessages list by an AiMessage plus one ToolExecutionResultMessage per tool call per iteration, and nothing between the loop head and chatModel.chat ever inspected its size, character count or token count. The only in-loop ceilings were the iteration counter (default 10) and the dollar budgets. Per-result truncation did not save it: ToolResponseTruncator.truncateIfNeeded returns the result unchanged when limits == null, and Task.toolResponseLimits has no default — so on an ordinary agent every tool result entered the context in full. A tool-heavy turn could therefore blow past the model context window mid-loop and hard-fail with a provider 400, after the tool side effects had already fired.
The fix
New per-task config field maxToolContextTokens (default 60000) on LlmConfiguration.Task. Before each model call the loop meters the accumulated tool traffic and, while it exceeds the ceiling, evicts the oldest complete tool exchange — a requesting AiMessage together with all of its ToolExecutionResultMessages — until the traffic fits or only the most recent exchange remains.
The pairing invariant is the whole reason eviction is subtle. Dropping a result without its requesting
AiMessageleaves a danglingtool_call_id; dropping theAiMessagewithout its results leaves an unanswered tool call. Either half is itself a provider400— the eviction would cause the failure it exists to prevent. Exchanges are therefore located as[AiMessage, results…]index ranges and removed whole. Pinned by its own test.The most recent exchange is never evicted. When it alone exceeds the ceiling the request goes through unchanged (the model asked for those results and must see them) and the overrun is logged
still_over_budget. That case is whattoolResponseLimitsis for.Only tool traffic is counted. System / user / assistant-prose messages are never candidates — conversation history is governed by
maxContextTokens/conversationHistoryLimit, and nothing in this guard may drop a history message.
Token accounting extended, not duplicated
TokenCounterFactory.extractText returned "" for both tool message shapes — an AiMessage announcing tool calls has a null text(), and ToolExecutionResultMessage hit the default arm — so the entire in-turn tool context weighed zero tokens to every caller. Extended so an AiMessage contributes its prose plus each requested tool name and argument JSON, and a ToolExecutionResultMessage contributes its tool name plus the whole payload. The SAME TokenCounterFactory LlmTask uses for history windowing is injected into AgentOrchestrator and reused — one accounting rule for both halves of the request, not a second estimator. LlmTask.resolveModelName was widened from private to package-private so the orchestrator picks the same estimator (tiktoken for OpenAI/Azure, chars÷4 elsewhere) rather than a drifting copy.
Design decisions
Config field, not a constant (Golden Rule 1). The ceiling is an agent-designer knob with a sensible default;
-1/0disables it and restores pre-6.1 unbounded behaviour.Default 60000 preserves today's behaviour. High enough that no ordinary tool-using turn is touched — the eviction path is byte-for-byte inert below the ceiling, guarded by a test that compares the default-budget message lists against the guard-disabled (
-1) lists — and low enough to keep a runaway loop inside a 128k window after the system prompt, history and completion are added.Eviction over refusal. Refusing the turn would strand the side effects already committed by earlier tool calls; evicting the oldest results lets the loop finish with the freshest evidence. The loss is made observable rather than prevented.
No gap-marker message injected. A mid-transcript
SystemMessageis not portable across the twelve providers (several hoist system content to a top-level field) and aUserMessagewould fabricate a turn; the loss is surfaced through trace + metric + WARN instead.Per-attempt
IdentityHashMaptoken memo on the call stack — the orchestrator is an@ApplicationScopedsingleton and stays stateless; without the memo a 10-iteration loop retokenizes the first result ten times.Estimator resolution fails safe. An unresolved global-variable model type or an unknown model name that makes a provider tokenizer refuse to construct falls back to the approximate estimator rather than aborting the turn — a safety ceiling that throws is worse than an approximate one.
Operator-visible behaviour changes
A tool-heavy turn that used to hard-fail on a provider context-window
400now completes, dropping its oldest tool exchanges once in-turn tool traffic passesmaxToolContextTokens.New trace entry
tool_context_evicted(token counts before/after, exchanges + messages dropped,withinBudget), new countereddi.llm.tool_context.evictions(tagoutcome=within_budget|still_over_budget), and aWARNllm.tool_context.evictedwith conversation id + remediation hint. A steady stream signals: lowermaxToolIterations, settoolResponseLimits, or raisemaxToolContextTokens.Default budget is inert for normal turns. Agents whose in-turn tool traffic stays under 60000 tokens are byte-for-byte unchanged;
-1/0disables the guard entirely.
New field only; nothing removed. FAIL_ON_UNKNOWN_PROPERTIES=false keeps rolling deploys and ZIP imports safe both directions. No ExtensionDescriptor change (it exposes only uri), so no Manager UI work.
Tests
AgentOrchestratorToolContextBudgetTest (new): the end-to-end regression drives the real loop with a mock model that requests a tool every iteration and asserts every request reaching chatModel.chat is within the ceiling (fails today — unbounded growth); the pairing invariant across every captured request; byte-identical message lists under the default budget vs. the guard disabled; and trace/metric observability. Plus isolation tests of enforceToolContextBudget for oldest-first order, whole-exchange (multi-call) eviction, history never evicted, and the unfittable-newest report. TokenCounterFactoryTest gains a tool messages nest asserting names/arguments/payloads are now counted.
Every behavioural test was mutation-checked by reverting the corresponding production change and confirming failure: disabling the guard call (request 3 carried 636 tokens … <=500 expected: <true> but was: <false>); shrinking exchange ranges to the AiMessage alone and, separately, clearing only the AiMessage index on removal (orphan ToolExecutionResultMessage id=c1 … its requesting AiMessage was evicted without it); and reverting extractText to the pre-fix "" for tool messages (the payload is what fills the context window … expected: <true> but was: <false>, and the guard measuring 0 tokens so it never evicts).
The nine existing AgentOrchestrator*Test constructors and three historyBuilder-style call sites gained the new TokenCounterFactory argument.
Files: AgentOrchestrator.java, TokenCounterFactory.java, LlmConfiguration.java, LlmTask.java, docs/langchain.md, plus AgentOrchestratorToolContextBudgetTest (new), TokenCounterFactoryTest, and the nine AgentOrchestrator*Test constructor call sites.
Out of scope: the cross-turn half of D6 (tool messages absent from ConversationHistoryBuilder) — an improvement, not a bug; summarizing evicted tool results instead of dropping them; a provider-reported hard context limit feeding the default.
💸 maxBudgetPerConversation can finally bind — canonical tool names at the executor boundary (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D2. A built-in tool has two names, and the engine only ever carried one of them past the dispatch loop.
ToolCostTracker's price table and ToolCacheService's smart-TTL table are keyed on the eight whitelist slugs (websearch, pdfreader, calculator, …). The only production call path passed toolRequest.name() — the bare @Tool method name (searchWeb, extractTextFromPdf, calculate), because no priced tool class declares @Tool(name = …). Zero keys overlapped. Consequences, all live until now:
Every built-in priced at $0.00, so
maxBudgetPerConversationcould never trip no matter how much tool work a conversation did, andeddi.tool.costsnever carried a non-zero value.toolRateLimitsin its documented slug form was inert.docs/langchain.md,docs/security.mdanddocs/agent-father-langchain-tools-guide.mdall show{"websearch": 30}; the lookup only ever sawsearchWeb, so the entry was discarded anddefaultRateLimit(100) applied.Cache TTLs mostly fell through to the flat 300s default.
getSmartTTLsubstring-matches, which is why the mismatch went unnoticed for so long:getCurrentDateTimehappens to contain "datetime" and resolved correctly, whilecalculatedoes not contain "calculator" and did not.
The fix
New ToolNameResolver — stateless, all-static, an exhaustive and exact switch from the 16 built-in tool class simple names to their whitelist slugs. No substring fallback: a lookup that is right by coincidence for some inputs is worse than one that is wrong for all of them, because it never looks broken. AgentOrchestrator.buildToolSetup populates a toolCanonicalNames map (dispatch name → slug) from the already proxy-unwrapped toolClass; using tool.getClass() would yield …_ClientProxy and reintroduce the same defect in a new coat. The map rides on ToolSetup, so the live loop and the HITL resume path share it.
A new ToolInvocation(dispatchName, canonicalName, priceOverride) record carries both names through ToolExecutionService. The split is load-bearing:
Resolved from the canonical slug
Resolved from the dispatch name
per-call price, cache TTL
cache key, rate-limit bucket, metric tags, per-tool and per-conversation cost breakdown, failure logs
Canonicalising the cache key would collapse searchWeb, searchNews and searchWikipedia onto one entry and serve each other's results for identical arguments — a correctness bug traded for a naming tidy-up. The legacy String-first overloads of executeToolWrapped, trackToolCall and put are retained and delegate via ToolInvocation.of(name).
Design decisions
enforceBudgetis opt-in, defaultfalse(deployment fallbackeddi.tools.budget.enforce-by-default). Enforcement is deliberately not inferred frommaxBudgetPerConversationbeing set. That ceiling has been inert for its entire shipped life, so no stored config has ever had a tool call refused by it; switching enforcement on together with the prices would newly abort tool calls on live agents. Cost tracking runs regardless of the flag. Read throughConfigProviderin astatic final, not@ConfigProperty—AgentOrchestratoris constructed withnewbyLlmTaskand is not a CDI bean, so an injection annotation would never fire while looking configurable.Metric tags keep the dispatch name.
eddi.tool.calls{tool=…}andeddi.tool.costs{tool=…}could have moved to slugs to aggregate the three web searches into one series. They did not: every othertool-tagged meter in the module (eddi.tool.execution.success,eddi.tool.cache.hits.by_tool,eddi.tool.execution.duration) reports the dispatched method name, so moving these two would split the tag vocabulary in half, make the series un-joinable and break existing dashboards for a marginal analytical gain. Slug-level totals stay available as a PromQL sum.toolRateLimitsandtoolPricingaccept either name, dispatch first. A dispatch-name entry is the more specific statement ({"searchNews": 5}pins one operation) and wins over the tool-wide slug entry.Rate-limit buckets stay per dispatch name; only the limit value is slug-resolved.
{"websearch": 30}therefore yields three independent 30/min buckets rather than one shared allowance. Documented explicitly rather than left for an operator to discover.Operator prices are clamped at
Math.max(0.0, …).toolPricingvalues come from agent JSON; a negative one would credit the conversation and make any ceiling unreachable by construction.
Operator-visible behaviour changes
toolRateLimitsslug keys start binding. A config carrying{"websearch": 30}was previously ignored; from this release it applies — 30/min to each ofsearchWeb,searchNews,searchWikipedia. Method-name keys are unchanged. Review any agent that has been running underdefaultRateLimitwhile believing it was rate-limited.Built-in cache TTLs change from a mostly-flat 300s to the intended per-tool values.
datetimeoperations tighten:convertTimezone,addTime,listTimezonesandcalculateDateDifferencego 300s → 60s. Others loosen to their configured values:calculator300s → 7d,searchWeb/searchWikipedia300s → 1800s,webscraper300s → 1h,pdfreader/dataformatter/textsummarizer300s → 24h. One value loosens unintuitively:searchNewsused to substring-match thenewsentry at 600s and now takeswebsearch's 1800s.weather,getCurrentDateTimeandformatDateTimeare unchanged.eddi.tool.costsbecomes non-zero for priced built-ins andGET /llm/toolhistory/costsstarts reporting real numbers:websearch$0.001,webscraper$0.002,pdfreader$0.001,weather$0.0005 per call. Tag values are unchanged (see above), so dashboards keep working — a series that previously only ever recorded zero now carries a value.maxBudgetPerConversationcovers TOOL cost only and stays inert unlessenforceBudget: trueis added. LLM token spend remains run-scoped under the cascade'smaxCostPerRun; the two are not summed.
New config fields enforceBudget (Boolean) and toolPricing (Map<String, Double>) on LlmConfiguration.Task. No field removed; FAIL_ON_UNKNOWN_PROPERTIES=false keeps rolling deploys and ZIP imports safe in both directions. No ExtensionDescriptor change (it exposes only uri), so no Manager UI work.
Tests
ToolNameResolverTest (new) asserts all 16 classes against their real getSimpleName() — renaming a tool class without updating the resolver now fails here instead of silently reverting that tool to $0.00 and a 300s TTL — plus CalculatorTool_ClientProxy → null and exact-match-only guards.
AgentOrchestratorToolCostTest (new) is the load-bearing one: a real ToolCostTracker and a real ToolExecutionService driven through the actual dispatch loop. It is the only test that can show maxBudgetPerConversation is reachable at all. The pre-existing budget test stubs isWithinBudget to false on a mock, so its refusal comes from the stub and it passed just as happily against the broken behaviour; it is kept for gate wiring and now says so in its javadoc.
Every new behavioural test was mutation-checked by reverting the corresponding production change and confirming failure: pricing by dispatch name (expected: <0.001> but was: <0.0>), dropping the slug rate-limit fallback (expected: <7> but was: <100>), removing the enforceBudget gate (Wanted but not invoked: isWithinBudget), removing the negative-price clamp (expected: <0.0> but was: <-10.0>), keying the cache on the slug (expected: <2> but was: <1> distinct keys), dropping the canonical map on the HITL resume path (expected: <calculator> but was: <calculate>), and adding a substring fallback to the resolver (expected: <null> but was: <calculator>).
Verifications that would have gone vacuous after the signature change were repointed rather than left to pass silently: the four executeToolWrapped(anyString(), …) stubs in the orchestrator tests (an unmatched stub returns null and nulls out tool results without failing) and verify(costTracker, never()).trackToolCall(anyString(), anyString()), which after the change verified an overload production no longer calls.
Files: ToolNameResolver.java and ToolInvocation.java (new), ToolCostTracker.java, ToolCacheService.java, ToolExecutionService.java, AgentOrchestrator.java, LlmConfiguration.java, docs/langchain.md, docs/security.md, docs/agent-father-langchain-tools-guide.md, plus ToolNameResolverTest and AgentOrchestratorToolCostTest (new) and 7 updated test classes.
Out of scope: folding cascade LLM-token cost into ToolCostTracker so maxBudgetPerConversation becomes a true total ceiling (separate follow-up); rate-limit bucket sharing across a tool's operations; the <=/pre-check budget boundary; getSmartTTL's substring fallback, which still serves unmapped http/mcp/a2a tool names.
⏳ Cache entries finally expire — CacheImpl honours per-entry TTLs (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D5b, the follow-up to D5. Every TTL-bearing ICache overload in CacheImpl dropped its lifespan argument and delegated to the untimed variant. Caffeine's standard builder has no per-entry expiry, so the wrapper simply threw the number away. Consequences, all live in production until now:
ToolCacheService's entire smart-TTL table was decorative.weather300s,websearch1800s,calculator7 days — every one of those values was computed, passed tocache.put(key, value, ttl, unit)and discarded. Tool results were removed by 10 000-entry size eviction alone, so a stale or poisoned result could be served indefinitely.GET /llm/tools/cache/ttl/{toolName}reported numbers that governed nothing.PaginatedResponseStore's documented "15-minute TTL" never existed. Pages lived until the cache filled.A2A replay nonces had no expiry at all.
NonceCacheServicecarried a comment claiming expiry was "configured externally via CaffeineexpireAfterWriteinapplication.properties". There is no such configuration anywhere in the repo.
The fix
CacheFactory now builds every cache with Caffeine.expireAfter(Expiry) instead of leaving expiry unconfigured (size-only caches) or using expireAfterWrite(ttl) (TTL caches). That is what exposes Caffeine's policy().expireVariably() view, and CacheImpl writes every TTL-bearing overload through it. Two entries in one cache can now carry two different lifespans.
New
WriteExpiry(package-private,engine/caching) — theExpiryimplementation.never()for size-only caches (an entry written without a lifespan still never expires on its own);of(ttl)for TTL caches (identical observable behaviour toexpireAfterWrite).expireAfterReadreturnscurrentDuration, so expiry stays expire-after-write and a read never keeps an entry alive.CacheImpl—put/putIfAbsentroute throughVarExpiration, which is atomic and returns the replaced value in one operation.putAllapplies the lifespan per entry. Bothreplaceoverloads do the replace and thensetExpiresAfteron success — not atomic, and the javadoc says so, because Caffeine has no replace-with-duration primitive. A negative lifespan means unlimited per theICachecontract and is translated into an effectively infinite duration; forwarding it would make Caffeine throwIllegalArgumentException. The false javadoc aboutCachedResult.expiresAtis gone.NonceCacheService— asks forgetCache(name, maxAge + clockSkew + 60s)(390s with the defaults) instead of the size-only cache, and the fictional comment aboutapplication.propertiesis deleted.CacheFactory.CACHE_SIZES—paginated-tool-responsespinned at 1 000 (entries are whole oversized tool responses; the TTL is now the primary eviction path and the cap only bounds memory) andnonce-replay-protectionraised to 100 000. The 1 000 default was a security hole: on a busy A2A endpoint a nonce could be size-evicted while its timestamp still passed the freshness check, re-opening the replay window. 100 000 covers ~300 signed requests/second sustained across the whole ~5.5-minute window.
Design decisions
Variable expiry in the factory, not a per-cache workaround. The alternative — one Caffeine instance per distinct TTL value, keyed like the existing
name:ttl=…scheme — would have giventool-resultsnine separate caches with nine separate size budgets and no shared eviction. Per-entry expiry is what theICachecontract already promised.expireAfterreplacesexpireAfterWrite, it is not added to it. Caffeine throwsIllegalStateExceptionatbuild()if both are configured, which would have failed the@PostConstructofSlackEventHandler,ChannelTargetRouterandNonceCacheServiceon startup.CacheFactoryTestpins this.maxIdleTimeon the two six-argument overloads is explicitly unsupported. Caffeine cannot combine a per-entry write duration with a per-entry idle duration. Thelifespanis honoured (previously both were discarded) and the javadoc states the limitation. These overloads have no callers in EDDI.A
CacheImplover a cache with no variable expiry degrades instead of throwing. Only reachable by constructing the class directly rather than throughCacheFactory; the constructor logs a WARN naming the cache.15 minutes is comfortably longer than a tool-calling loop. The
PaginatedResponseStoreTTL runs fromstore(), and a loop is bounded by the LLM request timeout — seconds to a couple of minutes.FetchToolResponsePageToolalready returns "It may have expired (15 minute TTL)" for an unresolvableresponseId; that message is finally true.
⚠️ Operator-visible behaviour change
Tool results that were cached forever now expire on the smart-TTL table. Cache hit rate will fall and real tool invocations — with their external API spend — will rise. The effect is sharpest for the short-TTL tools:
weather(300s),news(600s),websearch(1800s), and anything with no table entry (300s default).calculator/pdfreader/dataformatter/textsummarizer(24h–7d) barely move. Watcheddi_tool_cache_hits_total/eddi_tool_cache_misses_total.This is a correctness fix, not a regression. Serving a 5-day-old weather reading was never intended behaviour.
Paginated tool responses now expire 15 minutes after they are stored. An LLM that sits on a
responseIdpast that gets the existing "may have expired" error instead of a page.A2A replay nonces now expire ~6.5 minutes after first use instead of living until size eviction. Steady-state nonce memory drops; replay protection gets stronger, not weaker, because the cache is also 100× deeper.
No configuration change is required or available. All three TTLs were already the documented intent; nothing new is exposed.
⚠️ GraalVM native image — unverified. Caffeine picks a generated
BoundedLocalCachesubclass per feature combination, andquarkus-caffeineregisters a fixed list of those classes for reflection at build time. Variable expiry selects a different generated family that may not be on that list; if so,Caffeine.build()fails at first cache creation and every@PostConstructcallinggetCache(...)dies at startup. This cannot fail in JVM mode and is invisible tomvnw testand JVM-mode CI. Logged as a Phase 3 blocker inplanning/native-image-migration.md— the first native smoke test must exercise bothgetCache(name)andgetCache(name, ttl).
Tests
CacheImplTest's seven *_delegatesToPut cases asserted only that a value was readable immediately after a TTL put — true with or without the fix, which made them a codification of the defect. They are replaced by 26 ticker-driven cases covering all seven overloads: entry gone past the lifespan, per-entry (not cache-wide) expiry, expire-after-write rather than after-access, negative lifespan means unlimited, and the degraded no-variable-expiry path. ToolCacheServiceTest gains a Smart TTL is load-bearing group wired to a real CacheImpl over a ticker — a 60s datetime result dies at 61s while a 7-day calculator result does not — which is what finally makes TOOL_TTL_SECONDS more than a lookup table. PaginatedResponseStoreTest runs against a real cache too and winds past 901s. CacheFactoryTest non-vacuously protects the one expiry path that already worked. NonceCacheServiceTest captures the TTL and asserts it exceeds maxAge + clockSkew. New shared test seam: TestCaches (FakeTicker + a production-shaped cache).
Every behavioural test above was mutation-checked: with variableExpiry forced to null (the old behaviour) 10 CacheImplTest, 5 ToolCacheServiceTest and 2 PaginatedResponseStoreTest cases fail; forwarding a negative lifespan straight to Caffeine fails all 5 negative-lifespan cases with IllegalArgumentException; dropping expireAfter from getCache(name) fails CacheFactoryTest.perEntryTtlIsHonoured; making expireAfterRead reset the clock fails 11 cases; reverting NonceCacheService to the size-only cache fails both new nonce cases.
Docs
docs/security.md and docs/langchain.md said per-tool TTLs were "computed but not enforced" — corrected. planning/native-image-migration.md gains the Caffeine variable-expiry caveat above.
🔒 Scope the tool-result cache per identity — one user's tool result no longer reaches another (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D5, the highest-severity item in the langchain4j remediation backlog. ToolCacheService built its cache key as toolName + ":" + arguments and nothing else. That is a single global namespace: if user A asked getAccountBalance with {"account":"main"} and user B later made the byte-identical call, B was served A's cached result verbatim, with no execution and no authorization check in between. Every agent with enableToolCaching (default true) and a tool whose output depends on who is asking was affected.
The fix
Every cache key now starts with a scope tag:
user
u:<first 32 hex of SHA-256(userId)>
only the same authenticated user (the default)
conversation
c:<conversationId>
only the conversation that produced the entry
global
g
everyone — opt-in only
Resolution per tool call is task.toolCacheScopes[<tool>] → task.defaultToolCacheScope → user. Both are new lenient-String fields on LlmConfiguration.Task; the recognized tokens live in the new ToolCacheScope enum (fromConfig / resolve, following the existing CascadingStrategy pattern). Unrecognized or misspelled tokens fall through to user rather than failing the agent load — a typo must never silently widen a tool's audience.
Fail-closed identity handling. user scope with no usable user id degrades to the narrower c:<conversationId> partition. If neither a user id nor a conversation id is available, resolveScopeTag returns null and the cache is bypassed entirely — no get, no put. null is never turned into "" or "unknown": that would recreate exactly one shared partition for every anonymous request, i.e. the same bug under a new name. A new eddi.tool.cache.bypassed counter (tagged tool) makes the bypass visible on /q/metrics.
Changed files
ToolCacheScope(new,modules/llm/tools) — the recognized scope tokens plus lenient parsing and the per-tool → task-default →USERresolution chain.ToolCacheService—get/put/invalidatenow take the scope tag as their first argument. The old unscoped signatures were deleted, not overloaded: an overload lets a future caller silently reintroduce the global key.resolveScopeTag(...)is a static on this class so the orchestrator can build the tag without injecting the cache.ToolExecutionService.executeToolWrapped— takescacheScopeTag(3rd parameter, next to the other cache-key inputs) and gates both the read and the write on it being non-null.AgentOrchestrator.executeSingleToolCallResult— a four-line insertion above theexecuteToolWrappedcall resolves the tag fromtask+memory.getUserId()+conversationId. Because the live tool loop and the HITL resume path already share this one method, both are covered by that single change and no call site or method signature in the orchestrator moved.CacheFactory—tool-resultswas absent fromCACHE_SIZESand silently got the 1 000-entry default. Raised to 10 000: scoping multiplies the keyspace by the number of active users, and 1 000 would thrash.CacheImpl— javadoc and inline comment corrected. They claimed the TTL overloads were safe because "the ToolCacheService already tracks expiry internally viaCachedResult.expiresAt". No such field exists — the wrapper only recordscachedAt, for a debug log. The comments now state plainly that the TTL argument is discarded and size-based eviction is the only eviction strategy.Docs —
security.md(both the "SHA-256 key" and "within the same conversation" claims were factually false),langchain.md,agent-father-langchain-tools-guide.md,metrics.md,monitoring/monitoring-guide.md.
Design decisions
No name-based "pure tool ⇒ GLOBAL" default table. Agent-mode tool names are
@Toolmethod names (calculate,getCurrentWeather), not class names — the existingcontains-matcher ingetSmartTTLalready fails to recognize them. Guessing purity from a name would hand out cross-user reuse by accident.globalis opt-in via config only.No
TENANTscope.TenantQuotaServiceis still a single-tenant stub; a tenant partition today would be indistinguishable fromglobal.No global kill-switch flag. The opt-out is per tool:
"toolCacheScopes": {"<tool>": "global"}. A deployment-wide "disable scoping" switch is a foot-gun that re-opens the leak for every tool at once.Scope tag first in the key. Entries belonging to different identities cannot collide regardless of tool name or arguments.
The user id is hashed, not stored. Cache keys are visible in heap dumps and debug logs; 32 hex characters of SHA-256 is enough to partition without carrying the identifier around.
⚠️ Operator-visible behaviour change (deploy-time)
Cross-user tool-result sharing stops. This is the point of the change, and it is not configurable away except per tool.
Cache hit rate will drop and tool cost will rise. Entries that were previously shared by the whole deployment are now per user. Expect more misses, more outbound tool/API calls and a higher external spend, proportional to how much cross-user reuse the agent was silently relying on. Watch
eddi_tool_cache_hits_total/eddi_tool_cache_misses_totalafter deploying.Any agent that depended on cross-user reuse must opt that tool in explicitly with
"toolCacheScopes": {"<tool>": "global"}— and only where the result genuinely does not depend on the caller.Existing
tool-resultsentries are dropped. The cache is in-process and the key format changed, so the first requests after a restart are misses either way.New meter:
eddi_tool_cache_bypassed_total{tool="…"}. A sustained non-zero rate means tool calls are running with neither a user id nor a conversation id and are paying full tool cost every time — fix the caller, do not widen the scope.No config migration needed.
toolCacheScopesanddefaultToolCacheScopeare optional; stored configs without them behave asuserscope.
Not in scope (deliberately)
Cache TTLs still do not expire anything (next item, D5b).
CacheImpldiscards the lifespan argument, so scoped entries are removed by size eviction only. Nothing in this change or its tests implies otherwise.ToolRateLimiterremains cross-user. Its token buckets are keyed by tool name alone, so one user can exhaust another's budget for a tool. Same class of defect, separate fix — flagged here rather than folded in.
Tests
ToolCacheServiceTest gained the headline regression test differentUsers_produceDifferentKeys (same tool, same arguments, two identities ⇒ two distinct captured keys), a read-side twin (get_otherUser_missesEntry), the twelve-case scope-resolution table, and five null-scope bypass tests. Two pre-existing vacuous tests were replaced: longArgs_sha256Key asserted only startsWith("calculator:") and length < 200 — true for any truncation — and now asserts the exact key against an independently computed SHA-256 oracle; shortArgs_readableKey was a byte-identical duplicate of put_storesInCache and now asserts the full scopeTag|tool:args shape. A UnscopedApiRemovedTests tripwire fails if the deleted global-key signatures ever come back. ToolExecutionServiceTest covers the bypass at the wrapper level; AgentOrchestratorCoverageTest captures the scope tag the orchestrator actually passes for the default, global and no-identity cases.
Every behavioural test was mutation-checked: reverting the key scoping, replacing null with a placeholder, dropping either null-guard, removing the blank-user-id degradation, dropping the userId at the orchestrator, and ignoring the per-tool scope map each produce failures (e.g. expected: not equal but was: <getAccountBalance:{"account":"main"}> and expected: <fresh result> but was: <SOMEONE ELSES RESULT>).
🧹 Delete the dead enableParallelExecution config and the parallel tool machinery (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D10. Two LLM task knobs and ~250 lines of the machinery behind them were removed. Nothing in src/main read any of it.
What was dead, and why wiring it was not an option
LlmConfiguration.Task declared enableParallelExecution (default false) and parallelExecutionTimeoutMs (default 30000), with getters and setters and zero production readers — the only Java references outside the declaration were POJO round-trip assertions in two test classes.
The machinery they were meant to switch on was ToolExecutionService.executeToolsParallel / executeToolsParallelAndWait, which took (Object[] toolInstances, Method[] methods, Object[][] args) and fanned out over a ten-thread ExecutorService. That signature cannot be fed from the live dispatch path. Agent-mode tools are invoked by langchain4j through ToolExecutor.execute(ToolExecutionRequest, memoryId), which yields a (name, jsonArguments) pair; for MCP, A2A and dynamic tools there is no Java Method behind the tool at all, so no (instance, Method, Object[]) triple exists to pass. This was never a wire-up away from working — it was a second, incompatible execution model that had been left in the tree.
Concurrent tool calls remain a reasonable future feature. They belong at the AgentOrchestrator dispatch loop, batching ToolExecutionRequests, not in a reflection path.
Deleted
LlmConfiguration.Task— both fields and their four accessors. The class javadoc now carries a historical note explaining the removal and the back-compat guarantee.ToolExecutionService—executeToolsParallel,executeToolsParallelAndWait, and the full cascade they were the only callers of:executeTool(Object, Method, Object[], String, ToolExecutionTrace),serializeArguments, theIJsonSerializationinjection, the ten-threadExecutorServiceand its@PreDestroy shutdown().executeToolWrappedis now the single entry point, and the class no longer allocates a thread pool per bean for nothing.Five meters:
eddi.tool.execution.parallel(the only one registered eagerly, at@PostConstruct) plus…parallel.count,…parallel.duration,…parallel.timeoutand…parallel.error, all four of which were created lazily at increment sites that could not be reached and therefore never appeared on/q/metricsat all.The corresponding rows in
docs/metrics.mdanddocs/monitoring/monitoring-guide.md, and the "Parallel Execution" panel (id: 33) fromgrafana-data/dashboards/eddi-operations.json— an addition to the original scope, found while tracing the metric names. The two surviving panels in that dashboard row were widened fromw:8tow:12to fill it.
Design decisions
The cascade was taken deliberately, not stopped at the two named methods. Half-deleting would have left
executeTool,serializeArguments, an injected serializer and a thread pool alive with no production caller — the worst of both outcomes. The trade was ~15 reflection-based tests across three classes; every one of them exercised a path production cannot enter (see below).@PostConstruct init()is kept, now registering nothing and only logging. Every remaining meter in the class is per-tool (taggedtool) and created lazily on first use, so there is nothing left to pre-register; the startup log is pre-existing behaviour and removing it is an unrelated change.synchronizedstays onToolExecutionTrace.addToolCall/addFailedToolCall. Its javadoc justified the keyword by namingexecuteToolsParallelas the concurrent writer, so it would have become a comment pointing at deleted code. The rationale is restated on its true footing: the trace is a shared mutable accumulator that publishes no happens-before edge of its own; today's tool loop writes single-threaded, so the guard is defensive rather than load-bearing, and the cost of keeping it is nil against silent corruption of an audit artefact.No
ExtensionDescriptorchange was needed —LlmTask.getExtensionDescriptor()only ever exposeduri.
Stored configurations stay valid
Every mapper that reads an LLM configuration — REST, Postgres JSONB, @PersistenceMapper, and the Mongo BSON mapper — is built from SerializationCustomizer.configureObjectMapper, which sets FAIL_ON_UNKNOWN_PROPERTIES=false. A langchain.json already in MongoDB carrying "enableParallelExecution": true still deserializes; the key is ignored on read and dropped on the next save.
New LlmConfigurationParallelExecutionLegacyFieldsTest is the tripwire for that invariant: it loads a legacy document through both the JSON and the Mongo BSON mapper and asserts the surrounding live fields (defaultRateLimit, toolRateLimits, maxToolIterations, …) still populate. Verified non-vacuous by flipping FAIL_ON_UNKNOWN_PROPERTIES to true locally — all three deserialization tests then fail with UnrecognizedPropertyException. The same class asserts via java.beans.Introspector that Task exposes no enableParallelExecution / parallelExecutionTimeoutMs bean property, so a getter/setter pair cannot quietly put the knob back on the REST contract.
Tests
Removed: the
ParallelTests/ParallelAndWaitTests/ParallelExecnested classes and everyexecuteTool(Object, Method, …)andserializeArgumentstest acrossToolExecutionServiceTest,ToolExecutionServiceBranchTestandToolExecutionServiceExtendedTest, plus the twoshutdown()tests. None of them covered anything still live — they were the only callers of those methods anywhere outside the class, so they exercised a path production can never enter and stayed green whether the "feature" worked, was broken, or was disabled. Two were vacuous even on their own terms:parallelTimeoutpassed zero tools (new Object[0]), soallOf()completed immediately and theTimeoutExceptionbranch it claimed to cover was never taken; andconcurrentToolsShareTraceWithoutCorruptionspent 50 rounds × 32 tasks of CI wall-clock defending against a shared-trace race that no production code path can produce. The four POJO assertions ongetEnableParallelExecution() == false/getParallelExecutionTimeoutMs() == 30000Lpinned defaults nothing read.Added (fails before the deletion, passes after):
ToolExecutionServiceTest.ParallelMachineryRemovedTests— noexecuteToolsParallel*and noexecuteToolon the public API, and no meter whose id starts witheddi.tool.execution.parallelafterinit(). Mutation-checked by restoring the oldToolExecutionService: all three fail, the meter one reportingexpected: <[]> but was: <[eddi.tool.execution.parallel]>.LlmConfigurationParallelExecutionLegacyFieldsTestmutation-checked by restoring the oldLlmConfiguration:rewriteDropsRemovedParallelKeysandtaskExposesNoParallelExecutionBeanPropertyboth fail.
Operator-visible changes
Five Prometheus series disappear rather than reading zero:
eddi_tool_execution_parallel_total,…_parallel_count_total,…_parallel_duration_seconds,…_parallel_timeout_total,…_parallel_error_total. Only the first was ever exported, and it could only ever be0. Any dashboard or alert rule referencing them outside this repo will show "no data" — no signal is lost.No agent behaviour changes. Both deleted fields were inert at every value, so no configuration executes differently.
Follow-up required in EDDI-Manager (different repo) — the one user-visible risk
The Manager's LLM editor renders a "Parallel Tool Execution" checkbox (i18n key llmEditor.parallelExecution, data-testid enable-parallel-execution) plus a conditional parallelExecutionTimeoutMs input. That checkbox is now backed by nothing on the server.
Worse than inert: because the mapper uses JsonInclude.NON_NULL and the field no longer exists, the key is accepted on POST, silently dropped, and never echoed back — so the checkbox visibly resets to unchecked on every reload. Before this change it at least persisted while doing nothing. Removing the control in EDDI-Manager should land in the same release.
Yes — the checked-in Manager bundles in this repo also carry it. Both src/main/resources/META-INF/resources/assets/index-B36D6B8M.js (the one manage.html loads) and index-CHgQ1fX-.js (a second bundle graph, reachable only through chunk imports such as cssMode-BPLGavJr.js) contain one enableParallelExecution occurrence and one data-testid="enable-parallel-execution" each. They are built artifacts and were deliberately not hand-patched — the fix is to remove the control in the EDDI-Manager source, rebuild, and re-vendor both bundles. Same follow-up shape as the injectionStrategy <select> recorded in the D8 entry below.
📝 State plainly that TenantQuotaService.recordCost has no callers (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D9, part 3 of 3 — documentation only, no behaviour change.
TenantQuotaService.recordCost(tenantId, cost) is the post-call half of the monthly cost budget. It has zero production callers — the only references outside its own declaration are in TenantQuotaServiceTest. Its pre-call twin checkCostBudget is wired (AgentOrchestrator calls it before each LLM turn), but it reads ITenantQuotaStore.getMonthlyCost, which nothing ever writes. So eddi.tenant.quota.max-monthly-cost-usd currently cannot deny anything, at any value, on any backend — while the code reads as if cost budgets work.
Decision: keep the method, document the gap loudly — do not delete it, and do not wire it here.
Not deleted, because removing it would take away the seam without taking away the gap:
checkCostBudgetstays wired,ITenantQuotaStore.tryAddCoststays on the interface, all three stores implement it, and it is now covered byTenantQuotaStoreParityTest. A reader would be left with a half-system and no marker.Not wired, because there is nothing meaningful to meter yet. Built-in tool executions are priced at $0.00, and there is no token-cost metering for LLM turns at all — wiring today would add write load and record zeros. The candidate call sites are the
ChatResponse-holding seams tracked as C5 inplanning/manager-coverage-backend-design.md, whose own notes named the Mongo E11000 (fixed in part 1 of this item) as its blocker. That blocker is now gone.
Both recordCost and checkCostBudget carry javadoc saying this outright, including the two things that must land first.
🐛 Deny tenant quotas at the limit on the Postgres store (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D9, part 2 of 3 — on PostgreSQL the daily conversation cap and the per-minute API-call cap were never enforced at all.
The defect
Both increment methods had a fast path (UPDATE … WHERE window = ? AND counter < ? RETURNING counter) and, on a miss, a fallback INSERT … ON CONFLICT (tenant_id) DO UPDATE SET counter = CASE WHEN window_start < ? THEN 1 ELSE counter END … RETURNING counter, guarded by if (rs.next() && rs.getInt(1) <= limit).
When the window was not stale, the CASE took its ELSE branch and returned the counter unchanged — i.e. exactly limit, because the fast path only ever increments while counter < limit. limit <= limit is true, so the store returned OK. And because that path never increments, the counter stayed pinned at limit and every subsequent request also returned OK. The cap leaked without bound; only the fast path's own < limit was ever consulted, and it had already failed.
The two unit tests covering this stubbed the fallback to return 11 against a limit of 10 (and 61 against 60) — values production can never produce at that point — so they asserted the right verdict through an impossible fixture and stayed green while the quota did nothing.
The fix
The ON CONFLICT … DO UPDATE now carries WHERE tenant_usage.day_start < ? (resp. minute_start), and resets the counter to zero instead of one; the increment statement is then re-run. This mirrors the Mongo store exactly:
conditional increment (fast path, the only statement in steady state);
materialise-or-roll — creates the row or resets an expired window, and is a strict no-op for a row whose window is still current;
retry the conditional increment.
The cap is now enforced by a single predicate, counter < limit, in every path. A request at the limit with a current window falls through steps 2 and 3 and is denied, and limit = 0 denies without a special case.
tryAddCost got the same materialise-first treatment, which also fixes a latent bug in its INSERT: it seeded day_start / minute_start with a raw wall-clock timestamp rather than a truncated window start. A row first created by a cost write therefore had day_start greater than every truncated value the increment paths compare against — so neither day_start = ? nor day_start < ? could ever match and that tenant's conversations would have been denied until the next UTC day.
Tests
TenantQuotaStoreParityTest(new, Testcontainers) — runs one boundary sequence through all threeITenantQuotaStoreimplementations (in-memory, Mongo, Postgres) and asserts identical verdicts: allowed below the limit, denied at it, denied forlimit = 0, unlimited forlimit < 0, cost denied once spend reaches the budget, and all three counters coexisting. 18 tests. Against the reverted Postgres store exactly the threepostgres:rows fail (conversationsDenyAtLimit,apiCallsDenyAtLimit,zeroLimitDeniesEverything); the Mongo and in-memory rows stay green, which is the point — this is the only shape that catches a divergence.PostgresTenantQuotaStoreTest— the two impossible fixtures are replaced with faithful ones, plus a structural guard per method asserting the materialise-or-roll SQL carries its expired-windowWHERE. The behavioural at-limit proof cannot live in a mocked JDBC layer (whetherDO UPDATEfires is Postgres's decision, not the code's), so the structural guard is what fails there against the old SQL; the parity test carries the behaviour.PostgresTestBase.createDataSourceInstance()widened fromprotectedtopublicso the parity test can reuse the single shared container instead of starting a second one.
Operator note — behaviour change
Deployments running eddi.tenant.quota.enabled=true on PostgreSQL were not enforcing max-conversations-per-day or max-api-calls-per-minute at all. They will now start returning quota denials (HTTP 429) once a tenant reaches its configured limit. If limits were tuned against the leaky behaviour, they will need revisiting — this reads as a regression but is the cap doing its job for the first time. MongoDB and in-memory deployments are unaffected by this part.
🐛 Keep all tenant quota counters in one tenant_usage document (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D9, part 1 of 3 — MongoTenantQuotaStore was raising E11000 duplicate-key errors on a live request path, not silently mis-counting.
The defect
tenant_usage holds one document per tenant carrying all three counters, and it has a unique(true) index on tenantId. But all three mutating methods issued an upsert whose filter also pinned a rolling window:
tryIncrementConversations→and(tenantId, dayStart >= …, conversationsToday < limit)tryIncrementApiCalls→and(tenantId, minuteStart >= …, apiCallsThisMinute < limit)tryAddCost→and(tenantId, costMonth == …)
Whenever the extra predicate did not match — which is always the first time a second counter is touched, because the document written by the first method has no minuteStart / costMonth field at all — the upsert found nothing, tried to insert a second document for the same tenant, and the unique index rejected it. The exception propagates out of the store.
Concretely, with eddi.tenant.quota.enabled=true and both max-conversations-per-day and max-api-calls-per-minute set, ConversationService.acquireConversationSlot() succeeded and the very next acquireApiCallSlot() threw — a 500 on a user request, not a quota denial. (Earlier notes described this as "monthly cost silently reads 0.0 forever". That was true before the unique index was added; since then the failure mode is the hard error above.)
The design
Every write that can insert now filters on the unique-index key and nothing else. Each operation is:
Fast path — one conditional
findOneAndUpdate(window current AND counter < limit), no upsert. In steady state this is the only round trip, so the hot path is unchanged.Materialise —
ensureUsageDocument(tenantId), the single write in the class allowed to insert. Filter iseq("tenantId", …); all counters are seeded together via$setOnInsert, so whichever operation runs first for a tenant, the other two find their fields present.Roll + retry — reset an expired window to zero (conditional, no upsert), then re-run the fast path.
Steps 2–3 only run when the fast path misses (first call for a tenant, window rollover, or a real limit breach). Because the roll resets to zero and step 3 does the counting, the limit is enforced by exactly one predicate — counter < limit — in every path; a limit of 0 therefore denies without a special case, matching InMemoryTenantQuotaStore.
The stale-window filter is or(exists(field, false), lt(field, windowStart)), which doubles as repair for legacy documents written by the previous code that are missing a window field entirely.
Migration hazard, handled
createIndex(tenantId, unique=true) was called unguarded in the CDI constructor. Any instance that ran the pre-index build with quotas enabled may already hold duplicate tenantId rows, in which case index creation fails and the whole application fails to start. It is now wrapped: on failure it logs an ERROR naming the collection and the remediation, and continues. Safe, because correctness no longer depends on the index — it is a safety net against upsert races, not a precondition.
Tests
MongoTenantQuotaStoreContainerTest(new, Testcontainers) — 13 tests. Against the reverted store, 10 of them fail withE11000 duplicate key error collection: eddi_test.tenant_usage index: tenantId_1. Covers: conversations→api-calls, conversations→cost, interleaved three-counter accounting, at-limit denial, zero limit, day/minute window rollover, stale cost month, and a legacy document missing its window fields.MongoTenantQuotaStoreTest.UpsertFilterDiscipline(new, pure unit, no Docker) — renders every captured filter to BSON and asserts eachupsert(true)write is keyed ontenantIdalone. Against the reverted store it fails withexpected: <[tenantId]> but was: <[$and]>.TryAddCost.staleMonthwas renamed tousageDocumentDisappears: its premise (both conditional updates return null) is no longer "stale month" — it is now a concurrentresetUsage, which must not cost the caller a denial.
Operator note
No configuration change. The subsystem ships dormant (eddi.tenant.quota.enabled=false, all limits -1), so only deployments that opted in were affected — for those, this turns 500s back into correct allow/deny.
🧹 Delete dead RAG injectionStrategy / contextTemplate config (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D8 — two RAG configuration knobs on LlmConfiguration that never had any effect, removed rather than implemented.
What was dead, and how it was verified
LlmConfiguration.KnowledgeBaseReference declared injectionStrategy and contextTemplate; LlmConfiguration.RagDefaults declared injectionStrategy (defaulting to "system_message"). Every one of them was write-only:
RagContextProvider.retrieveContextreads exactly two override fields off aKnowledgeBaseReference/RagDefaults—maxResultsandminScore. It never callsgetInjectionStrategy()orgetContextTemplate().LlmTask(the provider's only production caller) unconditionally doessystemMessage += "\n\n## Relevant Context:\n" + ragContext. There is no branch, so no strategy value could have changed anything.Formatting is fixed in
RagContextProvider.formatRagContext(### Source: <kb>headings); no templating engine is reachable from that path, socontextTemplatehad nowhere to be applied even in principle.Grep across
src/mainreturns zero reads of either accessor. No REST resource, noExtensionDescriptor(LlmTask.getExtensionDescriptor()registers only theuriConfigValue), no MCP tool, no migration, no sample agent config insrc/main/resources/initial-agentsordocs/agent-configsreferenced them. The only Java references were POJO getter/setter round-trip assertions inLlmConfigurationTestandLlmConfigurationModelsTest— tests that assert a setter stores what you set and would stay green under any behaviour whatsoever.
Design decision — delete, not wire
Wiring injectionStrategy would have meant shipping user-message injection, and that is not a free "finish the feature" change: RAG context routed to the system message is not counted against maxContextTokens (ConversationHistoryBuilder budgets only the assembled history), while context routed into the user message is. Turning the knob on would therefore silently change how much conversation history survives windowing for anyone who had already saved user_message, with no config edit on their part. contextTemplate was worse still — it had zero reads and zero UI, and honouring it would have required introducing a templating engine into RagContextProvider plus inventing the {{context}} semantics the field name implied. Deleting a knob nothing honours is cheaper and more honest than inventing the semantics it advertised. The intended behaviour (system-message injection) is exactly what already happens, so nothing observable changes for any agent.
⚠️ Follow-up required in the EDDI-Manager repo (not fixable here)
The Manager UI bundles checked into this repo — src/main/resources/META-INF/resources/assets/index-B36D6B8M.js and index-CHgQ1fX-.js, loaded by manage.html — render an Injection <select> (System Message / User Message) bound to injectionStrategy, for both knowledgeBases[] and ragDefaults. Relevant strings: the i18n key llmEditor.injectionStrategy, the value expression injectionStrategy??"system_message", and the two onChange writers injectionStrategy:<e>.target.value. contextTemplate has no UI presence in either bundle.
These bundles were deliberately not edited. They are minified Vite build artifacts (~7 MB, single ~319 000-character lines, mangled identifiers) produced by a separate repo; hand-patching a built artifact is both unsafe and immediately undone by the next Manager release. The dropdown must be removed in the EDDI-Manager source and the rebuilt bundles re-committed here.
Until that lands, the shipped dashboard still offers the control, and its value is discarded on save — the same thing that happened before this change (the backend ignored it then too), so this is a pre-existing cosmetic defect made no worse, not a regression introduced here. Agent designers should ignore the Injection dropdown.
Stored-configuration compatibility
No migration, no operator action. Every mapper that deserializes an LLM configuration is built from SerializationCustomizer.configureObjectMapper, which sets FAIL_ON_UNKNOWN_PROPERTIES=false — the REST/CDI mapper (customize), the @PersistenceMapper used for Postgres JSONB (PersistenceMapperProducer), and the MongoDB BSON mapper (PersistenceModule.buildMongoClientOptions) all share that one static recipe. Existing langchain.json documents in MongoDB/Postgres, and agent ZIPs exported before this change, keep loading unchanged; the leftover keys are ignored on read and dropped on the next save.
Regression test
New LlmConfigurationRagLegacyFieldsTest (src/test/java/ai/labs/eddi/modules/llm/model/), 3 tests, pure-unit (no Testcontainers), mirroring the production mapper wiring the way HitlTimeoutPolicySerializationTest does. It deserializes a realistic stored langchain.json carrying all three removed keys across both RAG modes:
storedConfigWithRemovedRagKeysDeserializesViaJsonMapper— loads through the REST / Postgres-JSONB /@PersistenceMapperrecipe, with the surviving fields (name,maxResults,minScore,enableWorkflowRag) asserted intact.storedConfigWithRemovedRagKeysDeserializesViaBsonMapper— the raw document is encoded to BSON first, so the decoder sees exactly what an existingllmscollection holds.rewriteDropsRemovedRagKeys— re-serializing a legacy config must not resurrect either key, and the result must still round-trip (so the twoassertFalses cannot pass on an empty/broken rewrite).
This class is the tripwire for the whole deletion. (1) and (2) fail the moment anyone flips FAIL_ON_UNKNOWN_PROPERTIES on the shared recipe — which would break every stored configuration, not just RAG ones. All three were mutation-checked: flipping that flag to true fails all three with UnrecognizedPropertyException: Unrecognized field "injectionStrategy" ... (3 known properties: "minScore", "maxResults", "name"); re-adding the fields to KnowledgeBaseReference fails (3) with AssertionFailedError: injectionStrategy was deleted because nothing honoured it; a config rewrite must not resurrect it; re-adding only RagDefaults.injectionStrategy fails (3) the same way.
The pre-existing POJO round-trips in LlmConfigurationTest / LlmConfigurationModelsTest were trimmed to the surviving fields — they were the vacuous coverage that let these two fields look tested for as long as they did.
Documentation
docs/rag.md never documented either field, so nothing was wrong there — but it also never said where retrieved context goes, which is the gap that made an unread "injection strategy" knob plausible in the first place. Added a Context Injection subsection stating that vector-RAG context is always appended to the system message under ## Relevant Context: with no per-KB or per-task switch, plus a note for operators whose stored configs still carry the removed keys. The historical docs/changelog.md entry that listed this pair under "Feature exists but knob unwired" was left as-is — it is an append-only record of past work.
Files: src/main/java/ai/labs/eddi/modules/llm/model/LlmConfiguration.java, src/test/java/ai/labs/eddi/modules/llm/model/LlmConfigurationRagLegacyFieldsTest.java (new), src/test/java/ai/labs/eddi/modules/llm/model/LlmConfigurationTest.java, src/test/java/ai/labs/eddi/modules/llm/model/LlmConfigurationModelsTest.java, docs/rag.md
🧹 Remove dead eddi.audit.retentionDays; correct MCP-client and GDPR-retention docs (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Backlog item D15 — one dead configuration property plus two pieces of documentation that described behaviour the code does not have.
(a) eddi.audit.retentionDays was a silent no-op — removed
⚠️ Operator-visible. src/main/resources/application.properties shipped eddi.audit.retentionDays=-1 (and a %dev. twin) since the per-category-retention work. No Java code has ever read it. AuditLedgerService declares four eddi.audit.* @ConfigProperty values — enabled, flush-interval-seconds, dead-letter-path, agent-signing-enabled — and this is not one of them. Anyone who set eddi.audit.retentionDays=90 (or EDDI_AUDIT_RETENTIONDAYS=90) expecting old audit entries to be purged was getting nothing at all: no sweep ran, no entry was ever deleted, and no warning was logged. Drop the property from your deployment; if you need time-limited audit storage, do it at the operational/database layer (archival job, partition drop, storage-level TTL).
Design decision — do NOT implement the sweep; delete the knob. The audit ledger is append-only on purpose. IAuditStore's contract states that implementations "MUST NOT provide update or delete operations", justified by EU AI Act Arts. 17/19 (immutable decision traceability) and GDPR Art. 17(3)(e); pseudonymizeByUserId is documented as the sole permitted mutation, and both the MongoDB and PostgreSQL stores enforce insert-only semantics. Wiring a retention sweep would have required breaking that contract to satisfy a property nobody asked for. Removing the property makes the config honest instead.
Why this one mattered more than a typical dead key: its sibling eddi.usermemories.deleteOlderThanDays is read and does drive a real scheduled sweep. Two adjacent, identically-shaped keys where one works and one silently does nothing is exactly the kind of asymmetry an operator cannot detect from the outside.
Regression test: new AuditRetentionConfigTest (src/test/java/ai/labs/eddi/engine/audit/), three assertions, no Quarkus container required:
Every
eddi.audit.*key declared inapplication.properties(with any%profile.prefix stripped) must appear as a quoted string literal somewhere insrc/main/java— i.e. something actually reads it. This is the general guard: it fails if anyone re-adds an unreadeddi.audit.*property, not just this one. It carries a vacuity guard — the scan must first prove it can find"eddi.audit.enabled", a key production demonstrably declares, otherwise a broken scanner would make the whole assertion pass for the wrong reason.eddi.audit.retentionDaysspecifically is absent, with a failure message explaining why it can never be implemented.IAuditStoredeclares nodelete*/remove*/purge*/drop*/truncate*/expire*method — this pins the append-only design decision the docs now state, so a future contract change has to be deliberate and update the compliance doc alongside it.
All three were mutation-checked: re-adding the two property lines fails (1) and (2); adding a default long deleteOlderThan(long) to IAuditStore fails (3).
(b) docs/mcp-server.md documented an MCP-client config shape that no longer exists
The "MCP Client — Agents as MCP Consumers" section still described an inline mcpServers array on a langchain task. That field and its getMcpServers() accessor were deleted in 43ba59811 ("Remove inline mcpServers from LlmConfiguration.Task") without a matching docs update, so the documented JSON has been silently unusable ever since. The section also named a setup_agent(mcpServers:) parameter; the real one is mcpServerUrls (McpSetupTools.java).
Rewrote the section against the code rather than the old prose. External MCP servers are configured as mcpcalls workflow extensions — a versioned configuration resource (eddi://ai.labs.mcpcalls, POST /mcpcallsstore/mcpcalls), the MCP equivalent of httpcalls — referenced from a workflow step ahead of the LLM step. The field table now matches McpCallsConfiguration: mcpServerUrl (not url), name, transport (default "http", and documented honestly as informational only — McpToolProviderManager.createTransport unconditionally builds a StreamableHttpMcpTransport and never branches on it, so the previously documented "streamableHttp" default and "only streamableHttp supported" note were both wrong in different directions), apiKey, timeoutMs, toolsWhitelist, toolsBlacklist, mcpCalls. Added the dual-mode explanation the docs never had: agent mode (AgentOrchestrator.discoverMcpCallTools() traverses agent → workflow → every mcpcalls step at execution time and applies each config's whitelist/blacklist; gated by enableMcpCallTools on the LLM task, default true) versus pipeline mode (McpCallsTask matches behavior-rule actions against mcpCalls[].actions and invokes tools deterministically, no LLM involved). Corrected the setup_agent block to mcpServerUrls and documented what AgentSetupService actually does with it: one mcpcalls config per comma-separated URL (transport: "http", timeoutMs: 30000, no whitelist/blacklist, no mcpCalls bindings) plus a matching eddi://ai.labs.mcpcalls workflow step — nothing is written inline into the LLM configuration.
Also fixed the same drift in docs/architecture.md, which claimed "MCP server connections are configured per LLM task".
Left alone deliberately: the four other mcpServers occurrences in docs/mcp-server.md (lines ~179/193/211/224) are the client-side mcpServers key in Claude Desktop / Antigravity config files — correct as written and unrelated. HANDOFF.md and the historical changelog entries that mention the old inline field are append-only records of past work and were not rewritten.
(c) docs/gdpr-compliance.md promised deletion that cannot happen
The retention block documented eddi.audit.retentionDays as "delete entries older than N days" — a published compliance statement describing an operation the store forbids. Removed it and replaced it with what is actually true: the audit ledger has no retention property by design, EDDI never time-expires audit entries, and erasure requests are satisfied by pseudonymizing the userId via IAuditStore.pseudonymizeByUserId on the cascading-erasure path. Added an operator note that the old property was never read, and reworded the per-category bullet list so it no longer implies a configurable audit purge exists.
Note for reviewers: this narrows a published compliance claim (EDDI no longer offers even a nominal "time-limited audit retention" option). That is the honest direction — it aligns the doc with IAuditStore's contract — but it is a documentation-visible policy statement and deserves a human read, not a silent patch.
Blast radius: zero runtime behaviour change. No Java production code was modified; the only non-doc edit is the removal of a property nothing read. ./mvnw clean compile clean, ./mvnw validate clean, 145 tests green across AuditRetentionConfigTest, AuditLedgerServiceTest, AuditLedgerServiceExtendedTest, AuditLedgerServiceBranchTest, AuditHmacTest, McpSetupToolsTest, McpToolProviderManagerTest, McpCallsTaskTest.
🗑️ Delete unused EddiChatMemoryStore — dead since Phase 6E (2026-07-22)
Repo: EDDI (fix/backlog-defect-remediation)
Removed src/main/java/ai/labs/eddi/modules/llm/memory/EddiChatMemoryStore.java and its two test classes (EddiChatMemoryStoreTest, EddiChatMemoryStoreExtendedTest). The package ai.labs.eddi.modules.llm.memory is now gone entirely — those three files were its only occupants.
Why it was dead. The class's own javadoc describes it as "a quarkus-langchain4j ChatMemoryStore", but that extension was dropped in Phase 6E (2026-03-15) in favour of plain dev.langchain4j core. Nothing in pom.xml pulls io.quarkiverse.langchain4j, so the machinery that would have discovered and used a ChatMemoryStore bean — AiServices, ChatMemoryProvider, @MemoryId — is not on the classpath at all. A repo-wide grep confirms none of those four symbols appears anywhere else in the codebase, and ChatMemoryStore itself occurred only in the deleted class's own implements clause and import. The bean was @ApplicationScoped, so CDI instantiated it, but nothing ever injected it and no code path called getMessages/updateMessages/deleteMessages.
What actually does this job. The production LLM history path is ConversationHistoryBuilder (used by AgentOrchestrator/LlmTask), which performs the same EDDI-snapshot → langchain4j-ChatMessage conversion via the same ConversationLogGenerator, but with windowing, token budgeting and multimodal content that the deleted stub lacked. If EDDI ever re-adopts quarkus-langchain4j (see planning/native-image-migration.md), the bridge would be rebuilt from ConversationHistoryBuilder, not from this stub — so nothing of value is lost.
The 14 deleted tests asserted nothing about shipped behaviour. Both test classes only pinned a bean with zero production consumers, and several were vacuous on their own terms: getMessages_emptySnapshot_returnsEmpty and GetMessages#returnsMessagesFromSnapshot both fed in an empty conversationSteps list and asserted an empty result, never once exercising the role/ContentType conversion loop; the two updateMessages "no-op" tests called verifyNoInteractions on a mock that a single-LOGGER.trace method could not possibly touch. Net effect on the suite: 14 fewer tests, and the coverage gate should move neutral-to-positive because the deleted class contributed an entirely uncovered conversion block.
Design decision — the changelog is append-only. Five earlier entries in this file (from the 2026 test-coverage pushes that created and tidied these test classes) mention EddiChatMemoryStore. Those were left untouched: they are accurate records of past work, and rewriting history to hide a since-deleted class would make the log unreliable. This entry is the forward record of the removal.
Operator impact: none. No configuration key, REST endpoint, agent JSON field, or persisted document references the class or its package, and no runtime code path could reach it. Nothing to migrate.
Shared collaborators deliberately untouched: ConversationLogGenerator (5 other production consumers), IConversationMemoryStore, ConversationMemorySnapshot.
🧪 LLM — make AgentOrchestrator injectable; cover agent-mode response metadata (2026-07-22)
Repo: EDDI (refactor/agent-orchestrator-injectable; branched from fix/orphan-scan-and-quota-defects, which has since merged to main — this branch now targets main directly)
Follow-up to the token-usage fix below, which shipped without unit coverage.
Correction to the entry below
Its "Verification limits" paragraph states that LlmTask constructs its orchestrator internally, so executeIfToolsEnabled cannot be stubbed and "every existing LlmTask test therefore exercises the legacy branch". That is wrong. LlmTaskCoverage2Test and LlmTaskResumeModeTest already substituted a mocked AgentOrchestrator — via LlmTask.class.getDeclaredField("agentOrchestrator") + setAccessible(true) — and roughly 20 tests drove the agent branches through it, including the legacy-fallback case.
The branches were covered. What was never exercised was the metadata dimension: every stub built its result with the two-argument ExecutionResult convenience constructor, which hardcodes Map.of(). Against an always-empty map, a task that ignores responseMetadata() is indistinguishable from one that honours it. Branch coverage hid a data-flow gap — the more useful lesson than "the seam was missing".
What changed
AgentOrchestrator→@ApplicationScopedwith an@Injectconstructor. Signature unchanged, so the 8AgentOrchestrator*Testclasses that construct it directly are untouched.ConversationHistoryBuilder→@ApplicationScoped; it is an orchestrator constructor parameter and had to become resolvable. Stateless (only a static logger).LlmTask.agentOrchestratoris aprivate finalconstructor-injected collaborator. Tests pass a mock as an argument.LlmTask's constructor went from 41 parameters to 20. 22 of them existed only to feednew AgentOrchestrator(...), plusattachmentStore; all 23 left with the orchestrator, andAgentOrchestrator+ConversationHistoryBuildercame in. All 10 call sites across 9 test files were rewritten, along with the mocks, imports and locals the removal orphaned.Attachment services are now injected into
AgentOrchestratordirectly (@Injectfields), andLlmTask.wireAttachmentServices()is gone.setAttachmentServicessurvives only as a test seam for directly-constructed orchestrators.All three reflection hacks deleted (
LlmTaskCoverage2Test,LlmTaskCoverageTest,LlmTaskResumeModeTest— the third was missed on the first pass and found by grepping the whole test tree rather than the two files already in hand).
Statelessness check (AGENTS.md §4.1 rule 2)
Every AgentOrchestrator field is final except the two volatile attachment services, which are write-once deployment-scoped collaborators, not per-conversation state; ToolApprovalGate and ChatTranscriptCodec are stateless helpers. All conversational state travels through the IConversationMemory argument. Safe as a shared singleton — which it already effectively was, being owned by the singleton LlmTask.
Tests
New LlmTaskAgentModeMetadataTest (7 tests) covering the standard agent branch, the skipCascade agent branch, the legacy fallback when the orchestrator declines, the HITL resume continuation (non-null and null), and a null tool trace.
Validated by mutation rather than by observing green: reverting 15b7a08a7 and re-running turns five of the seven red. The two that survive (agentMode_emptyMetadata_publishesEmptyMap, agentReturnsNull_fallsBackToLegacyChatExecutor) pass with and without the fix by design — they pin behaviour the fix did not alter, and are regression guards, not discriminators. Each says so at its own assertion, so the distinction is visible in the file a future reader actually opens, not only here.
Correction, and a lesson about arithmetic-by-inspection. This paragraph twice carried a wrong count, and a Copilot review on PR #604 caught the inconsistency (
four+the other two≠ seven). Both the number and the suggested repair were wrong, in opposite directions: the review proposed listingresumeMode_nullResult_publishesEmptyMetadataas a third guard, but that test does discriminate — pre-fix,executeResumepublished no metadata at all, so itsverifyfails outright. Re-measuring gave five, not four. The stale figure came from updating the totals by hand across two changes that moved the score in opposite directions: the defensive copy demotedagentMode_emptyMetadata_publishesEmptyMapfrom discriminator to guard (identity assertion →assertNotSame), and the later null-trace test added a discriminator back. Mutation scores are cheap to measure and expensive to infer — re-run them.
Review pass — what it changed
Two independent reviewers (one general, one adversarial and instructed to refute) read the diff, the full orchestrator, and 15b7a08a7. Neither found a Critical issue; both independently refuted the sharpest hypothesis (that an immutable Map.of() could reach a mutation site — production always returns a mutable HashMap, and every post-assignment site is read-only). What the pass did produce:
A missing third call site. The fix has three separately-revertable assignments — the
skipCascadebranch, the standard branch, andexecuteResume— and the original five tests reached only the last two. Deleting theskipCascadeline left all five green. Test six closes it; surgically reverting that one line now fails exactly that one test.A false claim in this entry, retracted in place above.
A near-vacuous test.
agentMode_emptyMetadata_publishesEmptyMapasserted only emptiness, which the bug satisfies as readily as the fix.A lazy-init hazard introduced by the refactor itself. With the orchestrator injectable but its attachments still pushed in by
LlmTask's@PostConstruct, any other future injector ofAgentOrchestratorwould get one with null attachment services, because@ApplicationScoped LlmTaskis created lazily and might never have run. Fixed by injecting the services into the orchestrator directly. This had been part of the agreed design and was simply not implemented on the first pass.Defensive copy at all three sites:
responseMetadatais nownew HashMap<>(...)rather than an alias of the orchestrator's map. Dormant today, but the published map lands in conversation memory and the{{llmMeta}}namespace, and the two-argumentExecutionResultconstructor yields an immutable map — so a later metadata write would have thrown on the agent path only, in production only.Fixture fidelity: tasks now set
enableBuiltInTools, soisAgentMode()is true. Previously the fixtures stubbed a non-null agent result onto a config for which the real orchestrator always returnsnull.
Fourth review round — CodeRabbit + Copilot on PR #604
Assert the published metadata is writable, not merely a distinct instance (applied).
agentMode_emptyMetadata_publishesEmptyMapassertedassertNotSame— whichMap.copyOf(...)would also satisfy, while still throwing the instant downstream code adds a metadata key. That is exactly the failure the defensive copy exists to prevent, so the assertion was checking the wrong property. Now probes an actualput.Inline FQNs replaced (applied). 25 occurrences of
new io.micrometer.core.instrument.simple.SimpleMeterRegistry()across 9LlmTask*Testclasses, swapped for a top-level import per AGENTS.md §4.7. Pre-existing, but carried forward by this branch when the constructor call sites were rewritten, so they belong to it now.@Injectfields left package-private (declined, with evidence). The suggestion was to privatiseAgentOrchestrator's two injected attachment fields. Declined on three counts:src/maincontains 43 package-private@Injectfields and zero private ones, so this would be the sole exception;GroupConversationService.attachmentStoreis the identical shape, package-private and written directly byGroupConversationServiceTest; and Quarkus recommends package-private precisely to keep ArC from injecting reflectively, which matters for the native-image work on the roadmap. The stated risk — accidental in-package mutation — is also not removed by the change, since the package-privatesetAttachmentServicessetter next to the fields exists so the eightAgentOrchestrator*Testclasses can write them.
Third review round — Copilot on PR #604
Null
trace()guard, applied.executeTaskassignedagentResult.trace()straight intotoolTracewhileexecuteResumenull-guarded it, so!toolTrace.isEmpty()could throw. Both agent branches now guard. Flagged by all three reviewers, and previously deferred here as "pre-existing" — a thin defence given this change edits those exact lines. Mutation-verified: removing the guard yieldsNullPointerException: Cannot invoke "java.util.List.isEmpty()" because "toolTrace" is nulland fails exactly the newagentMode_nullTrace_doesNotThrow. Worth stating precisely, since the review overstated it: no production path returns a null trace — bothExecutionResultconstruction sites pass a fresh list — so this is hardening and consistency, not a live bug fix.PersistenceMapperProducer(rated High), not applied — the finding is incorrect. It claims the producer omits a productionWRITE_DATES_AS_TIMESTAMPSsetting. That feature defaults totruein Jackson, and a hand-builtnew ObjectMapper()is untouched byquarkus.jackson.*either way, so the suggestedconfigure(..., true)is a no-op. Verified by pointingSerializationCustomizerInstantFormatTestat the real producer: all 7 pass, including "Instant stays NUMERIC". But there is a real defect underneath it, in the test rather than production. That test's helper is documented as "Exactly what PersistenceMapperProducer builds" and is not — it adds aconfigure(WRITE_DATES_AS_TIMESTAMPS, true)the producer never makes, so it reconstructs the mapper instead of exercising it and cannot catch producer drift. That is precisely the driftdc117cddchad to revert once. Fix belongs on the branch that owns the file (#603): make the helperreturn new PersistenceMapperProducer().persistenceMapper();— confirmed green.
Verification limits
CDI wiring is validated locally, by ./mvnw package -DskipTests. Augmentation completes and ArC emits AgentOrchestrator_Bean + AgentOrchestrator_ClientProxy and ConversationHistoryBuilder_Bean + ConversationHistoryBuilder_ClientProxy into target/quarkus-app/quarkus/generated-bytecode.jar. That answers the only genuinely uncertain part of this change: these are the repo's first package-private @ApplicationScoped beans, and ArC proxies them without complaint. An unsatisfied injection point anywhere in the orchestrator's 29-dependency constructor would have failed the build outright, so resolution of the whole graph is confirmed too.
Worth remembering, because it cost three wrong claims in this entry's drafts: build-time augmentation binds no sockets and needs no MongoDB. The local limitation recorded elsewhere in this file is about tests that bind a loopback socket (
*IT.java, HTTP-server tests) — it does not extend toquarkus:build../mvnw package -DskipTestsis the right local gate for any CDI change and takes ~13s of augmentation.
@ApplicationScoped was chosen over @Singleton to match repo convention (AGENTS.md §4.1 rule 4) and because a normal scope's client proxy is the safer way to introduce a 29-dependency bean into a dense graph — lazy resolution tolerates ordering that a pseudo-scope resolves eagerly.
Correction (same-day review): an earlier revision of this entry justified the choice by claiming the chain
LlmTask → AgentOrchestrator → IConversationService → IAgentFactory → lifecycle tasksis cyclic and that a pseudo-scope could break it. That reasoning does not hold and should not be relied on:ILifecycleTaskis only ever consumed throughInstance<ILifecycleTask>/Provider<ILifecycleTask>(LlmModule,WorkflowStoreClientLibrary,ApiCallsModule), which is lazy and already breaks any cycle at that edge, andAgentFactoryinjects onlyIAgentStoreClientLibrary+IDeploymentListener. The conclusion stands; the stated reason was wrong.
🔍 More PR review follow-ups (2026-07-22)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
UsageSnapshotSerializationTest.productionMapper()stopped mirroring the real REST mapper. It calledSerializationCustomizer.configureObjectMapper(...)directly — which, since the persistence/REST mapper split earlier in this branch, builds the persistence recipe (noInstantoverride), not the REST/CDI one. The assertions still passed, becausecostMonthis aYearMonthwith@JsonFormaton the field directly and is unaffected by theInstantconfigOverride— but the test's stated purpose ("pins the REST wire shape") was no longer true, and it provided zero coverage of the actual production mapper. Same class of defect fixed earlier inSerializationCustomizerInstantFormatTest(a186dc903): the fixture now builds the mapper vianew SerializationCustomizer(false).customize(mapper), matching how Quarkus actually constructs it.Inline FQN in
DescriptorStoreTest.new java.util.ArrayList<>()wherejava.util.Listwas already imported. Added the top-level import; this file came in via theorigin/mainmerge, not authored on this branch, but the convention applies regardless of origin.
🔀 Merge origin/main into fix/orphan-scan-and-quota-defects — changelog conflict resolution (2026-07-21)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
Brought the branch up to date with origin/main, which had picked up the fix/descriptor-store-limit-semantics PR (limit=0 now means "unlimited" in DescriptorStore.readDescriptors, with a MAX_RESULT_LIMIT safety ceiling) via a background task spawned earlier in this effort. Both that PR and this branch touch DescriptorStore.readDescriptors, so docs/changelog.md conflicted at the top (both sides prepend); DescriptorStore.java and its test merged automatically without conflict — this branch's includeDeleted inclusion-flag fix and main's resolveDescriptorLimit/ceiling-warning logic touch disjoint parts of the same method. Kept both changelog blocks whole, this branch's newest-first on top, main's independent entry below.
🔍 PR review follow-ups — a vacuous test found and fixed (2026-07-21)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
Addressing CodeRabbit and Copilot review comments. One of them exposed a test that could not fail.
The MAX_PAGES ceiling test was vacuous. CodeRabbit noted the new page ceiling had no coverage. Adding a test made it pass immediately — but mutation-checking it (disabling the ceiling so the walk truncates silently) showed it still passed. Cause: the fixture left agentStore.read unstubbed, so the traversal NPE'd on a null config, marked the scan incomplete, and produced the expected 409 for the wrong reason. Stubbing the agent read so the ceiling is the only possible failure source, plus asserting the refusal message names it, makes the test load-bearing — the mutant now dies with "Expected WebApplicationException to be thrown, but nothing was thrown."
Copilot's WRITE_DATES_AS_TIMESTAMPS finding: premise wrong, instinct right. It claimed the produced @PersistenceMapper could change the on-disk shape. It cannot — Jackson enables that feature by default, so the producer already emitted numeric. But the real defect was next door: SerializationCustomizerInstantFormatTest reconstructed the producer instead of calling it, and set the flag itself — so it would have passed even if the producer were broken. The test now builds the mapper via new PersistenceMapperProducer().persistenceMapper(), and the producer states the flag explicitly. Relying on a library default for a persistence-format guarantee is precisely what dc117cddc was reverted for. Mutation-checked: flipping the producer to ISO now fails two tests.
TOCTOU on the agent quota — documented, not fixed, and the earlier claim corrected. CodeRabbit correctly flagged that concurrent deploys observing count == limit - 1 all pass. An internal note had called this "self-correcting"; that was wrong — once over, the gate merely refuses further deploys until an undeploy brings the count down. The javadoc now states the bound honestly and explains why a per-tenant lock is not used: it would serialize within one JVM while the count spans the shared store and every node's registry, giving the appearance of a hard guarantee exactly where it would not hold. Accepted because deploys are rare admin operations and the gate's purpose — stopping runaway growth such as an LLM creating sub-agents in a loop — survives a small transient overrun.
Log injection. sanitize(conversationId) added to the new quota-denial log line in RestAgentEngine, matching the rest of the class (lines 331, 368, 389, 436).
Not actioned: the advisory note that the orphan endpoint blocks a request thread. Pre-existing, explicitly raised as advice rather than a blocker, and moving it to AsyncResponse with a polling status endpoint is a separate change.
🔢 LLM — stop discarding agent-mode token usage (2026-07-21)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
AgentOrchestrator sums TokenUsage across every model call in the tool loop and returns it on ExecutionResult.responseMetadata() — on both the live path and the resume path. LlmTask then read .response() and .trace() from that result and never .responseMetadata(), so agent-mode token accounting was computed and dropped on the floor. Only the legacy-chat and cascade branches surfaced theirs.
Net effect: any agent with tools enabled reported {} for responseMetadataObjectName, and no per-turn token figure existed for the paths that dominate real usage. This is the prerequisite for monthly cost metering — there was nothing to meter.
Both agent branches now read the metadata, and executeResume surfaces it the same way executeTask does (it previously built no metadata map at all).
Known gap, deliberately documented rather than papered over
A turn that pauses for tool approval loses its pre-pause usage: ToolApprovalRequiredException escapes before the metadata is assembled and carries no usage, and resumeToolLoop starts a fresh accumulator. So a paused turn under-reports by its pre-pause segment. Closing that requires threading the step into runToolCallLoop so usage is written incrementally — out of scope here.
Safety check
responseMetadata also feeds applyResponseValidation, which branches on warning and streamingTimeout. AgentOrchestrator puts only tokenUsage into the map (AgentOrchestrator.java:470,826), so both of those keys stay absent exactly as they were with the previously-empty map — validation behaviour is unchanged.
Verification limits
Not unit-covered: LlmTask constructs its AgentOrchestrator internally rather than receiving it injected, so executeIfToolsEnabled cannot be stubbed to return a non-null result at the LlmTask unit level, and every existing LlmTask test therefore exercises the legacy branch. The change was verified by reading both sides of the contract and by the safety check above; all 184 LlmTask tests stay green. Making this properly testable means injecting AgentOrchestrator — a worthwhile refactor of an already 29-argument constructor, but a separate change.
🕐 Serialization — split the persistence mapper from the REST mapper; Instant is ISO-8601 on the wire (2026-07-21)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
⚠️ REST contract change, and it requires a companion EDDI-Manager fix (below).
Every java.time.Instant on every endpoint rendered as a 1970 date in the Manager. With quarkus.jackson.write-dates-as-timestamps=true (application.properties:174) plus JavaTimeModule, an Instant serializes as fractional epoch seconds (1719964800.123), while clients call new Date(value), which expects millis. Affected nextFire, lastFired, pausedAt, createdAt, updatedAt, transcript timestamps — essentially every timestamp in the UI.
The obvious fix — a configOverride(Instant.class) in the shared configureObjectMapper — is the trap. JsonSerialization @Injects the same CDI ObjectMapper, and it backs DocumentBuilder for every Mongo write, every Postgres JSONB column, the backup/export writer, and the {json:serialize} Qute extension available to agent authors. That would change on-disk formats. Concretely, GroupConversation.lastModified is a persisted Instant that GroupConversationStore sorts server-side on: Mongo's BSON cross-type ordering ranks all Doubles before all Strings, and Postgres does ORDER BY data->>'lastModified' lexicographically — so old numeric rows and new ISO rows would interleave wrongly, silently, with no backfill. Commit dc117cddc ("keep numeric date format") already reverted a broader version of this once for breaking findDueSchedules.
So the two mappers are now separated:
New
@PersistenceMapperqualifier +PersistenceMapperProducer, built from the sameconfigureObjectMapperrecipe without the date override.JsonSerializationinjects that.The
Instant→ ISO override moved intoSerializationCustomizer.customize()— the REST/CDI path only, never the shared static.
Design decisions
java.util.Datedeliberately untouched. It already emits epoch millis and its consumers (e.g.DocumentDescriptor.lastModifiedOn) are correct today; widening the override would break working paths. The wire therefore carries two encodings by design:Instant= ISO-8601 string,Date= epoch-millis number.application.properties:174must stay. Quarkus's own default forwrite-dates-as-timestampsisfalse, so that line is what prevents Quarkus disabling the feature globally. Deleting it as "redundant documentation" would flip persistence to ISO — the exact break this change avoids.A qualified CDI producer, not
new ObjectMapper()insideJsonSerialization. Keeps the persistence mapper a first-class, overridable bean and leaves the existing direct-construction test fixtures working unchanged.Deserialization is unaffected in both directions —
InstantDeserializerdispatches on the JSON token type, not the shape hint — so rows written numerically still parse.
Required companion change (EDDI-Manager repo)
The Schedules dashboard sorts arithmetically: (a.nextFire ?? 0) - (b.nextFire ?? 0), which yields NaN on ISO strings, leaving the "next fire" tile showing an arbitrary schedule. Change it to new Date(a.nextFire) - new Date(b.nextFire), which works with both encodings and can therefore land before or after this commit. Every other consumer (new Date(x).toLocaleString()) becomes correct automatically. The vendored bundle under META-INF/resources/assets/ is a build artifact and was deliberately not hand-edited.
Verification limits
CDI wiring for the new qualified producer cannot be validated locally — quarkus:build augmentation needs a loopback socket, which this environment refuses, and the repo has only one non-IT @QuarkusTest. CI is the gate for that specific aspect. The format behaviour itself is fully covered by unit tests.
Tests
New SerializationCustomizerInstantFormatTest (7 tests) asserts both halves: the REST mapper emits ISO and keeps Date numeric; the persistence mapper stays numeric and does not inherit the REST override despite sharing configureObjectMapper; and both still read the old numeric form.
🎫 Tenancy — enforce maxAgentsPerTenant on deploy (2026-07-21)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
TenantQuota.maxAgentsPerTenant was persisted end-to-end by all three stores and round-tripped through the REST API, but nothing ever read it — TenantQuotaService had no agent method at all. Operators could set the limit and get silent no-enforcement.
New TenantQuotaService.checkAgentQuota(tenantId, currentDistinctAgents) gates RestAgentAdministration.deployAgent, denying with QuotaExceededException → 429 {"error":"quota_exceeded"} via the existing mapper.
Design decisions
A read-only gate, not an atomic counter. The deployed-agent count is a stock derived by counting current deployments, not a per-window flow. A stored counter would drift irrecoverably: the 10s re-deploy sweep, the 24h old-version undeploy,
TeardownAgentTool,GroupConversationServiceand the lazy re-deploy on first use all add or remove deployments without passing any acquire/release point. Modelled oncheckCostBudget, so noITenantQuotaStoremethod was added and the three store implementations are untouched.Placement is forced, not stylistic. The gate sits between the null-checks and the
try. Inside thetry,catch (Exception) → InternalServerErrorExceptionwould convert the 429 into a 500; inside the submittedCallableit runs off the request thread and could never produce a status code at all.Counts distinct agent ids, so redeploys and version bumps are free — required because the old-version undeploy sweep legitimately keeps two versions of one agent deployed while the previous drains.
The count unions persisted rows with live agents, and needs both.
autoDeploy=falsenever writes adeployedrow, but the in-memory deploy is unconditional — so a rows-only count let a caller deploy unlimited agents with one query parameter. Those agents are genuinely live:getLatestReadyAgentserves them without consulting the store, andConversationService.getAgentlazily re-deploys them after a restart, so the bypass is durable, not merely transient. Conversely, a live-agents-only count would be per-JVM and would miss other cluster nodes.Fails open. A store outage must not block deployments; the denial is logged and metered either way.
The two CDI callers surface the reason.
AgentSetupService.deployAndWaitandMcpAdminTools.deployAgentcall the bean directly, so the exception mapper never runs — both now return the quota reason instead of "check server logs", which an agent designer (or a model drivingcreate_sub_agent) cannot act on and would retry in a loop.Scheduled sweeps stay ungated by construction — they call
IAgentFactorydirectly, so lowering the limit never undeploys anything. The limit gates new agents only.
Not addressed
Single-tenant only: the gate resolves getDefaultTenantId(), exactly as the conversation and api-call quotas already do. This is a deployment-wide cap, not per-organisation, until the multi-tenancy plan's Phase 1 lands. Do not describe it as multi-tenant enforcement.
Tests
New RestAgentAdministrationQuotaTest (8 tests): denial throws QuotaExceededException and submits nothing to the runtime; two versions of one agent count once; redeploy skips the quota entirely; live agents with no deployed row still count (the loophole); no double counting; non-READY agents skipped; null agent ids skipped; store error fails open. Mutation-checked — reverting to a rows-only count fails exactly the loophole test. TenantQuotaServiceTest gains 5 tests including parity with checkCostBudget on not inflating the allowed counter. Both existing RestAgentAdministration*Test classes updated for the new constructor arg in the same commit.
🧹 Orphan admin — includeDeleted becomes a true inclusion flag; purge default flipped (2026-07-21)
Repo: EDDI (fix/orphan-scan-and-quota-defects)
⚠️ REST behaviour change on DELETE /administration/orphans.
DescriptorStore.readDescriptors treated includeDeleted as an equality filter — eq("deleted", includeDeleted) — so includeDeleted=true matched only soft-deleted descriptors rather than adding them to the live ones. Two consequences:
The parameter did not mean what its name, its
@Parametertext, anddocs/deployment-management-of-agents.mdall said it meant.The shipped Manager scans with
includeDeleted=falseand purges withincludeDeleted=true, so the set shown to the user and the set deleted were disjoint. The UI listed live orphans and then purged soft-deleted ones.
true now drops the deleted constraint entirely (live and soft-deleted); false constrains to live only. Every other caller in src/main passes a literal false — verified by enumerating all ~35 call sites — so their behaviour is bit-for-bit unchanged.
purgeOrphans's @DefaultValue flipped from true to false. Left at true, the semantics fix would have made the parameterless DELETE dramatically more destructive — combined with the page-walk fix two commits earlier, from "purge ≤200 already-soft-deleted rows" to "permanently wipe every unreferenced resource, unbounded". Flipping the default makes the bare call the conservative one and matches scanOrphans, so a scan and a purge with no parameters now describe the same set.
Client impact
A client that relied on the old default now purges less: live-but-unreferenced orphans only, not soft-deleted ones. Pass includeDeleted=true explicitly to also purge soft-deleted resources. EDDI-Manager should send the flag explicitly, matching whichever set it is displaying.
Tests
DescriptorStoreTest gains two tests capturing the actual QueryFilters and asserting the deleted constraint is present for false and absent for true; mutation-checked by restoring the equality filter, which fails the second. RestOrphanAdminSafetyTest gains two tests pinning the flag's propagation. All 86 tests across every descriptor-store consumer green.
🔍 Review follow-ups on the orphan/tenancy fixes (2026-07-21)
Repo: EDDI (claude/eddi-backend-manager-coverage-0598fe)
Self-review of the three preceding commits found three defects, all fixed here.
The 409 refusal carried no body.
purgeOrphansthrewnew WebApplicationException(String, Response.Status), whose response has no entity — so an operator saw a bare 409 and the reason existed only in the server log. Confirmed empirically by mutation: reverting to that constructor makes the newhasEntity()assertion fail. Now builds theResponseexplicitly with{"error":"incomplete_scan","message":…}.Broken javadoc link. The page-walk javadoc still referenced
buildReferencedUrisSet(), renamed toscanReferencedUris()in the same commit. Also updated three now-inaccurate@DisplayNames inRestOrphanAdminBranchTest.Postgres had no at-limit test. The
>→>=change was mutation-verified on Mongo but not Postgres. AddedPostgresTenantQuotaStoreTest.exactlyAtLimitand mutation-checked it too.
Also documented why a MAX_PAGES trip during orphan collection is safely swallowed while the same failure in the reference scan blocks the purge: a type that cannot be enumerated yields fewer delete candidates (under-delete), whereas a missing reference promotes a live resource to "orphan" (over-delete).
Verification: full clean suite tests=11273 failures=8 errors=288 skipped=3 across the same 15 classes as the pre-change baseline — all environmental (loopback sockets, network egress, model downloads). Zero regressions. Checkstyle clean on every changed file; javadoc:javadoc builds.
🚦 Engine — return 429 instead of 500 when the api-call quota denies a turn (2026-07-21)
Repo: EDDI (claude/eddi-backend-manager-coverage-0598fe)
ConversationService.say/sayStreaming throw QuotaExceededException when acquireApiCallSlot() denies, and QuotaExceededExceptionMapper maps that to 429 with {"error":"quota_exceeded"} and Retry-After: 60. But say() is resumed through a JAX-RS AsyncResponse, so the exception is caught inside RestAgentEngine.sayInternal and never reaches the @Provider mapper. Its catch chain lists AgentMismatchException, AgentNotReadyException, ConversationEndedException, ConversationAwaitingApprovalException, ProcessingRestrictedException and ResourceNotFoundException — but not QuotaExceededException — so the denial fell through to catch (Exception e) and surfaced as 500 "An internal error occurred".
Only the conversation-start quota ever produced a real 429: startConversationWithContext is synchronous and its narrower catch block lets the exception escape to the mapper. The per-minute API rate limit — the quota an operator is most likely to actually hit — was indistinguishable from a server fault, so clients had no way to back off correctly.
Added an explicit catch (QuotaExceededException) that resumes with the same status, body and Retry-After header as the mapper, so both quota denials look identical on the wire.
Design decisions
Mirror the mapper rather than re-throw. There is no way to route an already-captured async exception back through the provider chain, so the branch duplicates the mapper's three-line response. Kept adjacent constants and a comment so the two stay in sync.
Streaming is not covered here.
sayStreamingthrows the same exception, but by then the SSE response has already committed HTTP 200, so no status code can be sent — it currently emits a genericerrorevent. Giving that event a distinguishable quota type is a separate, client-visible change and is tracked, not slipped in.
Tests
New RestAgentEngineTest.quotaExceeded asserts 429, the Retry-After: 60 header and the exact entity map. Mutation-checked: replacing the new catch with an unrelated exception type makes it fail with InternalServerError An internal error occurred — reproducing the original bug — so the test is not vacuous. All 43 RestAgentEngineTest tests green.
💵 Tenancy — align the at-limit cost comparison and fix costMonth JSON shape (2026-07-21)
Repo: EDDI (claude/eddi-backend-manager-coverage-0598fe)
Two independent defects in the tenant cost-budget surface.
1. The two production quota stores disagreed with the gate at exactly the limit. TenantQuotaService.checkCostBudget denies on currentCost >= limit, and InMemoryTenantQuotaStore.tryAddCost matches it — but MongoTenantQuotaStore and PostgresTenantQuotaStore used totalCost > limit. At exactly the budget the pre-call gate denied while post-call accounting allowed. docs/changelog.md records the in-memory store being deliberately moved to >= for this reason; the two DB stores were never updated. Both now use >=.
Verified by mutation: reverting the Mongo comparison to > makes the new exactlyAtLimit test fail with expected: <false> but was: <true>, so the test is not vacuous.
2. UsageSnapshot.costMonth serialized as a JSON array. Under quarkus.jackson.write-dates-as-timestamps=true (application.properties:174) Jackson's YearMonthSerializer takes its useTimestamp branch and emits [2026,7] instead of "2026-07". Both stores already persist the value as an ISO string (YearMonth.toString() / YearMonth.parse, never through Jackson), so the REST representation was the only place that disagreed. Annotated the record component with @JsonFormat(shape = JsonFormat.Shape.STRING).
Design decisions
Annotation, not a mapper-wide override. A
configOverride(Instant.class)onSerializationCustomizerwould have fixed every temporal field at once, but that customizer's mapper is also the persistence mapper —JsonSerialization@Injects the same CDIObjectMapper, which backsDocumentBuilderfor every Mongo write and Postgres JSONB column. Changing it would alter on-disk formats. Concretely,GroupConversation.lastModifiedis anInstantpersisted through it andGroupConversationStoresorts server-side on that field, so mixed old-numeric/new-string rows would sort wrongly and silently in both backends. Commitdc117cddc("keep numeric date format") already reverted a broader version of this change once, for breakingfindDueSchedules. The wider Instant-format cleanup is therefore left out and tracked separately — it needs the persistence mapper decoupled from the REST mapper first, plus a coordinated EDDI-Manager change (the Schedules dashboard sortsnextFirearithmetically, which yieldsNaNon ISO strings).costMonthis safe in isolation precisely because neither store round-tripsYearMonththrough Jackson — verified before changing it.
Tests
New UsageSnapshotSerializationTest asserts the shape on the real record (not a stand-in holder) through a mapper wired exactly as production is, plus a round-trip. New MongoTenantQuotaStoreTest.exactlyAtLimit. All 131 tenancy tests green.
🛡️ Orphan admin — fix page-walk truncation and refuse to purge on an incomplete scan (2026-07-21)
Repo: EDDI (claude/eddi-backend-manager-coverage-0598fe)
Two defects in RestOrphanAdmin that together could permanently delete live configuration.
1. The descriptor page walk never advanced past page 0. readAllDescriptors advanced its cursor with index += batch.size(), but DescriptorStore.readDescriptors treats that argument as a page index (skip = index * effectiveLimit, DescriptorStore.java:61). The second iteration therefore asked for page 200 — skip = 40 000 — which always came back empty, so every store type was silently truncated at 200 rows. Fixed to pageIndex++.
The dangerous half of this was not the orphan list but buildReferencedUrisSet: the referenced set was truncated the same way, so on any deployment with more than 200 agents or 200 workflows, live in-use resources were classified as orphans — and purgeOrphans deletes with permanent=true, which is deleteAllPermanently(id) (current document and all history).
A MAX_PAGES ceiling (100 pages / 20 000 rows per type) now bounds the walk, since the scan is a synchronous one-read-per-descriptor traversal on a blocking JAX-RS method. Hitting the ceiling raises ResourceStoreException rather than truncating silently — a truncated scan must not be used to decide what to delete.
2. The reference scan failed open. Every read error while building the referenced-URI set was swallowed (two at debug level), and the partially-built set was returned as if complete. Because that set is what protects a resource, each swallowed error made more things look orphaned. One unreadable workflow could mark every rules/apicalls/output/llm/property/dictionary/parser resource in the database as an orphan.
scanReferencedUris() now returns a ReferenceScan record carrying a complete flag, and purgeOrphans refuses with 409 Conflict when the scan is incomplete, naming the cause. scanOrphans (read-only) still returns its best-effort report — only the irreversible path is gated. A ResourceNotFoundException is explicitly not treated as incompleteness: a descriptor whose resource is gone is a genuine orphan.
Design decisions
Fail closed on the destructive path only. Read-only callers tolerate a partial picture; a permanent delete may not.
Ceiling raises rather than truncates. Silent truncation is what made the original bug invisible.
No change to
includeDeletedsemantics in this commit. It is currently an equality filter (Filters.eq("deleted", flag)), soscanOrphans(defaults tofalse) andpurgeOrphans(defaults totrue) operate on disjoint sets. Fixing that is a real behaviour change to an irreversible endpoint and is deliberately left to its own commit — with the page-walk now complete, redefining the flag without also flipping thetruedefault would turn the defaultDELETE /administration/orphansfrom "purge ≤200 already-soft-deleted rows" into "permanently wipe every unreferenced resource, unbounded".
Tests
New RestOrphanAdminSafetyTest (7 tests): page index advances by 1 and a second page is actually requested; walk stops on the first partial page; an unreadable Agent aborts the purge with 409 and deletes nothing; a missing Agent resource does not abort; a complete scan purges normally; scanOrphans tolerates an incomplete reference set and never deletes; a workflow referenced by an Agent is never purged. Fixtures use 24-char hex ids because RestUtilities.extractResourceId returns a null id otherwise, which would make the assertions pass vacuously. All 27 pre-existing orphan tests unchanged and green.
🐛 Fix: readDescriptors(limit = 0) silently returned only 20 descriptors (2026-07-21)
Repo: EDDI (fix/descriptor-store-limit-semantics)
Summary
DescriptorStore.readDescriptors resolved its limit with (limit == null || limit < 1) ? 20 : limit, so a caller passing 0 to mean "give me everything" silently received the first 20 rows. The same limit < 1 ? 20 fallback was duplicated in MongoResourceStorage.findResources, PostgresResourceStorage.findResources, and ResourceFilter.readResources — four independent copies of a magic default, none of them documented.
Three production call sites were affected in a user-visible way:
PromptSnippetService.loadAllSnippets— a deployment with more than 20 prompt snippets never got the rest into the{snippets.*}template namespace. Silent: templates just rendered empty.RestExportService.exportSnippets— agent export dropped any referenced snippet outside the first 20, producing an incomplete ZIP that imports without error.RestAgentStore.populateCapabilityRegistry— only the first 20 agents were scanned for capabilities at startup, so capability-based discovery/delegation silently missed agents.
Three further call sites (ChannelTargetRouter, ChannelConnectorMigration, RestChannelIntegrationStore) passed a magic 1000 to dodge the cap; the channel-uniqueness check among them would have let a duplicate channelId through past that many configs.
Fix — explicit sentinel plus a hard ceiling
Option (a) from the two candidates, because the storage layer supports a bounded unlimited query cleanly and paging every caller would have been six copies of RestOrphanAdmin's batch loop:
IDescriptorStore.NO_LIMIT(0) andDEFAULT_LIMIT(20) define the contract in one place, withresolveDescriptorLimit(Integer):null→ default page,<= 0→ unlimited,> 0→ honoured.IResourceStorage.MAX_RESULT_LIMIT(10_000) is the hard safety ceiling, applied through the shared staticIResourceStorage.resolveLimit(int)that both backends now call — the two implementations can no longer drift apart.Truncation is no longer silent:
DescriptorStorelogs a WARN naming the descriptor type when a result set hits the ceiling. The original defect was not the number 20, it was that nothing said anything.All six "give me everything" call sites now pass
IDescriptorStore.NO_LIMITinstead of0or1000, so intent is readable at the call site.RestOrphanAdminkeeps its explicit 200-row batch loop — it genuinely pages.
Key Design Decisions
nulland0deliberately mean different things.nullis "caller expressed no opinion" → default page;0is "no limit". Collapsing them would have made?limit=omission return everything.Redefining
0is safe for the REST API because every REST endpoint already declares@QueryParam("limit") @DefaultValue("20"). JAX-RS, not the store fallback, supplies the REST default, so no endpoint's behaviour changes when the parameter is omitted. Only internal callers and an explicit?limit=0are affected — and?limit=0now means what it reads like.Ceiling over true unbounded: an unbounded query on a large
descriptorscollection is a memory risk, andreadDescriptorsissues oneread()per descriptor. 10,000 is high enough that no realistic deployment hits it, and the WARN makes it loud if one does.Legacy
mongo/ResourceFilterupdated too, so both descriptor-store implementations obey one contract rather than diverging.The ceiling warning sanitizes
type. Self-review caught that the new WARN loggedtypeunsanitized — andtypereachesreadDescriptorsstraight from@QueryParam("type")onIRestDocumentDescriptorStore. That is the CWE-117 log-injection pattern that commitd71de742remediated across 13 files (the commit that also last touched this very method), so the new log line now uses theLogSanitizer.sanitizehelper that commit introduced. Reachability is hard (10,000 matching descriptors), but CodeQL taint analysis is flow-based, not reachability-gated, and the repo's convention is to sanitize unconditionally.The warning states impact, not just cause. Its reader is an operator who cannot "page" anything — the actionable fact for them is that the returned list is incomplete and dependent features are missing entries, so the message leads with that.
Review follow-ups (Copilot + CodeRabbit)
indexjavadoc overstated the contract. It claimed anyindex > 0withNO_LIMIT"yields an empty list". It does not: the effective limit is the ceiling, soindex=1returns rows 10,000–20,000. The claim was only true for collections smaller than the ceiling — the common case mistaken for the contract. Reworded to say a non-zero index pages in ceiling-sized chunks.Integer overflow in the legacy
ResourceFilterskip.index * limitwas int arithmetic; raising the effective limit from 20 to 10,000 dropped the overflow threshold fromindex > ~107Mtoindex > ~214,748, where the skip goes negative. Now uses the samelongcast +Math.minguard thatd71de742added toDescriptorStore.readDescriptors, so both paths match. Reachability is low (this is the legacy store path), but the fix is three lines and removes a CodeQL-shaped pattern.Removed a duplicate test. The added
nullLimitUsesDefaultPageSizere-tested exactly what the pre-existingdefaultsLimitWhenNullalready covered. Deleted it and updated the pre-existing test to assert againstDEFAULT_LIMITinstead of a literal20, keeping coverage and dropping the magic number.
Verification
./mvnw clean compile— clean976 unit tests across the datastore package and every affected call site — 0 failures, 1 pre-existing skip
New tests pin the semantics:
null→ 20,NO_LIMIT→ ceiling (with an explicitassertNotEquals(DEFAULT_LIMIT)regression guard), negative → ceiling, oversized → clamped,skip == index * effectiveLimit, results not post-truncated; plus helper-level tests and backend-level assertions (Mongoiterable.limit(...), PostgresLIMITin the generated SQL)Two pre-existing tests asserted the old
limit=0 → 20behaviour (ResourceFilterTest,MongoResourceStorageBranchTest) and were updated to the new contract — they were pinning the bug./mvnw formatter:format+./mvnw validateclean
🔎 Auto-approve workflow — Copilot pagination review comment is a false positive (2026-07-20)
Repo: EDDI (chore/auto-approve-copilot)
Copilot flagged the CI-gate's github.paginate(github.rest.checks.listForRef, …) call, claiming it returns page objects rather than check-runs (so the gate would never approve) and suggested adding a (response) => response.data.check_runs map function. Verified against the exact runtime this workflow pins (actions/github-script@v7.1.0 → @octokit/plugin-paginate-rest@9.2.2): the claim is backwards. Octokit auto-normalizes list responses that carry total_count (check-runs do), so paginate() already returns the flattened check_runs array — the current code is correct and yields APPROVE. The suggested map function would read .check_runs off that already-flattened array → undefined → TypeError: Cannot read properties of undefined (reading 'name'), crashing the job. Both outcomes were reproduced with a mocked-HTTP harness on the pinned versions. Left the call unchanged and added an inline comment documenting why no map function is used, so the concern isn't re-raised.
🔀 Merge origin/main into chore/auto-approve-copilot — conflict resolution (2026-07-20)
Repo: EDDI (chore/auto-approve-copilot)
Brought the auto-approve-workflow branch up to date with main to clear the open PR's merge conflict. The branch adds a single new file (.github/workflows/auto-approve-copilot.yml), so the merge pulled in all of main's post-branch work — including the HITL framework, multi-model cascade enterprise hardening, error-handling recovery, and group-conversation follow-ups — with a single textual conflict: docs/changelog.md. Both sides had prepended entry blocks; both kept whole — this merge entry plus the branch's auto-approve entry on top, main's newer history below. No source files conflicted: the branch modifies no .java that main also touched, so the merged tree's source is byte-identical to origin/main.
⚙️ Chore: Auto-approve workflow for Copilot-reviewed PRs (2026-07-08)
Repo: EDDI (chore/auto-approve-copilot)
Summary
New GitHub Actions workflow (.github/workflows/auto-approve-copilot.yml) that converts a clean GitHub Copilot code review into an actual PR approval once CI is green, so PRs from trusted authors can satisfy a required-review gate without a human reviewer on hand. Built as a self-owned workflow using actions/github-script (SHA-pinned, v7.1.0) instead of the marketplace strspc-pr-review action, which was rejected for low provenance (2 stars, no contributor data) and requiring a broad classic PAT — incompatible with this repo's OpenSSF-Gold security posture.
Key Design Decisions
Self-owned, not marketplace: All logic lives in-repo; the only external dependency is GitHub's own
actions/github-script. Bot credential is a fine-grained PAT scoped to one repo.Public-repo safety: Only PRs from authors with
author_associationin{OWNER, MEMBER, COLLABORATOR}are eligible — external contributors are never auto-approved.workflow_runtrigger, notcheck_suite: GitHub suppressescheck_suite/check_runevents for check suites produced by GitHub Actions. Since EDDI's CI runs entirely as Actions jobs, acheck_suite: [completed]trigger would never fire.workflow_run: { workflows: ["CI/CD"], types: [completed] }is used instead.Single-token design: All API calls (reads + approval) use the single
githubobject authenticated viagithub-tokenset to the bot PAT. AvoidsgetOctokit/require('@actions/github')differences betweenactions/github-scriptversions.Graceful skip when unconfigured:
env.BOT_TOKENpattern avoids thegithub-token: requirederror when the secret doesn't exist — the step is simply skipped, no red check.
Gates (all must pass for approval)
PR is open, not draft, targets
mainAuthor is trusted (
OWNER/MEMBER/COLLABORATOR)No outstanding
CHANGES_REQUESTEDfrom anyoneCopilot has reviewed the current head commit (not stale)
Copilot's latest review has zero inline/line comments
All CI check-runs on head commit (excluding this workflow's own check) are completed and successful
Bot has not already approved at this exact head SHA (idempotency)
Known Scope Limitations (documented, accepted)
CI gate checks only check-runs, not legacy commit statuses. EDDI's CI only emits check-runs (confirmed via
ci.yml); branch protection remains the actual merge gate regardless."Clean Copilot review" is defined as zero inline comments; a summary-only body with concerns in prose would still pass. This is an intentional proxy.
Verification
YAML syntax validated (
yaml.safe_load)Embedded JavaScript validated (
node --checkwrapped in async function)SHA pin format matches repo convention (
sha # tagcomment)Adversarial code review (3 dimensions × 2 agents, 7 findings, all addressed or documented)
One-Time Manual Setup Required
A repo admin must: (1) create/designate a bot GitHub account with write access, (2) generate a fine-grained PAT scoped to this repo with Pull requests: R/W, Checks: Read, Metadata: Read, (3) store it as repo secret AUTO_APPROVE_BOT_TOKEN, (4) set repo variable AUTO_APPROVE_BOT_LOGIN to the bot's username.
🔁 Fix: PR-review response — interrupted streaming, FQN imports, changelog accuracy (2026-07-20)
Repo: EDDI (feat/error-handling-recovery)
Four review comments on PR #593 (CodeRabbit + github-code-quality); all valid, all fixed. Two turned out to be broader than reported.
1. An interrupted streaming attempt was reported as a success (Major)
When latch.await(...) threw InterruptedException, neither timedOut nor errorRef was set, so execution fell through both guards to break and returned new StreamingResult(responseText, metadata) — an empty string presented as a completed answer, with no signal to the caller.
Corroborating evidence the review did not cite: AgentOrchestrator already treats a set interrupt flag as a hard abort (its test describes the scenario as "simulate a cascade per-step timeout cancel"). The streaming executor was therefore contradicting an established convention inside the same subsystem — a cancelled step could be accepted as a real, empty answer, and on the cascade's last step it would win outright.
Interruption is now a distinct outcome: streamingInterrupted is recorded in the metadata, the salvaging (LlmTask) path returns whatever text arrived with a streaming_interrupted_partial warning, and every other path throws. It is never retried — a retry would ignore the very cancellation being signalled. Four tests, each observed failing first.
2. Inline fully-qualified names in LifecycleManagerTest (Minor)
Flagged on one line; the file actually had eight — four IAuditEntryCollector and four ConversationEventSink. All replaced with two top-level imports per AGENTS.md §4.7.
3. The full-suite failure totals did not add up (Minor)
Correct, and the fault was in the measurement rather than the prose: the categorisation was run over target/surefire-reports without a preceding clean, so it also swept up XML from earlier targeted runs. The per-bucket figures were therefore drawn from a superset of the run they were attributed to. The numbers have been withdrawn (with an explicit correction note in the merge entry) rather than quietly adjusted, since the original claim was already pushed.
Re-measured from a genuinely clean run of the final code: 11,658 tests, 8 failures, 287 errors, and zero assertion failures. Every failing test case in the surefire XML carries a blocked-loopback or socket-selector message; not one is a code assertion.
4. Useless parameter in the new error-path test helper (Note)
executor(ChatModelRegistry, ITemplatingEngineStub) never used its second argument, and the ITemplatingEngineStub marker interface existed only to make that argument look intentional — introduced in the previous commit as a "readability" device that conveyed nothing. Parameter and interface removed, all seven call sites simplified to executor(registry).
One caveat, stated rather than papered over: the XML yields 301 distinct failing test cases against the console's 295 failures+errors, and that ~6 gap is unexplained (it is not reruns, not suite-level entries, and not skipped-plus-failed). The load-bearing claim deliberately does not depend on the count — "no entry in this set is an assertion failure" is a property of the set, unaffected by how its members are tallied. CI remains the source of truth for the socket- and Docker-dependent suites.
🔧 Fix: close the response-validation gaps between streaming, cascade and validation (2026-07-20)
Repo: EDDI (feat/error-handling-recovery)
Clears the four follow-ups recorded in the entry below. Each was pre-existing, and together they meant responseValidation — a headline feature of this PR — silently did nothing on the paths users are most likely to run it on. Every fix is TDD'd: the test was written first and observed failing.
1. Streaming now derives warning from finishReason
LegacyChatExecutor maps LENGTH → truncated and CONTENT_FILTER → content_filter; the streaming executor captured finishReason but never derived the warning, so onTruncation and onContentFilter could never fire on a streaming task. buildMetadata now mirrors the buffered executor. A later timeout/error warning deliberately overwrites it — a transport failure is the more urgent signal.
2. The cascade now carries the winning step's validation metadata
CascadeResult and the internal StepResult carried only tokenUsage; the producing executor's warning/streamingTimeout/finishReason were dropped on the floor. With modelCascade and responseValidation both enabled, only onEmpty and onRefusal could fire — a truncated or content-filtered cascade answer reached the user even with action: "error" configured. Both records gained a responseMetadata component, threaded from the legacy, streaming and agent-mode step paths, and merged into responseMetadata in LlmTask.
3. A timed-out live-streamed final step no longer wins
executeCapturing returns the (possibly empty) partial text on timeout instead of throwing, so a timed-out final step was accepted on the same footing as a real answer — and being last, it beat a good earlier response. A mid-stream error already fell back correctly; a timeout did not. A step whose metadata reports streamingTimeout is now treated as failed: it never becomes bestSoFar, it escalates when it is not the last step, and on the last step it falls back to bestSoFar (marked streamedLive so LlmTask does not re-emit different text over tokens the client already received).
Enabling this required executeCapturing to honour the task's streamingTimeoutSeconds, which it previously ignored by passing task = null — so the cascade was always pinned to the 120s default. The task's retry config is still deliberately not applied on the cascade path: the cascade owns escalation, and retrying inside a step would multiply spend against the very model it is about to escalate away from.
4. Restored the lapsed cascade error-path coverage
New CascadingModelExecutorErrorPathTest — written against the current implementation rather than restored verbatim — covering what was lost when CascadingModelExecutorExtendedTest was deleted and CascadingModelExecutorCoverageTest was rewritten on main: last-step timeout with and without a bestSoFar, timeout escalation firing onCascadeEscalation("timeout"), LifecycleException and plain-RuntimeException cause handling, aggregated all-steps-failed errors, and enableInAgentMode=false never consulting the orchestrator.
5. Cascade failures are now diagnosable (found while writing #4)
Two of the restored tests failed for a reason the tests were right about: the retry wrapper throws a generic "Chat model execution failed after N attempts", and the cascade recorded only e.getMessage(). So a fully failed cascade reported Step 0 (cheap): Chat model execution failed after 1 attempts; Step 1 (expensive): … — byte-identical whether the cause was a rate limit, an auth failure, a malformed request or a network outage, in both the thrown message and the audit trace. A new describeFailure() appends the root-cause message (bounded cause-chain walk, so a cyclic chain cannot hang the error path).
Verification
Full LLM module: 2,260 tests, zero assertion failures — every reported failure/error message is a blocked-loopback or socket-resource error from this sandbox, none a code assertion. All 73 cascade tests and 33 streaming-executor tests green. (Counts of each bucket are deliberately not quoted here; see the correction note in the merge entry below for why the earlier per-bucket figures were unreliable.)
Design decisions
Metadata threaded as a record component, not a side channel.
CascadeResultis the cascade's public contract withLlmTask; anything the caller must validate belongs on it rather than in a mutable out-parameter.A timed-out step is a failed step, not a low-confidence one. Demoting it via confidence would still let it win when no earlier step scored higher; excluding it from
bestSoFaroutright is what makes the fallback correct.Retry stays off inside cascade steps. Escalation is the cascade's retry mechanism; stacking both multiplies cost in a way no config expresses.
🐛 Fix: two latent retry-loop defects in StreamingLegacyChatExecutor (2026-07-20)
Repo: EDDI (feat/error-handling-recovery)
Surfaced by an adversarial post-merge review of the streaming executor (see the merge entry below). Both defects pre-date the merge — they came in with this branch's retry loop, not with the conflict resolution — but both live inside the method the merge rewrote, and both are squarely in this PR's own subject area (error handling and recovery), so they are fixed here rather than deferred.
1. A failed attempt's metadata leaked into a successful retry
metadata is declared outside the retry loop and mutated inside it. If attempt 1 timed out with no tokens (setting streamingTimeout=true) and attempt 2 succeeded, the successful response was returned with the stale streamingTimeout flag still attached. LlmTask.applyResponseValidation reads that flag and fires responseValidation.onStreamingTimeout — so with action: "fallback" a perfectly good answer was silently replaced by the "I wasn't able to generate a complete response" string, and with action: "error" the turn threw. The loop body now clears the map at the start of each attempt, so each attempt reports only its own outcome.
2. maxAttempts <= 0 skipped the model entirely and returned a null response
RetryConfiguration.setMaxAttempts does no clamping, so a config of retry: {maxAttempts: 0} — a natural way to write "don't retry" — made for (attempt = 1; attempt <= 0; …) never execute. The model was never invoked, responseText stayed null, and StreamingResult(null, …) propagated into LlmTask and on into a TextOutputItem(null, 0): no exception, no log, a completely silent turn. Now clamped with Math.max(1, …) — "no retries" means one attempt.
Deliberately not clamped in RetryConfiguration.setMaxAttempts itself: the shared executeWithRetry already fails loudly on maxAttempts=0 (throws LifecycleException), and changing the setter would silently alter that contract for the MCP and HTTP-call consumers and their existing tests. The clamp belongs at the streaming call site, which is the one that was failing silently.
Tests
Three regression tests added to StreamingLegacyChatExecutorRetryTest, each confirmed to fail before the fix and pass after: timeoutThenSuccess_doesNotLeakTimeoutMetadata, zeroMaxAttempts_stillRunsOnce, negativeMaxAttempts_stillRunsOnce. A fourth, executeCapturing_propagatesErrorDespitePartialTokens, pins the merge's central contract at the executor level — until now it was guarded only indirectly, by a cascade-level test.
155 tests green across the streaming, cascade, orchestrator and LlmTask suites.
Known follow-ups (not fixed here — pre-existing, out of scope for this PR)
Cascade drops the validation-signal metadata.
CascadeResultcarries onlytokenUsage, not the producing executor'swarning/streamingTimeout. WithmodelCascadeandresponseValidationboth enabled, onlyonEmptyandonRefusalcan fire;onTruncation,onContentFilterandonStreamingTimeoutare unreachable. Coverage was worse before the merge (the cascade path left the metadata map empty), so this is an exposure, not a regression.Streaming never derives
warningfromfinishReason.LegacyChatExecutormapsLENGTH→truncatedandCONTENT_FILTER→content_filter;buildMetadatadoes not, so those two policies are inert on the streaming path even without the cascade.A timed-out live-streamed final cascade step is accepted as the winner rather than falling back to
bestSoFar— a mid-stream error falls back correctly, a timeout does not.~500 lines of cascade error-path tests lapsed on
mainbefore this merge (CascadingModelExecutorExtendedTestdeleted; 15 of 16 tests inCascadingModelExecutorCoverageTestreplaced), covering exception-unwrapping and timeout-escalation. Worth re-adding against the current implementation.
🔀 Merge origin/main into feat/error-handling-recovery — conflict resolution (2026-07-20)
Repo: EDDI (feat/error-handling-recovery)
Second merge of main into the error-handling branch (PR #593), bringing in 43 commits (group conversations, MCP/REST ownership hardening, multi-model cascade enterprise work). Seven files conflicted; beyond those, two files were silently mis-merged and one behavioural regression was introduced by the first resolution — both classes of problem are recorded below, since neither is visible in git status.
Conflicts resolved (7)
RestAgentEngine— main refactored the descriptor-based ownership check intoConversationAccessGuard.requireConversationOwner(); ours addedIConversationMemoryStorefor the new adminresetState()endpoint. Resolution keeps both:conversationMemoryStorestays (still used byresetState),conversationDescriptorStoreis dropped (main removed its last usage — verified no remaining references). Constructor is now(conversationService, conversationMemoryStore, identity, ownershipValidator, conversationAccessGuard, hitlAccessGuard, hitlToolJournalStore, agentTimeout).RestAgentEngineTest/RestAgentEngineHitlTest/RestAgentEngineToolPauseDetailsTest— constructor-arity fallout from the above, updated to the merged signature. The HITL and tool-pause tests also swapped their inlinemock(ai.labs.…IConversationMemoryStore.class)FQN for a top-level import per AGENTS.md §4.7.CascadingModelExecutorCoverageTest— ours removed theLlmConfiguration.RetryConfigurationshim in favour ofai.labs.eddi.configs.shared.RetryConfiguration; main's newer version of the file still used the nested type. Resolved onto the shared type, keeping ours'task.setParameters(...)line.StreamingLegacyChatExecutor— the substantive one; see below.docs/changelog.md— both sides prepended entry blocks; both kept whole, main's block (which carries the newest entry) first.
StreamingLegacyChatExecutor — two overlapping features unified
Both branches rewrote the same method for different reasons:
ours added a configurable timeout, a retry loop driven by
RetryConfiguration, andfinishReasonmetadata, returningStreamingResult;main added
executeCapturing()returningStreamResultwith fullChatResponsemetadata (token usage), so the cascade keeps cost evidence when it streams the final step live.
Neither could be dropped — LlmTask calls the retry-aware overload, CascadingModelExecutor calls executeCapturing, and each has its own tests. The merged class keeps one core implementation (retry + configurable timeout) that now captures the whole ChatResponse and builds metadata via main's buildMetadata() (finishReason and tokenUsage); both public entry points delegate to it.
Regression caught during merge review: the first resolution had executeCapturing delegate with ours' error semantics, which salvage partial text on a mid-stream error instead of throwing. CascadingModelExecutor has no try/catch at that call — it relies on the throw to fall back to the best previous step — so a failed final step was silently accepted as successful (CascadingModelExecutorEnterpriseTest.streamingFinalStepFailsMidStream_fallbackMarkedStreamedLive: expected: <0> but was: <1>). The core now takes a salvagePartialOnError flag: true for the LlmTask path (keep whatever the model produced rather than fail the turn), false for executeCapturing (always propagate, so the cascade can fall back). Timeout handling is unchanged — both sides already salvaged partial text there.
Silent auto-merge breakage (fixed)
Ours deleted the nested LlmConfiguration.RetryConfiguration shim while main added new usages of it. Git merged both sides cleanly and the result did not compile:
AgentOrchestratorExtendedTest(2 usages) andCascadingModelExecutorEnterpriseTest(1 usage) — both moved toai.labs.eddi.configs.shared.RetryConfiguration, import added.
Only a clean test-compile surfaced these; they were not reported as conflicts.
Auto-merges verified, not assumed
The five remaining files touched by both sides were diffed against each parent to confirm neither side's changes were dropped:
IConversationService/ConversationService/RestAgentEngineStreaming— ours'onTaskFailedcallback andresetConversationStatealongside main'sonCascadeStepStart/onCascadeEscalation; orthogonal additions to the same interface and anonymous handler.LlmConfiguration— ours'responseValidation+streamingTimeoutSecondsalongside main'sjudgeModel/heuristic/maxTotalDurationMs.LlmTask— ours' retry-aware streaming call andapplyResponseValidationalongside main's cascade wiring andMeterRegistry. ConfirmedapplyResponseValidationsits outside theif (cascadeActive)branch, so response validation still runs on every path including the cascade.
Design decisions
Keep both streaming entry points rather than collapsing to one. Their error contracts genuinely differ (salvage vs. propagate) and each has a real production caller; one core plus an explicit flag beats duplicating ~50 lines of streaming boilerplate or silently changing one caller's behaviour.
conversationDescriptorStoredropped, not re-plumbed.ConversationAccessGuardis the single ownership-check path now; keeping a second route to the descriptor store would reintroduce exactly the drift the guard exists to remove.Changelog blocks kept whole rather than interleaved by date — the file is already organised in per-branch blocks, and re-sorting would produce a large, unreviewable diff.
Verification
mvnw clean test-compile green — a clean build deliberately, since the RestAgentEngine signature change would be masked by a stale incremental one.
Full unit suite: 11,409 tests run, 8 failures and 304 errors, none of them an assertion failure. Every failure/error message in the surefire XML falls into one of three buckets — Unable to establish loopback connection, Docker/Testcontainers unavailable (the Mongo + Postgres store tests), and socket-selector/event-loop creation errors — i.e. this sandbox's inability to open loopback sockets or run Docker. CI remains the source of truth for those suites.
Correction (post-review): an earlier draft of this entry gave a per-bucket breakdown (297/16/5 = 318) that did not reconcile with the run's 312 failures+errors. The categorisation had been run over
target/surefire-reportswithout a precedingclean, so it also counted XML left behind by earlier targeted runs — a superset of the full run. The conclusion is unaffected (a superset containing zero assertion failures still contains zero), but the counts were not defensible as stated and have been removed rather than quietly adjusted.
The 56 tests directly covering the merged paths pass locally: StreamingLegacyChatExecutor{,Retry,Coverage}Test, CascadingModelExecutor{Coverage,Enterprise}Test, RestAgentEngine{,Hitl,ToolPauseDetails}Test, AgentOrchestratorExtendedTest.
Next
Merge-ready for PR #593. Remaining follow-up unrelated to the merge: ours' onTaskFailed SSE handler in RestAgentEngineStreaming still hand-builds its JSON via String.format + escapeJson, while main introduced a sendJsonEvent(...) Jackson helper on the same class and documents it as preferred. Correct as-is (the payload is escaped), but worth converting for consistency.
🔀 Merge origin/main into feat/group-followups — conflict resolution (2026-07-20)
Repo: EDDI (feat/group-followups)
Merged origin/main (multi-model cascade enterprise hardening + MCP/REST conversation-ownership security hardening) to resolve PR #595's merge conflicts against main. Two files conflicted, both non-substantive:
ToolExecutionTrace.java: both branches independently documented the samesynchronizedrationale onaddToolCall/addFailedToolCall— ours as a Javadoc block above each method,origin/main's as an inline comment restating it. Kept the existing Javadoc, dropped the redundant inline comment.docs/changelog.md: both branches appended new entries directly below the file header. Resolved as a union, newest-first per branch: this branch's own entries (group follow-ups work, 2026-07-08 to 2026-07-15) kept first, followed byorigin/main's entries (MCP/REST conversation-ownership hardening + multi-model cascade, 2026-07-03 to 2026-07-15).
🧵 ToolExecutionTrace — thread-safe recording (fixes CI flake) (2026-07-15)
Repo: EDDI (feat/group-followups) — a pre-existing bug on main, unrelated to the group work, that surfaced as an intermittent full-suite CI failure on this branch: ToolExecutionServiceBranchTest.executeMultipleInParallel → expected <Hello, Alice!> but was <Error executing tool: ConcurrentModificationException>. Committed here to unblock this branch's CI (per decision), clearly scoped as an independent fix.
Root cause (systematic-debugging, root-caused before any fix): ToolExecutionService.executeToolsParallel shares one ToolExecutionTrace across every concurrent task by design (it's the accumulator). But the trace's addToolCall / addFailedToolCall mutated a plain ArrayList, a plain HashMap (toolMetrics), and non-atomic counters with no synchronization. Under real parallelism, HashMap.computeIfAbsent detects concurrent structural modification and throws ConcurrentModificationException; executeTool catches it and returns "Error executing tool: " + e.getClass().getSimpleName() (CME's message is null) — the exact observed string. This is a genuine production bug: executeToolsParallelAndWait is a live API.
Fix: synchronized on both trace mutators, serializing concurrent recording (covers the collection adds, the computeIfAbsent, and the primitive accumulations). updateMetrics is private and only called under those locks. Reads happen after CompletableFuture.allOf(...).join() (a happens-before edge), so writer synchronization is sufficient.
Coverage: new concurrentToolsShareTraceWithoutCorruption stress test — 50 rounds × 32 parallel tasks sharing one trace, asserting no task returns an error string and every call is recorded exactly once (no lost ArrayList updates). It fails reliably on the unsynchronized version (reproduced the CME at round 41) and passes 5/5 with the fix. The original 2-task test had too small a race window to be a reliable regression guard.
🔍 Group conversations — Copilot PR-review response (2026-07-15)
Repo: EDDI (feat/group-followups)
Copilot flagged 5 items on the pushed branch. Each was adversarially verified against the actual code (a 6-agent verification workflow plus independent reading) before implementing — external review is evaluated, not rubber-stamped. All 5 confirmed, all fixed:
MCP raw-exception leak (×3, Medium) —
followup_with_member,continue_group_discussion,close_group_conversationreturnederrorJson(e.getMessage()), forwarding raw internal exception text to MCP callers (information exposure). Now each logs the full throwable server-side (LOGGER.error(msg, e)) and returns a stable curatederrorJson("Failed to …", "INTERNAL", null)— matching the hardenedMcpHitlToolsconvention. (Verified the 3-argerrorJsonoverload exists, so this compiles; Copilot's suggested form was correct despite its own hedge.)REST 400 missing
TEXT_PLAIN(Medium) —rejectAttachmentsOnContinue()was the sole error response inRestGroupConversationnot setting.type(TEXT_PLAIN). Added, for content-type consistency with every sibling 400/404/409/504.Stale concurrency comment (Low) — the
operationsInProgressfield comment calledcompareAndSetStatea "best-effort read-check-update". That is now false:GroupConversationStore.compareAndSetStatedoes a fast-path read then an atomic storage-layer conditional write (storeIfFieldEquals→ single MongoupdateOne/ PostgresUPDATEfiltered on the currentstate). Corrected the comment: the in-memorySetis a single-node fast-fail optimization; the cluster-wide guard is the storage CAS.
Coverage (mutation-verified): three new McpGroupToolsTest cases drive each tool's generic catch (service throws a recognizable boom-internal-detail-42) and assert the response does not contain the raw text and does contain the curated message + "errorCode":"INTERNAL". Reverting all three curations fails exactly those three tests (and nothing else) — proving they pin the non-leak contract. 46 McpGroupToolsTest cases pass (was 43); full group/MCP suite green; Checkstyle clean.
Deliberately out of scope: the same errorJson(e.getMessage()) pattern exists in ~10 pre-existing catch blocks in McpGroupTools (and other MCP tool classes), and two existing tests depend on those messages (start_group_discussion → "Group not found", delete → "Not found"). A blanket sweep would break tested behaviour and expand well beyond this PR, so it is left as a separate follow-up rather than bundled here.
🌐 Group follow-up/continue — status-split completion: mid-round timeout → 504 (2026-07-15)
Repo: EDDI (feat/group-followups)
An adversarial re-review of the status-code split found one path the split had missed. executeAgentTurn (the per-member turn used by the initial discussion and re-run by every continuation) still threw a base GroupDiscussionException on an ABORT-policy member timeout — so a genuine member-agent timeout mid-round mapped to 502 Bad Gateway, not the 504 Gateway Timeout the split documents. Every other timeout site was already GroupTimeoutException; this was the last one.
Fix:
GroupConversationService.executeAgentTurnnow throwsGroupTimeoutExceptionon theonAgentFailure=ABORTtimeout branch.executeDiscussion's re-wrap preserves the subtype, so it surfaces as504.REST body hedge:
continueDiscussion's502(GroupExecutionException) body previously claimed only "a member agent could not be reached". That catch also covers unrunnable-config failures, so the body now reads "a member agent or a required dependency could not be reached, or the group is misconfigured" — no longer asserting a single cause it cannot verify.Coverage (mutation-verified): new
GroupConversationServiceExtendedTest.FailurePolicies#abortPolicy_agentTimesOut_throwsGroupTimeoutExceptiondrives a real member timeout (adoNothing()say stub so the future never completes) underABORTand assertsGroupTimeoutException. Reverting the fix to the parentGroupExecutionExceptionflips exactly this one test to failing ("expected GroupTimeoutException but was GroupExecutionException") — proving it pins the 504-vs-502 distinction, not just "some 5xx".
250 group/MCP unit tests pass (52 extended, 82 service, 39 REST, 3 routing, 31 HITL, 43 MCP); Checkstyle clean.
🌐 Group follow-up/continue — HTTP status-code split (2026-07-14)
Repo: EDDI (feat/group-followups)
A final pre-review pass flagged that followUpWithMember / continueDiscussion mapped every GroupDiscussionException to 409 Conflict — including an unknown target agent (a client error) and mid-round server/upstream failures (LLM/DB down, agent timeout). A 409 tells the client "retryable conflict", which is wrong for a typo'd agent or a provider outage. This was pre-existing feature behaviour, not a regression, but it is a real API-semantics issue a reviewer would raise, so it was fixed properly.
Cause-differentiated exception subtypes (all extend
GroupDiscussionException, so existingcatch (GroupDiscussionException)in the MCP tools and tests keeps working):GroupMemberNotFoundException(→ 404),GroupExecutionException(→ 502),GroupTimeoutException extends GroupExecutionException(→ 504). The base type now means only a state/concurrency conflict (→ 409).Single interception point for execution failures:
executeDiscussionre-throws every failure caught in its phase loop asGroupExecutionException(preservingGroupTimeoutException), so the many deep agent/quota/config throw sites did not each need editing.followUpWithMember's own agent-call/timeout throws are re-typed directly.REST mapping (most-specific catch first): unknown member →
404, agent timeout →504, agent/model failure →502(Bad Gateway — an upstream dependency failed, logged with its stack trace), state/concurrency →409. The@APIResponseannotations now list the full set.closeis unchanged (only ever a state conflict → 409).Nits swept: the new
400bodies now set.type(TEXT_PLAIN)for consistency; the follow-upInterruptedExceptionpath restores the interrupt flag.
Coverage: added REST tests (followUp/continue → 404/502/504, and still-409 for a genuine conflict) and service tests asserting the specific subtypes are thrown (unknown member → GroupMemberNotFoundException, agent failure → GroupExecutionException, and the executeDiscussion failure-policy tests now assert GroupExecutionException). Each new mapping was mutation-verified: reverting the split makes the corresponding test fail (404/502/504 → 409; specific subtype → base). 654 group/MCP unit tests pass; Checkstyle clean.
🧬 Group conversations — mutation-audited regression coverage (2026-07-14)
Repo: EDDI (feat/group-followups)
The question "do we have coverage for everything we fixed?" was answered with mutation testing rather than by reading test names: each fix was reverted in turn and the suite re-run. A fix whose mutant survives (suite still green) has no coverage and can silently regress.
Mutants that survived — i.e. bugs with ZERO coverage
JAX-RS routing — the /{groupId}/conversations prefix on the four post-discussion endpoints
The unit tests invoke resource methods directly and never exercise JAX-RS path binding. This was the single highest-severity bug of the whole effort (every follow-up/continue/close would have 404'd), and it could be reintroduced with the suite still green.
SSE error curation — the service pushing raw e.getMessage() into GroupErrorEvent
Nothing asserted what actually goes out over the wire to the browser.
Malformed-id reflected value — Mongo's ObjectId parser echoing the raw caller string
No test drove an unparseable id.
Curated exception messages (store + loadInGroup)
The existing test asserted the response body, which the REST layer curates anyway — so the message itself (which read/delete surface through the global mapper) was unguarded.
Continuation startPhaseIndex = 0
Nothing asserted that a continuation re-runs from the FIRST phase; a mutant that skipped phase 0 passed.
finally cleanup condition (defer COMPLETED, reclaim on FAILED/CANCELLED)
Nothing asserted that a COMPLETED round keeps its ephemeral agents for follow-ups.
resumeQuestion read side
Only the write was tested; nothing asserted that resumeDiscussion actually uses it.
The three new metrics counters
Never asserted.
Coverage added (each verified to KILL its mutant)
IRestGroupConversationRoutingTest(new) — reflective assertions on the JAX-RS annotations: every@PathParammust have a matching{template}segment (a mismatch bindsnull— the exact production failure), every per-conversation route stays under/groups/{groupId}/conversations/{groupConversationId}, and the four endpoints resolve to their documented URLs. This is the invariant a direct-invocation test structurally cannot check.GroupConversationServiceExtendedTest.MergeRegressionGuards(new) — a failed discussion streams a curated error (never the raw exception text); a continuation re-runs from phase 0 and emitsround_start(notgroup_start); a COMPLETED round keeps its ephemeral agents.GroupConversationServiceHitlTest— a paused continuation resumes with the follow-up question, not the stale round-1 one.GroupConversationStoreTest/RestGroupConversationTest— the not-found message never embeds the caller id; a malformed id is answered 404 without reflecting the payload; the group-mismatch exception message is curated.GroupConversationServiceTest— the three operation counters, and the failure counter incrementing even when the CAS is lost.
Mutants already killed before this pass (genuinely covered): failConversation's upsert, the MCP ownership gate, CLOSED-blindness in persistedTerminalOverride, continueDiscussion's conditional write, the control-token pre-registration, cancelDiscussion's CLOSED guard, availableActions for CANCELLED, and the MCP list owner-filter.
647 group/MCP unit tests pass; Checkstyle clean. No production code changed in this commit.
🛡️ Group conversations — terminal-state integrity + error-body hardening (2026-07-14)
Repo: EDDI (feat/group-followups). Found by an adversarial review of the previous PR-review-response commit — which had hardened the success write in continueDiscussion while leaving the failure write, and the rest of the reflected-value surface, wide open.
Terminal states are now irreversible
failConversationwas an unconditional whole-document write — i.e. an UPSERT. It could re-create a group conversation another pod had deleted, and could overwrite a terminalCANCELLED(committed by a cross-pod cancel) withFAILED, clobbering that writer's transcript. It is now a conditional write.The CAS expectation is taken from the PERSISTED state, not the in-memory one. A first attempt CASed on
gc.getState()and was itself a blocker:executeDiscussionflips the conversation toSYNTHESIZINGin memory before the synthesis phase runs and only persists it afterwards, so a failure inside a synthesis phase would have CASedSYNTHESIZINGagainst a persistedIN_PROGRESS, lost the race, skipped the write, and stranded the conversationIN_PROGRESSforever — worse than the bug being fixed.failConversationnow re-reads the persisted state, skips the write when it is already terminal (aligning the in-memory state to it, so thefinallymakes the right ephemeral-agent decision), and counts the failure metric unconditionally so a lost race can never hide a failure from operators.
No exception text reaches the client (CodeQL: information exposure / reflected value)
The previous commit curated one handler. This closes the class:
Throw sites:
GroupConversationStoreandGroupConversationServiceno longer embed caller-supplied ids in exception messages ("Group conversation not found: {id}","Group not found: {groupId}","No phases defined for group: {groupId}").Sinks:
RestGroupConversationreturns no raw exception text in any body — every 400/404/409 is a curated, deliberately non-committal message (these exceptions cover several causes, so the body must not assert one), with the detail logged viaLogSanitizer.SSE: the service was pushing raw
e.getMessage()intoGroupErrorEvent, which the streaming listener forwards to the browser — so LLM/DB/driver detail (and the caller's own input) reached the client even though the REST catch sites were curated. Those events are now curated too, and the raw cause is logged with its stack trace.Malformed ids: a non-hex id reaches Mongo's
ObjectIdparser, whose message embeds the raw caller string — the most exploitable sink. It is caught insideloadInGroup, scoped to the id lookup only, and answered with a curated 404. It is deliberately not a blanketcatch (IllegalArgumentException)around the whole operation: that would mask a genuine internal bug as a false "not found" and hide its stack trace.
Verified: 636 group/MCP unit tests pass (incl. new regression tests for the persisted-state CAS and the already-terminal skip); Checkstyle clean. Remaining full-suite failures are environmental only (Testcontainers/Docker + loopback-socket suites).
🤖 Group follow-ups — automated PR review response (CodeQL / Copilot / CodeRabbit) (2026-07-14)
Repo: EDDI (feat/group-followups) — responses to the bot reviews on PR #595.
Correctness
Terminal-state resurrection in
continueDiscussion(Copilot, High): after theCOMPLETED → IN_PROGRESSCAS, the round/question mutation was persisted with an unconditional whole-documentupdate(). A cancel/close/delete winning the window would be overwritten and the conversation resurrected asIN_PROGRESS. Now a conditional write (updateIfState(gc, IN_PROGRESS)) →409on conflict. This is the same defect class already fixed infollowUpWithMember; the continue path had been missed.
Security — reflected input & error exposure
Reflected
groupIdin 404 bodies (Copilot, Medium):loadInGroup()embedded the caller-suppliedgroupIdin its exception message, which follow-up / continue / close echo verbatim into the404body (CodeQL reflected-value/XSS). Now a curated body; both ids are logged server-side viaLogSanitizer.Reflected
targetAgentIdin 409 bodies:followUpWithMember's "not a member" message echoed the caller-supplied agent id into the409. The id is no longer reflected (the available-member list — server data — is kept).Information exposure through an error message (CodeQL, Medium): the delete-conflict
409returned the rawe.getMessage(). Now a curated "busy, please retry" body with the detail logged server-side.
Observability & docs
Metrics for the new operations (CodeRabbit):
followUpWithMember/continueDiscussion/closeGroupConversationwere uninstrumented, contrary to AGENTS.md. Addededdi_group_followup_count,eddi_group_continue_count,eddi_group_close_count. Instrumented in the service (not the MCP tools, as suggested) so the counters cover the REST and MCP surfaces.Authorization denials now log at WARN (CodeRabbit) — a security-relevant event should be alertable.
getAvailableActions()javadoc corrected (Copilot, Low): it claimed "not persisted", but Jackson serializes it into stored documents. It isREAD_ONLY, so the value is never read back and is always recomputed fromstate— the javadoc now says so rather than making a false claim.
Tests
Post-CAS IN_PROGRESS read modelled so the FAILED recovery path is actually exercised; close now asserts a CLOSED result rather than assertSame on a stale instance; added the concurrent-terminal-transition conflict test, the SSE cancelled callback test, the admin-bypass tests (the other half of requireOwnerOrAdmin and the list-filter exemption), and assertions that neither the raw exception text nor the caller-supplied groupId reaches the client.
Not actioned (false positive)
CodeRabbit (Major) claimed
updateIfStatewrappingResourceNotFoundExceptionin the uncheckedGroupConversationGoneExceptionbypasses the REST layer's 404 handling and yields a 500. It does not:RestGroupConversationexplicitly catchesGroupConversationGoneExceptionalongsideResourceNotFoundExceptionin a multi-catch on every surface that exposes the operation (cancel / approve / approve-stream) and maps it to404. The unchecked type is a deliberate design from the HITL work (documented on the exception) so existing CAS call sites keep compiling;compareAndSetStateconverts it toResourceNotFoundExceptionfor its own callers. No change made.
🔐 Group merge — third-pass review: MCP authz, CLOSED-blindness, atomic CAS (2026-07-14)
Repo: EDDI (feat/group-followups)
A third critical review targeted what the earlier passes never looked at — the auto-merged files (git merged them without conflict, so nobody had reviewed them), the whole-branch PR surface, test coverage of the fixes, and security. 14 findings survived adversarial verification. All fixed:
Security — MCP was an authorization bypass (IDOR)
McpGroupToolsgated the conversation-scoped tools on a role check only (eddi-viewerforfollowup_with_member/continue_group_discussion/read_group_conversation,eddi-editorforclose/delete) with no ownership check, while the equivalent REST endpoints all enforcerequireOwnerOrAdmin(403). Any authenticated viewer could read another user's full transcript, append to it, re-run every phase against their conversation (burning their LLM budget), and an editor could close or delete it. InjectedOwnershipValidatorand added arequireConversationOwner()gate to all five tools, with a uniform non-leaking denial. Main's own HITL MCP tools already enforced this viaHitlAccessGuard— MCP is now consistent with REST.Owner resolution on creation (found reviewing the gate above): MCP recorded the owner as the literal
"mcp-client"(or any caller-supplieduserId), so the new gate would have locked the creator out of their own conversation whenever auth was enabled — and let a caller create a conversation owned by someone else.discuss_with_group/start_group_discussionnow resolve the owner viavalidateAndResolveUserId(the calling principal; impersonation rejected), falling back to"mcp-client"only when auth is off.list_group_conversationsis now owner-filtered (mirroring REST). It returns full conversation documents, so without this the per-conversation gate was pointless — a non-owner could simply list the group and read everyone's transcripts.
Correctness — a systemic CLOSED-blindness
Ours introduced CLOSED as a new terminal state; theirs' HITL/cancel code predates it. A sweep of every terminal-state check found exactly two blind spots:
persistedTerminalOverridetreated only{CANCELLED, FAILED, COMPLETED}as terminal, so a running leg that found the conversationCLOSEDdid not stop — it fell through to an unconditional whole-document write and resurrected the closed conversation toIN_PROGRESS→COMPLETEDafter its member conversations were ended and its ephemeral agents deleted. (The previous round's fix F — close-of-CANCELLED— made this materially more reachable.)cancelDiscussionlikewise ignoredCLOSED, so a cancel could CASCLOSED → CANCELLEDand un-terminalize an irreversible state.
Correctness — the cross-process guard wasn't atomic
compareAndSetStatewas a read-check-write, not a CAS: it read, compared in Java, then wrote unconditionally, so two racing callers could both pass the check and both write. It is the only cross-process guard behind follow-up/continue/close (the original changelog admitted "single-node only… would require a conditional update at the storage layer"). The merge made the fix available — it now uses theirs'storeIfFieldEquals, returningfalseon a lost race.
Contract / robustness
followUpWithMemberwith a null/blanktargetAgentIdNPE'd into a 500; now validated → 400 (service throwsIllegalArgumentException, REST rejects up front). Same for a blankquestionon follow-up and continue.POST /continueadvertisedattachmentson its body but silently dropped them. Now rejected with 400 rather than silently ignored. (A first attempt to honour them was reverted after review: attachments are granted and injected to a member only on its first-ever turn, and on a continuation every member conversation already exists — so the "fix" was a no-op that stored an orphaned blob and still returned 200. Actually sharing new files mid-conversation needs the attachment fan-out reworked — see What's next.)DELETEduring an in-flight follow-up/continue returned 500; now aGroupDiscussionException→ 409.OpenAPI updated for the new 400/409 responses and the
CANCELLED-closeable state.
Tests
Backfilled the previously untested fixes (they could each have been reverted with the suite still green): resumeQuestion persistence, control-token pre-registration + removal, persistedTerminalOverride state alignment across all four terminal states, cancel-of-CLOSED, conditional-CAS + lost-race, MCP ownership denial/allow, blank-input 400s, continue-attachment forwarding, delete 409, and the streaming listener forwarding HITL + round_start events.
Also fixed a latent broken test the merge introduced: GroupConversationHitlTest still asserted 7 enum states (the merge made it 8 with CLOSED) — it was never in the narrower test selections and had been failing since the merge commit.
What's next (deliberately not done here)
Attachments on a continuation round.
grantAndInjectAttachmentsruns only on a member's first-ever turn (privateConvId == null), so a continuation cannot share new files. Supporting it means reworking the fan-out to grant/inject per round (e.g. tracking which member conversations have been granted the current attachment set) — a change to shared attachment code, out of scope for a merge-response fix. Until then/continuerejects attachments with 400.Follow-up to a dynamically recruited agent.
GroupConversation.dynamicMembershas no production writer (sub-agent creation records onlycreatedAgentIds), so recruited agents are not addressable as follow-up targets at all. A first attempt to resolve them by display name was reverted as dead code; the real fix is to register recruited agents as members (roster +memberConversationIds).
Verified: mvnw test green — 758 group/MCP unit tests pass (0 failures). The remaining full-suite failures are environmental only (Testcontainers/Docker and loopback-socket suites — Mongo/Postgres stores, HTTP tool tests); no group or MCP test fails. CI covers those.
🩹 Group merge — cross-feature review-response fixes (2026-07-14)
Repo: EDDI (feat/group-followups)
A deep adversarial review of the merge (7 dimensions, 58 agents, each finding cross-examined by 3 skeptics) confirmed the conflict resolution was sound but surfaced cross-feature interaction bugs between ours (continue/follow-up/close) and theirs (HITL pause/cancel/resume) that neither branch could have had alone — none caught by the impl-level unit tests. Fixed:
A — stale question on continuation resume: a continuation round that paused at an HITL gate resumed with the round-1 question (
resumeDiscussionreadoriginalQuestion, whichcontinueDiscussionnever updated) — silent wrong multi-agent output. Added a dedicatedGroupConversation.resumeQuestionfield:continueDiscussionsets it,resumeDiscussionreads it (falling back tooriginalQuestionfor round 1 / legacy docs). Kept separate fromoriginalQuestionso the Manager conversation-list title (which rendersoriginalQuestion) is not rewritten by continuations.B — continue/stream dropped HITL events:
continueDiscussionStreamingused a hand-rolled inline SSE listener predating theirs' HITL callbacks, so a continuation that paused/cancelled emitted no event and hung the client + leaked the sink. Unified it on the sharedcreateStreamingListener(added anonRoundStartoverride there); removed ~55 lines of duplication.C — ephemeral-agent leak on cross-pod terminal race: the merged
finallycleans up only onFAILED/CANCELLED(to deferCOMPLETEDfor follow-up reuse), but the cross-pod terminal-override and lost-completion-CAS exits left in-memory state stale (running/optimistic-COMPLETED), skipping cleanup. Both exits now align in-memorystateto the actual persisted terminal value so thefinallydecides correctly.D — follow-up clobbered a racing cancel:
followUpWithMember's success path used an unconditionalupdate()that could overwrite a concurrentCANCELLED. Switched toupdateIfState(gc, IN_PROGRESS)(matching its own error path); a concurrent cancel/delete now yields a409instead of resurrecting the conversation.E — cancel race + latency on continuation:
continueDiscussiondidn't pre-register aDiscussionControlToken, so a cancel racing the CAS→executeDiscussionwindow took the DB branch and was overwritten, and cancel latency was a whole phase worse. Now pre-registers the token right after the CAS (mirrorsstartAndDiscussAsync/resumeDiscussion); removed on the pre-exec failure path.F —
CANCELLEDhad no reclaim path: a cancel landing in the follow-up/continue pre-exec window could reachCANCELLEDwith orphaned ephemeral agents and no recovery.closeGroupConversationnow acceptsCANCELLED → CLOSEDandgetAvailableActions()returns["close"]forCANCELLED, giving operators a reclaim path.
Added regression tests (CANCELLED available-actions, close-of-CANCELLED); updated the follow-up success-write assertion. Verified: mvnw test green — 189 group-conversation unit tests pass (0 failures); each fix re-verified by an adversarial pass (5/6 clean first time; A refined from overloading originalQuestion to the dedicated field per that review).
🔀 Merge origin/main into feat/group-followups — conflict resolution (2026-07-14)
Repo: EDDI (feat/group-followups)
Merged 170 commits of origin/main (HITL framework + multimodal-attachments group parity) into the group follow-up/continuation/close branch. Both sides evolved the group-conversation subsystem in parallel, so all 23 conflict hunks across 12 files were resolved as a union of the two feature sets. Key decisions:
GroupConversationService.executeDiscussionsetup (conflict): interleaved ours' member-display-name population + round-aware start events (onGroupStarton round 1,onRoundStarton continuation rounds) with theirs' attachment re-hydration, resume-seeded turn counter (pausedTurnCount), HITL granularity, and control-token registration. The start-event now fires only on fresh execution (startPhaseIndex == 0), branching round-1 vs continuation.executeDiscussionfinally-block cleanup (conflict): reconciled ours' "defer ephemeral cleanup for COMPLETED rounds so follow-ups can reuse dynamic agents" with theirs' "keep agents alive whileAWAITING_APPROVAL". Result: always remove the control token; drop the verification cursor unless paused; clean up ephemeral agents only on terminal states with no follow-up/close path (FAILED,CANCELLED).COMPLETEDcleanup stays deferred toclose/delete.GroupConversationStateis now 8 values (ours'CLOSED+ theirs'CANCELLED).getAvailableActions()gained aCANCELLEDarm (terminal, no actions — updated the exhaustive switch); the enum-count guard test was corrected 7 → 8.executeDiscussionsignature: theirs addedint startPhaseIndex; ours'continueDiscussioncall site now passes0(a continuation re-runs the full protocol from phase 0).Group-path guard unified on
loadInGroup(): all six endpoints (read, delete, followup, continue, continue/stream, close) route through ours'loadInGroup(); theirs' parallelrequireGroupMembership()helper was removed as dead code. HITL endpoints keep theirs'validateGroupConversationOwnership.JAX-RS routing fix (would-be regression): theirs flattened the class-level
@Pathfrom/groups/{groupId}/conversationsto/groups, moving the{groupId}/conversationsprefix onto each method. Ours' four methods (followup/continue/continue/stream/close) carried their old class-relative@Path("/{groupConversationId}/…"), so post-flatten they lost the{groupId}template segment while still declaring@PathParam("groupId")—groupIdwould bindnulland every call would 404 (invisible to the unit tests, which call the impl directly). Each of the four method paths was prefixed with/{groupId}/conversations, restoring the original external URLs. Caught by an adversarial merge review.Both feature APIs preserved: ours'
followUpWithMember/continueDiscussion/closeGroupConversation/compareAndSetState+ theirs'cancelDiscussion/resumeDiscussion/approveGroupPhase(/Streaming)/getGroupApprovalStatus/listGroup(All)PendingApprovals/updateIfState/findByState; all SSE events and listener callbacks from both sides retained.Review nitpick:
RestGroupConversationExtendedTestnow has an@AfterEachthat invokes the package-privateRestGroupConversation.shutdown(), so a full suite run no longer accumulates un-terminated virtual-thread executors.
Verified: mvnw clean test-compile green; ~187 group-conversation unit tests pass (GroupConversation 25, Store 19, Rest 25, RestExtended 21, Service 68, Hitl 29 — 0 failures/errors).
🔒 Group Conversation Follow-Ups — review-response hardening (2026-07-13)
Repo: EDDI (feat/group-followups)
Addresses static-analysis and code-review findings on PR #595 (CodeQL, Copilot, CodeRabbit, GitHub Code Quality).
Security & correctness
Log injection (CWE-117): sanitized user-controlled
groupConversationIdin log statements viaLogSanitizer.sanitize()(GroupConversationServicefollow-up recovery, close, delete-not-found, and timeout-resolution paths).Group-path validation:
followup/continue/continue/stream/closenow verify the conversation belongs to the{groupId}in the path — mismatches return404(SSEgroup_errorfor the stream) via a sharedloadInGroup()helper. Closes a "wrong group path" access gap and resolves the unused-groupIdfindings.403 on streaming continue:
continueDiscussionStreamingnow rethrowsForbiddenExceptionso ownership failures map to HTTP403instead of a200SSE error event.
Concurrency
Per-conversation operation guard:
followUpWithMember/continueDiscussion/closeGroupConversationacquire a fail-fast in-process guard (ConcurrentHashMap.newKeySet()) keyed by conversation ID; a second concurrent operation on the same conversation is rejected (409) rather than racing thecompareAndSetStateread-check-update. NOTE: single-node only — cluster-wide atomicity would require a conditional update at the storage layer (documented as future hardening, not built here).
Resource lifecycle
@PreDestroyonRestGroupConversation: the virtual-thread executor is now shut down on bean destroy.Ephemeral cleanup on delete:
deleteGroupConversationnow reclaims dynamically-created agents. Deferred cleanup previously ran only onclose, so deleting aCOMPLETEDconversation orphaned them — this regression is introduced-and-fixed within the same feature branch. ExtractedcleanupEphemeralAgentsForGroup()shared by close + delete.Known limitations (ephemeral-agent reclamation): (a) a
COMPLETEDconversation that is never closed or deleted still retains its ephemeral agents; (b) if cleanup fails ondelete(group config already gone, or a transient undeploy error), the record is still hard-deleted, so thecreatedAgentIdsmapping is lost and a future reaper cannot reclaim those agents (they remain operator-recoverable via the agent store). A TTL reaper (built onScheduleFireExecutor) is the planned mitigation for (a) — tracked as a separate item.
API cleanup
Removed dead
userIdparam fromfollowUpWithMember/continueDiscussion(service interface, impl, REST, MCP tools). Ownership is validated via the stored conversation owner; the param was never used downstream.Configurable follow-up timeout: the follow-up agent call now uses the group's
protocol.agentTimeoutSeconds()(default 60, consistent with discussion turns) viaresolveAgentTimeoutSeconds(), instead of a hardcoded 120s.Model encapsulation:
GroupConversation.getMemberDisplayNames()returns an unmodifiable view; population goes through the newaddMemberDisplayName()method (the getter can no longer be mutated); the setter defensively copies.OpenAPI: added
404responses tofollowup/continue, and200/404+ event listing (incl.round_start) tocontinue/stream.
Round 2 — multi-agent adversarial review + CI (2026-07-13)
Second pass after an adversarial multi-dimension review (concurrency / security / REST / lifecycle / test-coverage) plus the CI test run.
CI green: updated the
GroupConversationState(6→7,CLOSED) andTranscriptEntryType(14→15,FOLLOW_UP) enum-count guard tests. Added the three follow-up MCP tools (followup_with_member,continue_group_discussion,close_group_conversation) toMcpToolFilter— they were filtered out entirely (never exposed to MCP clients) because the whitelist was never updated, so this is a functional fix, not just a test tweak.Delete race (flagged by two review dimensions):
deleteGroupConversationnow participates in the per-conversation guard. Because deferred cleanup made delete a terminal operation, an unguarded delete could tear down member conversations / ephemeral agents while acontinue/follow-upwas mid-run and then be resurrected as a "zombie" document via the store's upsert-by-idupdate().closestatus codes: business conflicts (in-progress / wrong-state) now throwGroupDiscussionException→409; a genuineResourceStoreException(DB failure) falls through to500via the global mapper, instead of every store error mapping to409. Alignsclosewithfollowup/continue.Consistent group-scoping:
readGroupConversationanddeleteGroupConversationnow also route throughloadInGroup(), so every endpoint under/groups/{groupId}/conversations/{id}verifies the conversation belongs to the path group (404 on mismatch). Previously only the new endpoints did.CWE-117 in MCP tools: the three new MCP follow-up tools now sanitize
e.getMessage()before logging (targetAgentIdis user-controlled and flows into exception messages).Test coverage: added unit tests for
getAvailableActions()(per state),memberDisplayNamesencapsulation, therounddefault,compareAndSetState()(all branches),followUpWithMember/continueDiscussion/closeGroupConversationservice logic (display-name resolution, wrong-state, concurrency guard, state restore, round increment), and the new REST endpoints including theloadInGroup404 guard.
Deliberately not done
SSE listener factory extraction (CodeRabbit nitpick): skipped — a pure DRY refactor of working, integration-only streaming code with no behavior gain.
What's next
TTL reaper for abandoned
COMPLETEDconversations (ephemeral-agent reclamation).Optional cluster-wide atomic state transition at the storage layer if concurrent group operations become a real deployment concern.
Optional: sweep the remaining pre-existing
McpGroupToolscatch blocks for the sameLogSanitizertreatment (≈11 older tools still use the unsanitizederrorf(..., e.getMessage())pattern — lower priority, outside this PR's scope).
✨ Group Conversation Follow-Ups — member follow-up, continuation rounds, explicit close (2026-07-08)
Repo: EDDI (feat/group-followups)
Summary
Adds three new interaction patterns for completed group conversations:
Follow up with any member — ask a specific agent (including the moderator) a question; both the question and response are appended to the group transcript as
FOLLOW_UPentriesContinue the full group — re-run all discussion phases with a new question; agents retain conversation memory from prior rounds via reused private conversations; round counter increments
Explicit close — end member conversations, run ephemeral agent cleanup, lock the conversation permanently (
CLOSEDstate)
Changes
Model layer:
GroupConversation.java: Addedroundfield (1-based counter),CLOSEDtoGroupConversationState,FOLLOW_UPtoTranscriptEntryTypeIGroupConversationStore.java: AddedcompareAndSetState()for optimistic concurrency controlGroupConversationStore.java: ImplementedcompareAndSetState()(read-check-update pattern)
Service layer:
IGroupConversationService.java: AddedfollowUpWithMember(),continueDiscussion(),closeGroupConversation()+onRoundStart()listener methodGroupConversationService.java: Implemented all three methods; modifiedexecuteDiscussion()to emitround_startSSE event (instead ofgroup_start) for continuation rounds; deferred ephemeral agent cleanup tocloseGroupConversation()for successful rounds (only immediate cleanup on failure)GroupConversationEventSink.java: AddedEVENT_ROUND_STARTconstant andRoundStartEventrecord
REST + MCP layer:
IRestGroupConversation.java: Added 4 endpoints (POST /{gcId}/followup,POST /{gcId}/continue,POST /{gcId}/continue/stream,POST /{gcId}/close) +FollowUpRequestrecordRestGroupConversation.java: Implemented all 4 endpoints with ownership validation and SSE streaming support for continuationMcpGroupTools.java: Addedfollowup_with_member,continue_group_discussion,close_group_conversationtools
Design decisions
Concurrency:
compareAndSetState(COMPLETED → IN_PROGRESS)is a best-effort guard against overlapping follow-ups; state restored toCOMPLETEDon error. (Hardened on 2026-07-13 with a per-conversation in-process guard — see that entry.)Deferred cleanup: Ephemeral agents survive until explicit close so follow-ups can use dynamically-created agents; immediate cleanup only on failure
No TranscriptEntry.round field: Round boundaries are inferred from
QUESTIONentries in the transcript — avoids churn on the 13-field record with 4 constructorsPlain-text follow-up input: The follow-up input is the plain question; the full transcript is still provided via the
groupTranscriptcontext variable (not re-injected into the input text), and the agent's private conversation retains prior turns
Client experience improvements (follow-up commit)
Consistent response shapes: All endpoints (
followup,continue,close) now return the fullGroupConversation— same shape as the initialdiscussendpointDisplay name resolution:
followUpWithMemberaccepts either an agent ID or a display name (case-insensitive). Error messages list available members if target not foundmemberDisplayNamesmap: New field onGroupConversationmaps agentId → displayName, populated at discussion start from group config. Eliminates client-side transcript scanningavailableActionscomputed property: JSON response includes["followup", "continue", "close"]when COMPLETED,["close"]when FAILED,[]otherwise. Clients can discover available operations without reading docsClose returns body:
/closenow returns the closedGroupConversationwithstate: CLOSEDandavailableActions: []Lifecycle documented in OpenAPI: Close endpoint description includes
discuss → COMPLETED → [followup|continue]* → close → CLOSED (terminal)
🔭 Security — conversation-listing scan: enforce the budget per-descriptor + changelog accuracy (2026-07-15)
Repo: EDDI (fix/mcp-conversation-ownership)
Second CodeRabbit pass on this branch (its first pass predated the metrics/doc commit and was already moot). Two valid, current items:
Enforce
MAX_OWNER_SCANper-descriptor, not per-page. The budget was only checked in thedo-whilecondition (after a full page), so a non-admin scan could reachMAX_OWNER_SCAN + limit - 1before stopping — over the documented bound. Added an in-loopbreak. Impact is small (the overrun rows are within an already-fetched page, and foreign rows skip the snapshot load either way), but it makes the bound exact and theowner_scan_exhaustedmetric fire at 500 rather than up to a page late. Not separately unit-tested: with all-foreign pages the store returns the same empty list and the same page-read count with or without the break, so the tightening isn't observable through the store interface; the existing bounded-scan tests guard against regression.Changelog accuracy. The
009ca0f20entry's "Fix" paragraph said the ownership check "runs afterpopulateDataToDescriptor" — stale since8bb304b4csplit it (common case decides before the snapshot load; only a legacy null-owner row is re-checked after). Corrected. Also dropped a second stale "mirroring the MCP twin's budget" reference (MCP has no scan cap since009ca0f20).
🔭 Security — MCP/REST conversation ownership: PR-review response (metrics + doc accuracy) (2026-07-15)
Repo: EDDI (fix/mcp-conversation-ownership)
Triaged the Copilot + CodeRabbit review of this branch. Most bot findings targeted the MCP-side list_conversations over-fetch/scan loop that 009ca0f20 already deleted (its page-index bug was the reason for the delete), so they were moot against current HEAD. The substantive, still-valid items:
Observability (Micrometer). Per the project convention "always add metrics to new features", the new authorization paths were operationally invisible. Added two counters via field-injected
MeterRegistry(AGENTS.md metrics pattern,SimpleMeterRegistrydefault so unit tests that construct the bean directly stay non-null):eddi.mcp.conversation.access.denied{tool}incremented on every MCP ownership denial (the six gated read/drive tools viaaccessDenied, pluschat_managed's impersonation denial — MCP denials return a 200 error-body, so unlike REST 403s they are not visible inhttp.server.requests); andeddi.conversations.listing.owner_scan_exhaustedincremented when a non-admin listing stops on theMAX_OWNER_SCANbudget with fewer thanlimitresults, so a persistently-truncated user is not invisible.Fail-open on a missing descriptor — kept, documented. CodeRabbit flagged that
requireConversationOwnerreturnsnull(operation proceeds → 404) rather than denying when a descriptor is absent. Kept deliberately: a missing descriptor means the conversation is genuinely not found, and 404 is correct; flipping the shared guard to deny would change REST 404→403 and contradict its documented "let the operation handle the 404" contract. The only residual is orphaned memory with no descriptor — a deletion-path integrity concern, not something the read gate should mask. SoftenedaccessDenied's javadoc, which had over-claimed that a denial is indistinguishable from "does not exist".Doc accuracy.
RestConversationStore.MAX_OWNER_SCANjavadoc no longer says it "mirrors the MCP owner-scan cap" (MCP has none since009ca0f20; this is now the sole budget); the previous entry's "never starved" line is qualified to "within the scan budget".
Declined: the suggestion to take userId out of chat_managed. The cited rule exempts "external interfaces (MCP, REST) that operate outside a conversation", which is exactly what chat_managed is (it routes to a per-intent+userId managed conversation rather than running inside one); resolveOwnerUserId already rejects impersonation.
🔒 Security — RestConversationStore listing: owner-filter the conversation store enumeration (2026-07-15)
Repo: EDDI (fix/mcp-conversation-ownership)
Closes the first residual gap filed two entries below. RestConversationStore.readConversationDescriptors — the GET /conversationstore/conversations endpoint declared on IRestConversationStore — carried no @RolesAllowed and no ownership filter, so it fell through to the global authenticated policy. With authorization.enabled=true, any authenticated caller could enumerate every user's conversation descriptors (id, agent, state, and the descriptor's userId). It is the REST twin of the MCP list_conversations gap fixed in the entry two below.
Fix (RestConversationStore): inject the existing ConversationAccessGuard and filter the listing inside the endpoint's existing paging do-while. seesAllConversations() is resolved once up front (admins, and any caller when authorization is disabled, skip filtering entirely); otherwise each descriptor is dropped unless canAccessConversation(descriptor.getUserId()) admits it. The check runs before populateDataToDescriptor for the common case (every conversation since v5.1.6 records its owner on the descriptor), and only after populate for a legacy null-owner row, where populate resolves the owner from the snapshot (the pre-v5.1.6 fallback) — see the bounded back-fill note below. An unowned/legacy conversation stays visible, matching OwnershipValidator.requireOwnerOrAdmin.
Design decisions
Owner-filter, not admin-only. The endpoint backs the EDDI-Manager conversation views (its bundled UI calls
conversationstore/conversations). Owner-filtering keeps admins' full visibility and still lets a non-admin Manager user see their own conversations; a blanket@RolesAllowed("eddi-admin")would 403 the listing for every non-admin and diverge from the MCP twin, which chose owner-filtering. TheConversationAccessGuardwas purpose-built for a listing (canAccessConversation/seesAllConversations), so this reuses it rather than adding a check.No starvation, no dedup needed. The store's
readDescriptorstreats itsindexas a page number (ResourceFilter:skip = index * limit), and the endpoint'sdo-whilealready re-pages (index++) until it fillslimitor the store is exhausted. So a filtered-out row is naturally back-filled from a later, non-overlapping page — a caller's own conversations are not starved just because newer pages belong to others (bounded by the scan budget in the next bullet), and (unlike the MCP over-fetch) no resource-URI dedup is required.Bounded, cheap back-fill (self-review fix). Two costs had to be contained before that back-fill was safe on a large multi-tenant store, since this is the default EDDI-Manager list view: (1) the ownership check is split around
populateDataToDescriptor— for the common case (every conversation since v5.1.6 records its owner on the descriptor) the decision is made ondescriptor.getUserId()before the snapshot load, so a foreign row is skipped without loading its full memory document; only a legacy null-owner row falls through to the post-populate re-check that resolves the owner from the snapshot. (2) the back-fill is capped atMAX_OWNER_SCAN = 500descriptors — the sole owner-scan budget in the system, since the MCP listing delegates here rather than scanning itself — so a caller who owns few/none of a huge store cannot force a full-collection scan. Admins / auth-disabled callers are never filtered and never reach the budget. Tradeoff: for a non-admin, a very old owned conversation buried beyond the 500-descriptor (most-recent-first) window may not appear; theListreturn type can't signal truncation the way the MCP tool'sincompleteflag did. The proper long-term fix is an owner-scoped descriptor query (index onuserId) rather than in-memory filtering of a global scan — filed as a follow-up; the same unbounded-scan shape pre-existed for theagentId/state/viewStatefilters.
MCP list_conversations simplified (same branch). McpConversationTools.listConversations now issues a single store call and relays the result, dropping its per-caller seesAll/over-fetch branch, the 100-row chunking, the resource-URI dedup, and the incomplete/note signal. This is safe once the reality of the internal hop is accounted for — and that reality is why the removed loop was effectively dead code:
The MCP→store call is an unauthenticated loopback.
RestInterfaceFactorybuilds a bare REST client tohttp://127.0.0.1:<port>with noClientHeadersFactoryand no header propagation anywhere, so the caller's identity does not cross that hop.Auth off (the default):
DisabledAuthControllerreports authorization disabled, so the loopback is allowed and the store'sseesAllConversations()istrue— it returns everything and the tool relays it. This matches the prior behavior exactly (the tool's own guard also admitted everything with auth off), so the simplification is behavior-neutral here.Auth on: the
authenticatedHTTP policy rejects the token-less loopback with 401 before the endpoint method runs, so the tool's internal listing is non-functional underauthorization.enabled=true— and was already so, independently of this change. That is also why the removed scan-loop is safe to delete: its only runtime path (auth-on, non-admin) 401s upstream and never executes. (For the record, that loop also carried a latent bug — it passedscanned += page.size(), a row count, as the store's pageindex, so a second page wouldskip = 100 * 100; the ownership unit test masked it by mockingIRestConversationStoredirectly.)
Tests
New
RestConversationStoreOwnershipTest(real guard over a mockedSecurityIdentity,authorization.enabled=true): a caller sees only their own conversations and never another user's; a non-owner enumerating the store gets nothing of the owner's; an admin sees all; an unowned/legacy (null-owner) conversation stays visible; a personal list is back-filled across foreign pages (first page all-foreign, the caller's own on the next) rather than starved; a foreign row is skipped without loading its memory snapshot (guards the cheap pre-check); a legacy null-owner descriptor whose snapshot resolves to a foreign owner is filtered out (guards the post-populate ordering — added per self-review, so a reorder that moved the check abovepopulateDataToDescriptorwould fail); and a sparse owner's scan is bounded atMAX_OWNER_SCAN(5 page reads, no snapshot loads) rather than scanning the whole store.RestConversationStoreTestandRestConversationStoreFilterTestupdated to construct with the guard (stubbedseesAllConversations() → true, so their filter/paging assertions are unchanged).McpConversationToolsOwnershipTest.ListConversationsreduced to a single delegation test (one store call, no scan loop); the ownership-filtering assertions moved toRestConversationStoreOwnershipTest.McpConversationToolsExtendedTest'slist_conversationscases already exercised the single-call path (its guard has auth disabled) and are unchanged.
Remaining residual gaps (deliberately out of scope): the single-conversation REST reads (readRawConversationLog / readSimpleConversationLog) and getActiveConversations carry no ownership check; internal loopback REST calls via RestInterfaceFactory do not authenticate, so MCP tools that call them are non-functional under authorization.enabled=true (a pre-existing, cross-cutting gap affecting every internal REST caller and the auth model, not just this endpoint — filed as a follow-up); read_agent_logs without a conversationId was closed in the entry immediately below.
🔒 Security — read_agent_logs: admin-gate unscoped/agent-only log reads (2026-07-15)
Repo: EDDI (fix/mcp-conversation-ownership)
Closes the residual gap filed by the entry below. The ownership pass gated read_agent_logs only when a conversationId was supplied; without one (unfiltered, or filtered by agentId alone) it still returned BoundedLogStore entries — workflow execution logs, LLM provider errors, internal diagnostics that can quote other users' conversation data — to any caller holding the coarse eddi-viewer role. That unscoped read pulls from a single shared server-side ring buffer that mixes every user's activity: an operator surface, not one user's data.
Why this is the same class of bug the ownership pass fixed: the REST equivalent, IRestLogAdmin (/administration/logs — recent, /history, and /stream), is @RolesAllowed("eddi-admin") at the interface level. Every log read over REST already requires admin; the MCP tool was the more permissive door. This aligns MCP with REST.
Fix (McpConversationTools.readAgentLogs): require eddi-admin when no conversationId filter is present; the conversation-scoped path is unchanged (ConversationAccessGuard.requireConversationOwner → owner-or-admin). The admin check is placed before the try so a role denial surfaces as an honest role error, not the ownership accessDenied(...) "you do not own this conversation" message (there is no conversation to own).
Design decisions
Owner-or-admin kept for the conversation-scoped path, rather than admin-only parity with REST.
BoundedLogStore.getEntriesfilters by exactconversationId, so a scoped read returns only that one conversation's log lines — no cross-user leakage. That preserves the self-service diagnostics the ownership pass deliberately added forread_conversation/read_audit_trail. The cross-user exposure was only ever in the unscoped path, and that is what is now closed. (An admin-only-for-all-log-reads posture was considered and rejected as an unnecessary regression of that self-service capability.)agentId-only counts as unscoped. An agent filter still spans every user of that agent, so it is admin-gated too — only aconversationIdnarrows the read to a single owner's data.
Tests: extended McpConversationToolsOwnershipTest.ReadAgentLogs — a viewer is denied the unscoped buffer and the agent-only buffer (ForbiddenException, and BoundedLogStore is never reached); owning one conversation does not grant the unscoped firehose; an admin may read both the unscoped and agent-scoped buffers; the existing owner/non-owner conversation-scoped cases still pass.
🔒 Security — MCP conversation tools had no ownership check (2026-07-14)
Repo: EDDI (fix/mcp-conversation-ownership, from main)
Gap (pre-existing on main): every conversation-scoped tool in McpConversationTools was gated on the coarse eddi-viewer role and nothing else, while the equivalent REST endpoints on RestAgentEngine all enforce requireOwnerOrAdmin (403). With authorization.enabled=true, any caller holding eddi-viewer could — over MCP — read any user's conversation memory (read_conversation) and transcript (read_conversation_log), enumerate conversations across all users (list_conversations), read another conversation's prompts/tool-calls/costs (read_audit_trail, whose REST surface is @RolesAllowed("eddi-admin")) and its server logs (read_agent_logs), inject turns into someone else's conversation and read the agent's reply (talk_to_agent, chat_with_agent with a foreign conversationId), and take over another user's managed conversation by simply naming their userId (chat_managed). The read half also defeats the group-conversation ownership gate: group members' conversations are ordinary conversations, so list_conversations + read_conversation_log reached transcripts that read_group_conversation denies.
Two findings that shaped the fix:
A naive ownership gate would have broken MCP outright. MCP created conversations with
userId = null, andConversationSetup.computeAnonymousUserIdIfEmptyturns that into a generatedanonymous-<uuid>— a non-blank owner matching no principal. Gating reads onrequireOwnerOrAdminalone would therefore have locked every MCP-created conversation away from its own creator (admins only). REST never had this problem because it resolves the owner at creation. So the fix has to stamp the caller as owner at MCP conversation creation — that is what makes the gate both effective and non-regressive.The read gap had a write-side twin (
talk_to_agent/chat_with_agent/chat_managed), which is strictly worse than reading and lives in the same file, so it is closed here too.
Fix — new ConversationAccessGuard (engine.security), the non-HITL sibling of HitlAccessGuard:
requireConversationOwner(conversationId)— owner-or-admin via the conversation descriptor; skips when the descriptor is absent (the operation itself 404s); fail-closed on a store error. This isRestAgentEngine's privatevalidateConversationOwnershiplifted out verbatim.canAccessConversation(ownerId)/seesAllConversations()— non-throwing predicates for listings; admit exactly what the read gate admits (admin, owner, or unowned legacy data), so a caller never lists what they cannot read, nor reads what they cannot list.resolveOwnerUserId(requestedUserId)— delegates tovalidateAndResolveUserId, stamping the caller and rejecting impersonation.
RestAgentEngine now delegates to the guard (behavior identical; its IConversationDescriptorStore dependency became dead and was dropped). McpConversationTools gates all eight conversation-scoped tools, each catching ForbiddenException ahead of its generic catch and returning a uniform, non-leaking accessDenied(...) that never distinguishes "not yours" from "does not exist".
list_conversations owner-filtering (reworked after self-review). The first cut filtered a single 100-row page, which silently starves a personal list: on a shared agent the newest page is often entirely other users' conversations, so the caller would get count: 0 — indistinguishable from "you have none" — and the requested limit stopped meaning anything. It now scans forward page by page until the limit is filled or a 500-descriptor budget is spent, dedupes by resource URI (the store's own paging skips deleted rows, so its cursor can outrun the rows it returns and an offset scan can re-read one), and sets incomplete: true with a note when it stops on the budget rather than on the store running out — per AGENTS.md "no silent caps".
Design decisions
Shared guard, not a third copy.
McpGroupTools(onfeat/group-followups) had already duplicated this logic once; a third copy in the MCP conversation tools would guarantee drift. One@ApplicationScopedguard is now the single answer to "who may read or drive a conversation", exactly asHitlAccessGuardis for "who may decide an approval".No auth-disabled short-circuit inside the guard. It reads the descriptor unconditionally, as
RestAgentEnginealways has;OwnershipValidatoralready no-ops when authorization is off. Short-circuiting would have quietly changed REST semantics (and its tests mockOwnershipValidator).read_audit_trailgets the ownership gate, not admin-only. REST's audit surface is admin-only; matching that would have been a role-policy change beyond this fix. The ownership gate is a strict tightening either way.
Behavior change (auth on only). With authorization.enabled=false — the default — nothing changes: every check no-ops and new conversations still get their anonymous-* id. With auth on, pre-existing anonymous-* conversations become invisible and unreadable to non-admins over MCP: they provably belong to nobody. That is the intended tightening.
Residual gaps, deliberately out of scope (filed as follow-ups): RestConversationStore.readConversationDescriptors — the REST store listing — is unfiltered for every caller, and read_agent_logs without a conversationId still returns cross-user server logs to any viewer.
Tests: new ConversationAccessGuardTest (owner / non-owner / admin / missing descriptor / store error / unowned / auth-off, plus the listing predicates) and McpConversationToolsOwnershipTest, which asserts for every gated tool that a non-owner is denied and the underlying service is never reached — no data leaves, not even inside an error message — while owner and admin pass. Existing McpConversationTools*Test and RestAgentEngine*Test constructors updated to wire a real guard from the same mocks.
🔬 Multi-Model Cascade — Merge-readiness review: fixes + coverage backfill (2026-07-14)
Repo: EDDI (feat/model-cascade-enterprise-hardening)
A critical whole-branch review (6 parallel high-effort reviewers, then adversarial confirm/refute on each finding) declared the branch merge-ready — every unit "ready-with-nits", no blockers. Acted on the confirmed nits and backfilled test coverage for new/adapted paths the existing suite missed.
Fixes
Validator ↔ runtime parity (
CascadeConfigValidator). TheconvertToObject-incompatibility warning now usesEvaluationStrategy.fromConfigOrDefault, so an unknownevaluationStrategy— whichresolveEffectiveStrategyalso resolves tostructured_outputand then downgrades at runtime — warns too (previously onlynull/structured_outputwarned).Live-stream mid-failure de-dup (
CascadingModelExecutor). If the live-streamed final step fails after emitting partial tokens, the fallback to the buffered best is now markedstreamedLive=true, soLlmTaskdoes not re-emit the best's (different) text as a duplicate token stream after the partial tokens the client already received — the correct full response still arrives via the finaldonesnapshot. Added awithRun(…, streamedLive)overload and a per-stepstepStreamedLiveflag read in the catch.
Coverage backfill (new tests; all green)
AgentOrchestrator: token accumulation into
ExecutionResult.responseMetadata(the cascade-cost feed — previously 0% exercised because every test mocked null response metadata), directsumTokens/tokenUsageMapunit tests (helpers made package-private), and the before-tool cooperative-cancellation check.CascadingModelExecutor: step-param templating + credential skip (
TEMPLATE_SKIP_PARAMS— previously 0%, all tests used a null templating engine), a deterministic duration-ceiling test (replacing a timing-flaky one), and the streaming mid-failure de-dup above.LlmTask: the skipCascade legacy-fallback SSE emit and cascade token-usage surfacing (
responseMetadata+audit:cascade_token_usage).ConfidenceEvaluator:
stripJsonWrapperfallback,extractFirstBalancedObjectbackslash-escaped-quote handling, and the judge-model readTree-throw → regex-fallback path.CascadeConfigValidator: cascade-level negative-pricing hard-fail and the
convertToObject+ unknown-strategy warn path.
Flagged (pre-existing, out of scope)
A HITL tool-approval pause originating inside an agent-mode cascade step resumes on the base model, not the cascade step's (cheaper) model — misattributing cost/audit. Confirmed real but pre-existing (baseline already threaded the tool-approval params; the resume path predates cascade-step pauses and stores only the outer task's model). Tracked as a follow-up.
🧊 Multi-Model Cascade — PR-review follow-ups: type-safe SSE events + strategy enums (2026-07-14)
Repo: EDDI (feat/model-cascade-enterprise-hardening)
Addressed two @niedch review comments on PR #587, after merging origin/main (tool-level HITL) into the branch.
Typed SSE cascade events (comment #1).
RestAgentEngineStreamingbuilt thecascade_step_start/cascade_escalationSSE payloads with hand-writtenString.formatJSON (manual escaping,%.4fformatting). Replaced with tworecordpayloads serialized through the existing JacksonMAPPERvia a newsendJsonEventhelper (graceful{}fallback on the unexpected serialization failure). Non-finiteconfidence/thresholdare still sanitized viafinite()before serialization. TheescapeJson/finitehelpers remain (still used by the task/error events).Strategy enums (comment #2). Introduced
EvaluationStrategy(structured_output/heuristic/judge_model/none) andCascadingStrategy(cascade/parallel) as the single source of truth for the recognized strategy tokens.ConfidenceEvaluator(exhaustive enum switch),CascadingModelExecutor(resolveEffectiveStrategy+ gating checks), andCascadeConfigValidator(valid-set + warn logic) now resolve to these enums instead of scattered magic strings.Design note (answers "is there a reason it's a String?"): the config wire fields (
ModelCascadeConfig.strategy/.evaluationStrategy) deliberately stay lenientStrings. An unrecognized value (a typo, or one written by a newer engine) still loads, the validator warns, the runtime falls back to the enumDEFAULT, and the original token round-trips unchanged through export/import — behavior a strict enum field would regress. Parsing to the enum happens at the boundary viafromConfig/fromConfigOrDefault. If the field type itself should become an enum, that's a separate contract decision (see the HITL enums for the pattern).Behavior is byte-for-byte preserved (verified:
ConfidenceEvaluator*Test,CascadeConfigValidatorTest,CascadingModelExecutor*Test,LlmTask*Test— 324 tests green); newStrategyEnumsTestlocks the lenientfromConfigcontract (case-insensitive, trimmed, unknown→null, default fallback).
🚀 Multi-Model Cascade — Enterprise Hardening (2026-07-03)
Repo: EDDI (feat/model-cascade-enterprise-hardening) What changed: Full enterprise pass over the multi-model cascading feature. A review found two documented-but-dead capabilities (SSE events, judge model), a compliance bug (audit recorded the wrong model), and discarded token/cost/metrics that made the cost-savings pitch unmeasurable. This lands all of it. Plan: planning/model-cascade-enterprise-hardening-plan.md.
Correctness / compliance
Audit records the real model (#5).
LlmTasknow writesaudit:model_nameandaudit:cascade_modelfrom the cascade-selected step (provider/model (step N)), not the task-level default. Addedaudit:cascade_costandaudit:cascade_token_usage. An auditor can now reconstruct which model produced an answer.Agent-mode confidence (#6).
structured_outputcannot be injected around the tool-loop, so agent mode auto-routes tojudge_model(if configured) elseheuristic. A single deploy-time warning replaces the previous per-turn WARN.convertToObject + cascade (#7). The cascade now honors native
jsonMode, and forces a non-wrapper confidence strategy whenconvertToObject=true(the wrapper contradicts the raw-JSON instruction).Global-var / Qute consistency (#8). Step
typeis resolved throughGlobalVariableResolverand step param values are run through the template engine — parity with the standard path.
Broken promises made real
SSE cascade events (#1).
StreamingResponseHandlergained defaultonCascadeStepStart/onCascadeEscalation; the anonymous sink inConversationService.sayStreamingforwards them;RestAgentEngineStreamingemitscascade_step_start/cascade_escalationSSE events. The plumbing is now live end-to-end.judge_model implemented (#2). New
judgeModel: {type, parameters}config block, built once viaChatModelRegistry(vault + global-var resolution), passed intoConfidenceEvaluator.evaluationStrategy: judge_modelwithout a judge logs a deploy-time warning and falls back to heuristic at runtime.strategy: parallel(#3) / budget javadoc (#4). Unknown/parallelstrategy logs a deploy-time warning and runs sequentially; the false "budget exhausted" javadoc replaced with real ceiling docs.
Observability & guardrails
Token + cost evidence. Per-step
tokenUsageandcostUsdin the trace; aggregate run cost + token usage surfaced viaresponseMetadataObjectName(was{}). Agent-mode token usage is accumulated across tool-loop iterations.Micrometer metrics under
eddi.llm.cascade.*: executions, escalations (tagreason), accepted step, step latency, confidence distribution, step errors (tagsprovider,type), tokens, cost, ceiling exceeded (tagkind).Cascade ceilings.
maxTotalDurationMs(wall-clock) andmaxCostPerRun(dollars, from configurable per-stepinputPricePer1M/outputPricePer1M) stop escalation and return the best response so far. Per-step timeout is capped by the remaining duration budget for buffered steps only — a live-streamed step is exempt (see the "Live-stream timeout" fix below).Configure-time validation (
CascadeConfigValidator): invalid new numeric fields (negative pricing, non-positivemaxTotalDurationMs, negativemaxCostPerRun) fail fast at deploy; legacy conditions (empty steps, unknownevaluationStrategy/strategy,judge_modelwithout a judge, thresholds ∉ [0,1], dead non-last null thresholds, non-positivetimeoutMs) emit deploy-time warnings but still load (backward-compatible).
Robustness
Confidence parsing tries a real Jackson parse first (only reads
confidencefrom an identified wrapper object, so a stray"confidence":in answer content is ignored), regex as fallback.Heuristic i18n.
heuristicConfigmakes phrases/thresholds config-driven (English defaults); the no-phrase-match fallback is language-agnostic (keeps the default score rather than mis-scoring).Cancellation safety (#9).
AgentOrchestratorchecks interruption between tool-loop iterations and before each tool, so a timed-out cascade step stops launching further side-effectful tools. Residual risk: a tool already in-flight when the timeout fires may complete.Streaming the final step live. The always-accepted final step (legacy mode, non-wrapper strategy, streaming-capable provider) streams token-by-token via the event sink instead of buffering.
StreamingLegacyChatExecutor.executeCapturingpreserves token usage while streaming.returnBestAcrossSteps(opt-in): return an earlier step's response if it scored strictly higher than the finally-accepted step.Base-model laziness. The base
ChatModelis no longer built when the active-cascade branch owns the request.
Architecture
CascadingModelExecutorconverted from a static utility to an instance (constructed byLlmTask) holdingChatModelRegistry,GlobalVariableResolver,ITemplatingEngine,LegacyChatExecutor,StreamingLegacyChatExecutor, andMeterRegistry.AgentOrchestrator.ExecutionResultgained aresponseMetadatafield (2-arg constructor retained for compatibility).Backward compatible: all new config fields optional with today's defaults; configs without
modelCascadeandenabled:falseare unaffected;StreamingResponseHandlercascade methods aredefault.
Cross-provider credentials
Because step/judge parameters are merged over the task parameters, a step (or judge) targeting a different provider than the task would silently inherit the task's
apiKey— wrong for that provider, failing at runtime as a 401 that looks like an escalation.CascadeConfigValidatornow emits a deploy-time warning for a different-provider step/judge that omits its ownapiKey. Not a hard error (Ollama/Bedrock don't useapiKey); documented indocs/model-cascade.md.
Tests & coverage
Updated the 3 executor test classes to the instance API and the 6
LlmTasktest classes to the new constructor. Removed the backward-incompatiblelanguageAgnosticScoreband that regressed the default heuristic score.New coverage:
CascadingModelExecutorEnterpriseTest,CascadingModelExecutorCoverageTest(agent mode, live streaming, cost/duration ceilings, timeout + retryable escalation, convertToObject downgrade),ConfidenceEvaluatorEnterpriseTest,StreamingLegacyChatExecutorCoverageTest, and expandedCascadeConfigValidatorTest. New-code coverage ≈ 92% instruction / 78% branch (residual branches are the 120s streaming-timeout guard and typed-exception variants); the project aggregate stays above the 90%/80% gate.CascadingModelExecutor.isRetryableErrormessage matching collapsed to a single regex (fewer branches, same behavior).
Adversarial-review fixes
A multi-lens adversarial review (7 reviewers → independent skeptics) surfaced several real defects, now fixed:
returnBestAcrossStepsvs. live streaming (high): a final step already streamed live is no longer superseded by an earlier higher-scoring step — that would have replaced text the client had already received. The trace marks the superseded step accordingly.Agent-mode cascade streaming (medium): the cascade now emits the agent-mode final response to the SSE stream as a single chunk, matching the standard (non-cascade) agent path (it was silently dropped before); docs corrected.
Validator backward-compat (medium):
CascadeConfigValidatornow warns (instead of hard-failing) for conditions older releases tolerated at load — unknown strategy/evaluationStrategy, out-of-range thresholds, dead non-last steps, judge_model without a judge, empty steps — so upgrading cannot stop a previously-loading agent from deploying. Only the new pricing/ceiling fields hard-fail on an invalid value.Heuristic clamping (medium): config-supplied heuristic scores are clamped to [0,1] so a mis-set value can't produce an out-of-range confidence.
unescapeJsonString(low): rewritten as a single-pass scanner so an escaped backslash is consumed before the following char (chainedreplacecorrupted\\n). Judge regex fallback scoped to the extracted object.Streaming-timeout caveat documented (partial tokens of an abandoned final step).
New regression tests for all of the above, plus the previously-missing SSE-forwarding and cooperative-cancellation tests. New-code coverage ≈ 92% instruction / 79% branch.
Second-pass review fixes
A lean second adversarial pass (5 reviewers → synthesizer) found five more real issues, now fixed:
Live-stream timeout (high): a live-streamed step is no longer subject to the per-step/duration timeout — cancelling it couldn't stop the provider's callback thread, so tokens leaked to the client while the cascade re-emitted a different response (concurrent SSE writes). A streamed step now runs under the streaming executor's own ~120 s bound and its result (even if partial) is the accepted answer; no re-emit, no mid-stream cancel.
streamLivealso tightened to guaranteed-accept steps only (last, null-threshold, ornone≤1.0).Judge confidence regression (medium): the judge regex fallback runs over the full judge text again (scoping it to the first balanced object dropped the score when a reasoning object preceded the rating).
Docs vs. validator (medium): the Configure-time Validation section now states the real two tiers (hard-error only for new pricing/ceiling fields; warnings for legacy conditions).
Single-line code fence (low):
stripCodeFencesnow unwraps```{...}```(no newline), which was being discarded.returnBestAcrossStepstrace (low): the earlier winning step's trace entry is relabeledaccepted_as_bestso the trace agrees withstepUsed.
Regression tests added for each. Full touched-area suite green.
PR-review fixes (bots)
CodeRabbit + Copilot + github-code-quality on PR #587 flagged further items, now addressed: retry token usage accumulated across all attempts (not just the last); the cancellation interrupt flag is cleared (Thread.interrupted()) so it can't leak; the accepted.step metric + trace status name the actual returned step under returnBestAcrossSteps; unknown evaluationStrategy normalized to structured_output at runtime (matches the validator + evaluator default); SSE cascade_escalation guards non-finite confidence/threshold; the structured-output regex fallback uses the fence-stripped text; the cascade-disabled agent path forwards its buffered response to the stream; an unused parameter removed; a dead @Disabled test deleted; and the docs/changelog/plan corrected to say the validator warns (not "rejects/fails fast") on legacy conditions.
Status
Complete and merged-ready on feat/model-cascade-enterprise-hardening (PR #587). No open items; the branch is the terminal state of this feature — next planned work is unrelated (see Section 3 of AGENTS.md).
🔒 Fix: CodeRabbit review — LifecycleManager failure-path hardening (2026-07-16)
Repo: EDDI (feat/error-handling-recovery)
Summary
Addressed four CodeRabbit findings on the error-handling PR's own LifecycleManager failure path (PR #593). All four verified as valid against the code; each fix reuses existing infrastructure rather than adding a new utility.
Key Changes
Audit must not bypass strict-write recovery (Major). If
auditCollector.collect()threw, the strict-write rollback was skipped — leaving the partial task writes it exists to remove — and the audit exception propagated out of the catch, replacing the original task failure. Strict-write recovery now runs first (it is integrity-critical), and audit collection is shielded in a try/catch that attaches any reporting error to the original exception viaaddSuppressedinstead of masking it.Redact credentials from audit/SSE summaries (Major).
summarizeForAudit()only truncated; its output is persisted to the audit ledger and streamed to admins overtask_failedSSE. It now applies the existingSecretRedactionFilter.redact()before truncating — cutting first can split a secret so the pattern no longer matches, leaving a fragment behind. URLs and class names are deliberately retained: the audience is privileged and needs them to diagnose (this is what distinguishes it fromsummarizeException, the LLM-facing path).Typed causes outrank wrapper messages (Minor).
classifyError()checked each level's message before descending, so a"429"wrapper around aSocketTimeoutExceptionclassified asrate_limit. It now scans the whole chain for typed causes first (authoritative), and only then falls back to message heuristics — substring matching is easily fooled (e.g."failed after 429ms").SSE failure logging (Minor). The
task_failedemission catch logged at DEBUG and dropped the throwable plus all context. Now WARN, with the throwable, the sanitized conversation id (LogSanitizer, CWE-117) and the task id.
Tests
Four regression tests added, each of which fails under the previous behavior: typed-cause precedence, credential redaction, redact-before-truncate ordering, and audit-failure-does-not-mask-the-original.
🐛 Fix: two stale tests red since the error-handling PR (2026-07-16)
Repo: EDDI (feat/error-handling-recovery)
Summary
CI on this branch had been red since 2026-07-08 (commit c054b430, "Tests run: 9776, Failures: 1, Errors: 1") — both failures pre-date the origin/main merge and were surfaced again by it. Each is a stale test asserting a contract the error-handling PR itself deliberately superseded. Test-only changes; no production behavior altered.
Key Changes
StreamingLegacyChatExecutorTest.execute_error_throwsRuntimeException— asserted that a streaming error always throws. The PR intentionally changed this: an error arriving after partial tokens now returns the partial text with astreaming_error_partialwarning, and only a zero-content error throws. Retargeted the test at the zero-content case (matching its name) and addedexecute_errorAfterPartial_returnsPartialContentto cover the partial contract, preserving the original token-forwarding assertion.ConversationExtendedTest.saySucceeds— stubbedgetConversationState()with a call-count-sensitive consecutive-return sequence (READY, thenIN_PROGRESS). The PR'sEXECUTION_INTERRUPTEDauto-recovery added a state read at the top ofrunStep, consuming theREADY, so the in-progress guard sawIN_PROGRESSand threwConversationNotReadyException. The mock now tracks state like real memory (returns whatever was last set), making it robust to how often production reads it.
Design Decisions
Fixed the tests, not the production code: both behaviors (partial-response salvage, interrupted-state auto-recovery) are intentional, documented features of this PR and are covered by
StreamingLegacyChatExecutorRetryTest. The tests simply encoded the pre-feature contract.
🧹 Refactor: Remove duplicate RetryConfiguration shim in LlmConfiguration (2026-07-16)
Repo: EDDI (feat/error-handling-recovery)
Summary
Follow-up to the error-handling PR (#593): resolved a code-quality finding (nested class with the same simple name as its superclass). The LlmConfiguration.RetryConfiguration nested class was an empty subclass of the extracted ai.labs.eddi.configs.shared.RetryConfiguration, kept as a backward-compat shim. It overrode nothing and shadowed the imported shared type within LlmConfiguration's body.
Key Changes
LlmConfiguration.java: Deleted the empty nestedRetryConfigurationsubclass. Theretryfield, getter, and setter now bind directly to the imported sharedRetryConfiguration(import already present).9 test files: Replaced
new LlmConfiguration.RetryConfiguration()with the sharedRetryConfiguration(added theconfigs.sharedimport). IncludesLlmConfigurationTest, which had pulled the nested type in via aLlmConfiguration.*wildcard import.
Design Decisions
Deleted the shim rather than renaming it (a reviewer suggested
LegacyRetryConfiguration). The subclass added zero fields/overrides, so a rename would keep a misleadingly-named dead class for the same test churn. Per project philosophy, internal-API removal is safe — the only backward-compat concern is stored JSON, and because the removed subclass added no fields, theretryJSON structure is byte-for-byte identical (existing MongoDB/ZIP configs deserialize unchanged).
Verification
mvnw clean test-compileclean; 123 affected unit tests pass (0 failures).Repo-wide grep confirms zero remaining references to the nested type (dotted, JVM binary-name, reflection strings, or wildcard imports).
⚡ Feat: Holistic Error Handling and Recovery Infrastructure (2026-07-07)
Repo: EDDI (feat/error-handling-recovery)
Summary
Complete overhaul of error handling across LLM, HTTP, and MCP call subsystems plus cross-cutting infrastructure for admin visibility, recovery, and monitoring. 19 source files changed, 7 test files added (83+ new tests). Code-reviewed and all review findings addressed before commit.
Key Changes
Shared RetryConfiguration: Extracted reusable retry logic with exponential backoff, retryable error classification, configurable per subsystem.
LifecycleManager: Error classification, failure audit entries, SSE
task_failedevents, Micrometer counters tagged byerror.type.Admin state reset:
PATCH /{conversationId}/stateendpoint to recover stuck conversations.HTTP error body storage: 4xx/5xx response bodies stored in memory; JSON parse softened.
MCP continueOnError + retry + circuit breaker: Config-driven error resilience per MCP call.
LLM ResponseValidation: Config-driven policies for empty/truncated/filtered responses.
Streaming retry: Zero-token failures retried; partial responses returned with metadata.
Design Decisions
Retry at call site (not pipeline level) per user directive.
Strict-write default kept as
false(opt-in) to avoid breaking existing agents.No
"retry"validation action — retry handled by RetryConfiguration at call level.
🧹 Multimodal Attachments Completion — Remove dead config knob reattachTurns (2026-07-13)
Repo: EDDI (feat/multimodal-attachments-completion)
LlmConfiguration.Task.reattachTurns (@since 6.1.0, added on this branch) was a no-op: getReattachTurns() is called nowhere in src/main, so setting it changed nothing at runtime. Past-turn PDFs/docs already reach the model via text-extract stitching (attachments:extracts), never native re-attachment. Removed the field, getter/setter, and its round-trip test.
Surfaced by a codebase-wide dead-config audit (adversarial multi-agent sweep). The audit flagged ~26 other candidate no-op knobs; rather than mass-delete, they were triaged and tracked as follow-ups:
Genuinely dead —
ModelCascadeConfig.strategy("parallel = future", never built),dream.batchSize.Feature exists but knob unwired —
enableParallelExecution+parallelExecutionTimeoutMs(orphanedToolExecutionServiceparallel machinery), RAGinjectionStrategy/contextTemplate,McpServerConfig.transport,autoRecallCategories,dream.schedule/maxUsersPerRun.⚠️ Unenforced guardrails —
DynamicAgentConfig.allowRecruitment/allowDelegation/maxRecruitedAgentsPerDiscussion/maxDelegationsPerTask/inheritParentModelare read nowhere; the guardrails silently don't apply (tracked as its own security/cost fix).Roadmap scaffolding — keep —
sessionManagement/autoSnapshot/maxCheckpointsPerConversation(Session Forking is in-progress per roadmap).Audit blind spot — operator knobs selected via Quarkus
@IfBuildProfile/@LookupIfProperty(e.g.eddi.messaging.type) are not dead; a getter-grep can't see build-time bean selection. Those need per-item verification, not deletion.
🔍 Multimodal Attachments Completion — PR #588 review-comment fixes (2026-07-13)
Repo: EDDI (feat/multimodal-attachments-completion)
Addressed CodeRabbit + Copilot review comments.
Correctness
Download 404-vs-500 (High):
IAttachmentStore.load/getMetadatathrew a bareAttachmentStoreExceptionfor both a missing blob and an internal store failure, soRestAttachmentUpload.downloadAttachmentmapped SQL/backend errors to 404 (at DEBUG) — hiding outages. Added a typedAttachmentNotFoundException(symmetric withAttachmentAccessDeniedException); both stores throw it for genuinely-missing blobs; the endpoint returns 404 for it and 500 (ERROR log,ATTACHMENT_STORE_ERROR) for any other store exception. +regression test.GDPR export isolation (Major): the attachment-metadata export wrapped the whole conversation loop in one try/catch, so one failing
listByConversationtruncated the export for every remaining conversation. Each conversation is now isolated (mirrors the conversation-snapshot block above it).URL group attachment without mimeType (Medium):
RestGroupConversation.toAttachmentskept URL refs with null/blank mimeType thatAttachmentContextExtractorsilently drops later; now skipped up front so the loss is explicit.
Observability
AttachmentForwarder: reusableCounters initialized once (in the constructor — the registry is constructor-injected, so@PostConstructwouldn't fire in the direct-construction unit tests) instead of resolved perforward();MeterRegistry/Counterimported.AttachmentTextExtractor: per-extraction PDF logs lowered INFO → DEBUG (they run on every user turn / tool call).Conversation: the attachment-issue warning now includes the conversation id.
Style (the import guideline just added to AGENTS.md §4.7)
LlmTask(@jakarta.inject.Inject→@Inject),GroupConversation(Attachmentimported),GroupConversationServiceTest(Contextimported), and the FQNMeterRegistryinAttachmentForwarder.GridFsAttachmentStoreTest.whenFindIterategeneralized to any file count (was hardcoded to the 0/1/2-file cases).
Declined / documented
LlmConfiguration.MultimodalOverridekept as a mutable Jackson POJO (not a record) for consistency with every sibling nested config type in the file — converting only one would be inconsistent and need@JsonCreatorwiring.URL-only group attachments still aren't recovered after a HITL resume — a deliberate, documented limitation (the blob store is the durable source; URLs aren't blob-backed). The PR description should note this.
All affected unit tests green.
🐛 Multimodal Attachments Completion — Fix: group attachments lost on HITL resume (2026-07-13)
Repo: EDDI (feat/multimodal-attachments-completion)
Found by a critical adversarial re-review of the origin/main merge (10-dimension workflow + per-finding refutation). A merge-emergent bug — neither parent could exhibit it alone: our branch added group-shared attachments; origin/main added group HITL pause/resume; combined, they interact badly.
Bug: GroupConversation.attachments is @JsonIgnore transient (the durable copy is the blob store). resumeDiscussion() reloads a fresh GC from the store, so getAttachments() is null; executeDiscussion() re-seeded the sibling transient field dynamicAgentConfig but not attachments. Result: a member speaking for the first time after a HITL resume got neither the blob-store grant nor the attachment_* context — blind to the shared files. Compiles cleanly; runtime-only.
Fix: new package-private rehydrateAttachmentsFromStore(gc), called in executeDiscussion right after the dynamicAgentConfig re-seed (so the two transient fields are handled symmetrically in one place). It rebuilds the metadata list from IAttachmentStore.listByConversation(gc.getId()) when the in-memory list is empty — keeping the blob store as the single source of truth (no dangling refs after erasure) with no persistence-schema change. Guarded by null/empty (not startPhaseIndex, since a task-level pause in phase 0 resumes at index 0). Known limitation: URL-only attachments are not blob-backed and are not recovered on resume (documented in code; a follow-up can persist those if it becomes a real need).
4 unit tests added (rehydrate_*); GroupConversationServiceTest + RestGroupConversationTest green. The rest of the merge review came back clean — 9/10 dimensions no findings, and the integration sweep confirmed the conflict resolutions themselves are correct (clean unions, no mis-picked sides, consistent call sites).
📎 Multimodal Attachments Completion — Human review fixes: FQN → imports (2026-07-13)
Repo: EDDI (feat/multimodal-attachments-completion)
Addressed @niedch's human review comments on PR #588:
GroupConversationService— the field-injected attachment store used a fully-qualified@jakarta.inject.Injectandai.labs.eddi.engine.attachments.IAttachmentStoretype.jakarta.inject.Injectwas already imported, so the annotation is now@Inject; added anIAttachmentStoreimport and the field readsIAttachmentStore attachmentStore;.ConversationService— same FQN smell on the injected field and the anonymousgetAttachmentStore()override (reviewer flagged the override; the field had it too). Added theIAttachmentStoreimport and simplified both usages.
Compile clean (mvnw compile → exit 0). No behavior change — pure import hygiene.
Also codified the convention in AGENTS.md §4.7 (new Imports subsection): always import types/annotations and reference them by simple name; the only acceptable inline FQN is disambiguating two same-named classes used in one file. Prevents this review comment from recurring.
Deferred (tracked separately): @niedch also suggested a "general solution for the authorization to avoid duplicating it in multiple places" on PostgresAttachmentStore.authorize. Verified as a real duplication — the access policy is copy-pasted across 4 sites (read owner-or-grant + delete owner-only, in both the Postgres and GridFS stores) with an identical denial message, and the read path has already drifted for the null-owner edge case (Postgres denies, Mongo allows; the delete path stays consistent). Because the reviewer framed it as future work and unifying the read path is a security-behavior change that deserves its own tested PR, it was not folded into this PR — spun off as a dedicated follow-up (extract a shared AttachmentAccessPolicy, standardize null-owner reads to deny-by-default, add a two-backend regression test).
📎 Multimodal Attachments Completion — Automated review fixes (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion)
Addressed the GitHub code-quality / Copilot review of PR #588:
(High)
readAttachmentcouldn't see group-shared blobs.listByConversationreturns only owned blobs, so a group member — whose shared attachments are owned by the group conversation and merely granted to it — got an empty list and couldn't recall them via the tool. AddedIAttachmentStore.listAccessible(conversationId)(owned OR granted) in both backends (GridFSmetadata.grantsarray match / Postgres? = ANY(grants)), andReadAttachmentToolnow lists/resolves through it.(Medium) URL attachments dropped when no store configured.
GroupConversationService.materializeAttachmentsreturned early on a null store, discarding hosted-urlattachments that don't need a store. Restructured to skip only the inline-base64 (store-requiring) path.(Medium) Brittle access-denied detection. The download endpoint keyed 403-vs-404 off
message.contains("denied"). Added a typedIAttachmentStore.AttachmentAccessDeniedException(thrown by both backends' authz/delete paths); the REST layer catches it for 403 and treats other store exceptions as 404/500.(Note) Unused local variable removed from a GridFS test.
Tests updated + added (grant-aware listing, url-without-store materialize, typed-exception 403 paths); 277 green across the affected classes, coverage gate still met.
📎 Multimodal Attachments Completion — Adversarial review + fixes (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion)
A multi-agent adversarial review of the whole implementation surfaced two real high-severity defects (both verified by an independent refutation pass, both missed by the unit tests because they stubbed getLatestData directly and used single-turn memories):
Prefix-collision silent data loss.
IConversationStep.getLatestDatais a prefix scan, and theATTACHMENTSkey"attachments"is a prefix of theattachments:extracts/attachments:errorskeys the forwarderpersist()s. A second forwarder (orreadAttachmentauto-add, orContentTypeMatcher) read in the same step reverse-scanned and returned aList<String>instead of theList<Attachment>→ zero attachments forwarded, no error note. Reachable with two langchain tasks sharing an action or two langchain workflow steps. Fixed by reading the exact key viagetData(MemoryKey)inAttachmentForwarder,AgentOrchestrator, andContentTypeMatcher.Mirror-inverted history stitching.
ConversationLogGenerator.withAttachmentExtractspassed the forward conversation-output index intoIConversationStepStack.get(), which is reverse-ordered (get(0)= newest). In a 3-turn conversation, turn 1's extract surfaced on turn 3 and turn 1 lost it; only the middle turn aligned. Fixed by converting the forward index to the reverse accessor index (size-1-index).
Regression tests added for both (a real ConversationMemory with persisted extract/error keys proving the forwarder still forwards; a 3-turn stitching test proving extracts land on the correct turn). All new/changed classes remain above the >90% instruction / >80% branch gate; 654 tests green across the touched surface.
📎 Multimodal Attachments Completion — Phase 6 (partial): Metrics + GDPR portability (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion) Plan: planning/multimodal-attachments-completion-plan.md (Phase 6 of 6, partial).
What changed
Forwarder metrics —
AttachmentForwardernow takes aMeterRegistryand recordseddi.attachment.forwarded(content items sent to the LLM) andeddi.attachment.errors(dropped/gated/failed) per turn, satisfying AGENTS.md's "always add metrics" rule for the multimodal hot path.GDPR portability —
UserDataExportgains anattachmentslist (AttachmentExportEntry= conversationId/storageRef/fileName/mimeType/sizeBytes, metadata only, never bytes) plus a backward-compatible constructor.GdprComplianceService.exportUserDatacollects attachment metadata across the user's conversations viaIAttachmentStore.listByConversation, and the compliance audit event recordsattachmentsExported.
Deferred (documented follow-ups)
Still open in Phase 6: nightly reaper (orphaned blobs / stale grants via ScheduleFireExecutor), CostTracker multimodal token estimates, and an attachmentsForwarded audit-ledger entry. Phase 5 (multipart 1:1 say, SSE/output chips, and the EDDI-Manager / eddi-chat-ui frontend in their own repos) is likewise a follow-up — the two-step upload→say flow already works end-to-end.
Tests
Forwarder metrics assertion + GDPR attachment-metadata export test. Both green.
📎 Multimodal Attachments Completion — Phase 3: Group parity (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion) Plan: planning/multimodal-attachments-completion-plan.md (Phase 3 of 6).
What changed
DiscussRequestcarries attachments —IRestGroupConversation.DiscussRequestgains an optionalList<AttachmentRef> attachments(AttachmentRef = {mimeType, data, url, fileName}) plus a two-argument compat constructor, so existing JSON clients and call sites are unaffected.IGroupConversationService.discuss(...)andstartAndDiscussAsync(...)gain attachment-carrying overloads (default methods → real impl overrides).Materialize + bind at fan-out —
GroupConversationService.materializeAttachmentsstores inline base64 files inIAttachmentStorebound to the group conversation id (so they can be granted and reaped with it) and passes hostedurlrefs through, stashing the result on the (transient)GroupConversation.attachments.Grant + inject per member — on each member's first turn,
grantAndInjectAttachmentscallsIAttachmentStore.grantAccess(storageRef, memberConversationId)(the only place grants are minted — trusted server code, D2) and injectsattachment_*context into the member'sInputData. Stored refs are granted; URL refs are forwarded without a grant. Later phases rely on the Phase-2 extract-stitching and the Phase-4readAttachmenttool. Nested groups receive the parent's attachments and re-grant down the chain.REST routing —
RestGroupConversationconvertsAttachmentRef → Attachmentand routes through the attachment overload only when attachments are present (so the no-attachment path — and its existing mock-based tests — is untouched).
Design note
Group members run in their own conversations, so strict per-conversation ownership would block them from reading a group-uploaded blob — grants are exactly the primitive that makes this safe without opening cross-conversation access generally. Transport is JSON inline (base64/url); a multipart file-part variant of the endpoint is a thin follow-up (the capability and service path are complete).
Tests
7 service tests (materialize base64/url/no-store/empty; grant+inject stored-ref/url/grant-failure/none) + 2 REST routing tests (attachment overload vs plain). Group ITs (member observes content, grant-before-turn, nested) stay CI-only.
📎 Multimodal Attachments Completion — Phase 4: readAttachment tool (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion) Plan: planning/multimodal-attachments-completion-plan.md (Phase 4 of 6).
What changed
ReadAttachmentTool(modules/llm/tools/impl,@Vetoed) — the multi-turn recall path. Two@Tools:listAttachments()(name/type/size/ref of every attachment in the conversation) andreadAttachment(nameOrRef, page)(loads one attachment, extracts text — 1-based PDF page or 0 for whole doc — else a "no extractable text" note). The conversation id is implicit (constructor-injected), so the LLM never supplies a userId/conversationId and can only reach its own (or granted) attachments — enforced byIAttachmentStore.Auto-add wiring —
AgentOrchestratorgainssetAttachmentServices(store, extractor)(wired byLlmTaskin a new@PostConstruct, after CDI injection, so the long constructor + its six direct-construction tests are untouched).addReadAttachmentToolIfEnabledadds the tool in the no-whitelist branch when the turn has attachments, and in the whitelist branch under keyreadattachment; skipped when the services are unset (isolated tests) or the turn has no attachments. The forwarder's fallback notes already point the model at this tool.
Tests
ReadAttachmentToolTest (11 — list/read by name & ref, case-insensitive, PDF page, not-found, non-extractable, denied load, empty text, blank ref) + 5 orchestrator auto-add branch tests (no-whitelist, whitelisted, whitelist-excluded, services-unset, no-attachments). Existing orchestrator/LlmTask tests unchanged.
📎 Multimodal Attachments Completion — Phase 2: Unified forwarder (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion) Plan: planning/multimodal-attachments-completion-plan.md (Phase 2 of 6). Forwarder core.
What changed
AttachmentForwarder(modules/llm/impl, new) — the single place attachments become langchain4jContenton the outgoing user message. Replaces the image-onlyMultimodalMessageEnhancer(deleted, with its tests). Per attachment it resolves bytes from any source (stored blob →store.load, URL →SafeHttpClientdownload, base64 → decode) under uniform per-file (10 MB) and aggregate (20 MB) byte caps across all sources (the base64 path was previously unguarded), gates onModelCapabilityService(provider, model), and emits:image/*→ImageContentwhen vision-capable (URL passed through when the provider fetches URLs, else downloaded and inlined — provider URL normalization, D7), else a note;application/pdf→ hybrid: nativePdfFileContentwhen the model supports documents, else PDFBox text extraction inlined asTextContent;text-like (
text/*, JSON, XML, CSV, YAML) → decoded + inlined, no capability required (always works);audio/*→AudioContentwhen supported, else a note;everything else → a metadata note pointing at the (Phase 4)
readAttachmenttool.
Extracted text is persisted to
attachments:extracts(for Phase-2 history stitching) and every drop/skip/gate is appended toattachments:errors— never silent; each also leaves a note the LLM can relay.LlmTasknow calls the forwarder with the resolved(provider, model)instead of the static enhancer (field-injected + null-guarded so the six direct-constructionLlmTasktests are untouched).
Design decisions
Capability service uses the real defaults, not mocks, in tests — the forwarder test drives the true
ModelCapabilityServicematrix (OpenAI URL-image fast path, Gemini download-and-inline, Anthropic native PDF, OpenAI PDF text-fallback, jlama no-vision note).Skip ≠ silence — a per-file/aggregate cap hit, store-load failure, or download failure records to
attachments:errorsand emits aTextContentnote so the model can tell the user, rather than dropping the attachment invisibly.
Tests
AttachmentForwarderTest (18) covers the full branch matrix incl. URL-passthrough vs download-inline, base64/stored images, PDF native vs text-fallback (with extract persistence), text inline, audio on/off, unsupported note, per-file cap, store-load failure, and no-source skip. Enhancer tests removed.
Phase 2 tail (completed same branch)
Per-task multimodal override + reattachTurns —
LlmConfiguration.Taskgains an optionalmultimodal { vision|documents|audio: auto|on|off }block andreattachTurns(default 0). Old JSON configs deserialize cleanly (FAIL_ON_UNKNOWN_PROPERTIES=false).AttachmentForwarder.forwardgains aSupport-parameterized overload;LlmTaskparses the task block and passes the overrides (per-task > deployment > default precedence).History stitching —
ConversationLogGenerator.generategains an opt-institchAttachmentExtractsflag (only the LLM-facingConversationHistoryBuilderpath passestrue, so the visible transcript stays clean). Per turn it appends that step'sattachments:extractsto the rebuilt user message; verified aligned 1:1 with conversation outputs and that non-public step data survives snapshot persistence/reload, so a turn-2 follow-up sees turn-1's PDF/text extracts.reattachTurnsis schema-ready; extract-stitching + thereadAttachmenttool (Phase 4) are the primary multi-turn continuity mechanisms.
What's next (Phases 3–6)
Phase 3 (group parity), 4 (readAttachment tool), 5 (UX), 6 (ops).
📎 Multimodal Attachments Completion — Phase 1: Storage unification + secure upload (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion) Plan: planning/multimodal-attachments-completion-plan.md (Phase 1 of 6).
What changed
One blob store. Collapsed the two parallel abstractions onto
IAttachmentStore. Uploads already wrote to it (GridFS / Postgres*Store), but conversation-deletion and GDPR erasure cascaded through a different store (IAttachmentStorage→Mongo/PostgresAttachmentStorage), so uploaded blobs were never actually deleted. Ported both consumers (RestConversationStoredelete cascade,GdprComplianceServiceerasure) toIAttachmentStore, then deletedIAttachmentStorage+ both impls + their 4 tests (verified write-dead — only the delete cascades referenced them).Grants + owner-or-grant authz. New
IAttachmentStore.getMetadata()(server-validated metadata, no bytes),grantAccess()(trusted-caller-only cross-conversation read grant), single-itemdelete()(owner-only).load()/getMetadata()authorize owner OR an explicit grant; grants die with the blob. This is what lets group members read a blob uploaded to the group conversation (Phase 3) without opening cross-conversation access generally.UUID ref hardening (open decision #4). GridFS now returns an unguessable random-UUID
storageRefheld in file metadata (legacy ObjectId-hex refs still resolve); Postgres already used UUIDs. Both backends unified on one opaque ref format.Quotas. Per-conversation count + total-byte caps enforced in
store()(eddi.attachments.max-per-conversation= 50,eddi.attachments.max-total-bytes-per-conversation= 100 MB; non-positive disables).storageRefextraction branch (defect #2 — upload was orphaned).AttachmentContextExtractornow parses{storageRef}(precedence storageRef > url > data) andresolveAndGuard()resolves each stored ref's authoritative MIME/size viagetMetadata(owner/grant authorized) before behavior rules run, enforces the per-turn cap (eddi.attachments.max-per-turn= 5), and records every drop/failure toattachments:errors— never silent. Wired intoConversationinit viaIPropertiesHandler.getAttachmentStore()/getMaxAttachmentsPerTurn()(populated byConversationService).Secure REST surface.
RestAttachmentUploadgains aforwardableInlinehint on upload (upload cap 20 MB > forward cap 10 MB — warn at upload, not silently at forward), a single-item download endpoint (GET /conversations/{id}/attachments/{storageRef}, owner/grant-checked, Content-Disposition sanitized) and single-itemDELETE.
Design decisions
Auth model fits EDDI's anonymous-capable conversations. No other conversation endpoint uses
@RolesAllowed(only admin endpoints do), and anonymous deployments must keep working (D2). Enforcement is therefore store-level owner-or-grant authorization on everyload/getMetadata/delete, plus unguessable UUID refs — not an OIDC role gate.@RolesAllowedcan be layered on when a deployment makes OIDC mandatory.tenantIdstays advisory (sanitized, not an access boundary).Field injection for the two new
ConversationServicedeps (attachment store + per-turn cap) so the numerous direct-construction unit tests need no change.
Tests
161 unit tests across the affected classes: GridFsAttachmentStoreTest rewritten for UUID refs + grants + quota (26), AttachmentContextExtractorTest +storageRef/resolveAndGuard (27), RestAttachmentUploadTest +download/delete-one/forwardableInline/CD-sanitization (21), re-typed consumer tests. Postgres store IT and full ITs stay CI-only.
What's next
Phase 2 — the unified AttachmentForwarder (replaces MultimodalMessageEnhancer + convertMessage): hybrid PDF (native PdfFileContent vs PDFBox text), universal text inline, uniform per-file/aggregate caps across all sources, provider image-URL normalization, capability gating via ModelCapabilityService, and extracts-in-history stitching.
📎 Multimodal Attachments Completion — Phase 0: Foundations & bug fixes (2026-07-03)
Repo: EDDI (feat/multimodal-attachments-completion) Plan: planning/multimodal-attachments-completion-plan.md (Phase 0 of 6). Low-risk foundations that ship alone.
What changed
@JsonIgnoreonAttachment.getBase64Data()(engine/memory/model/Attachment.java) — thetransientkeyword did not stop Jackson (getter-based serialization, noPROPAGATE_TRANSIENT_MARKER), so inline base64 payloads were being serialized into Mongo conversation documents. Now excluded; metadata still persists. Serialization tests prove the payload never reaches persisted JSON.Scrub inline base64 from persisted context copies (
engine/memory/AttachmentContextExtractor.java+engine/runtime/internal/Conversation.java) — newAttachmentContextExtractor.scrubInlinePayload()returns a metadata-only copy of anattachment_*context when it carries adatapayload.Conversation.createContextData()builds the persisted copy (step data +context.*conversation output) through it, so the raw base64 (~1.33× file size/turn against the 16 MB doc limit) never lands in Mongo and is never exposed via{context.attachment_*.data}. The live payload still rides ATTACHMENTS memory for the turn. Mirrors secret-input scrubbing.AttachmentTextExtractor(modules/llm/tools/impl/, new) — shared PDFBox + plain-text extraction behind a uniform, configurable cap (eddi.attachments.extraction.max-chars, default 10k).extractText(bytes, mime[, maxChars])dispatches PDF + text-like (text/*, JSON, XML, CSV, YAML); PDF full/page-range/info methods;canExtractText().PdfReaderToolnow delegates all extraction to it (download/SSRF/formatting unchanged). Reused by the Phase 2 forwarder and Phase 4 readAttachment tool.ModelCapabilityService(modules/llm/capability/, new) — resolves vision/documents/audio/image-by-URL support for a(provider, model)pair. Precedence: per-task override > deployment override (eddi.multimodal.<provider>.<cap>theneddi.multimodal.<cap>) > conservative model-aware defaults (plan §5). Unknown ⇒ unsupported ⇒ fallback. Injectable via MicroProfile Config; Function-based constructor keeps it unit-testable.Body-size alignment (
application.properties) — addedquarkus.http.limits.max-body-size=25M(was Quarkus' 10 MB default, below the 20 MB attachment cap → 10–20 MB uploads died with a bare 413), plus documentededdi.attachments.max-size-bytesandeddi.attachments.extraction.max-chars.
Design decisions
Scrub is a copy, not a mutation — the original context map keeps its payload so the current turn's extraction/forwarding is unaffected; only the persisted derivative is stripped.
Extractor owns extraction, tool owns presentation —
PdfReaderTool.getPdfInfostill formats the human-readable string; the extractor returns a structuredPdfInfo, so the shared service stays presentation-free and reusable by the forwarder.Capability defaults are conservative and model-aware — vision-first providers (OpenAI/Anthropic/Gemini/Mistral) default on but downgrade for known text-only models; model-dependent providers (Ollama/Bedrock/Oracle) default off but upgrade for known vision models; image-by-URL only for OpenAI/Azure (everything else inlines).
Tests
146 new/covered unit tests: AttachmentTest (serialization no-payload), AttachmentContextExtractorTest (scrub matrix), AttachmentTextExtractorTest (PDF/text/caps/corrupt), ModelCapabilityServiceTest (74 — default matrix across 11 providers + override precedence). PdfReaderToolTest remains CI-only (SafeHttpClient opens a loopback selector local JVMs may block).
What's next
Phase 1 — storage unification (collapse IAttachmentStorage into IAttachmentStore, port conversation-delete + GDPR cascades), grants (grantAccess/grant-aware load), authenticated upload/list/download/delete, quotas, storageRef extraction branch, UUID ref hardening.
🐛 schedule — correct poll-batch-size comment (at-least-once, not exactly-once) (2026-07-13)
Repo: EDDI (feat/hitl-framework)
Copilot PR review flagged that the eddi.schedule.poll-batch-size comment in application.properties claimed "cluster-wide CAS still guarantees exactly-once" — which contradicts IScheduleStore's documented contract: firing is at-least-once, not exactly-once (an expired lease can be stolen, so schedule targets must be idempotent). Corrected the comment to state that per-lease CAS gives a single claimant but delivery is at-least-once, and the HITL timeout handler (resumes/cancels via CAS on conversation state) is idempotent.
Two other Copilot nits were declined as inconsistent with established codebase convention: (a) the exact GroupConversationState.values().length == 7 assertion is a deliberate tripwire matching its sibling TranscriptEntryType test — a lower-bound guard would lose the "did you mean to change the state set?" protection; (b) the // MINOR-2: label on OwnershipValidator is consistent with a pervasive plan-reference convention (9 MINOR-/MAJOR- labels plus hundreds of #NN/Hn/Task N markers) — a one-off removal would be inconsistent.
📝 HITL enum refactor — documentation audit (2026-07-13)
Repo: EDDI (feat/hitl-framework)
Audited all documentation for the enum refactor (changelog accuracy, user-doc coverage, code comments). User-facing docs correctly need no change: README.md, AGENTS.md, and docs/hitl.md reference timeoutPolicy only at the config/REST layer (the JSON string values AUTO_APPROVE/AUTO_REJECT/ABORT/WAIT_INDEFINITELY), which the internal String → enum retype leaves byte-identical — no config-schema or wire-format change to document. Two accuracy fixes made:
HitlCrashRecoveryObservercomment (group re-arm site): the comment claimed the inline null-default avoids "the String overload the regular surface shares" — stale after the regular surface also became an enum. Corrected to state that both bookmarks are now enum andparsePolicy(String)survives only for thePendingApprovalSummaryprojection scan (stillString).Changelog (regular-surface entry): it listed the
McpHitlToolsregular read site among the sites updated to.name(), but that site was the one missed in that commit and fixed in the follow-up — corrected to say so.
Also re-verified HitlTimeoutPolicySerializationTest passes directly (Tests run: 15, Failures: 0).
✅ HITL enum refactor — round-2 review clean + serialization regression guard (2026-07-13)
Repo: EDDI (feat/hitl-framework)
After the round-1 review caught McpHitlTools:185, ran a second, deeper adversarial review (4 orthogonal angles: complete call-site re-inventory, runtime serialization across both stores, end-to-end timeout-fire path, and an explicit "find one more bug" hunt — each finding verify-gated, plus a completeness critic). Result: zero findings, verdict CORRECT_AND_COMPLETE. Every remaining String touchpoint is a deliberate guarded boundary conversion (PendingApprovalSummary stays String via guarded .name(); schedule metadata stays String and HitlTimeoutHandler parses it back via valueOf; REST/MCP/Slack summaries via .name()); crash-recovery null-defaults faithfully mirror the old parsePolicy; parsePolicy(String) remains live for the projection-scan path.
Added HitlTimeoutPolicySerializationTest (pure-unit, no Testcontainers) as a permanent regression guard for the invariant the whole refactor rests on. It replicates BOTH production mappers — the JSON mapper (Postgres JSONB + REST) and the BSON mapper (MongoDB, built like PersistenceModule) — across BOTH surfaces (ConversationMemorySnapshot, GroupConversation) and asserts: enum ⇄ name()-string round-trip for all four values; BSON encodes a string, not an ordinal; a null policy is omitted (NON_NULL) and round-trips to null; and legacy pre-refactor documents (policy as a bare JSON string) still deserialize into the enum. 15/15 pass locally — closing the residual runtime/persistence risk that the CI-only Testcontainer store tests would otherwise be the sole coverage for.
🐛 HITL enum refactor — fix missed McpHitlTools read site (clean-compile break) (2026-07-13)
Repo: EDDI (feat/hitl-framework)
A thorough adversarial code review (5 dimensions + verify + completeness critic) of the two enum-refactor commits below found one real, CRITICAL defect: the regular-surface MCP read site McpHitlTools.getApprovalStatus (McpHitlTools.java:185) still put snapshot.getHitlTimeoutPolicy() (now the enum) into a Map<String,String> without .name(). Because that map's value type is String (unlike the Map<String,Object> sibling at :359), the mixed enum : "" ternary is a hard javac error (bad type in conditional expression: HitlTimeoutPolicy cannot be converted to String). The group twin (:359) and RestAgentEngine:407 were fixed; this regular twin was missed.
Why it slipped past verification: the earlier ./mvnw test runs reported BUILD SUCCESS because Maven incremental compilation reused a stale McpHitlTools.class — the source file wasn't edited, so it wasn't recompiled even though its dependency (ConversationMemorySnapshot) changed type. A ./mvnw clean compile fails. The prior changelog claim that "the full main + test tree compiles" was therefore based on a false pass and is corrected here. Lesson: verify type-signature refactors with clean compile, not incremental.
Fix: append the guarded .name() to match its three siblings — summary.put("timeoutPolicy", paused && snapshot.getHitlTimeoutPolicy() != null ? snapshot.getHitlTimeoutPolicy().name() : ""). Wire output is byte-identical ("AUTO_REJECT" / "").
Verified with a clean build: ./mvnw clean test compiles the whole main + test tree from scratch and the affected suites pass (Tests run: 258, Failures: 0, Errors: 0). The review's other four dimensions (serialization/persistence, null-safety, behavior-preservation, test-fidelity) and the completeness critic returned no other defects — the refactor is otherwise correct and complete.
🎯 Regular surface — type hitlTimeoutPolicy as the HitlTimeoutPolicy enum (2026-07-13)
Repo: EDDI (feat/hitl-framework)
Follow-up to the group-surface enum change (below): applied the same String → HitlTimeoutPolicy retype to the regular (agent) conversation surface so both surfaces are consistent. The hitlApprovalTimeout field stays String on both, for the same reasons documented in the group entry (uniform convention + Duration would serialize as a number under write-dates-as-timestamps=true).
Model layer (IConversationMemory default methods, ConversationMemory impl field + accessors, ConversationMemorySnapshot field + accessors) now carry the enum. ConversationMemoryUtilities copies memory ↔ snapshot unchanged (both enum). Consumer (ConversationService): the four bookmark set-sites drop .name() (the source AgentConfiguration.HitlConfig.getTimeoutPolicy() / ToolApprovalsConfig.getTimeoutPolicy() / the computed effectivePolicy are all already the enum); scheduleHitlTimeout compares == WAIT_INDEFINITELY and emits .name() only into the Map<String,Object> schedule metadata. Read/display sites call .name(): the RestAgentEngine summary map, ConversationMemoryStore.collectPendingSummaries (feeds the String-typed PendingApprovalSummary), and SlackEventHandler.formatTimeoutInfo. (The parity McpHitlTools:185 regular read site was missed here and fixed in the follow-up above.) Crash recovery (HitlCrashRecoveryObserver): the regular IN_PROGRESS-recovery site inlines the null → WAIT_INDEFINITELY default; parsePolicy(String) stays intact for its remaining caller (the PendingApprovalSummary projection, still String).
Persistence — verified wire-safe. ConversationMemorySnapshot is stored as a JSONB/BSON blob (Jackson serializes the enum as its name()), so already-persisted AWAITING_HUMAN bookmarks deserialize unchanged. The Postgres bounded projection (data->>'hitlTimeoutPolicy' AS timeout_policy → rs.getString(...) → PendingApprovalSummary) reads the raw JSON name string and is unaffected by the model type change. The REST awaitingApproval summary and Manager UI contract are byte-identical.
Scope: 9 main files + 12 test files (setters/asserts moved to enum constants; String-param helpers convert via valueOf; store-test call-sites retyped). SimpleConversationMemorySnapshot does not carry this field, so it is untouched. Verified: Tests run: 258, Failures: 0, Errors: 0 across the regular + group HITL unit suites; full main + test tree compiles (the Testcontainer store tests compile and run in CI). IRestAgentEngine.java (formatter oscillation) restored/excluded again. Nothing pushed.
🎯 GroupConversation — type hitlTimeoutPolicy as the HitlTimeoutPolicy enum (2026-07-13)
Repo: EDDI (feat/hitl-framework)
A PR review comment (@niedch) on GroupConversation's HITL bookmark getters/setters said: "I would prefer to go with the actual enum type and Duration for this." Analyzed both halves (4-way investigation + adversarial verify) and split the decision: did the enum, deliberately skipped the Duration.
Enum — done. hitlTimeoutPolicy was a raw String copied from config at pause time, but the HitlTimeoutPolicy enum (WAIT_INDEFINITELY, AUTO_APPROVE, AUTO_REJECT, ABORT) already exists and is the declared type in all three HITL config POJOs (AgentConfiguration.HitlConfig, AgentGroupConfiguration.HitlConfig, ToolApprovalsConfig.timeoutPolicy) — GroupConversation was the lone raw-String outlier for the policy. The field is control-flow-relevant (gates/arms scheduleGroupHitlTimeout, drives crash re-arm in HitlCrashRecoveryObserver), so typing it removes stringly-typed valueOf/.name()/parsePolicy juggling. It is wire-safe: Jackson serializes enums by name(), so "AUTO_REJECT" round-trips identically in Mongo/Postgres JSON, over REST, and to the Manager UI — exactly how the sibling GroupConversationState state field already persists. Every set-site only ever wrote a valid name() or null, so deserializing already-persisted AWAITING_APPROVAL transcripts cannot throw.
Files:
GroupConversation.java— field + getter/setter →HitlTimeoutPolicy(import added).GroupConversationService.java—commitPause/restoreGroupPausepass the enum directly (dropped.name());restoreGroupPause'sfallbackTimeoutPolicyparam +resumeDiscussion'ssavedTimeoutPolicylocal retyped to the enum;scheduleGroupHitlTimeoutcompares== WAIT_INDEFINITELYand calls.name()only when writing the schedule-metadataMap<String,Object>;listGroupPendingApprovalsnull-guards.name()for theString-typedPendingApprovalSummary.HitlCrashRecoveryObserver.java— the group site inlines thenull → WAIT_INDEFINITELYdefault instead of routing through the sharedparsePolicy(String), which stays intact for its two regular-surface callers (PendingApprovalSummary,ConversationMemorySnapshot).RestGroupConversation.java/McpHitlTools.java— the summary map puts.name()(identical wire value, keeps the value aString).Tests (
GroupConversationServiceHitlCoverage2Test,…CoverageTest,HitlCrashRecoveryObserverTest,…CoverageTest) — reflectiverestoreGroupPausesignature + args updated to the enum, assertions compare the enum,gc.helpers convert.
Duration — skipped (deliberate). hitlApprovalTimeout stays String. Unlike the policy, approvalTimeout is uniformly String across every carrier (all three configs, the memory bookmark, PendingApprovalSummary, the transcript), so the convention favors String. And it is not wire-safe: the deployment sets quarkus.jackson.write-dates-as-timestamps=true with no WRITE_DURATIONS_AS_TIMESTAMPS override, so a java.time.Duration would serialize to a bare number (900.0) instead of "PT15M", breaking the raw-over-REST OpenAPI/Manager-UI contract (the frontend reads it as an ISO-8601 string with a PT15M placeholder). A correct migration would need a whole-surface change + a custom ISO-8601 serializer + coordinated frontend work — a separate cross-cutting effort, out of scope for a review nit.
The ConversationMemorySnapshot regular surface keeps String for both fields — it is a separate class with its own parsePolicy(String) path; only the group transcript was retyped. (A parallel enum change there is a possible follow-up but was left out to keep this diff scoped.)
Verified: the four affected unit-test classes pass (Tests run: 149, Failures: 0, Errors: 0); surefire compiled the entire main + test tree first, so all call-sites type-check. Neither change is a correctness bug and the comment was a "prefer", so this does not block the branch. IRestAgentEngine.java (reformatted by formatter-maven-plugin during the build, unrelated to this task) was restored and excluded. Nothing pushed.
🧹 ChannelTargetRouter — drop dead getPlatformConfig() null-checks + duplicate allocations (2026-07-06)
Repo: EDDI (feat/hitl-framework)
A Copilot review of the HITL PR flagged one site in ChannelTargetRouter.getIntegrationByApprovalChannel where getPlatformConfig() was called twice behind a redundant != null guard. Verified against ChannelIntegrationConfiguration.getPlatformConfig(): it returns new HashMap<>(platformConfig) — a fresh defensive copy, with the field initialized non-null and the setter null-guarded — so it never returns null. That makes every getPlatformConfig() != null sub-check dead code, and each doubled call allocates a throwaway map per invocation.
The same pattern existed in five other methods Copilot did not flag; fixed all six for consistency:
getIntegrationByApprovalChannel(the flagged one) — cache the copy once, drop the dead guardgetBotToken, the config-load loop,ResolvedTarget.botToken,ResolvedTarget.signingSecret— drop the redundant&& getPlatformConfig() != nullclausedeepCopyConfig— drop the always-trueifwrapper, single call
Every genuine guard is preserved (integration != null, config != null, getChannelType() != null); only the provably-dead getPlatformConfig() != null sub-checks and the duplicate allocations were removed. Behavior is identical because the getter cannot return null.
Copilot's second comment — rejected (verified against tooling): it wanted IRestAgentEngine.listPendingApprovals collapsed to one line. The formatter-maven-plugin (Eclipse formatter, bound to the build) produces exactly the two-line split it objects to and auto-reverts the single-line form on every mvnw compile, so the nit conflicts with the project's enforced format — no change made.
Verified: ./mvnw compile exits 0 (checkstyle at validate, formatter at process-sources, javac all clean). Change isolated to ChannelTargetRouter.java. Nothing pushed.
🐛 mcpcalls — register McpCallsTask via startup module (2026-07-06)
Repo: EDDI (feat/hitl-framework)
McpCallsTask (the MCP-client httpcall lifecycle task) was tracked and committed, but its bootstrap registration was not — so on a fresh checkout the task was never inserted into the lifecycle-task provider map (@LifecycleExtensions) and the mcpcalls feature silently failed to wire into the pipeline. Added McpCallsModule (@Startup(1000) + @PostConstruct), mirroring the seven sibling bootstraps (ApiCallsModule, LlmModule, OutputGenerationModule, PropertySetterModule, RulesModule, SemanticParserModule, TemplateEngineModule), which registers McpCallsTask.ID. The file existed but was untracked in the working tree; surfaced while auditing the tree during the MCP-whitelist review and committed here as a wiring bug relevant to this branch. Compiles clean.
🔌 MCP tool filter — expose HITL/memory/GDPR tools + build-time regression guard (2026-07-06)
Repo: EDDI (feat/hitl-framework)
A client reported that new HITL MCP tools were "implemented but not available." Confirmed: McpToolFilter is a name whitelist (ToolFilter SPI — quarkus-MCP only surfaces a tool's name, not its declaring class/annotation, so filtering must be by name), and it exposes only the intended MCP tools while hiding the langchain4j built-in agent tools (calculator, websearch, etc.) that leak into the same scan. Three Mcp*Tools classes had shipped @Tools that were never added to the whitelist, making them unreachable dead code — a quarkus-MCP @Tool has no other invocation path:
McpHitlTools(9):list_pending_approvals,get_approval_status,resume_conversation,cancel_conversation,list_group_pending_approvals,list_all_group_pending_approvals,get_group_approval_status,approve_group_phase,cancel_group_discussion— documented as the MCP HITL surface indocs/hitl.mdbut invisible.McpMemoryTools(8):list_user_memories,get_visible_memories,search_user_memories,get_memory_by_key,upsert_user_memory,delete_user_memory,delete_all_user_memories,count_user_memories.McpGdprTools(2):delete_user_data,export_user_data.
Why it slipped through: the existing regression test (McpToolFilterTest.test_allMcpToolMethods_areWhitelisted) scanned only a hardcoded array of 4 Mcp*Tools classes — HITL, memory, and GDPR were not in it, so CI stayed green.
Fix:
McpToolFilter.java— added all 19 names toMCP_TOOLS(whitelist 55 → 74 = every declared quarkus-MCP@Tool). Verified there is no name collision with any langchain4j built-in tool (effective names cross-checked), so whitelisting a name cannot accidentally expose an internal agent tool. All three classes already enforce their own authz (requireRoleviewer/admin + per-userOwnershipValidator; GDPR delete is admin-only +CONFIRMarg), identical to their REST counterparts — MCP is a transport, not new authority.McpToolFilterTest.java— rewrote the guard to auto-discover every class in theai.labs.eddi.engine.mcppackage by scanning the compiled-classes directory (no hardcoded class list), resolve each@Tool's effective name (explicitname, else method name — theMcpGroupToolsconvention), and fail the build if any is not whitelisted. Anchor tools (one perMcp*Toolsclass) guard against a broken scan passing vacuously. Any future MCP tool that isn't whitelisted now turns CI red.docs/mcp-server.md— corrected the stale tool count (63 → 74), documented the name-onlyToolFilterconstraint and the new build-time guard.
Decision: whitelist (not delete) memory/GDPR — they were intended MCP tools (Phase 11a persistent memory, GDPR/CCPA framework) that were simply never wired into the filter; the annotation encodes intent to expose.
Follow-up — adversarial code review + fixes: the commit was then put through a 4-dimension adversarial review (whitelist-correctness, test-robustness, security/authz, docs-completeness), each finding skeptic-verified. Whitelist-correctness and test-robustness came back clean; 3 low-severity findings survived and 2 were addressed here:
Doc role-name fix (
docs/mcp-server.md): the "Recommended Role Mapping" table named non-existent rolesmcp-user/mcp-adminand cited@RolesAllowed; the code actually enforceseddi-viewer/eddi-editor/eddi-admin(viarequireRole) andeddi-approver/owner (viaHitlAccessGuard). Rewrote the section with the real role strings and mechanism; this became load-bearing now that 10 role-guarded memory/GDPR tools are reachable.Collision-guard test (
McpToolFilterTest.test_noLangchain4jBuiltinToolIsWhitelisted): the "no name collision with langchain4j built-ins" property was a one-time manual check. Added the inverse build-time guard — auto-discovers everydev.langchain4j.agent.tool.Toolundermodules.llm.toolsand fails if any effective name is whitelisted (would leak an internal agent tool to MCP). Also hardened both discovery helpers to load classes without static init (Class.forName(name, false, …)).Not fixed (decision deferred): the memory/GDPR mutation tools lack an independent MCP mutation kill-switch like HITL's
eddi.mcp.hitl.mutations.enabled— flagged low, consistent with the pre-existing posture of other whitelisted destructive tools (delete_agent, etc.); left for the maintainer to decide whether to add symmetric kill-switches across the MCP mutation surface.
Method: verified the whole diagnosis against source (annotation imports, ToolInfo/langchain4j @Tool APIs via javap, collision analysis) before changing anything; both regression guards were proven to fail on an injected regression, then restored. ./mvnw -o test -Dtest=McpToolFilterTest → 90 green; ./mvnw -o validate clean. Nothing pushed — that stays the maintainer's call.
🔧 Dependency bumps — Quarkus 3.37.1, quarkus-mcp-server 1.13.1 (2026-07-06)
Repo: EDDI (feat/hitl-framework)
Patch bumps in pom.xml: quarkus.platform.version 3.37.0 → 3.37.1 and quarkus-mcp-server.version 1.13.0 → 1.13.1 (used by io.quarkiverse.mcp:quarkus-mcp-server-http). Both are single-property changes; the version is defined only in pom.xml, so no other current-state reference needed updating (historical changelog/release-note mentions left as-is). Verified locally with ./mvnw -B compile — BUILD SUCCESS against the new BOM (quarkus:3.37.1:generate-code ran) and quarkus-mcp-server-http:1.13.1 resolved into the local repo; full test suite runs in CI.
📝 AI-agent docs audit — AGENTS.md overhaul + linked-doc consistency fixes (2026-07-06)
Repo: EDDI (feat/hitl-framework)
Audited AGENTS.md (the instruction file AI coding assistants load; CLAUDE.md just delegates to it) against every authoritative source it relies on, then rewrote it for correctness and frictionless cold-checkout onboarding. Verification ran as five parallel research agents cross-checking claims against pom.xml, ci.yml, Dockerfile, README.md, docs/project-philosophy.md, docs/architecture.md, docs/hitl.md, and the Agent Father config, plus an independent "fresh contributor" friction review.
AGENTS.md — factual fixes: Quarkus 3.34.1 de-pinned (versions now reference pom.xml as the single source of truth, per the file's own rule 7); MCP 33 tools → 60+; CostTracker → ToolCostTracker (real class name); SafeMathParser clarified as a static inner class of CalculatorTool; HITL tool-source count 8 → 7 (verified against ToolApprovalPatterns.KNOWN_SOURCES); §5.6 corrected (Agent Father uses scope: "conversation", not secret); HITL moved Upcoming → Completed with residual work (Manager approvals UI, inGroupTurns: INBOX) kept in Upcoming; Multi-Channel narrowed to Teams (Slack ships via HITL); five broken docs/planning/ → planning/ links.
AGENTS.md — onboarding & policy: added a table of contents, a Build & Test Commands section (Windows .\mvnw.cmd note, sandbox/IT caveat, mise.toml toolchain, prerequisites → README, pre-push hook activation), an external-contributor fork-model pointer, and a pillar cross-reference. Codified two team policies: no AI co-authorship trailers or tool-advertising footers on commits/PRs (§2 rule 5) and ask before pushing (§2 rule 4).
Linked-doc consistency fixes: docs/hitl.md eight → seven tool sources; docs/architecture.md retired stale v5 package terminology across the Agent Composition section (packagestore → workflowstore, .package.json → .workflow.json, packageExtensions/WorkflowExtension → workflowSteps/WorkflowStep, configs.packages.model → configs.workflows.model, "packages" → "workflows") and fixed the LLM URI llmstore/llmconfigs → llmstore/llms — all verified against WorkflowConfiguration.java and the Agent Father config; CONTRIBUTING.md reconciled "squash fixup commits" with the never-rewrite-pushed-history rule.
Decisions: kept all content inline in AGENTS.md (no extraction to new files) per maintainer preference and because the architecture audit confirmed AGENTS.md's prescriptive content is high-value and, on workflow-vs-package naming, more current than architecture.md was; prefer referencing canonical sources over restating drift-prone numbers/versions; committed to feat/hitl-framework rather than a new branch off origin/main because docs/hitl.md exists only on this branch.
Method: six delegated research/critique agents, each finding verified against source before acceptance. Nothing pushed — that stays the maintainer's call.
🧭 HITL — whole-branch merge review (round 2) + all 22 findings fixed + fix-batch review (2026-07-05)
Repo: EDDI (feat/hitl-framework)
The entire branch (111 commits, 242 files, base 6f5f5dd68 → a5df6afd2) — including the ~90 commits of MCP-HITL pre-work the earlier 15-commit review never covered, plus the composition of the five follow-up fixes below — was put through a second whole-branch adversarial review (11 dimension reviewers → per-finding skeptic verification → gap round). It surfaced 25 confirmed defects (3 high, 6 medium, 13 low after de-dup) that the narrower per-task and 15-commit reviews had missed because they are cross-cutting. The two most safety-critical dimensions (durability/at-most-once, backward-compat) again came back clean. Verdict: not merge-ready until the 3 highs were fixed; the user chose to fix all 22. All are now fixed across 10 commits (b9a6c1263..499095fa4), each build+test-gated:
Blockers (high):
b9a6c1263—EXECUTION_INTERRUPTEDno longer bricks input (H1). The queued-say guard skippedEXECUTION_INTERRUPTED, but nothing returns that state to READY except a running turn — so an ordinary 60sagentTimeoutwatchdog expiry or aHitlCrashRecoveryObserver"unlock say()" recovery permanently locked the conversation's input (a non-HITL-scoped regression). It is a recoverable marker, not terminal: dropped from the guard so a fresh say re-runs and self-heals. Same commit fixes M1 (pause-commit CAS now from the actual pre-turn state, not hard-coded READY, so an ERROR/interrupted-retry pause commits), M2 (post-commitisCancelled()re-check converts a cancel-raced pause toEXECUTION_INTERRUPTEDinstead of stranding it with an armed timer), and two lows (undo/redo now CAS from the loaded state; the 2-min re-arm grace clamps only past-due deadlines, honoring sub-2minapprovalTimeout).056a02e13— tool-approval gate honored on theCONVERSATION_STARTinit turn (H2). The agent-leveltoolApprovalscarrier was populated only on say/resume, never at conversation start, so a gated tool invoked by a greeting-turn LLM task executed without approval (fail-open). TheAgentnow carries the config (likememoryPolicy) and sets it on memory beforeinit(); also covers scheduled and group-member conversations.6e4474552— PostgreSQL journal store + working TTL + GDPR erasure (H3, M4, M5, M6).IHitlToolJournalStorehad no Postgres impl, so oneddi.datastore.type=postgresevery tool-approval resume/approval-status read dialed a nonexistent Mongo — tool-level HITL was unusable on a first-class backend. AddedPostgresHitlToolJournalStore(INSERT … ON CONFLICT DO NOTHINGpreserves at-most-once) + aDataStoreProducersselector matching the 16 sibling stores. M4: the Mongo TTL was inert (executedAtstored as int64, which the TTL monitor ignores) — nowclaimedAt/executedAtare BSON Dates, the TTL is anchored onclaimedAt(so orphaned EXECUTING claims also expire), withIndexOptionsConflictdrop+recreate. M5: GDPR erasure now cascades to the journal (before conversation deletion, so ids resolve). M6: the vacuous unique-index test and tautologicaldifferentPauseEpochtest are now genuine assertions.
Medium/low (other commits): dcfe2c2f7 (M3 — the raw conversation-read REST endpoint no longer leaks argumentsRaw+transcript; names-only projection reused from fix #4), e7da12f8f (gate lows: null-tool-name NPE, resume kill-switch threading, null-id callId normalization at AiMessage reception, cascade/watchdog abandoned-thread guard), a7d0df601 (RULE-pause hitl:status output marker), c6745b6ae (MCP approve_group_phase returns BAD_REQUEST not INTERNAL for non-string values), d76b84869 (group cancel-signal remove-window + sub-2min timeout parity), 50c97387f (doc: outcome_unknown is WARN-logged not audited; errorCode set += CONFLICT/INTERNAL), 499095fa4 (LlmTask→orchestrator transcript-cap threading test).
Fix-batch critical review. The 10 fix commits were then themselves put through an adversarial review (4 concern reviewers — state-machine, gate, journal, cross-cutting — each finding skeptic-verified). The journal batch came back fully clean (CDI producer pattern verified byte-identical to the 16 siblings; Postgres at-most-once and Mongo TTL-anchor correctness confirmed). The gate abandoned-thread concern was refuted (fail-safe holds). Two low, fix-introduced defects were confirmed and fixed in 8ebf5b691: (a) storeConversationMemorySnapshotIfState could NPE on a null expectedState (M1/undo/redo now pass a live-looked-up state that is null if the conversation was deleted concurrently) — guarded to a clean CAS-miss in both stores; (b) the RULE-pause hitl:status marker was never cleared on resume, so a resolved turn kept advertising "awaiting approval" — now removed on resume via a new removeConversationOutput step API, with the key/value extracted to shared constants.
Method: two deterministic multi-agent review workflows (11 + 4 reviewers, each finding adversarially verified before acceptance); the large Postgres-journal batch was implemented by a delegated agent whose diff was reviewed against spec before commit. Full clean compile green; every batch's targeted tests green. Nothing pushed — that stays the maintainer's call.
🔬 Tool-level HITL — adversarial final review + follow-up fixes (2026-07-04)
Repo: EDDI (feat/hitl-framework)
After Tasks 5–17 landed (see the entry below), the whole tool-level HITL change (15 commits, 3e5da4345..dda7c644e) was put through a whole-branch adversarial review: six independent dimension reviewers (correctness, concurrency, security, backward-compat, durability/at-most-once, spec-completeness), each finding then handed to an independent skeptic instructed to refute it, then a synthesis pass. Verdict: merge-ready, no blockers — the two most safety-critical dimensions, durability/at-most-once and backward-compat, came back clean (no double-execution of an approved tool across crash/re-approval; null-config / RULE-pause / legacy-snapshot paths byte-identical). Five fail-safe defects survived verification and are now all fixed (each via a TDD fix + independent review gate):
fix(hitl)6413db97a— generic read surface no longer leaks raw tool args + transcript.SimpleConversationMemorySnapshot(the generic conversation-read DTO behind e.g. MCPread_conversationand the REST simple log) carried the fullPendingToolCallBatchincludingargumentsRawandchatTranscriptJsonwith no@JsonIgnore, so anyeddi-viewercould read unredacted tool arguments + the whole transcript of a paused conversation — broader than the deliberately approver-onlydetail=fullgate. Fixed with a names-only projection at the Simple-snapshot boundary (freshPendingToolCallobjects carrying onlycallId/toolName/source/gateReason/argsTruncated); the persisted fullConversationMemorySnapshotis untouched, so the at-most-once resume path still round-trips. (Introduced by the Task-13 Simple-snapshot extension.)fix(hitl)1d7ca72e7— say-path pause commit guarded by a state-CAS. The say-path fresh-pause persistence was an unconditional full-document store guarded only by an up-frontisCancelled()check; a concurrentend/cancellanding in the TOCTOU window could be lost, resurrecting an ENDED/cancelled conversation asAWAITING_HUMANwith an armed timeout. Now usesstoreConversationMemorySnapshotIfState(compare-and-store from the runningREADYstate); a miss discards the pause (no store, no counter, no schedule), mirroring the resume path's existing guard. Covers both RULE and TOOL_CALL pauses.fix(hitl)30e495b88— tool-pause policy resolved from the task-scoped effective config. Post-pause resolution of timeout policy, no-progress policy, auto-approval cap, and pending message read only the agent-levelhitlConfig.toolApprovals, ignoring the per-taskLlmConfiguration.Task.toolApprovalsoverride that the gate itself honors — so a task-scoped finitetimeoutPolicy=AUTO_REJECT/approvalTimeoutsilently degraded toWAIT_INDEFINITELY(waited forever instead of auto-rejecting). The gate's resolved effective config is now stamped onto thePendingToolCallBatchand read back by all four resolvers (agent-level fallback for legacy/RULE/null batches); Task 10's inherit-from-outer +AUTO_APPROVE-demotion semantics are unchanged.fix(hitl)69143a799— resumed tool-pause turn renders only the final answer. On a same-index TOOL_CALL resume the final answer was appended to the same step's"output"list that still held the "awaiting approval" placeholder, so the turn rendered both stacked. Resume now removes exactly the deterministic placeholder (identified viaresolvePendingMessage, stable across pause→resume through the persisted batch) plus its mirrorData<>; the placeholder still shows whileAWAITING_HUMAN, earlier multi-task output is preserved, and the RULE path is unchanged.fix(hitl)9d07c525d—eddi.hitl.tool.transcript-max-byteswired. The plan-mandated transcript-cap override property was never wired (hard-coded 2 MB). Now injected inLlmTask(the CDI seam, likeeddi.hitl.tool.enabled) and threaded through the standard + cascade branches tobuildPendingBatch; an absent property reproduces the unchanged 2 MB default.
Method: the review ran as a deterministic multi-agent workflow (12 agents); every finding was adversarially verified against HEAD before acceptance, and every fix was independently re-reviewed before landing. The full local suite is ~10,500 tests green with no logic regression (only the pre-existing Docker/Testcontainers/loopback-socket classes fail in the sandbox — they run in CI).
Tracked follow-ups (fail-safe, non-blocking): (a) resumeToolLoop's rare re-pause-during-continuation still caps the transcript at the 2 MB default rather than the configured value (commented at the call site); (b) no validation of a pathological 0/negative transcript-max-bytes (fail-safe always-omit). The Testcontainers ITs and the BSON/JSONB round-trip of the new effectiveToolApprovals batch field are CI-verified only.
🛠️ Tool-level HITL — complete feature + documentation (Tasks 5–17 of 17) (2026-07-04)
Repo: EDDI (feat/hitl-framework)
Completes and documents tool-level HITL approval gating: the conversation pauses for human approval when the LLM invokes a gated tool (any of the 8 sources — built-in @Tool, http, mcp, a2a, dynamic, memory, recall), gated before the tool executes (fail-safe), configured via allow/exempt glob patterns, coexisting with the behavior-rule PAUSE_CONVERSATION turn gate. Builds on the Tasks 1–4 foundation (see the earlier 2026-07-03 entry — do not duplicate). Full plan: planning/hitl-tool-approval-plan.md.
Task 17 — documentation (this entry). No production code changed. Docs/markdown only:
docs/hitl.md— removed the now-false "Tool-level HITL … is deferred" from Known Limitations (it contradicted the rest of the same doc) and replaced it with an implemented note; scoped the Slack data-minimization sentence to RULE pauses and documented the TOOL_CALL exception (the approval channel renders redacted, 300-char-truncated tool arguments so a reviewer can see what they are approving; the in-thread notice stays pause-reason-only); added a Tool-Level Approval Gating section (config schema + defaults, pattern language, precedence, effective-timeout-policy rule, per-call verdict/amendment REST bodies with a JSON example,pauseDetailsreference, the write-ahead journal + outcome-unknown contract, Slack all-or-nothing buttons, group-member REJECT, frozen-transcript semantics, and theeddi.hitl.tool.enabledrolling-upgrade note). Reconciled with the existing tool-level content earlier tasks had already added (thepauseDetailsshapes, MCP Surface table,eddi.mcp.hitl.mutations.enabledkill-switch) — no duplicate sections.AGENTS.md§5.3 — extended the HITL note: behavior rules gate turns (PAUSE_CONVERSATION);hitlConfig.toolApprovalsgates individual LLM tool calls (hitlPauseType: "TOOL_CALL"); both share the same pause/timeout/audit/Slack machinery; link todocs/hitl.md.planning/hitl-framework-plan.md— flipped the decision-table line "Tool-level HITL: Deferred" to "Implemented — seeplanning/hitl-tool-approval-plan.md".
The 5 product decisions (from the plan's decision record), now locked and documented:
AUTO_APPROVEnever applies to tool pauses implicitly — explicit opt-in only. Agent-levelAUTO_APPROVEcovers RULE pauses; for a tool pause it is demoted toWAIT_INDEFINITELYunlesstoolApprovals.timeoutPolicysetsAUTO_APPROVEexplicitly (a silent timeout must never auto-execute a gated tool).Crash inside an approved tool yields an honest
EXECUTION_OUTCOME_UNKNOWN— never silent re-execution. The write-ahead journal (IHitlToolJournalStore, keyed byconversationId + pauseEpoch + callId) replaysEXECUTEDresults and reportsEXECUTING(crashed mid-tool) as genuinely unknown, audited and surfaced inpauseDetails.outcomeUnknown.Group-member tool pauses auto-reject gracefully (
system:group) — a group has no reviewer, so the member's gated call is REJECTED through the normal resume path and its LLM produces a coherent tool-less contribution (fallback: SKIP + auto-cancel).inGroupTurns: "INBOX"is reserved (400 in v1).Ungated calls in a mixed batch execute before the human sees the pause — the approver is then shown which ones already ran via
pauseDetails.executedUngatedCalls.A multi-day pause resumes against pause-time prompt state — the exact in-flight langchain4j transcript is frozen at pause time and replayed on resume (same task index), never rebuilt from current memory.
Cross-checked every documented fact against source (ToolApprovalsConfig, ToolApprovalPatterns, ToolApprovalGate, HitlDecision/ToolCallDecision, HitlConfigValidation, ConversationService.applyEffectiveToolTimeoutPolicy + validateToolDecisions, RestAgentEngine.buildToolCallPauseDetails, AgentOrchestrator.resumeToolLoop, IHitlToolJournalStore, SlackHitlSupport, GroupConversationService.tryResolveMemberToolPause, and the eddi.hitl.tool.enabled flag in LlmTask/application.properties) — field names, defaults (maxPausesPerTurn 3/1..10, maxAutoApprovalsPerTurn 2/0..10), ranges, note caps (top-level 4096, per-call 1024), the 300-char Slack display truncation, and the precedence/timeout rules all match the implementation.
Next: the tool-level HITL feature (Tasks 1–17) is complete on feat/hitl-framework. Remaining HITL roadmap items: EDDI-Manager approvals UI (separate repo/PR) and the reserved inGroupTurns: "INBOX" mode.
🔐 MCP HITL surface — resolve approval gates over MCP (2026-07-03)
Repo: EDDI (feat/hitl-framework)
Exposes the HITL approval operations over the MCP server so an external MCP client (agent / orchestrator / ops console) can list, read, resume/approve, and cancel paused conversations and group discussions — at full parity with the REST endpoints, for both the regular (1:1) and group surfaces. Closes the loop chat_managed/talk_to_agent already open: they return PAUSED_FOR_APPROVAL but, until now, the client had to drop to REST to resolve it. Full plan: planning/mcp-hitl-surface-plan.md.
Design — human authority preserved: MCP is a transport, not a new authority; no tool lets an agent approve its own gate. Authorization mirrors REST exactly via a new shared HitlAccessGuard (extracted from RestAgentEngine/RestGroupConversation, so who may decide lives in exactly one place): per-conversation owner / eddi-admin / eddi-approver, owner-scoped listings, fail-closed on a missing descriptor. Decisions are attributed server-side as mcp:<principal> (mirroring the existing system:timeout convention). A global kill-switch eddi.mcp.hitl.mutations.enabled (default true) can make MCP a read-only HITL surface without touching REST.
Method: brainstorm → adversarial design critique (four subagent workflows: verify-assumptions + security + architecture + completeness; 28/28 code assumptions verified, ~26 findings triaged — folded in owner-scoped group listings, structured error codes, the discoverability hint, and metrics; rejected agent-level config, dev-mode fail-closed mutations, and a by-intent resume variant, each with reasons) → TDD execution, one commit per task. All new tests are plain Mockito (locally runnable, no Quarkus boot).
Landed (each task = its own commit, all tests green locally):
McpToolUtils.errorJson(msg, code, details)— structured error JSON (errorCode∈NOT_FOUND | WRONG_STATE | FORBIDDEN | DISABLED | BAD_REQUEST), manual construction so it never throws on the error path.HitlAccessGuard— shared HITL ownership check + owner-scoped pending-approval listing (regular + group).RestAgentEngine/RestGroupConversationrefactored to delegate thehitlOperation=truepath (non-HITL paths untouched); existing REST HITL tests pass unchanged via a real guard wired with the same mocks.McpHitlTools— 9@Tools:list_pending_approvals,get_approval_status,resume_conversation,cancel_conversation,list_group_pending_approvals,list_all_group_pending_approvals,get_group_approval_status,approve_group_phase(optionaltaskApprovalsJSON for TASK granularity),cancel_group_discussion.@Blocking, JSON returns,eddi.mcp.hitl.*metrics (verdict-tagged).chat_managed/talk_to_agentPAUSED_FOR_APPROVALpayload now names"suggestNextTool": "resume_conversation"so an LLM client can chain the approval over MCP.
Pause-type-agnostic: because the tool-level HITL layer (see below) reuses the same AWAITING_HUMAN state + /resume + single-verdict HitlDecision, the regular tools resolve both RULE and TOOL_CALL pauses unchanged; get_approval_status reports pauseType and exposes the pending tool-call batch via detail=full. No SSE/streaming variant over MCP, no autonomous approver, no new realm role.
Docs: docs/hitl.md (new MCP Surface section), docs/mcp-server.md (new HITL Tools category).
🛠️ Tool-level HITL — foundational layer (Tasks 1–4 of 17) (2026-07-03)
Repo: EDDI (feat/hitl-framework)
Implementing tool-level HITL approval gating: pausing a conversation for human approval when the LLM invokes a gated tool (any source — built-in @Tool, MCP, A2A, httpcall, dynamic-agent, memory, recall), configured via allow/disallow pattern lists, co-existing with the behavior-rule PAUSE_CONVERSATION mechanism. Full plan: planning/hitl-tool-approval-plan.md. This closes the deferred "Tool-level HITL" limitation (docs/hitl.md Known Limitations).
Architecture — Durable Re-entry (ToolGate-DR): a batch gate in AgentOrchestrator.executeWithTools() intercepts gated calls before execution (fail-safe), serializes the exact in-flight langchain4j message list, persists it + pending-call metadata on ConversationMemorySnapshot, and aborts the LLM loop with an unchecked ToolApprovalRequiredException that LifecycleManager converts into the existing ConversationPauseException (new pauseOrigin=TOOL_CALL). Resume re-enters the same task index, replays the transcript, applies verdicts (write-ahead journal → at-most-once), and continues the loop. Chosen over a turn-completing "pending result" design after a 3-way design panel + 2 adversarial judges: durable replay uniquely preserves exactly the state the human is approving against (a multi-iteration turn that rebuilds from memory loses intermediate tool results).
Method: two adversarial subagent workflows (design-judge, then fact-verification against the codebase); 20 verification findings (1 blocker, 5 major) folded back into the plan before execution.
Landed (each task = its own commit, TDD, all tests green locally):
Task 1 — the architectural gate.
ChatTranscriptCodecwraps langchain4j 1.17.0ChatMessageSerializer/Deserializerwith a size cap + typed failure. Its test empirically proves the round-trip the whole design depends on (AiMessage+ToolExecutionRequest+ToolExecutionResultMessage+multimodal content survive serialize→deserialize). 6 tests. If this had failed, execution was gated to stop-and-escalate — it passed.Task 2 — pattern engine + gate.
ToolApprovalPatterns(ReDoS-safe*-only glob, source-prefix validation with typo suggestions),ToolApprovalGate(batch classify; precedence exempt-beats-require;source:namethen bare-name matching = fail-safe),ToolApprovalsConfigPOJO. 9 tests.Task 3 — config homes + validation.
toolApprovalsonAgentConfiguration.HitlConfig(agent-level default) andLlmConfiguration.Task(per-task full-replace override).HitlConfigValidation.validateToolApprovals(actionable 400s: bad pattern w/ index, both-lists conflict, duplicates, exempt-without-require, range checks, reservedINBOX, timeout/reason length); wired intoLlmStorecreate/update; agent-levelAUTO_APPROVE-inheritance WARN. 15 tests + all existing validation suites still green.Task 4 — memory model.
PendingToolCallBatch(+PendingToolCall, size caps);ConversationPauseException.PauseOrigin(RULE default / TOOL_CALL, backward-compatible 3-arg ctor); transienthitlPauseType/hitlPendingToolCalls/agentToolApprovalsConfig/hitlResumeDecisiononConversationMemory+IConversationMemory; persisted mirrors onConversationMemorySnapshot; both-directions copy inConversationMemoryUtilities. Round-trips through Jackson; legacy documents (null pauseType) treated as RULE. 4 tests + all existing HITL/resume/lifecycle suites green.
Design decisions locked for the remaining tasks (flag if wrong): (1) AUTO_APPROVE never applies to tool pauses implicitly — explicit per-gate opt-in only; (2) crash-inside-an-approved-tool yields honest EXECUTION_OUTCOME_UNKNOWN, never silent re-execution; (3) group-member tool pauses auto-reject gracefully (system:group); (4) ungated calls in a mixed batch execute before the human sees the pause (approver is shown which ran); (5) a multi-day pause resumes against pause-time prompt state.
Adversarial review of the foundation (5 dimensions → per-finding verification): 14 raw findings → 9 refuted, 5 survived, all minor. Fixes applied: (a) reject leading/trailing-colon patterns (:foo, mcp:) at save time — previously accepted but inert; (b) terminal cleanup (ConversationMemoryStore + PostgresConversationMemoryStore clearHitlBookmark) now also drops hitlPauseType/hitlPendingToolCalls so no stale tool-pause state lingers on ended/cancelled docs; (c) ConversationMemoryUtilitiesHitlTest extended to assert the two new fields round-trip both directions; (d) PendingToolCallBatchSnapshotTest doc clarified as a structural proxy (production uses BSON-backed JacksonCodec) with the real BSON round-trip routed to a CI Testcontainers IT (added to Task 14). No blockers, no majors.
Next: Task 5 (the gate hook + signal plumbing + pause commit in AgentOrchestrator/LifecycleManager/Conversation — the heaviest single task), then 6 (journal store), 7 (per-call verdict REST model), 8 (same-index re-entry), 9 (resumeToolLoop), 10 (timeout/no-progress), 11–13 (approver surfaces, Slack, delegated/group parity), 14 (crash recovery), 15 (lints), 16 (ITs), 17 (docs). Tasks 8→9 share a resumeToolLoop stub to keep each commit building.
🐰 CodeRabbit review triage — 21 fixes, adversarially verified (2026-07-03)
Repo: EDDI (feat/hitl-framework, PR #585)
CodeRabbit posted 23 actionable findings (16 inline + 7 outside-diff) plus observability nitpicks. Each was adversarially re-verified against HEAD (two parallel review passes) before any change — several overlapped fixes already made, some were stale/unreachable, and one CRITICAL-tagged item turned out already-mitigated-or-worse than described. Fixed the 21 that survived verification; skipped 2 as INVALID; deferred 1 as a scoped follow-up. No PR threads were replied to or resolved (standing instruction).
Scheduler / crash-recovery:
(HIGH) Lease-expired
CLAIMEDschedules were never reclaimable.findDueSchedulesreturns lease-expiredCLAIMEDrows, but bothtryClaimimpls only matchedPENDING/FAILED— so a crashed/wedged pod's claim was fetched every poll and never re-fired.tryClaimnow takes aleaseExpiryand steals aCLAIMEDrow withclaimedAt <= leaseExpiry(Mongo + Postgres, mirroringfindDueSchedules).(MEDIUM)
dispatchClaimedper-future timeout could stack to N×leaseTimeout. Now bounded by one shared batch deadline.(MEDIUM) Retention sweep ignored group conversations.
HitlCrashRecoveryObserver.sweepExpiredPendingApprovalsnow also cancels expired groupAWAITING_APPROVALpauses (newIGroupConversationServicedependency).(MEDIUM) Crash-recovery re-arm could keep a stale timeout across pause→resume→pause. The re-arm re-check now compares the pause bookmark (
pausedAt), not just the awaiting state, at all three sites.(HIGH)
PostgresScheduleStoremetadata (de)serialization failed open (returnednull, silently stripping the HITL contract). Now fails closed withResourceStoreException.(HIGH)
RestScheduleStore.requireAdminForHitlfailed open on any read error. Now onlyResourceNotFoundExceptionfalls through; other failures return 500 and stop the mutation.(doc) "exactly-once" was an overclaim — the design is at-least-once with idempotent HITL fire targets (the lease-steal above makes this explicit). Corrected
SchedulePollerService/IScheduleStorejavadoc +docs/hitl.md.
Engine / conversation:
(LOW)
endConversationonly disarmed the timeout whenAWAITING_HUMAN— a resume-in-flightIN_PROGRESSwindow could leave a stale timer. Now disarms unconditionally (idempotent).(nitpick)
ConversationMemoryStore.compareAndSetStatenow usesgetMatchedCount()(consistency withstoreConversationMemorySnapshotIfState; avoids a no-op-CAS false negative).
Group surface:
(MEDIUM) Synchronous member-pause exception stranded the member approval.
executeAgentTurnnow catchesConversationAwaitingApprovalExceptionand routes tohandleMemberPause(cancel + SKIPPED) instead ofhandleAgentFailure.(MEDIUM) Cancel window between the resume CAS and control-token registration. The
DiscussionControlTokenis now registered immediately after the CAS, so a concurrent cancel takes the signal path and stops before any phase runs.(LOW)
RestGroupConversationreflected raw ids inrequireGroupMembership/validateGroupConversationOwnershipNotFoundExceptionmessages, and (MEDIUM) the streaming approve endpoint echoed raw exception text over SSE. Both now curated (generic message + sanitized server-side log), matching the non-streaming hardening.(MEDIUM)
GroupConversationStore.findByStateaborted the whole batch on one record'sResourceStoreException. Now logged-and-skipped per record (mirrorslistByGroupId).
Slack / MCP tools:
(HIGH) Slack approval-notification idempotency was too coarse (keyed by
conversationId, marked before the post). Now keyed per-pause (hitlPausedAt) and cleared on failed delivery, so retries deliver and a second distinct pause is not suppressed.(HIGH) Slack HITL
resolveOwningIntegrationfell back to a by-approval-channel lookup for unbindable (bare) action values, reintroducing shared-channel cross-integration ambiguity. Removed — bare values now resolve to empty and are rejected (403).(HIGH) MCP
talk_to_agent/chat_with_agentreported a deliberateAWAITING_HUMANpause as BUSY (andchat_with_agentlost a freshly-created conversation id on skip). Both now return a structuredPAUSED_FOR_APPROVALand preserve the created id.(MEDIUM)
CreateSubAgentTooltreated a skipped initial turn as a real reply. Now mirrorsConverseWithAgentTool'sonSkippedhandling.(LOW)
RestSlackWebhookmalformed percent-encoding threw → 500 before signature check. Now caught → 400.(HIGH)
SecretRedactionFilterBearer rule only matched dotted JWTs; opaque tokens leaked. Now redacts opaque tokens too (possessive, ReDoS-safe).
Skipped/deferred (with reason):
INVALID: F9 (fractional-second read compat) — moot for unreleased/disposable schedule rows and would reintroduce the seconds-heuristic the epoch-millis fix removed; F14 (
ConverseWithAgentToolERROR-skip) —onSkippedprovably never receivesERROR.DEFERRED (follow-up task): owner-scoped group pending-approvals query (owner filter applied after the limit → possible starvation). The safe fix needs a DB-agnostic exact-match for
userId(the query layer treats string filters as regex on both backends;Pattern.quoteis Postgres-incompatible), so it warrants its own focused change rather than a rushed regex that could over-match.Observability nitpicks (Micrometer counters,
SafeHttpClientfor the fixed Slack host, managed executor for one-shot startup recovery,CREATE INDEX CONCURRENTLY, test-style suggestions) — intentionally out of scope for this correctness/security pass.
Every fix has a regression test; all touched unit suites are green locally (Testcontainers ITs run in CI).
🔬 Critical adversarial re-review — 7 findings fixed (2026-07-03)
Repo: EDDI (feat/hitl-framework, PR #585)
A final 6-reviewer / adversarial-verify pass over the HITL branch surfaced seven confirmed defects (1 CRITICAL, 1 HIGH, 4 MEDIUM, 1 LOW). All fixed with regression tests:
CRITICAL —
MongoScheduleStoretruncated everyInstantto epoch-SECONDS (1000× too small).toDocument()round-trips through the shared Jackson mapper, which (withwrite-dates-as-timestamps=true+JavaTimeModule, defaultWRITE_DATE_TIMESTAMPS_AS_NANOSECONDS) serializes anInstantas fractional seconds (1719964800.123). Deserialized into aDocumentthat's aDouble;convertInstantField'snum.longValue()then stored epoch seconds (~1.7e9), not millis (~1.7e12). SincefindDueSchedulescomparesnextFire <= nowMs(millis), every future-armed schedule looked immediately due — HITL approval timeouts (and Dream/maintenance) fired on the very next poll instead of after their configured duration, and it diverged fromPostgresScheduleStore(which stores millis). This is why the earlier ISO-string "defence in depth" (commitdc117cddc) never actually fixed it: the numericNumberbranch's millis assumption was the real bug, and no test asserted a futurenextFire. Fix:MongoScheduleStorenow writes every date field as an epoch-millisLongstraight from the getters (writeScheduleInstants/writeFireLogInstants) and reads them back viareadEpochMillis(Instant.ofEpochMilli), stripping them before the Jackson build — both directions are now independent of the mapper's date format (the seconds-basedconvertInstantFieldand its ISO branch are gone), mirroringPostgresScheduleStore.setNullableEpoch/instantFromEpoch. NewMongoScheduleStoreInstantRoundTripTestexercises the REAL serialization (the sibling test mocks it, which is why the bug hid) and asserts a futurenextFireis stored as millis and round-trips; the Mongo/Postgres store ITs gain an epoch-millis assertion; branch-coverage tests rewritten for the new helpers.HIGH —
SchedulePollerService.dispatchClaimedcould still pin the poll thread forever. The per-firefuture.get(leaseTimeout)bound was undermined by the try-with-resourcesExecutorService.close(), which awaits termination indefinitely;future.cancel(true)only unblocks tasks that honor interruption, but the real fire path can stall on a NON-interruptible synchronous DB socket read. Replaced try-with-resources with an explicitfinally { executor.shutdownNow(); }(noawaitTermination) so a wedged fire leaks a single cheap virtual thread instead of freezing all scheduling. New regression test uses a fire stub that swallows interruption and asserts the poll cycle still returns.MEDIUM — resume
onCompletecould clobber a terminal state (lost update / resurrection). The resume persist was an unconditional full-document store guarded only by a single up-frontisCancelled()check; a concurrentend/cancellanding between the check and the store overwroteENDED/EXECUTION_INTERRUPTEDwithREADY, resurrecting a terminated conversation that then accepted newsay()input. AddedIConversationMemoryStore.storeConversationMemorySnapshotIfState(snapshot, expectedState)— an atomic compare-and-store (Mongo:replaceOnewith a state predicate; Postgres:UPDATE … WHERE conversation_state = ?) — and the resumeonCompletenow persists only while it still ownsIN_PROGRESS, discarding its outcome (no schedule/notify) when a terminal writer won.MEDIUM — a transient snapshot-load failure during resume permanently dropped the finite-timeout schedule.
resumeConversationdeleted the HITL timeout schedule before loading the snapshot; a transient load failure then restoredAWAITING_HUMANwithout re-arming, so anAUTO_REJECT/AUTO_APPROVE/ABORTpolicy silently degraded to wait-forever until the next restart. The delete now runs only after the snapshot loads and the agent is confirmed deployed — a pre-execution failure leaves the original timer armed. (TheAWAITING_HUMAN→IN_PROGRESSCAS already prevents the timeout firing concurrently, so nothing races on the deferred window.)MEDIUM — Slack group HITL pause leaked a parked virtual thread per paused discussion.
SlackGroupDiscussionListener.completionLatchwas counted down only in the terminal callbacks;onHitlPause(a pause is terminal for this listener — resume flows through a different instance) did not, soSlackEventHandler.registerAgentThreadMappingsparked onawaitCompletion(300s)for every paused expanded-mode discussion and follow-up thread routing was unavailable for that window.onHitlPausenow counts the latch down in afinally.MEDIUM — dead-letter endpoints bypassed the HITL admin guard.
RestScheduleStore.dismissDeadLetter/retryDeadLetterwere the only schedule-by-id mutations lackingrequireAdminForHitl(), so a non-admin editor could disarm anABORT/AUTO_REJECTsafety timeout (dismissDeadLetteron a one-shot →markCompleted(id, null)→ disabled). Both now carry the guard; regression tests assert 403 for a non-admin on a HITL timeout schedule.LOW — group HITL REST error bodies reflected raw ids/exception text. The CodeQL XSS/info-exposure hardening applied to
RestAgentEnginewas not mirrored onRestGroupConversation;cancelDiscussion/approveGroupPhase/getGroupApprovalStatusechoede.getMessage()(which embeds the caller-suppliedgcIdfor the gone/404 case). Now curatedtext/plainbodies with the detail logged server-side (id sanitized), same HTTP status codes.
🔍 Copilot PR review — 5 findings (2026-07-03, post-push)
Repo: EDDI (feat/hitl-framework, PR #585)
Automated PR review (copilot-pull-request-reviewer) flagged 5 issues in the already-pushed HITL work; all fixed:
SchedulePollerService.dispatchClaimed: each claimed fire'sFuture#get()had no timeout — one stalled downstream call (e.g. a hung LLM call inside a fired schedule) could block the@Scheduledpoll loop forever, stopping this instance from claiming or firing any further schedule. Now bounded byleaseTimeout(the same window after which another instance may reclaim the schedule anyway) — a timeout cancels the future (best-effort interrupt) and logs, instead of hanging. New regression test asserts the poll cycle returns promptly under a stalled fire.SchedulePollerService.claimSchedule: after a successful CAS claim, the in-memoryScheduleConfigurationstill carried its pre-claimfireId(oftennull) — bothtryClaim()implementations (Mongo, Postgres) persist a freshfireId(scheduleId + "_" + now) but only return aboolean, so the caller never saw it.ScheduleFireExecutoruses this field for fire-log correlation and injects it into the agent context, so every claimed fire's correlation id was wrong. The poller now derives the identical value and sets it on the in-memory object after a successful claim. New regression test asserts the fired schedule carries a non-null, correctly-derivedfireId.ConversationPauseException: the exception message was built as"Conversation paused: " + pauseReason, producing the confusing"Conversation paused: null"in logs/clients when nopauseReasonwas configured (the common case, sincepauseReasonis optional). Now falls back to"human approval required"for the message only —getPauseReason()still returns the raw (possibly null) value for callers that need it.GroupConversationStore.listByGroupId: calledSAFE_ID.matcher(groupId)without a null guard — anullgroupId (a defensive REST layer, or an internal caller) threw NPE instead of returning an honest empty list. Fixed; regression test added. (The siblingfindByStatealready guarded this correctly.)UserConversationStore/PostgresUserConversationStore:readUserConversationByConversationId(added by the Slack HITL continuation-push work) queried byconversationIdwith no supporting index — a full collection/table scan on every reverse lookup (which happens on every HITL resume that needs to notify a Slack thread). Added a dedicated index on both backends (Mongo: ascending index; Postgres: expression index on the JSONB field) and a regression test asserting the Mongo index is created on construction.
🛡️ CI gate fixes — CodeQL (XSS/ReDoS) + Postgres Instant serialization (2026-07-03, final gate)
Repo: EDDI (feat/hitl-framework, PR #585)
CodeQL (high-severity) — both blockers fixed:
RestAgentEngineHITL error bodies reflected the rawconversationIdpath param (java/xss) and echoed internal exception messages to the client — nowtext/plainwith curated, non-reflecting messages; detail logged server-side (id sanitized). New-in-PR, HITL-introduced.SecretRedactionFilterReDoS (java/polynomial-redos, pre-existing onmain): the redaction patterns used ambiguous/unbounded backtracking quantifiers, quadratic on adversarial inputs (long${vault:repetitions). Made the quantifiers possessive (++/*+/{n,}+) — behavior-preserving (every quantified class is followed by a literal outside the class, so no backtrack is ever needed for a correct match) and linear-time.SecretRedactionFilterTest6/6 still green. Addressed here because it was the sole remaining CodeQL gate blocker.
Final full-suite verification surfaced a latent Postgres-only defect: the JSONB-backed resource storage serializes snapshots through the shared Jackson mapper (SerializationCustomizer.configureObjectMapper), which did not explicitly register JavaTimeModule — it relied on Quarkus auto-registration. The HITL bookmark carries an Instant (hitlPausedAt); a non-null Instant (i.e. an actual pause) would fail serialization ("Java 8 date/time type not supported") on the Postgres backend, so HITL pause persistence was one module-registration-order change away from breaking on Postgres (Mongo was unaffected — its BSON codec handles Instant; null Instants serialize fine, which is why it stayed latent). SerializationCustomizer now registers JavaTimeModule explicitly — the date format is deliberately left at the default numeric timestamps (an initial attempt to also switch to ISO-8601 strings was reverted: MongoScheduleStore normalizes date fields to epoch-millis for numeric range queries and expects numbers, so an ISO string silently broke findDueSchedules). As defence in depth, MongoScheduleStore.convertInstantField now also parses ISO-8601 strings → epoch-millis, so schedule storage is correct regardless of the mapper's date format. The Postgres store tests (and the MCP HITL test) now build their mapper via configureObjectMapper instead of a bare new ObjectMapper(), so they exercise the production serialization path — PostgresConversationMemoryStoreTest goes 20/22 → 22/22.
🛠️ HITL Round-2 Engine-Core Remediation (2026-07-03, WS-G)
Repo: EDDI (fix/hitl-r2-engine, branched from feat/hitl-framework) Scope: Confirmed findings from a second adversarial review of the HITL remediation, engine-core only (no integrations/slack/**, no tool/mcp bridges).
G1 (HIGH) — queued say resurrecting a terminated conversation:
ConversationService.processConversationStep's queued-turn skip set only covered AWAITING_HUMAN/IN_PROGRESS. Added ENDED and EXECUTION_INTERRUPTED so a say queued behind a running turn that terminates (endConversation / cancel) before it runs is routed throughonSkippedwith the persisted terminal state instead of executing the pipeline and persisting READY over the terminal state.G2 (HIGH) — say-path onComplete ignoring cooperative-cancel (group member-pause stranded approval):
runGuardedConversationStep's onComplete now checksmemory.isCancelled()(parity with the resume path). A concurrent cancel/end (e.g. grouphandleMemberPause → cancelConversationthat loses both state CAS races because the pause isn't persisted yet) makes the completing turn skip pause persistence/schedule/counter and CAS the running state to EXECUTION_INTERRUPTED — the approval is never stranded.G3 (HIGH) — approval gate bypass via schedule create/update:
RestScheduleStore.createSchedule/updateSchedulenow reject any request BODY whose metadata is ahitl_timeout(viaHitlSchedules.isHitlTimeout) for EVERYONE (even admin) with 400 — these schedules are minted internally only. Closes the forge/convert path that let an editor mint a timeout schedule the poller would fire to force-resume/abort a victim's approval unauthenticated.G4 (MEDIUM) — endConversation terminating a pause with no audit/actor: added
endConversation(String, String endedBy); the AWAITING_HUMAN branch now writes thehitl.approvalcancellation audit with the actor. Callers attribute: RestAgentEngine → principal, RestConversationStore delete/bulk-end paths →system:admin-end, 1-arg overload →system:end. AgentDeploymentManagement already SKIPs paused conversations (verified — no change).G5 (MEDIUM) — cancel/end never fired HitlResumeCompletedEvent: cancelConversation's pauseCancelled branch and endConversation's AWAITING_HUMAN branch now fire
HitlResumeCompletedEventwithverdict=null, the cancelling/ending actor, and the terminal snapshot (newfireHitlResumeCompletedTerminalhelper, async + fully isolated). The Slack observer renders these; the event's fields/signature are unchanged.G6 (MEDIUM) — retention sweep attributed to "unknown":
HitlCrashRecoveryObserver.sweepExpiredPendingApprovalsnow calls the 3-argcancelConversation(id, CANCEL_GRACEFUL, "system:retention").G7 (LOW) — restored pause re-armed at now+timeout:
ConversationService.scheduleHitlTimeoutandGroupConversationService.scheduleGroupHitlTimeoutnow anchorfireAttopausedAt + timeout(clamped tonow + 2mgrace, mirroring crash recovery) so restore-after-failed-resume re-arms at the original deadline instead of extending it.G8 (test) — vacuous Postgres zombie regression: added
loadReportsColumnStateOverForgedDivergentDocument(Testcontainers, CI-only) that forges TRUE document/column divergence via raw SQL and asserts bothloadConversationMemorySnapshotandloadActiveConversationMemorySnapshotreport the COLUMN state — deletingapplyStateColumnnow fails a test.Cheap extras:
RestAgentEngine.resumeConversationnull-guardsidentity.getPrincipal()(parity with cancel/end — no NPE for anonymous). The resumecatch (IllegalStateException)carve-out is narrowed via a privateAgentNotDeployedForResumeExceptionsentinel so ONLY the deliberate agent-not-deployed ISE re-throws without a double restore; any other ISE (e.g. from continueConversation) now restores the pause and maps to 500.
Verification: ./mvnw -q compile test-compile clean. Plain-Mockito tests for all touched classes run green locally (G1/G2 in ConversationServiceSayHitlTest, G4/G5 end in ConversationServiceTest, G5 cancel in ConversationServiceHitlTest, G3 in RestScheduleStoreTest, G6 in HitlCrashRecoveryObserverTest, plus reconciled RestAgentEngineTest/RestConversationStoreTest). The G8 Postgres test is Testcontainers → CI-only.
🔐 HITL Slack round-2 remediation — IDOR fix + approval-flow correctness (2026-07-03, WS-H r2)
Repo: EDDI (fix/hitl-r2-slack-impl, branched from feat/hitl-framework) Scope: Ten confirmed re-review findings (H1–H10) plus cancellation-rendering (H-consume) on the new Slack HITL surface + tool bridges. Ownership: integrations/slack/**, integrations/channels/ChannelTargetRouter.java, modules/llm/tools/ConverseWithAgentTool.java, engine/mcp/McpConversationTools.java, and their tests. No engine/internal/* or engine/api/* touched.
H1 (HIGH, security) — cross-integration IDOR on /interactive
/interactive previously verified the raw-body signature against the pooled set of all Slack signing secrets (incl. legacy per-agent ChannelConnector secrets) and only checked authz afterward — so a holder of ANY one Slack secret could forge an approval on another integration's paused conversation. Fix: the decision is now bound to the integration that owns it, carried explicitly in the button value. New SlackSignatureVerifier.verifyWithSecret(ts, body, sig, secret) verifies against exactly ONE secret. RestSlackWebhook.handleInteractive resolves the owning integration's secret from the payload first (SlackInteractivityHandler.resolveSigningSecretForDecision), then verifies against only that; an unbindable decision (legacy/unknown → no owning new-style integration) is rejected. /events keeps pooled verification (unchanged).
H2 (MEDIUM) — shared-approval-channel nondeterminism
The approval button value format changed from <subject> to <integrationName>|<subject> (<integrationName>|<conversationId> and <integrationName>|group:<gcId>). The handler resolves the owning integration by NAME (ChannelTargetRouter.getIntegrationByName), not by an arbitrary-first channel lookup — so authz + verification are deterministic even when integrations share one hitlApprovalChannel. New SlackHitlSupport.buildActionValue/parseActionValue (+ ActionValue record). Legacy bare values (no name) parse with a null integration and are rejected up-front (acceptable per spec).
H3 (MEDIUM) — group double-click idempotency
SlackInteractivityHandler.resolveGroup caught only IllegalStateException; resumeDiscussion signals a non-paused group with the CHECKED GroupDiscussionException, a race with ResourceModifiedException, and a deleted group with the unchecked GroupConversationGoneException — all fell into the generic catch (warn-spam + live buttons). Now all four route to finalizeAlreadyResolved.
H4/H7 — dropped-turn (onSkipped) discrimination
SlackEventHandler.sendAndWait now uses a full ConversationResponseHandler overriding onSkipped, mapping a skip to sentinel snapshots: AWAITING_HUMAN → STILL_AWAITING notice (no second approval card, H4); else → new CONVERSATION_NOT_ACTIVE_NOTICE (H7). ConverseWithAgentTool and McpConversationTools (talk_to_agent/chat_with_agent/chat_managed) detect onSkipped and return busy/not-active (chat_managed: AWAITING_HUMAN-skip → PAUSED_FOR_APPROVAL) instead of replaying the previous turn's output (mirrors RestAgentEngine's 409 discrimination).
H5 + H-consume — resume ERROR / cancellation rendering (defensive)
SlackHitlResumeObserver.decisionSummary now takes the snapshot: an approved resume that ended in ERROR renders a failure ("continuation failed…") not "continuing" (H5), and a null-verdict event (cancel/timeout-abort/end, terminal EXECUTION_INTERRUPTED/ENDED) renders "⛔ cancelled or expired" — the previously-dead branch is now live and correct (H-consume). Implemented defensively: works whether or not the engine's parallel change to fire HitlResumeCompletedEvent with verdict==null on cancel/end has landed. (Approval-card button-removal on cancel is not done — the approval card's message ts is not resolvable from the conversationId in the current data model; the thread message, which IS resolvable, is delivered.)
H6 — "Bearer null" guard
notifyApprovers now skips the call and logs an explicit "no bot token — HITL approval notification NOT delivered for " error when neither the resolved integration token nor the approval-channel lookup yields a non-blank token (mirrors postMessage's guard).
H8 — init-turn (CONVERSATION_START) pause never notified approvers
An init-turn pause happens inside getOrCreateConversation → startConversation; the first user say then throws ConversationAwaitingApprovalException and no approval card was ever posted. The exception branch now calls notifyApprovers, made idempotent via a slack-hitl-approval-notified cache (putIfAbsent) so re-message-while-paused never posts a second card.
H9 — follow-up conversations get the resume continuation push
SlackHitlResumeObserver now recognizes the channel:followup:<channelId>:<parentTs> intent shape (in the prefix guard and parseIntent) in addition to channel:slack:..., so agent-thread follow-ups receive the verdict/continuation in their thread.
H10 — pause card read the bookmark before it was persisted
The say callback completes before ConversationService persists the HITL bookmark, so getConversationMemorySnapshot re-reads returned the previous turn (null pause reason/timeout). New loadHitlBookmark retries the read (5×100ms) until state==AWAITING_HUMAN and is loaded ONCE per pause, shared by the in-thread notice and the approval card.
New contracts
SlackSignatureVerifier.verifyWithSecret(String ts, String body, String sig, String secret)— single-secret (integration-bound) verification for/interactive.SlackInteractivityHandler.resolveSigningSecretForDecision(String payloadJson)— resolves the owning integration's signing secret from the button value; null → endpoint rejects.ChannelTargetRouter.getIntegrationByName(String channelType, String name)— deterministic by-name lookup.Approval button
valueformat:<integrationName>|<conversationId>/<integrationName>|group:<gcId>(was bare<conversationId>/group:<gcId>).
Tests
Plain JUnit/Mockito (compile + test-compile pass; touched suites run green locally): SlackSignatureVerifierTest (+verifyWithSecret), SlackHitlSupportTest (+buildActionValue/parseActionValue), SlackInteractivityHandlerTest (rewritten for value-binding + H1 cross-integration + H3 group double-click + resolveSigningSecretForDecision), SlackHitlResumeObserverTest (3-arg decisionSummary, ERROR/cancellation/followup delivery), RestSlackWebhookTest (new interactivity flow), SlackGroupDiscussionListenerTest (integration-bound group value), ConverseWithAgentToolHitlTest/McpConversationToolsHitlTest (H7 skip), ChannelTargetRouterRefreshTest (getIntegrationByName).
🧪 HITL Coverage Closure + Schedule-Contract Consolidation (2026-07-03, WS-F + merge)
Repo: EDDI (feat/hitl-framework, PR #585)
Coverage (WS-F, findings 10/22/23/24/41/43): 13 test files, +1469 lines, tests only. Queued-say guard + say fast-fail (finding 10 — previously zero coverage), zombie-pause discard guard, the entire finite-timeout leg (schedule creation/metadata routing/fire-log parity/error isolation/delete-on-resume+cancel, initial say-path arming), regular-surface endpoint authz incl. fail-closed missing-descriptor, resume robustness against REAL workflow lists (config drift → ERROR, multi-workflow continuation order), HitlConfigValidation wiring at AgentStore/AgentGroupStore CRUD + the import seam, storage regressions (Postgres zombie: post-CAS load must report the column state; jsonb_set convergence; owner-filtered summaries; storeIfFieldEquals deleted-404 vs mismatch-409 on both backends; anchored group filters + SAFE_ID rejection), case-insensitive verdict round-trip, REJECTED-path ConversationOutput visibility + ACTIONS strip. Testcontainers classes (PostgresConversationMemoryStoreTest, MongoConversationMemoryStoreTest) execute in CI; everything else ran green locally (202 tests). Known residual gaps documented in the test agent's report: no wall-clock end-to-end timeout IT (every seam unit-covered), full ZIP pipeline (validation seam covered).
Consolidation: new ai.labs.eddi.engine.hitl.HitlSchedules — single source of truth for the HITL timeout-schedule contract (names hitl-timeout-*/hitl-timeout-group-*; metadata keys hitlType/policy/surface/conversationId; isHitlTimeout predicate) — adopted by ConversationService, GroupConversationService, ScheduleFireExecutor, HitlTimeoutHandler, HitlCrashRecoveryObserver, RestScheduleStore. Closes the "HITL lifecycle glued by magic strings across five classes" review finding.
🔌 HITL — Slack integration + nested-consumer bridges (2026-07-03, WS-E)
Repo: EDDI (feat/hitl-ws-e-slack, branched from feat/hitl-framework) Scope: Human-in-the-loop support in the Slack channel adapter, plus finding 25 (nested/managed/delegated conversation consumers stranded by a pause).
Slack HITL surface
In-thread pause notice (
SlackEventHandler): when a say returns anAWAITING_HUMANsnapshot, the output-so-far is posted followed by a pause notice ("⏸️ This conversation is awaiting human approval" + pause reason from the HITL bookmark). AConversationAwaitingApprovalExceptionon subsequent messages posts "Still awaiting approval — a reviewer must decide…" instead of the generic error. Follow-up (group-thread) replies get the same handling.Approver notification + Approve/Reject buttons (config-driven, fail-closed): two new optional
ChannelIntegrationConfiguration.platformConfigkeys —hitlApprovalChannel(Slack channel id for approval notifications) andhitlApproverUserIds(comma-separated Slack user ids allowed to decide). On pause, an interactive Block Kit message is posted to the approval channel (conversationId, agent, reason, timeout policy/deadline). Buttons are rendered only whenhitlApproverUserIdsis set (otherwise notification-only). Data minimization: only the pause reason is included, never the user's message.Interactivity endpoint (
RestSlackWebhook): newPOST /integrations/slack/interactive(form-urlencoded,payloadparam). Verifies the Slack signature over the RAW body with the existingSlackSignatureVerifier+ router signing secrets, acks 200 within 3s, processes async on a virtual thread (SlackInteractivityHandler). Handlesblock_actions(hitl_approve/hitl_reject). AUTHZ is fail-closed against the owning integration's approver list;decidedByis always derived server-side (slack:<userId>). On success itchat.updates the message ("✅ Approved by …" / "⛔ Rejected"), removing buttons; an already-decided/timed-out click resolves to theIllegalStateExceptionpath and updates the message without error-spam (idempotent double-click).Continuation push after resume (CDI event):
ConversationServicefires a new async CDI eventai.labs.eddi.engine.events.HitlResumeCompletedEventwhen a resume settles to a non-paused state (inresumeFinished.onComplete, afterstoreConversationMemory,fireAsyncso observers never block the engine; failures isolated).SlackHitlResumeObserverobserves it, resolves the conversation's Slack routing via the new reverse-lookupIUserConversationStore.readUserConversationByConversationId(implemented on both Mongo + Postgres for parity), and posts the verdict + continuation output to the originating channel/thread. Timeout (system:timeout) and cancellation outcomes flow through the same event.Group discussions (
SlackGroupDiscussionListener): implemented the HITL listener callbacks —onHitlPause(thread notice + approval-channel buttons whose action value carriesgroup:<groupConversationId>, routed toresumeDiscussion),onHitlResume(verdict),onMemberPauseSkipped, andonCancelled.Shared helpers:
SlackHitlSupport(config keys, Block Kit builders, approver authz, and the Slack-friendly response-text extraction refactored out ofSlackEventHandler);SlackWebApiClientgainedpostBlocksMessage+updateMessage(chat.update), reusing a shared send/parse helper with the existing retry/backoff classification.
Finding 25 — nested/managed/delegated consumers
ConverseWithAgentTool,McpConversationTools#chat_managed, andCreateSubAgentToolnow detect the delegated conversation pausing (AWAITING_HUMAN snapshot orConversationAwaitingApprovalException) and return a structured, actionablePAUSED_FOR_APPROVALresult (with the conversationId and the/resumeinstruction) instead of "[no response]" or a 60s hang. The nested pause is not auto-cancelled (a delegated approval may be intended) and managed mappings are preserved so re-invoking after approval continues the same conversation.
Tests
Plain JUnit/Mockito (no Quarkus boot): SlackHitlSupportTest, SlackInteractivityHandlerTest, SlackHitlResumeObserverTest, ConverseWithAgentToolHitlTest, McpConversationToolsHitlTest, plus additions to SlackEventHandlerTest, SlackGroupDiscussionListenerTest, RestSlackWebhookTest. Covers signature rejection on /interactive, unauthorized user cannot decide, authorized resume with decidedBy=slack:…, double-click no error-spam, buttons omitted without approvers, observer posts to the right channel/thread and ignores non-Slack conversations, and the bridges' PAUSED_FOR_APPROVAL result. docs/hitl.md needs a new "Slack" section (docs phase — not edited here).
🔧 HITL Production-Readiness Remediation — storage parity + say-path contract (2026-07-03, session 5)
Repo: EDDI (feat/hitl-framework, PR #585) Trigger: A 92-agent adversarial review of the branch confirmed 46 findings (1 CRITICAL, 7 HIGH, 18 MEDIUM, 20 LOW). This entry covers the storage-layer and regular-surface batches; parallel batches (schedule security/sweeps, group surface) land as separate commits from their own branches.
CRITICAL — PostgreSQL conversation-state duality (the Postgres zombie)
State was persisted twice on Postgres: in the indexed conversation_state column (updated by CAS/cancel) AND inside the JSONB snapshot (read by loads). A cancelled or ABORT-timed-out pause kept reporting AWAITING_HUMAN from the stale document — wedging say(), showing phantom pauses in approval-status, and letting the next user message resurrect the terminated approval as a zombie (full-document store flips the column back, re-arms a timeout; a later approve fails into ERROR). Fixes, defense in depth:
Column wins on load —
loadConversationMemorySnapshot/loadActiveConversationMemorySnapshotoverwrite the deserialized state with theconversation_statecolumn (applyStateColumn).Writers converge the document —
setConversationStateandcompareAndSetStatealsojsonb_setthe JSONBconversationStatein the same statement.Say-path zombie guard —
runGuardedConversationStep.onCompletediscards a turn result whoseAWAITING_HUMANstate was already present at submit time (a stale pause the turn did not produce is never re-persisted or re-armed).
HIGH — say() into a paused conversation: honest 409 instead of a 60s hang
say()/sayStreaming() now fast-fail with a new ConversationAwaitingApprovalException → REST 409 with an actionable body (matches the docs' "say() is rejected" promise, mirrors ENDED→410). The queued-say race backstop now completes the response via a new ConversationResponseHandler.onSkipped(snapshot) (default: delegates to onComplete) instead of dropping the turn into the 408 watchdog — and the processingConversationReferences gauge entry is removed on every exit path (was a permanent leak per dropped request). RestAgentEngine maps skipped turns to 409 ("awaiting approval" vs "busy — retry"). Callback consumers (group, Slack, MCP) are unaffected: the default onSkipped delivers the snapshot whose state they already inspect.
Storage-layer fixes (DB-agnostic parity)
Postgres regex 500s
GroupConversationStore built filters with Pattern.quote (\Q…\E) — valid in Mongo's PCRE, rejected by PostgreSQL's regex engine → group listing + pending-approvals 500'd on PG. Replaced with charset-validated plain anchoring (^id$; ids are hex/UUID, no metacharacters). Non-id input → honest empty result.
Projected pending summaries
Postgres now runs ONE projected query (JSONB field extraction, hitlPausedAt round-tripped through the same Jackson mapper) instead of 1+limit full-document deserializations; Mongo now runs ONE projected query instead of N+1 point-reads.
Owner-scoped inbox
findPendingApprovalSummaries(ownerUserId, limit) implemented on both backends — the owner filter is INSIDE the query, so the limit applies after the restriction (a non-admin's inbox can no longer be starved by other users' backlog). New Mongo compound index (conversationState, userId). RestAgentEngine uses it for non-admin/non-approver callers.
CAS 404-vs-409
storeIfFieldEquals (both backends) now distinguishes "document deleted" (ResourceNotFoundException) from "field mismatch" (ResourceModifiedException) via an existence check on zero-match. GroupConversationStore.updateIfState surfaces deletion as unchecked GroupConversationGoneException (kept unchecked so existing CAS call sites compile; surfaces map it to 404). The IResourceStorage default no longer silently degrades the CAS to an unconditional store — it throws UnsupportedOperationException.
Truncation visibility
findByState WARNs when it hits its limit (pending listings / crash recovery must never truncate silently).
Regular-surface fixes
End-vs-resume race: the resume pre-execution guard now also aborts on persisted
ENDED(previously onlyEXECUTION_INTERRUPTED) — an accepted resume can no longer resurrect an ended conversation.Cooperative-cancel integrity: all
inFlightConversations.remove(key)calls are now value-conditionalremove(key, memory)— a finishing leg can no longer evict a newer execution's registration.Cancel attribution (EU AI Act):
cancelConversation(id, mode, cancelledBy)threads the actor into thehitl.approvalaudit entry (decidedBy+automated); REST passes the principal,HitlTimeoutHandlerpassessystem:timeout. Old 2-arg signature delegates (unknown).Configurable pause reason: new optional
hitlConfig.pauseReason(agent-level, ≤500 chars, validated at save/import) flows into the bookmark → pending-approvals/approval-status answer "what am I approving?". Falls back to the generic constant.approval-status payload now includes
approvalTimeoutso UIs can render the auto-decision deadline.Fail-closed HITL authz: resume/cancel/approval-status on a conversation whose descriptor is missing now require admin/approver (was: ownership check silently skipped).
REJECTED-path visibility: the rejection message is now written to
ConversationOutput["output"](UIs/log generator actually render it) and the stalePAUSE_CONVERSATIONaction is stripped on the REJECTED path (as on APPROVED).Verdict parsing is case-insensitive on all surfaces (
HitlVerdict.fromString@JsonCreator); note-length cap single-sourced asHitlDecision.MAX_NOTE_LENGTH.IConversation.resume(decision)— deadcontextsparameter removed (CodeQL).Misleading "transient — not serialized" comment on the HITL bookmark fields corrected (they ARE persisted via the snapshot).
Deliberately deferred
Bookmark value-object refactor (6 flat fields → 1 object): cosmetic, touches the persisted snapshot shape late in the branch — deferred.
RestGroupConversation's duplicated note-length constant: consolidation phase (group-surface files owned by a parallel batch).
🔒 HITL Schedule Security, Sweeps & Retention — WS-C (2026-07-03)
Repo: EDDI (fix/hitl-ws-c-schedules, branched from feat/hitl-framework) Trigger: Confirmed code-review findings (5, 7, 17, 26, 32, 44) on schedule security, idle/undeploy sweeps, poller scalability, and pause retention.
#5 (HIGH, security)
Editor could bypass the HITL approval gate via the schedule REST surface. RestScheduleStore now: (a) fireNow refuses any schedule with metadata.hitlType=="hitl_timeout" — for everyone including admins — returning 409 Conflict directing to /agents/{id}/resume or /cancel (manual firing side-steps the /resume owner/admin/approver audit gate); (b) updateSchedule/deleteSchedule/enableSchedule/disableSchedule on a HITL schedule require eddi-admin, else 403 Forbidden (detected via the STORED schedule so a doctored request body can't hide the marker); (c) readAllSchedules redacts HITL schedules for non-admins so they can't be enumerated. Internal firing via SchedulePollerService is unaffected (it bypasses REST). Parity enabler: PostgresScheduleStore never persisted metadata (Mongo did via full-doc serialization) — added a metadata JSONB column (+ idempotent ADD COLUMN IF NOT EXISTS upgrade, IJsonSerialization round-trip). Without this the HITL timeout fast-path never fired on Postgres AND the security guard couldn't recognize HITL schedules there.
#7 (HIGH)
AgentDeploymentManagement.endOldConversationsWithOldAgents now skips AWAITING_HUMAN conversations (mirrors the deliberate getActiveConversationCount exclusion) instead of force-ENDing them with a raw non-CAS write (which left armed schedules, stale bookmarks, and no audit). Logs at INFO how many paused conversations were spared. The pre-existing agent-document-age heuristic for non-paused conversations is left unchanged with an explanatory comment.
#17 (MEDIUM, scalability)
Poll batch size is now configurable (eddi.schedule.poll-batch-size, default 100) in both stores; SchedulePollerService claims all due schedules on the poll thread (CAS before dispatch) then fires the claimed ones concurrently on virtual threads (newVirtualThreadPerTaskExecutor) with per-fire error isolation — a mass HITL-timeout burst no longer serializes behind one thread and starves Dream/maintenance schedules. Exactly-once cluster semantics preserved.
#26 (MEDIUM)
RestConversationStore.endActiveConversations routes AWAITING_HUMAN conversations through the HITL-aware IConversationService.endConversation (schedule disarm + bookmark clear + audit + in-flight-resume signal) instead of a raw ENDED write; non-paused conversations keep the raw path.
#44 (LOW)
RestConversationStore.deleteConversationLog(deletePermanently=true) now calls endConversation for an AWAITING_HUMAN conversation before deleting the document — disarming the leaked one-shot schedule, clearing the bookmark, auditing, and invalidating the cached state — via the existing public service method.
#32 (LOW)
New optional pause-retention sweep in HitlCrashRecoveryObserver (@Scheduled): eddi.hitl.pending.max-age (ISO-8601, default empty=OFF) auto-cancels pauses older than the threshold via cancelConversation (audited, schedule-disarmed); eddi.hitl.pending.sweep-interval (default 6h). Reuses the existing poller/scheduling infra — no new scheduler.
CodeQL
HitlCrashRecoveryObserver.onStartup(@Observes StartupEvent event) — silenced the "unused parameter" alert with @SuppressWarnings("unused") + comment; the param is the required CDI observer trigger.
REST status codes chosen: manual fire of a HITL schedule → 409 Conflict (operation not permitted for anyone; directs to /resume|/cancel). Non-admin mutate/disable/delete of a HITL schedule → 403 Forbidden.
New config properties: eddi.schedule.poll-batch-size (default 100), eddi.hitl.pending.max-age (default empty/OFF), eddi.hitl.pending.sweep-interval (default 6h) — documented in application.properties. docs/hitl.md (owned by a later phase) should document the retention property for operators.
Tests (pure JUnit/Mockito): RestScheduleStore HITL guards (editor/admin fire+mutate+redact); AgentDeploymentManagement paused-skip; SchedulePollerService concurrent dispatch + per-fire error isolation + claim-before-dispatch; RestConversationStore paused end/delete routing; HitlCrashRecoveryObserver retention sweep (OFF-by-default, cancels-expired, non-positive=OFF); PostgresScheduleStore metadata round-trip.
🔧 HITL PR Review Response — Copilot + CodeRabbit (2026-07-02, session 4)
Repo: EDDI (feat/hitl-framework, PR #585) Trigger: Automated review on the open PR (GitHub Copilot + CodeRabbit) surfaced ~24 findings. Each was independently, adversarially re-verified against the actual code (not the bot's paraphrase); the confirmed ones are fixed here, deliberate skips are recorded with rationale.
Cross-group leak (MAJOR)
GroupConversationStore.findByState(state, groupId, limit) passed groupId as a raw filter value; MongoResourceStorage.findResources turns String filters into unanchored regexes, so a group id that is a substring of another matched across groups. Now anchored + Pattern.quoted (^\Q…\E$) in both findByState and the pre-existing listByGroupId.
Stuck resume (MAJOR)
resumeConversation's critical section (CAS → submitInOrder) only caught ServiceException/InstantiationException/IllegalAccessException; an unchecked exception from continueConversation escaped, leaving the conversation stuck IN_PROGRESS with a leaked inFlightConversations entry. The catch now also covers RuntimeException (restores the pause + drops the registry entry), while the deliberate agent-not-deployed IllegalStateException is re-thrown as-is so it still maps to 409.
Props on failed resume (MAJOR)
Conversation.resume()'s finally ran postConversationLifecycleTasks() whenever the state was not AWAITING_HUMAN — including ERROR, so a failed resume persisted long-term properties, unlike the say path. Now also skipped on ERROR.
Resume bookmark index (MAJOR)
LifecycleManager selective execution (rerun) ran a suffix sublist but stored the sublist-relative loop index as the "absolute" pause bookmark. Added an indexOffset threaded only into the pause-index computation (component-cache/telemetry indices unchanged), so a pause during selective execution records a true absolute index. Also added bounds validation to executeLifecycleFromIndex (negative → LifecycleException; strictly-past-end from a redeployed workflow → warn + skip; exactly size() remains valid).
End-vs-resume race (MAJOR)
endConversation wrote ENDED without signalling inFlightConversations, so a resume past the CAS could persist its snapshot back over the terminal state. It now sets the cooperative-cancel flag on any in-flight memory (mirrors cancelConversation), so the resume's onComplete skips persistence and ENDED wins.
Timeout fire-log parity
ScheduleFireExecutor's HITL fast-path returned before logFire() and had no exception isolation. It now records a ScheduleFireLog (with the conversationId + FAILED status on error) and wraps handleTimeout in try/catch, matching the normal path.
Consistency / hardening
OwnershipValidator.isAdmin null-guards identity (matches isApprover/isOwner); AgentGroupConfiguration.HitlConfig setters null-coalesce to their defaults (a JSON null can no longer wipe timeoutPolicy/granularity/onTaskRejection, mirroring setLifecyclePolicy); DiscussionControlToken wave loop re-checks isCancelled() after setActiveFuture so a CANCEL_IMMEDIATE landing mid-registration still cancels the new future; MongoScheduleStore uses a NAME constant; ConversationMemory drops redundant java.time.Instant FQNs; SharedTaskList.TaskStatus Javadoc corrected (AWAITING_APPROVAL is fully wired, not a placeholder).
Tests
New resetFromAnyToAssigned coverage (7 cases: valid transitions, ASSIGNED no-op, rejected states); new resume test for an unexpected RuntimeException from continueConversation (asserts pause restored + ResourceStoreException + never submitted); turnCounterSeedsFromPausedTurnCount de-flaked by capturing pausedTurnCount at the synchronous resume CAS instead of racing the async completion; bogus timeout-policy strings in memory round-trip tests replaced with real enum names.
Deliberately skipped
listPendingApprovals "max 1000 not enforced" — FALSE POSITIVE, the service already clamps Math.min(limit, 1000). Crash-recovery pagination (10k scan cap) — 10k concurrent pauses is unrealistic and the code already warns. Pending-summary N+1 (Mongo/Postgres) and findByState bulk-read — perf-only on bounded/one-time paths, a pre-existing IResourceStorage API limitation; deferred. HitlPauseType vs HitlGranularity unification — intentional pause-record vs config separation. Duplicated cancel-check refactor — declined to churn just-hardened control flow.
🔧 HITL Merge-Readiness Hardening (2026-07-02, session 2)
Repo: EDDI (feat/hitl-framework) Trigger: Second full-branch review after the phase 1–7 fixes; verified 22 tracked findings (10 FIXED, 7 PARTIAL, 5 UNFIXED) plus new regressions in the fix commits themselves. This session closes the remainder before PR.
Red tests
HitlTimeoutHandlerTest — inject SimpleMeterRegistry (new metrics field NPE'd all 7 tests); RestGroupConversationHitlTest.approveDenied — body validation runs before authz, so the test now sends a valid decision (plus a new test pinning the 400-before-authz ordering). The suite was red at HEAD; these commits restore green as the baseline.
| Crash recovery (C-A/F4/F6) | HitlCrashRecoveryObserver reworked from destroyer to repairer. It no longer transitions old paused conversations to ERROR/FAILED (the previous behavior destroyed legitimately-paused WAIT_INDEFINITELY conversations — the default policy — on every restart). New behavior: finite-policy pauses get their one-shot timeout schedule idempotently re-armed at the original due time (applies the configured policy through the normal handler, even after a crash); WAIT_INDEFINITELY pauses are never touched; regular conversations stuck IN_PROGRESS with an intact HITL bookmark (pod died mid-resume) are CAS-restored to AWAITING_HUMAN; IN_PROGRESS without bookmark → EXECUTION_INTERRUPTED. Stale-threshold config dropped (no longer meaningful); added eddi.hitl.crash-recovery.recover-in-progress (default true, multi-pod caveat documented). Prereq: the regular pause commit now populates hitlTimeoutPolicy/hitlApprovalTimeout bookmark fields (F6) — approval-status/pending-approvals finally report the effective policy on both surfaces. | | Regular cancel (F2) | cancelConversation is no longer a silent no-op: in-flight registry (inFlightConversations) lets cancel set the cooperative setCancelled flag the LifecycleManager checks at task boundaries (mode param now read; IMMEDIATE degrades to graceful, documented); returns typed CancelOutcome — REST maps CANCELLED→200, NOT_FOUND→404, NOTHING_TO_CANCEL→409 (was unconditional 200). | | Resume robustness (F4/F7) | Resume now: pre-checks existence → 404 (was 409 with misleading message); reports the current state in the 409 body; restores the pause (CAS back to AWAITING_HUMAN + re-arms timeout) when the agent is undeployed or a service error occurs, instead of destroying the approval with ERROR; wraps execution with the same watchdog as the say path (hung LLM → EXECUTION_INTERRUPTED, never stuck IN_PROGRESS). | | Undo/redo gate (F5/C-C) | Gate now reads the DB-loaded state (was per-pod cache — silently bypassed after restart/cross-pod) and returns false → 409 CONFLICT (was 500). |
| Group cancel (F9) | No-token branch now uses the updateIfState state-CAS (was plain read-modify-write racing approve/resume); CancelledEvent is finally emitted via new onCancelled listener method and the SSE stream closes on cancel (was: /discuss/stream clients hung forever after a cancel). | | Group approvals (F13/C-D) | taskApprovals validated up front (unknown taskId / task not awaiting → 400, no partial in-memory mutation, schedule untouched); RETRY rejection now passes the reviewer's note into the re-queued task and buildTaskExecutionInput surfaces it as feedback (was: blind retry loop); resetFromAnyToAssigned enforces its documented status contract. | | Group robustness (F11/5f) | Stranded IN_PROGRESS tasks are reset to ASSIGNED in ALL wave-abort branches (was: timeout only — cancellation/error still stranded tasks forever); config-drift guard now also fails when phases were REMOVED (bookmarked index out of range no longer silently skips the check); nested-group sub-pauses are cancelled instead of stranded with an armed schedule; commitPause reuses the in-scope config (no duplicate store read per pause). | | Group pending list (C-B) | GET /groups/{groupId}/conversations/pending-approvals now scopes to the path's group and applies the same ownership filter as the regular listing (admin/approver see the group's items, others only their own, anonymous nothing) — was a global unfiltered dump of all users' paused conversations incl. transcripts. | | Approver role | New eddi-approver role: OwnershipValidator.isApprover, added to @RolesAllowed on all HITL endpoints (approver-only accounts were blocked at RBAC), and approvers now see pending listings on both surfaces (they could approve but not list). SSE error events now serialize through the JSON serializer (no string-concatenation injection). |
| Audit (F15) | hitl.approval AuditEntry submitted on BOTH surfaces for every decision (verdict, decidedBy, automated flag, note) — covers human and system:timeout decisions; GroupConversationService now injects AuditLedgerService. Combined with the earlier resume audit-collector wiring, the EU AI Act human-oversight trail is complete. | | Quota (F16, plan §10a) | getActiveConversationCount excludes AWAITING_HUMAN on Mongo AND Postgres — paused conversations no longer block undeploy/old-version GC forever. Undeployed-while-paused conversations keep their pause; resume reports 409 and restores it. | | Pending listing scale (F17) | New findPendingApprovalSummaries(limit) store method: Mongo uses POJO-codec projection (never deserializes step data), Postgres a LIMIT-bounded loop; REST takes ?limit (default 200, max 1000); PendingApprovalSummary gains userId so the ownership filter no longer does N+1 descriptor reads. | | Config safety (F20/C-E/C-F) | Duplicate engine.lifecycle.model.HitlTimeoutPolicy enum deleted — AgentGroupConfiguration.HitlTimeoutPolicy is the single source (no more constants-drift between schedule metadata writer and parser). New HitlConfigValidation enforced in AgentStore/AgentGroupStore create+update: finite policy requires a valid positive ISO-8601 approvalTimeout, actionable 400 messages via the existing IllegalArgumentExceptionMapper. | | Input bounds | HITL decision note capped at 4 KB on both surfaces (400 on overflow). Dead counterHitlTimeout field removed (handler owns the timeout metric). |
| Test hardening (F22) | R2 cancel tests: else assertNotNull escape hatches removed — reverting the R2 fix now fails them. NEW R1 test: cancel racing a requiresApproval phase asserts CANCELLED + commitPause never ran. B1 test rebuilt: snapshot reflects post-CAS reality (IN_PROGRESS) and the memory state is captured at continueConversation time — deleting the B1 fix now fails it. New behavioral tests: stripPauseAction (stale action removed, others preserved), decision visibility (hitlDecision output + hitlVerdict property), Invariant 9 asserted via step-property survival (pause) vs purge (normal turn). | | Integration test (F22) | New HitlPauseResumeIT — full end-to-end with zero mocked seams: real behavior rule emits PAUSE_CONVERSATION → real Mongo persistence → REST resume completes the remaining pipeline tasks (the original BLOCKER's fail-on-revert), plus REJECTED path, cancel path, 400-does-not-consume-pause, 404 unknown id, 409 not-paused, and undo-blocked-while-paused. New tests/hitl/*.json agent fixtures. Runs in CI via mvnw verify -DskipITs=false (ci.yml:179); local Docker daemon was unavailable during this session, so first execution happens in CI — same as any IT change. | | Docs (F21) | New docs/hitl.md: both surfaces, config reference incl. onTaskRejection, real REST paths, template access (hitlDecision / {properties.hitlVerdict}), timeout policies, approver role, crash recovery config, operations notes (metrics, undeploy semantics, cancel matrix), known v1 limitations, requiresApproval upgrade note. AGENTS.md reserved-action list updated with PAUSE_CONVERSATION. planning/hitl-framework-plan.md committed. |
| Final sweep | Full 9,761-test suite executed: the only genuine branch defect was GroupConversationTest.groupConversationStates still asserting 6 enum values after the branch added CANCELLED (fixed: 7 + CANCELLED assertion). All other local failures are environmental (Docker unavailable for Testcontainers classes, sandbox-blocked loopback sockets for HTTP-server-based tool tests) — these classes are untouched by this branch and run in CI. Mongo findPendingApprovalSummaries reworked to bounded projected point-reads with explicit id mapping (the bulk POJO-codec projection could not be guaranteed to populate _id→conversationId). |
(End of session 2.)
🔧 HITL Final-Review Fixes — Round 3 (2026-07-02, session 3)
Repo: EDDI (feat/hitl-framework) Trigger: Third full-branch multi-agent review (70 agents, adversarial verification completed): 43 confirmed findings (12 MAJOR). This session fixes all of them.
Queued-say race (MAJOR)
processConversationStep re-reads the persisted state at execution time and DROPS a queued turn when the conversation is AWAITING_HUMAN/IN_PROGRESS — a stale pre-pause memory copy can no longer execute and full-document-overwrite a just-committed pause.
Zombie resume (MAJOR)
Resume persistence moved from the callable's finally into IFinishedExecution.onComplete — BaseRuntime's completed-after-cancellation discard now protects the resume path like the say path; a timed-out resume can never clobber state written after its watchdog fired.
Cancel-vs-resume window (MAJOR)
The live memory is registered in inFlightConversations synchronously after the resume CAS; the resume callable re-checks isCancelled()/persisted EXECUTION_INTERRUPTED before executing and skips persistence when cancelled. Cancel now also wins over a pause committed by the very task it interrupted (Conversation treats pause-while-cancelled as stop), on both say and resume paths.
Init pause (MAJOR)
startConversation performs the same HITL bookkeeping as the say path — a CONVERSATION_START pause now gets its policy bookmark, pause counter, and timeout schedule.
Resume rollback (MAJOR)
Every post-CAS failure restores the pause: snapshot-load failures, RejectedExecutionException from a saturated coordinator, and service exceptions. Wrong-state/agent-undeployed now throw IllegalStateException → 409; infrastructure failures throw ResourceStoreException → 500 (was: everything 409). Audit + resume counter moved AFTER the successful submit — rolled-back resumes no longer pollute the compliance trail or metrics; undeployed-agent restores skip schedule re-arm (kills the infinite timeout→restore→re-arm loop).
Bookmark hygiene
New clearHitlBookmark store op (Mongo $unset / Postgres jsonb key-removal) called when a pause is terminally resolved outside resume (cancel, end-while-paused) — stale bookmarks no longer round-trip forever, mislead approval-status (now also state-gated), or trick crash recovery into resurrecting dead pauses. Cancel of a pending approval writes an hitl.approval audit entry (verdict CANCELLED). endConversation on a paused conversation disarms the schedule and clears the bookmark (round-1 leftover).
Config resolution
HITL timeout config is read ONCE per pause at the conversation's PINNED agentVersion (fallback to latest); scheduleHitlTimeout derives from the memory bookmark — bookmark and schedule can no longer diverge, and draft config edits no longer change paused conversations' behavior. Re-pauses now increment the pause counter (metric parity with the group surface). Undo/redo additionally rejected during IN_PROGRESS (protects the resume-CAS invariant crash recovery relies on).
Group cancel window (MAJOR)
Control tokens are registered BEFORE the executor submit in startAndDiscussAsync and resumeDiscussion (executeDiscussion uses computeIfAbsent so a signalled pre-registered token is never wiped) — a cancel landing between the resume CAS (or async start) and thread startup now finds a signalable token instead of being overwritten by the running leg's unconditional updates. New convertPauseToCancelIfSignalled: a cancel that lands while commitPause is writing converts the just-committed pause to CANCELLED (CAS), disarms the schedule, audits, and emits the cancelled SSE event — cancel can no longer report success while the pause survives.
Group TASK-gate bypass (MAJOR)
The requiresApproval EXECUTE gate now also pauses when an aborted wave (timeout/error) left executable tasks behind with nothing awaiting approval — previously the phase loop fell through to VERIFY/synthesis over unexecuted work and silently skipped the remaining tasks.
Group resume resilience (MAJOR)
Config-drift aborts and pre-executeDiscussion failures now RESTORE the pause (restoreGroupPause: bookmark re-set, CAS IN_PROGRESS→AWAITING_APPROVAL, schedule re-armed) and fire group_error so SSE clients terminate — previously they persisted terminal FAILED, destroying the approval, without ever notifying the stream. executorService.submit failures roll back the same way (resume) or fail the conversation honestly (async start) instead of leaving IN_PROGRESS zombies. Failures INSIDE executeDiscussion are not double-handled (it owns its terminal states + events). REJECTED verdict now fires group_complete so streams close.
Group terminal cleanup (MAJOR)
New cleanupAfterTerminalState: ephemeral dynamic agents and lastVerifiedIndex entries are released when a paused discussion reaches a terminal state OUTSIDE the execution loop (cancel-of-paused, REJECTED resume) — previously they leaked forever because the in-loop finally only runs while a thread is executing.
Group cancel outcome
cancelDiscussion returns boolean (false = already terminal / lost CAS race) → REST maps to 409 (was unconditional 200); paused-cancels emit an hitl.approval audit entry (verdict CANCELLED) — parity with the regular surface. Timeout handler logs skipped aborts.
Group approvals
taskApprovals VALUES validated up front (only APPROVED/REJECTED, case-insensitive → else 400 before any mutation); an explicit {} map is treated as the approve-all shortcut instead of approving nothing and instantly re-pausing. approveGroupPhase returns a freshly-read copy — the HTTP layer no longer serializes the live object being mutated by the background thread.
Group timeout source
scheduleGroupHitlTimeout reads the pause bookmark on the conversation (set by commitPause/restoreGroupPause) instead of re-reading the group config — schedule and approval-status can no longer diverge after a config edit.
Group pending listing
listGroupPendingApprovals(groupId, limit) returns bounded PendingApprovalSummary objects (query-level group filter + limit, new findByState(state, groupId, limit) store variant) instead of unbounded full transcripts; summary gains groupId; REST takes ?limit (default 100).
Approver read scope (MAJOR)
detail=full on both approval-status endpoints is now gated for approver-only callers (not owner, not admin): full content is readable ONLY while the conversation is actually awaiting approval → 403 otherwise. The approver role exists to decide pending approvals, not as a universal read-everything grant over all conversations/transcripts. New OwnershipValidator.isOwner helper.
Group approval-status detail
GET .../approval-status on the group surface finally honors detail: default is a summary projection (state, pausedAt, phase, pauseType, reason, timeoutPolicy, awaiting task ids — stale fields suppressed outside AWAITING_APPROVAL, mirroring the regular surface) instead of always dumping the full conversation incl. transcript; detail=full returns the conversation, subject to the read-scope gate.
Crash recovery scale (MAJOR)
The recovery sweep runs on a background virtual thread — application readiness no longer blocks on repairing thousands of paused conversations. The paused-regular sweep reads bounded PROJECTED summaries (10k cap, logged if hit) instead of full multi-MB documents, and skips WAIT_INDEFINITELY pauses without any further read (PendingApprovalSummary gains approvalTimeout, projected on Mongo + Postgres and exposed on both listing surfaces). rearmSchedule re-checks the pause state AFTER creating the schedule and withdraws it if a resume/cancel landed in the window — no armed timeout on a no-longer-paused conversation.
Schedule + listing bounds
New name index on schedules (Mongo + Postgres) — HITL timeout delete/re-arm by name was a collection scan on every pause/resume/cancel. Mongo findPendingApprovalSummaries bounds the ids query with .limit() at the DB instead of materializing every paused id first.
HITL enum home
HitlTimeoutPolicy/HitlGranularity/HitlRejectionPolicy moved from nested types in AgentGroupConfiguration to the neutral ai.labs.eddi.configs.hitl package (HitlConfigValidation moved there too) — the regular-surface agent config and the whole engine no longer depend on the GROUP config class for shared HITL vocabulary. JSON compatibility unchanged (enum names serialize identically).
Dead surface removed
ControlSignal.PAUSE and DiscussionControlToken.isPaused()/shouldStop() deleted — HITL pauses are committed by the execution loop at the gates, never signalled through the token; the dead PAUSE path only invited misuse (a signalled "pause" would have been persisted as CANCELLED). Token checks now read isCancelled() explicitly.
hitl_resume event wired
HitlResumeEvent/EVENT_HITL_RESUME existed but were never fired. New onHitlResume listener method fires after the resume CAS commits; the SSE streaming endpoint forwards hitl_resume WITHOUT closing, so /approve/stream clients see an explicit resume marker before the resumed discussion's events.
ZIP import validation
RestImportService validates hitlConfig right after deserializing the agent file — an invalid config now fails the import up front with 400 (via IllegalArgumentExceptionMapper) instead of importing all workflows/extensions first and then failing agent creation with a 500 (partial import).
Javadoc drift
IConversationMemory.getHitlTimeoutPolicy no longer documents non-existent policy values ("expire"); AgentGroupConfiguration.HitlConfig no longer claims a "per turn / per discussion" granularity that never existed (actual: PHASE or TASK). Unused Mongo Updates import removed.
Test hardening (round 3)
New HitlConfigValidationTest (both surfaces, all rejection branches). taskApprovals validation tests: unknown id / wrong state / bad value → IAE with NOTHING mutated and no CAS; case-insensitive values; {} = approve-all. Audit emission tests (hitl.approval on resume with automated flag; verdict CANCELLED on cancel-of-paused; silent when ledger disabled). Note-cap tests on both surfaces (4097 → 400, 4096 → OK). Undo/redo HITL gate unit tests (AWAITING_HUMAN + IN_PROGRESS → false, nothing stored) + redo-409 IT. Group pending listing filter tests on summaries (admin/approver/owner/anonymous + default limit). Crash-observer tests rewritten for the projected sweep incl. schedule-withdraw races on both surfaces. Postgres container IT now covers the HITL store primitives (CAS, state query, projected summaries incl. approvalTimeout + limit, bookmark clearing). HitlPauseResumeIT.waitForState polls the DB-backed approval-status instead of the per-pod /status cache (flakiness fix).
Docs
docs/hitl.md updated: approver read-scope gate (403 semantics), group approval-status summary/full, pending summaries + ?limit, group cancel 409, hitl_resume SSE event + every-terminal-path-closes guarantee, drift-restores-pause behavior, cancel audit entries, async + projected crash recovery.
Group end-to-end IT
New GroupHitlIT — full group-surface HITL path with zero mocked seams: a requiresApproval phase commits a real pause through the store, /approve applies the decision, and the background resume re-enters the phase loop and runs the post-gate phase to completion (transcript asserted). Also: approval-status summary vs detail=full, pending-approvals summary listing (conversationId + groupId), REJECTED-is-terminal (later approve/cancel → 409), cancel-of-paused → CANCELLED (second cancel/late approve → 409), 400s that do NOT consume the pause (missing verdict, >4 KB note), 404 for unknown ids. Like all ITs, first execution happens in CI (-DskipITs=false); local Docker unavailable.
(End of session 3.)
🔧 HITL Review Fixes — Phases 1–5 (partial) (2026-07-02)
Repo: EDDI (feat/hitl-framework) Trigger: 7-phase implementation plan from code review (1 BLOCKER + 21 MAJORs).
Fixes Implemented
#1
BLOCKER
1a
Resume re-pause loop: checkIfPauseConversationAction is now delta-based — only throws if the just-executed task added PAUSE_CONVERSATION (not if it was stale from the prior turn). Belt-and-braces: Conversation.resume() strips PAUSE_CONVERSATION from step ACTIONS before re-entering the pipeline.
#1b
MAJOR
1b
Decision visibility: Verdict stored as conversation output (hitlDecision) and conversation-scoped property (hitlVerdict) for template/behavior-rule access. REJECTED emits public output.
#8
MAJOR
2a
Request body validation: Null/missing verdict → 400 on both REST surfaces (regular + group + streaming).
#12
MAJOR
5g
Double-approve → 409: GroupDiscussionException caught and mapped to 409 Conflict (was falling through to 500).
#9
MAJOR
3a
Group cancel state guard: No-token branch validates state before writing CANCELLED — terminal states (COMPLETED/CANCELLED/FAILED) cannot be overwritten.
#3
MAJOR
3b
Timeout rescheduling on re-pause: Resume callable's finally block arms a new HITL timeout if the conversation re-paused to AWAITING_HUMAN.
#5
MAJOR
4a
Undo/redo gate: Undo and redo blocked during AWAITING_HUMAN state (would corrupt the HITL bookmark).
Test Changes
pauseActionThrowsPause: Updated for delta-based semantics (sequential mock: null → actionData)fromIndexDetectsPause→ split intofromIndexIgnoresStaleAction(stale action = no re-pause) +fromIndexDetectsNewPause(new action = re-pause)New:
executeLifecycleDetectsFreshPause— verifies fresh pause onexecuteLifecycle
Files Changed
LifecycleManager.java— Delta-basedcheckIfPauseConversationAction, unconditionalactionsBeforesnapshotConversation.java—stripPauseActionhelper, decision visibility, rejection outputConversationService.java— Undo/redo gate, timeout rescheduling on re-pauseRestAgentEngine.java— Resume body validationRestGroupConversation.java— Approve body validation,GroupDiscussionException→ 409GroupConversationService.java— Cancel state guard in no-token branchLifecycleManagerHitlTest.java— 3 new/fixed delta-based pause tests
Completed (this session)
Phase 2b: HitlConfig string→enum typing (HitlGranularity, HitlTimeoutPolicy, HitlRejectionPolicy)
Phase 3d: Discriminating status codes — cancelDiscussion exception mapping (409/404 instead of 500)
Phase 4b: Strict ownership + eddi-approver role for HITL endpoints (requireOwnerAdminOrApprover)
Phase 4d: Micrometer HITL counters (eddi_hitl_pause/resume/timeout_count with surface tag)
Phase 6a: Deduplicated executeLifecycle/executeLifecycleFromIndex → shared executeTaskRange()
Phase 7b: HitlCrashRecoveryObserver unit tests (6 tests)
Phase 7b: OwnershipValidator approver role tests (6 tests)
🔧 HITL Review Fixes — Phases 5/6: Group Correctness + Config Surface (2026-07-02)
Repo: EDDI (feat/hitl-framework) Trigger: Continuing 7-phase implementation plan — group API correctness, config surface, and architecture.
Fixes Implemented
#10
MAJOR
5b
Non-EXECUTE + TASK fallback: TASK granularity only applies to EXECUTE phases (they have a SharedTaskList). Non-EXECUTE phases (OPINION, SYNTHESIS, etc.) now fall back to PHASE-style pause.
#5e
MAJOR
5e
Resume ordering: Timeout schedule deleted only AFTER the CAS succeeds (both approve + reject paths). If CAS fails, schedule preserved → timeout can still fire.
#5f
MAJOR
5f
Config drift guard: On resume, bookmarked phase name validated against loaded config. If config was edited while paused → FAILED + ERROR transcript entry.
#10
MAJOR
5d
Nested group HITL guard: Sub-group returning AWAITING_APPROVAL → SKIPPED entry instead of extracting partial answer. Nested HITL not supported in v1.
#11
MAJOR
5c
Timed-out task fixup: After wave timeout, IN_PROGRESS tasks reset to ASSIGNED (prevents permanent stranding).
#5a
MAJOR
5a
Task rejection policy: New onTaskRejection field in HitlConfig (FAIL/RETRY). RETRY resets rejected tasks to ASSIGNED for re-execution.
#15
MAJOR
4c
Audit trail: Audit collector added to resume path (same as say path).
#6c
MINOR
6c
Pause reason: hitlPauseReason field on GroupConversation — human-readable reason set at commitPause.
#6d
MINOR
6d
Bookmark timeout fields: hitlTimeoutPolicy + hitlApprovalTimeout copied from config at pause time for REST visibility.
Files Changed
GroupConversationService.java— HITL gate type check, drift guard, resume ordering, nested guard, timeout task fixup, rejection policy, bookmark populationGroupConversation.java— 3 new bookmark fields (hitlPauseReason, hitlTimeoutPolicy, hitlApprovalTimeout)AgentGroupConfiguration.java—onTaskRejectionfield in HitlConfigSharedTaskList.java—resetFromAnyToAssigned()method for RETRY policyConversationService.java— Audit collector on resume path
🔧 HITL Framework — Cancel Path Fixes: R1 + R2 MAJORs (2026-07-01)
Repo: EDDI (feat/hitl-framework) Trigger: Final merge verdict found 2 MAJORs in cancel path — cancel-vs-pause race and CancellationException misrouting.
Fixes
R1
MAJOR
Cancel-vs-pause race: Added isCancelled() guard immediately before the HITL gate. After the wave loop breaks on cancel, the HITL gate fired before the next phase-loop iteration's shouldStop() check, converting a cancel into a pause. Guard uses isCancelled() (not shouldStop()) so real PAUSE still routes to commitPause.
R2
MAJOR
CANCEL_IMMEDIATE → FAILED: Added explicit CancellationException catch in the wave allOf.get() handler. Forward-cancels all source agent futures (since allOf.cancel doesn't propagate). Both generic catch (GroupDiscussionException) and catch (Exception) now check token.isCancelled() and route to CANCELLED instead of FAILED. Also added source-future forward-cancel in the ExecutionException branch.
Regression Test Added
InFlightCancel.gracefulCancelDuringExecution— Concurrent latch-based test: launchesdiscuss()on separate thread, blockssay()with latch, firescancelDiscussion(GRACEFUL), asserts CANCELLED state.
Files Changed
GroupConversationService.java— R1 cancel guard before HITL gate + R2 CancellationException handling + cancel-aware generic catch blocksGroupConversationServiceHitlTest.java— In-flight cancel test with proper agent mock wiring
🔧 HITL Framework — Final Ship Fix: 2 BLOCKERs + 2 MAJORs in group TASK path (2026-07-01)
Repo: EDDI (feat/hitl-framework) Trigger: Ship/no-ship verdict found 2 BLOCKERs + 2 MAJORs, all in the group TASK surface.
Fixes
NEW-1
BLOCKER
Submit gate ≠ pause gate: submitForApproval now gates on taskLevelHitl && phase.requiresApproval(). Without both, completeTask is used. Prevents TASK_FORCE preset from stranding all tasks in AWAITING_APPROVAL when the phase doesn't require approval.
NEW-2
BLOCKER
Cancel-of-paused silent no-op: activeTokens.remove() is now unconditional in the finally block. Paused conversations have no running thread, so a lingering token caused cancelDiscussion to take the no-op signal branch. Resume re-registers a fresh token.
NEW-3
MAJOR
Control token write-only: Added token.shouldStop() safe-points at the top of both the phase loop and the wave loop. Registered the wave allOf future via setActiveFuture() so IMMEDIATE cancel can interrupt.
AUTO_APPROVE
MAJOR
TASK auto-approve infinite loop: When TASK granularity + APPROVED verdict + null taskApprovals (e.g., timeout handler), resumeDiscussion now synthesizes APPROVED for all AWAITING_APPROVAL tasks. Previously caused infinite reschedule.
Regression Tests Added
SubmitGateAlignment— TASK granularity + requiresApproval=false → tasks COMPLETED not strandedCancelOfPaused— Cancel of AWAITING_APPROVAL group does DB write to CANCELLEDAutoApproveTaskSynthesis— APPROVED + TASK + null taskApprovals auto-approves all tasksTaskResumeCompletesDependent— TASK resume re-enters same phase, clears hitlPauseType
Files Changed
GroupConversationService.java— All 4 fixes +taskLevelHitllocal variable inexecuteTaskExecutionPhaseGroupConversationServiceHitlTest.java— 4 new regression test classes (13 → 17 tests)
🔧 HITL Framework — Delta Code Review Fix #2: BLOCKER + MAJORs (2026-07-01)
Repo: EDDI (feat/hitl-framework) Trigger: Delta code review identified 1 BLOCKER + 7 MAJORs + 2 MINORs in group TASK surface.
Fixes
BLOCKER
BLOCKER
TASK resume index: resumeDiscussion now reads hitlPauseType before clearing. TASK resumes at same phase (re-entry idempotent via findExecutableTasks); PHASE resumes at +1.
MAJOR-1
MAJOR
Mutual exclusion of PHASE/TASK gates: TASK gate fires only when phase.requiresApproval() AND taskLevelHitl AND hasAwaitingApproval(). PHASE gate fires only when NOT taskLevelHitl. Eliminates double-pause.
MAJOR-2
MAJOR
Group timeout scheduling: commitPause now creates a one-shot IScheduleStore schedule for group HITL timeouts (reads approvalTimeout + timeoutPolicy from group config).
MAJOR-3
MAJOR
Schedule deletion on resume/cancel: Added IScheduleStore.deleteSchedulesByName() (MongoDB + PostgreSQL). Called in ConversationService.resumeConversation, cancelConversation, GroupConversationService.resumeDiscussion, and cancelDiscussion.
MAJOR-4
MAJOR
REJECTED branch CAS: Rejection now uses updateIfState(gc, AWAITING_APPROVAL) instead of plain update(gc) to prevent concurrent approve clobbering reject.
MAJOR-5
MAJOR
activeTokens lifecycle: executeDiscussion now registers control token at start (activeTokens.put) and removes it in finally block when discussion truly ends (not paused).
MAJOR-6
MAJOR
listPendingApprovals ownership filter: RestAgentEngine.listPendingApprovals() now filters by caller identity. Admin sees all; non-admin sees only their conversations. Added OwnershipValidator.isAdmin().
MINOR-1
MINOR
Metrics/events guard: counterGroupDiscussion.increment() and onGroupStart only fire when startPhaseIndex == 0 (fresh discussion, not resume).
MINOR-2
MINOR
Fail-closed on null owner: Added OwnershipValidator.requireOwnerOrAdminStrict() for state-changing ops where null-owner resources should deny access (admin exempted).
Files Changed
GroupConversationService.java— BLOCKER, MAJOR-1/2/3/4/5, MINOR-1 fixesConversationService.java— MAJOR-3 schedule deletion on resume/cancelRestAgentEngine.java— MAJOR-6 ownership filteringIScheduleStore.java— MAJOR-3 newdeleteSchedulesByNameAPIMongoScheduleStore.java— MAJOR-3 MongoDB implementationPostgresScheduleStore.java— MAJOR-3 PostgreSQL implementationOwnershipValidator.java— MAJOR-6isAdmin(), MINOR-2requireOwnerOrAdminStrict()6 test files — Constructor updated for new
IScheduleStoreparam + test fixes
🐛 Fix: PostgreSQL group conversations broken — JDBC ?| operator escape (2026-07-02)
Repo: EDDI (fix/postgres-group-conversation-jdbc-escape) Severity: Critical — all group conversations fail on PostgreSQL; MongoDB unaffected.
Root cause
PostgresUserMemoryStore.getVisibleEntries() builds a dynamic SQL query that uses PostgreSQL's ?| (array overlap) operator for group-scoped visibility filtering. The JDBC driver interpreted the ? in ?| as a bind parameter placeholder, inflating the expected parameter count by one. This caused PSQLException: No value specified for parameter 5 every time groupIds was non-empty.
Group conversations always pass a groupId context when starting agent sub-conversations (GroupConversationService.executeAgentTurn() → startConversation() with groupId in context), so this bug was triggered on every group conversation turn — making group conversations completely non-functional on PostgreSQL.
Fix
PostgresUserMemoryStore.java: Changed?|to??|(JDBC escape syntax for a literal?). Single character fix.
Regression guard
PostgresUserMemoryStoreTest.java: Added two integration tests (Testcontainers PostgreSQL) that exercisegetVisibleEntrieswith non-emptygroupIds:groupIdsDoNotBreakQuery— verifies the query doesn't throw with non-empty groupIds (would have caught this bug directly)groupScopedEntriesVisible— verifies group-scoped entries are correctly returned when groupIds match, and excluded when they don't
Why existing tests missed it
The unit test (
PostgresUserMemoryStoreUnitTest.getVisibleEntries_withGroupIds_includesGroupClause) tested with non-empty groupIds but mocked the JDBC PreparedStatement — never sent actual SQL to PostgreSQL, so the?parsing was invisible.The integration test (
PostgresUserMemoryStoreTest.selfAndGlobalVisible) ran against real PostgreSQL but only tested withgroupIds = null— never exercised the group visibility branch.
Last updated
Was this helpful?