Agent Testing — Golden Conversations
An agent test suite is a set of scripted conversations with an agent, and for each turn a statement of what must be true afterwards: which actions fired, which properties were set, which tools were called, what the output says, what it cost. EDDI runs the suite against an agent version and records a pass/fail result per case and turn.
Because EDDI is config-driven and records every turn in conversation memory, the strongest assertions do not depend on how an LLM words its answer: "the store_city action fired and city is Vienna" holds or it does not. Output regexes are there too, as the secondary signal.
Scope. This page describes suites, runs, tool mocks, REST, MCP and the Manager: suites are edited under Resources → Test Suites, and an agent's detail page has a Golden conversations section that starts a run, cancels one in progress and shows its results. Generating a suite from a recorded conversation, an LLM-as-judge expectation and a deployment gate are planned follow-ups. Suites are not yet included in agent export ZIPs.
Concepts
Suite
A versioned configuration document (/testsuitestore/testsuites), like a rule set or an LLM config
Case
One scripted conversation: a list of turns, optional context, repetitions and passRate
Turn
One user input and its expect block
Run
One execution of a suite against one agent version; a stored record with per-case, per-turn results
Mock
A substitute response for a tool call — a test never reaches a real endpoint unless the suite says so
How a run executes
POST /agenttests/runsnames a suite and an agent (versions optional — the current ones by default).The agent version is deployed to the
testenvironment if it is not already (through the normal deploy, so the caller needs deploy rights). Production is never touched. If a newer version of the agent is also deployed totest, the run is refused with 409 — new conversations always start on the newest ready version, so the run would silently test the wrong one.The run is recorded as
RUNNING, with the node executing it (ownerNode), and the call returns 202 with the run id. A node runs at mosteddi.agent-tests.max-concurrent-runsat once; that is checked before the deploy of step 2, so a request refused with 429 has deployed nothing.In the background, every case runs once per repetition, each in its own conversation (environment
test, user idagent-test:<runId>:<case>:<repetition>). Up toeddi.agent-tests.max-parallel-casesconversations run at once.After each turn the stored conversation memory is read and the turn's expectations are checked.
Every test conversation is ended and permanently deleted afterwards, unless the run asked for
keepConversations: true.The run record is updated to
PASSED,FAILED,ERRORorCANCELLED, and one summary entry is written to the audit ledger (taskId: ai.labs.agenttest). While it runs, the record'sheartbeatAtis refreshed every 30 seconds (see Known limitations).
What a test conversation does not do
It neither reads nor writes long-term user memory. A test must not change what real users have stored, and must not pass or fail because of it.
It never executes a tool call unless the suite opts in — see Tool mocking.
It never waits for a human. A HITL pause (
PAUSE_CONVERSATIONor a gated tool call) is an observable outcome — assert it with"conversationState": "AWAITING_HUMAN". Any later turn of that case fails with "the conversation is AWAITING_HUMAN", and the conversation is ended (which resolves the pause and is audited assystem:agent-test). Approval notifications go only to channel-started conversations (e.g. Slack), so none is sent for a test.Dream consolidation never runs for it — Dream runs on per-user schedules, and a test user has neither a schedule nor any stored memory.
Suite reference
Top level
name, description
—
For people
defaults
see below
Per-case defaults
maxCostPerRun
1.0
Dollar ceiling for the whole run, checked before every turn against the summed audit:cost of all turns so far (start turns included). Reaching it stops the run; it ends FAILED with costCeilingReached: true
cases
—
The scripted conversations
defaults
toolMode
mock
mock: tool calls get their mock; live: every call runs for real
unmockedTools
fail
With toolMode: mock, what a call without a mock does. fail: not executed, the case fails. live: it runs for real
repetitions
1
How often each case runs (1–20)
passRate
1.0
Fraction of repetitions that must pass, 0–1. repetitions: 5, passRate: 0.8 means "at least 4 of 5"
turnTimeoutSeconds
120
A turn that takes longer fails
A case may override repetitions and passRate. context on a case is sent with every turn; context on a turn is applied on top. A string becomes a string context, an object an object context, a list an array context.
Expectations (expect)
Every present field must hold. Every one that does not is reported, with the expected and the actual value.
actionsContain
every listed action fired this turn
actionsExact
exactly these actions fired, in this order
properties
for each named conversation property: {"equals": v} (value equal; 3, 3.0 and "3" are equal, objects and lists compare structurally), {"matches": "regex"}, {"exists": true} or {"absent": true}
outputContains
the turn's output text contains the string (case-sensitive)
outputMatches
the output text matches the regex (find, not a full match)
toolCalled
each listed tool was called this turn; with argsSubset, with arguments that contain those key/values
toolNotCalled
none of the listed tools was called
maxToolCalls
at most this many tool calls
maxCostUsd
the turn's audit:cost was at most this
conversationState
the state after the turn: READY, AWAITING_HUMAN, ENDED, ERROR, IN_PROGRESS, EXECUTION_INTERRUPTED
Tool expectations see every call of the turn — LLM tool calls of any source, rule-triggered httpcalls and rule-triggered MCP calls — mocked, refused or live.
Regexes are bounded. A pattern may be at most 500 characters (checked when the suite is saved) and one match may take at most 250 ms; a match that runs out of time fails its expectation with <regex timed out>. A suite therefore cannot hang the runner with a catastrophically backtracking pattern.
The time limit only sees work that reads the text, and a counted repetition of something that can match empty reads nothing: (?:(?:){99999}){99999} spins ten billion times against any output. So counted minimums ({n}, {n,}, {n,m}) applied to a group, an anchor or a back-reference may force at most 1,000 iterations in total — nested counts multiply, and the counts across the pattern add up. Counts on a single character, class or escape (\d{4}, [a-z]{2,8}) are not limited, since each iteration reads the text. The comments flag (?x) is not supported. Such a pattern is rejected when the suite is saved, and one saved before the check existed fails its expectation with <regex timed out> without being run.
Limits
Checked when a suite is saved (400 naming the field): at most 200 cases, 50 turns per case, 20 repetitions, 500 mocks, and 2,000 turns per run in total (cases × repetitions × turns). Invalid regexes, unknown modes and states are rejected too. Also: maxCostPerRun between 0 and 100 (USD), at most 20,000 characters per turn input, and turnTimeoutSeconds between 1 and 900.
Tool mocking
A test must never reach a production endpoint by accident, so tool calls of a test conversation are fail-closed:
a call that has a mock gets the mock's response and is never executed;
a call without a mock, with
unmockedTools: fail(the default), is not executed and answered with{"status":"NOT_EXECUTED","reason":"no mock defined in test mode"}— and the case fails, with anunmockedToolfailure on the turn and the tool named inunmockedToolCalls;only
toolMode: liveorunmockedTools: livelets a call run for real — an explicit opt-in per suite.
Mocks apply at the three places a pipeline reaches the outside world:
LLM tool calls of an httpcall tool
The mock is the HTTP response, exactly as for a rule-triggered httpcall below: no request is built or sent, and saveResponse, retries and postResponse property instructions run on the mocked {httpCode, body} — so a property an agent sets from a tool's response is set in a test too. The model gets the executor's result, {"httpCode": …, "body": …}
LLM tool calls of every other source (MCP, A2A, built-in, dynamic, recall)
Before the executor runs: the tool gets body (a string as-is, anything else as JSON)
Rule-triggered httpcalls (httpcalls step)
The mock is the HTTP response: saveResponse, retries and postResponse property instructions run on the mocked {httpCode, body} exactly as on a real one. An unmocked call is answered 501 with the NOT_EXECUTED body. Fire-and-forget calls are not dispatched
Rule-triggered MCP calls (mcpcalls step)
The mock's body is the tool result. With nothing allowed to run live, the MCP server is not contacted at all — not even to list its tools
Matching: tool is the name the agent calls — an httpcall's name, an MCP tool name, a built-in tool name. argsSubset narrows a mock to calls whose JSON arguments contain those key/values; the first matching mock wins. Rule-triggered calls have no model-chosen arguments, so only mocks without argsSubset match them. A mock's response is {httpCode, body}; httpCode defaults to 200.
Exceptions:
Tools that write long-term user memory never run live in a test, whatever the suite says; in a live suite they are answered NOT_EXECUTED without failing the case.
discover_tools(LAZY tool loading) is not an outside call and always runs.A tool call made on the start turn (
CONVERSATION_START) has no turn result of its own; an unmocked one still fails the attempt and is listed in the attempt'sunmockedToolCalls.Mocked calls cost $0, and are never served from or stored in the tool-result cache.
A call is recorded once, however often it is retried — a retried live call is one tool call too — so a retry does not count against
maxToolCallsagain. An LLM tool call that goes through its executor (an httpcall tool's mock, or any call the suite runs live) is recorded only once the loop's own gates — tool budget, tenant quota, turn deadline — have let it through; one they refuse was never made and is not counted.Listing an MCP server's tools to build an LLM's tool set still happens in a test (it reads a schema, it does not call a tool).
Results
A run record (GET /agenttests/runs/{runId}):
status is PASSED when every case met its pass rate, FAILED otherwise (or when the cost ceiling stopped the run), ERROR when the run could not execute, ABORTED when the node running it stopped before it finished, and CANCELLED when it was cancelled. error gives the reason for the last three. The newest eddi.agent-tests.max-runs-per-agent runs per agent are kept.
Cancelling a run
POST /agenttests/runs/{runId}/cancel (MCP: cancel_agent_test_run; Manager: the Cancel run button on a running run) stops a run that is RUNNING on the node that receives the call. It takes the rights starting one takes: EDIT on the agent. The stop is cooperative: no further conversation or turn starts, a turn waiting for the engine is abandoned, and every test conversation is ended and deleted (unless the run keeps them). The call returns 202 with the record as it is; it reads CANCELLED moments later, with the cases that finished and the reason (Cancelled by <principal> …). The cases that did not run to the end are failed. 409 means the run has already finished, or executes on another node — the message names it.
REST API
All endpoints take eddi-admin or eddi-editor — the same roles as deploying an agent, because a run deploys to test.
POST /testsuitestore/testsuites
Create a suite (201, Location carries id and version)
GET /testsuitestore/testsuites/{id}?version=N
Read a suite
PUT /testsuitestore/testsuites/{id}?version=N
Update (creates version N+1)
DELETE /testsuitestore/testsuites/{id}?version=N
Delete (&permanent=true removes it instead of marking it deleted)
GET /testsuitestore/testsuites/descriptors
List suites
POST /agenttests/runs
Start a run — body {"suiteId", "agentId", "suiteVersion"?, "agentVersion"?, "keepConversations"?}; 202 with the RUNNING record
POST /agenttests/runs/{runId}/cancel
Cancel a RUNNING run — see Cancelling a run; 202
GET /agenttests/runs/{runId}
Read a run
GET /agenttests/runs?agentId=…&index=0&limit=20
An agent's runs, newest first
Suite URIs follow the v6 scheme: eddi://ai.labs.testsuite/testsuitestore/testsuites/{id}?version=N.
Errors: 400 (invalid suite, missing ids, empty suite), 404 (suite, agent or run not found), 409 (the version could not be deployed to test, a newer version is deployed there, the deployment runs in cluster mode; on cancel: the run has finished or executes on another node), 429 (more than eddi.agent-tests.max-concurrent-runs runs at once).
CI recipe
MCP tools
run_agent_tests, get_agent_test_run, list_agent_test_runs and cancel_agent_test_run, at the same tier as deploy_agent — see MCP Server.
LLM nondeterminism
A run uses the live model. Deterministic assertions — actions, properties, tool calls — are the primary signal; output regexes are secondary. Where an LLM decides the path, express the tolerance you accept with repetitions and passRate ("at least 4 of 5 runs must pass"), and keep maxCostPerRun in proportion.
Evidence (EU AI Act)
Every run is stored with the suite version, the agent version, the models the turns reported, per-turn results and cost, and leaves one summary entry in the tamper-evident audit ledger. Together they document that an agent version was tested against a defined set of behaviours, what the result was and who ran it — the kind of testing and accuracy evidence Art. 9 (risk management) and Art. 15 (accuracy, robustness) ask a provider of a high-risk system to keep. A suite complements, it does not replace, the provider's own risk assessment.
Configuration
eddi.agent-tests.max-parallel-cases
4
Conversations of one run at once
eddi.agent-tests.max-concurrent-runs
2
Runs at once per node
eddi.agent-tests.max-runs-per-agent
50
Run records kept per agent
Metrics: eddi_agent_tests_runs_total{status}, eddi_agent_tests_turns_total, eddi_agent_tests_run_duration_seconds — see Metrics.
Known limitations
Not supported in cluster mode (
eddi.messaging.type=nats): starting a run is refused with 409. A run and its tool mocks live in the JVM that started it, and cluster mode may route a test conversation's turn to another node, where every tool call would be refused. Run suites against a single-node deployment. If a call is ever refused because the node holds no session for the conversation, its result says so —"reason":"no agent-test session on this node (…)"— rather than no mock defined in test mode, so the two causes cannot be confused.A run executes on the node that accepted it, and can be cancelled only there. If that node stops mid-run, the record is marked
ABORTED(with the reason inerror) and the run's test conversations are deleted. Start the run again. Every node sharing the database checks, every minute, whether theheartbeatAtof eachRUNNINGrun it does not execute itself has changed since it last looked; a run whose heartbeat has stood still for three minutes (90 seconds when the owner carries this node's id — a restart of this process, or a second process on the same host) is closed. Only equality of the owner's timestamp is compared, on the observer's own clock, so clock skew between nodes does not matter, and a live run is never closed while its heartbeat moves. The close is a compare-and-swap on the heartbeat that stood still: a heartbeat arriving in between wins. A run recorded before the owner and heartbeat fields existed is closed once it has been seen unchanged for three minutes.The flip side: an owner that cannot write its heartbeat for three minutes (a database outage on its side) loses the run. Another node closes it, the owner's next heartbeat finds the record changed, and the owner stops the run without writing anything back — the record stays
ABORTED, and no result is audited.A turn that times out, or is abandoned by a cancel, may still be running in the engine after its conversation was ended and deleted. When it completes (or after another turn timeout), the conversation is deleted again, in case the turn wrote it back.
Such a turn is still paid for. Its cost is charged when it completes. If that is after the run returned, the stored run record is adjusted:
totalCostUsdgrows by the late cost andmodelsgains any new model (a compare-and-swap on the record, retried if another adjustment lands first), and the audit ledger gets an additionalAGENT_TEST_LATE_CHARGEentry whose cost is just the added amount, so the ledger's sum for the run stays its real spend. The run's status, itscostCeilingReachedflag and the per-case results are not recomputed: a run that came in under budget and was pushed over it by a late turn keeps its verdict, and shows the true figure.Test conversations count against the tenant's conversation quota and, while a run is in progress, appear in conversation listings and conversation metrics — and in what is computed from them — like any other conversation of the agent (environment
test, user idagent-test:…). They are deleted when the run ends unless it keeps them.User ids starting with
agent-test:(in any case) are reserved: starting a conversation with one through REST, MCP or a group conversation is refused with 403, for an admin and with authorization off too — whether it is requested or is the caller's own identity — a schedule with one is refused when it is saved over REST (400), and any schedule with one, however it was stored, is not fired. A signed-in principal with such a name is refused at authentication. A run's mocks and its record of tool calls are keyed on those ids, so an outside conversation must not join them.A mocked httpcall — rule-triggered or an LLM tool — is never built: its
${vault:},${connection:}and${caller:}references are not resolved, so a mocked call whose headers need${caller:token}works in a test although there is no caller.
Last updated
Was this helpful?