> For the complete documentation index, see [llms.txt](https://docs.labs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.labs.ai/agent-configuration/model-cascade.md).

# Model Cascade

> Cost-optimized LLM execution via sequential model escalation with confidence-based routing.

## Overview

Multi-model cascading lets an LLM task try a fast, cheap model first and only escalate to a more expensive model if the confidence in the response is below a threshold. This reduces cost for queries that simpler models handle well, while preserving quality for hard queries.

**Flow:** `User query → Model A (fast/cheap) → Confidence check → if low → Model B (powerful) → Confidence check → ...`

The cascade is a self-contained branch inside `LlmTask` — no engine pipeline changes, and configs without `modelCascade` (or with `enabled: false`) behave exactly as before.

## Configuration

Cascading is configured per-task in a `langchain.json` resource:

```json
{
  "tasks": [
    {
      "id": "cascade-task",
      "type": "openai",
      "actions": ["*"],
      "parameters": {
        "systemMessage": "You are a helpful assistant.",
        "apiKey": "${vault:openai-key}",
        "logSizeLimit": "10"
      },
      "modelCascade": {
        "enabled": true,
        "strategy": "cascade",
        "evaluationStrategy": "structured_output",
        "enableInAgentMode": true,
        "maxTotalDurationMs": 45000,
        "maxCostPerRun": 0.05,
        "steps": [
          {
            "type": "openai",
            "parameters": { "model": "gpt-4o-mini" },
            "confidenceThreshold": 0.7,
            "timeoutMs": 10000,
            "inputPricePer1M": 0.15,
            "outputPricePer1M": 0.60
          },
          {
            "type": "openai",
            "parameters": { "model": "gpt-4o" },
            "confidenceThreshold": null,
            "timeoutMs": 30000,
            "inputPricePer1M": 2.50,
            "outputPricePer1M": 10.00
          }
        ]
      }
    }
  ]
}
```

### Cascade fields

| Field                                  | Type    | Default               | Description                                                                                                                                                                                                                                                                            |
| -------------------------------------- | ------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`                              | boolean | `false`               | Master toggle for cascading                                                                                                                                                                                                                                                            |
| `strategy`                             | string  | `"cascade"`           | Execution strategy. Only `cascade` (sequential) is implemented; `parallel` and any unknown value warn at deploy time and run sequentially.                                                                                                                                             |
| `evaluationStrategy`                   | string  | `"structured_output"` | How confidence is evaluated (see below)                                                                                                                                                                                                                                                |
| `enableInAgentMode`                    | boolean | `true`                | Whether cascade activates when tools/agents are configured                                                                                                                                                                                                                             |
| `judgeModel`                           | object  | —                     | Model for the `judge_model` strategy: `{ "type": "...", "parameters": {...} }`. Expected when `evaluationStrategy` is `judge_model`; if omitted or unbuildable, deployment logs a warning and confidence evaluation falls back to `heuristic` at runtime.                              |
| `heuristic`                            | object  | —                     | Overrides for the `heuristic` strategy (see below). Optional.                                                                                                                                                                                                                          |
| `maxTotalDurationMs`                   | long    | —                     | Wall-clock ceiling across the whole cascade. When reached, escalation stops and the best response so far is returned. Also caps each **buffered** step's timeout by the remaining budget — a step streamed live is exempt (see [Streaming the Final Step](#streaming-the-final-step)). |
| `maxCostPerRun`                        | double  | —                     | Dollar ceiling for a single run, computed from token usage × per-step pricing. When reached, escalation stops and the best response so far is returned.                                                                                                                                |
| `inputPricePer1M` / `outputPricePer1M` | double  | —                     | Cascade-level default token pricing (steps may override). Used for cost reporting and the cost ceiling.                                                                                                                                                                                |
| `returnBestAcrossSteps`                | boolean | `false`               | When true, if an earlier (escalated) step scored strictly higher than the finally-accepted step, the earlier step's response is returned.                                                                                                                                              |
| `steps`                                | array   | —                     | Ordered list of cascade steps (cheap → expensive)                                                                                                                                                                                                                                      |

### Step fields

| Field                                  | Type   | Default         | Description                                                                                                                                                                                                                                                                          |
| -------------------------------------- | ------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`                                 | string | task `type`     | Provider type (e.g., `openai`, `anthropic`, `ollama`). Resolved through global variables, like the task type.                                                                                                                                                                        |
| `parameters`                           | object | `{}`            | Provider-specific params. Merged over the base task parameters (step wins). Values are resolved for `${vault:...}` secrets, global variables, and Qute templates — parity with task params.                                                                                          |
| `confidenceThreshold`                  | Double | `null`          | Minimum confidence to accept this step. Below it, escalate. **A non-last step should set a threshold** (a null threshold there is always-accepted, making later steps unreachable — flagged with a deploy-time warning). The last step's threshold is ignored (always accepted).     |
| `timeoutMs`                            | long   | `30000`         | Per-step timeout in milliseconds, for **buffered** (non-streamed) steps — also bounded by the remaining `maxTotalDurationMs` budget. A step streamed live ignores this and instead runs under an internal \~120 s bound (see [Streaming the Final Step](#streaming-the-final-step)). |
| `inputPricePer1M` / `outputPricePer1M` | double | cascade default | Per-step token pricing (overrides the cascade-level default).                                                                                                                                                                                                                        |

> **Merge note:** Step parameters are merged over base task parameters (step wins). Steps only specify overrides (e.g., a different `model`); shared params like `systemMessage` are inherited.
>
> **⚠️ Cross-provider credentials:** Because parameters are inherited, a step (or `judgeModel`) that targets a **different provider** than the task must supply **its own credentials** — otherwise it silently inherits the task's `apiKey`, which is wrong for a different provider and fails at runtime as a 401 (which the cascade then treats as an escalation). A different-provider step/judge that omits its own `apiKey` is flagged with a deploy-time warning. Give each cross-provider step its own full parameter set (`apiKey`, `baseUrl`, etc.). Same-provider steps may safely inherit the task's credentials.

## Confidence Evaluation Strategies

### `structured_output` (default)

Appends a JSON-format instruction to the system prompt asking the model to respond with a single JSON object:

```json
{ "response": "The actual answer...", "confidence": 0.85 }
```

The evaluator tries a **real JSON parse first** (Jackson), and only treats the response as a confidence wrapper when the whole response is a single JSON object — so a stray `"confidence": ...` inside legitimate answer content (e.g. a code sample) is **not** mistaken for the score. A regex fallback handles a malformed-but-object-shaped wrapper. If the response is not a JSON-object wrapper, it falls back to `heuristic`.

> **Agent mode / convertToObject:** the wrapper cannot be used with tools or with `convertToObject: true` (it would collide with the raw-schema JSON). In those cases the cascade automatically uses `judge_model` (if a judge is configured) or `heuristic`. The `convertToObject` + `structured_output` combination is flagged with a deploy-time warning; the agent-mode downgrade is logged at debug level at runtime (agent mode is only known when the task runs). See below.

### `heuristic`

Analyzes the response text for uncertainty signals. Phrases and thresholds are **configurable** via `heuristic` (English defaults). When no configured phrase matches, a language-agnostic default score is used.

| Signal                                                  | Confidence (default)   |
| ------------------------------------------------------- | ---------------------- |
| Empty/null response                                     | `0.0`                  |
| Very short (< `shortLengthThreshold`, default 20 chars) | `shortScore` (`0.3`)   |
| Refusal phrase (e.g. "I cannot fulfill")                | `refusalScore` (`0.2`) |
| Hedging phrase (e.g. "I'm not sure")                    | `hedgingScore` (`0.4`) |
| No red flags                                            | `defaultScore` (`0.8`) |

`heuristic` config fields (all optional): `lowConfidencePhrases`, `refusalPhrases`, `shortLengthThreshold`, `shortScore`, `refusalScore`, `hedgingScore`, `defaultScore`. Localize the phrase lists for non-English deployments — without configured phrases, the evaluator cannot distinguish hedging from confidence and returns the default score. Configured score values are clamped to `[0.0, 1.0]`, so a mis-set value can't produce an out-of-range confidence.

### `judge_model`

A separate (typically cheap) model rates the response's confidence. Requires a `judgeModel` config block; the judge is built once via the model registry (with vault + global-variable resolution). If the judge cannot be built or the call fails, it falls back to `heuristic`.

```json
"judgeModel": { "type": "openai", "parameters": { "model": "gpt-4o-mini", "apiKey": "${vault:openai-key}" } }
```

### `none`

Always returns `1.0` — effectively disables confidence gating. The first step's response is always accepted. Useful for timeout/error recovery only, or A/B testing.

## Error Handling

| Error Type                          | Behavior                                                                                                                                                                            |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Rate limited (429) / 5xx**        | Retried **in-step** up to the task's `retry.maxAttempts` (with backoff) before escalating to the next step.                                                                         |
| **Timeout**                         | The step is cancelled and the cascade escalates; a warning is logged. A step streamed live is exempt from cancellation — see [Streaming the Final Step](#streaming-the-final-step). |
| **Other errors**                    | Logged; escalate to the next step.                                                                                                                                                  |
| **Duration / cost ceiling reached** | Stop escalating, return the best response so far.                                                                                                                                   |
| **All steps fail**                  | Return the best response seen so far, or throw `LifecycleException` if none produced a result.                                                                                      |

The cascade tracks the "best response" seen so far — if a later step fails but an earlier step produced a usable response, that response is returned rather than throwing.

## SSE Events

Two SSE event types provide real-time visibility, emitted through `ConversationEventSink` → `StreamingResponseHandler` → the `/agents/{conversationId}/stream` SSE endpoint:

| Event                | Fields                                                                  |
| -------------------- | ----------------------------------------------------------------------- |
| `cascade_step_start` | `stepIndex`, `modelType`, `modelName`, `totalSteps`                     |
| `cascade_escalation` | `fromStep`, `toStep`, `confidence`, `threshold`, `reason`, `durationMs` |

`reason` is one of `low_confidence`, `timeout`, `error`, `retryable_error`.

## Streaming the Final Step

When streaming (SSE), a **guaranteed-accept step** is streamed **live** token-by-token: the last step, a step with a null `confidenceThreshold`, or a `none`-strategy step whose threshold is `≤ 1.0` (its confidence is always `1.0`, so it will always accept). This requires legacy (no-tools) mode, a non-wrapper strategy (`heuristic`, `judge_model`, or `none`), and a streaming-capable provider. Any step that could still escalate is always buffered instead (its full text is needed to evaluate confidence before deciding). In agent mode, the cascade emits the final response as a single chunk rather than streaming it live.

> **Bounds & consistency:** a live-streamed step is **not** subject to the per-step `timeoutMs` / `maxTotalDurationMs` cap — cancelling it mid-stream cannot stop the provider from continuing to emit tokens to the client, so instead it runs under the streaming executor's own internal bound (\~120 s) and its result — even if partial at that bound — is the accepted answer. The client therefore never receives tokens for a response that is then replaced. `returnBestAcrossSteps` also never supersedes a step that was streamed live, for the same reason.

## Observability

### Trace

The full per-step trace is stored in conversation memory under `langchain:cascade:trace:<taskId>`. Each entry contains: `step`, `model`, `modelType`, `confidence`, `durationMs`, `tokenUsage` (`inputTokens`/`outputTokens`/`totalTokens`), `costUsd`, and `status` (`accepted`, `escalated`, `timeout`, `error`, `retryable_error`). When `returnBestAcrossSteps` overrides the outcome, the step that would have been accepted is relabeled `superseded_by_best` and the earlier winning step is relabeled `accepted_as_best`, so the trace always agrees with the returned `stepUsed`.

### Response metadata

If `responseMetadataObjectName` is set, the cascade populates it with real token usage plus `cascadeCostUsd`, `cascadeModel` (`provider/model`), `cascadeStep`, and `cascadeConfidence`.

### Metrics (Micrometer, `/q/metrics`)

`eddi.llm.cascade.executions` (tag `agentMode`), `eddi.llm.cascade.escalations` (tag `reason`), `eddi.llm.cascade.accepted.step` (tag `step`), `eddi.llm.cascade.step.latency` (timer, tag `provider`), `eddi.llm.cascade.confidence` (distribution), `eddi.llm.cascade.step.errors` (tags `provider`, `type`), `eddi.llm.cascade.tokens` / `eddi.llm.cascade.cost` (tag `provider`), `eddi.llm.cascade.ceiling.exceeded` (tag `kind` = `duration`|`cost`).

## Audit Trail

When the audit collector is active, the cascade writes:

| Memory Key                  | Content                                        |
| --------------------------- | ---------------------------------------------- |
| `audit:model_name`          | The **actual** winning model, `provider/model` |
| `audit:cascade_model`       | `provider/model (step N)`                      |
| `audit:cascade_confidence`  | Confidence of the accepted response            |
| `audit:cascade_cost`        | Aggregate run cost in dollars                  |
| `audit:cascade_token_usage` | Token usage of the accepted step               |

## Agent Mode

When `enableInAgentMode` is `true` (default), the cascade also works when tools (built-in, MCP, HTTP calls, A2A) are configured — each step can independently invoke the tool-calling loop. Because the `structured_output` wrapper cannot be injected around the tool loop, agent-mode confidence uses `judge_model` (if configured) or `heuristic`.

**Cancellation:** when a step times out, the orchestrator checks for interruption between tool-loop iterations and before each tool, so it stops launching further side-effectful tools. A tool already in flight when the timeout fires may still complete — keep cascade-in-agent-mode tools idempotent where possible.

When `enableInAgentMode` is `false`, cascading is skipped in agent mode and the standard single-model path is used.

## Configure-time Validation

Cascade configs are validated at deploy (`LlmTask.configure`), in two tiers so an upgrade never stops a previously-loading agent from deploying:

* **Hard error (deployment fails)** — only the **new** numeric fields, since no stored config predating this release can contain them: non-positive `maxTotalDurationMs`, negative `maxCostPerRun`, negative per-step / cascade `inputPricePer1M` / `outputPricePer1M`.
* **Warning (logged, deployment proceeds)** — conditions older releases tolerated at load and that still fail/degrade at runtime exactly as before: empty steps, unknown `strategy`, unknown `evaluationStrategy`, `judge_model` without a `judgeModel`, `confidenceThreshold` outside `[0.0, 1.0]`, a non-last step with a null threshold (dead-step trap), non-positive `timeoutMs`, a cross-provider step/judge missing its own `apiKey`, and `convertToObject: true` with `structured_output` (auto-downgraded at runtime).

## Backward Compatibility

* Configs without `modelCascade` work exactly as before.
* `enabled: false` (default) keeps standard execution.
* All new config fields are optional with today's behavior as defaults.
* The cascade lives entirely within `LlmTask`; `StreamingResponseHandler`'s cascade methods are `default`, so other implementers are unaffected.

## Example: Cost Optimization

A 3-tier cascade for a customer support agent, with pricing so savings are measurable:

```json
{
  "modelCascade": {
    "enabled": true,
    "evaluationStrategy": "heuristic",
    "maxCostPerRun": 0.02,
    "steps": [
      {
        "type": "ollama",
        "parameters": { "model": "llama3.2:3b", "baseUrl": "http://localhost:11434" },
        "confidenceThreshold": 0.7,
        "timeoutMs": 5000
      },
      {
        "type": "openai",
        "parameters": { "model": "gpt-4o-mini" },
        "confidenceThreshold": 0.8,
        "timeoutMs": 15000,
        "inputPricePer1M": 0.15,
        "outputPricePer1M": 0.60
      },
      {
        "type": "anthropic",
        "parameters": { "model": "claude-sonnet-4-20250514" },
        "confidenceThreshold": null,
        "timeoutMs": 30000,
        "inputPricePer1M": 3.00,
        "outputPricePer1M": 15.00
      }
    ]
  }
}
```

This routes simple FAQs to a local Ollama model (free), medium queries to GPT-4o-mini, and only complex queries to Claude Sonnet — with per-turn cost recorded in the trace, audit ledger, and metrics so the savings are provable.