> For the complete documentation index, see [llms.txt](https://docs.labs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.labs.ai/security-and-compliance/incident-response.md).

# Incident Response Plan

This runbook outlines the steps for detecting, assessing, and responding to a data breach involving the EDDI platform.

## 1. Detection

### Indicators

* Unexpected audit ledger HMAC validation failures (tamper detection)
* Unusual administrative actions — every `POST`/`PUT`/`PATCH`/`DELETE` outside the chat APIs is in the audit ledger with its caller and status (`taskId: ai.labs.admin`, see [audit-ledger.md](https://docs.labs.ai/security-and-compliance/audit-ledger.md#administrative-actions)), so a burst of endpoint `403`s or an unexpected `/secretstore` or `/backup` call is visible there. A request rejected by authentication before it reaches an endpoint is not — check Keycloak's event log and your reverse proxy's access log for those
* Failed authentication spikes in Keycloak logs
* Anomalous conversation volume or data export requests
* Alerts from infrastructure monitoring (Grafana/Prometheus)

### Monitoring

EDDI exposes metrics at `/q/metrics` (Prometheus format). The names below are the Prometheus spellings — Micrometer replaces `.` with `_` and appends `_total` to counters, so `eddi.tool.execution.failure` is scraped as `eddi_tool_execution_failure_total`:

* `eddi_processing_conversation_count` — conversations currently being processed (gauge). A sustained rise with flat throughput means the coordinator is backing up
* `eddi_coordinator_queue_depth` / `eddi_coordinator_active_conversations` — the coordinator's own view of the same backlog
* `eddi_tool_execution_success_total` / `eddi_tool_execution_failure_total` / `eddi_tool_execution_ratelimited_total` — tool execution volume, tagged by `tool`. A spike in one tool's `failure` or `ratelimited` is the usual first sign of an outbound-integration incident
* `eddi_audit_entries_dropped_total` — audit entries the ledger could **not** write, either because its bounded queue was full or after `MAX_FLUSH_RETRIES` (3) consecutive audit-store write failures. This must stay at zero, and it has two different causes: check the store's health, not just queue pressure. Entries dropped on the flush path are written to the dead-letter sink (`eddi.audit.dead-letter-path`, default `/opt/eddi/data/eddi-audit-deadletter.jsonl`) and can be recovered as evidence; an entry refused at ingest because the queue was full is genuinely lost, so the compliance trail has holes for the affected window
* `eddi_vault_errors_count_total` — vault resolve/store failures

There is no single "audit write rate" counter; use the ledger's own listing endpoint for volume, and the dropped counter above for integrity.

## 2. Assessment (First 4 Hours)

### Scope Determination

1. **Identify affected data**: Which stores were compromised?
   * Conversation content (chat history)
   * User memories (persistent facts)
   * API keys/credentials (vault)
   * Audit trail integrity
2. **Identify affected users**: Query the GDPR export endpoint to enumerate affected user data:

   ```bash
   GET /admin/gdpr/{userId}/export
   ```
3. **Determine attack vector**: Check application logs for unauthorized access:

   ```bash
   GET /administration/logs?level=ERROR&limit=100
   ```

   `/administration/logs/history` queries persisted logs by `agentId`, `conversationId`, `userId` or `instanceId`; `/administration/logs/stream` tails them live over SSE.

### Risk Classification

| Risk Level | Criteria                         | Response             |
| ---------- | -------------------------------- | -------------------- |
| **High**   | PII exposure, credentials leaked | Full breach protocol |
| **Medium** | System data exposed, no PII      | Containment + review |
| **Low**    | Failed attempt, no data access   | Log + monitor        |

## 3. Containment (First 24 Hours)

1. **Rotate compromised credentials**:
   * Rotate all LLM API keys in the Secrets Vault
   * Invalidate affected Keycloak sessions
   * Update any exposed database credentials
2. **Isolate affected systems**:
   * Undeploy compromised agents
   * Revoke affected user tokens
3. **Preserve evidence**:
   * Export audit trail for affected conversations
   * Snapshot database logs
   * Do NOT delete audit entries. The ledger is append-only by design: rows are never deleted, and the only in-place change is a GDPR erasure redacting the erased user's content (re-signed when the row verified) — see [audit-ledger.md](https://docs.labs.ai/security-and-compliance/audit-ledger.md#gdpr-erasure-redaction-not-deletion)

## 4. Notification

### GDPR (Art. 33-34)

* **Supervisory authority**: Within **72 hours** of becoming aware
* **Data subjects**: Without undue delay if high risk to rights/freedoms
* **Processor → Controller**: If you use EDDI as a managed service, ensure your hosting provider notifies you immediately upon discovering a breach. If you self-host, you are both controller and processor.

### CCPA

* **Affected consumers**: Notification required for certain categories of personal information
* **California Attorney General**: If breach affects 500+ residents

### HIPAA (§164.408)

* **HHS (Secretary)**: For breaches of 500 or more individuals, within **60 days** of discovery, contemporaneously with individual notice
* **Affected individuals**: Without unreasonable delay, no later than **60 days** after discovery
* **Media**: If breach affects ≥ 500 residents of a state, notify prominent media outlets in that state
* **Small breaches** (< 500 individuals): Logged and reported annually to HHS, within 60 days of the end of the calendar year

### Template

```
Subject: Data Breach Notification

Date of discovery: [DATE]
Nature of breach: [DESCRIPTION]
Categories of data: [conversation content / user memories / credentials]
Approximate number of affected users: [COUNT]
Measures taken: [CONTAINMENT STEPS]
Contact: [DPO / PRIVACY CONTACT]
```

## 5. Recovery

1. Deploy patched version of EDDI with vulnerability remediated
2. Re-validate audit ledger integrity (`/auditstore/verify/…` — HMAC and chain verification). It proves nothing for entries written while no signing key was configured: those report `UNSIGNED`
3. Conduct post-incident review
4. Update this runbook with lessons learned

## 6. Prevention

* Enable Keycloak authentication in production
* Use RBAC: grant `eddi-admin` only to administrators and the narrower roles (`eddi-editor`, `eddi-user`, `eddi-approver`, `eddi-viewer`) to everyone else — roles are enumerated per endpoint, with no hierarchy
* Review audit trail regularly for anomalies
* Keep EDDI updated to the latest version
* Use self-hosted LLM providers for sensitive deployments
* Enable TLS for all EDDI endpoints
* Regularly rotate API keys and vault credentials

## 7. Emergency Access Procedure (HIPAA §164.312(a)(2)(ii))

For healthcare deployments, maintain a documented "break glass" procedure:

1. **Emergency admin account**: Create a dedicated Keycloak account with `eddi-admin` role, stored in a sealed envelope or hardware security module (HSM)
2. **Activation**: Require two-person authorization to unseal the emergency credentials
3. **Audit**: Every administrative action taken with the emergency account is recorded in the audit ledger with that account as the actor, as are the conversation turns and tool invocations it causes. Reads are **not** recorded — use Keycloak's event log for the logins and your reverse proxy's access log for what was read
4. **Deactivation**: Rotate emergency credentials immediately after each use via Keycloak admin console
5. **Documentation**: Log the reason for emergency access, duration, and actions taken

## See Also

* [hipaa-compliance.md](https://docs.labs.ai/security-and-compliance/hipaa-compliance.md) — HIPAA deployment guide
* [gdpr-compliance.md](https://docs.labs.ai/security-and-compliance/gdpr-compliance.md) — GDPR/CCPA compliance
* [security.md](https://docs.labs.ai/security-and-compliance/security.md) — Security architecture
* [audit-ledger.md](https://docs.labs.ai/security-and-compliance/audit-ledger.md) — Immutable audit trail
