> For the complete documentation index, see [llms.txt](https://docs.labs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.labs.ai/security-and-compliance/compliance-data-flow.md).

# Compliance Data Flow

> **Audience**: Compliance auditors, DPOs, and deployers performing risk assessments. This is the **canonical data inventory** for EDDI: how data flows through it, where it is stored, what leaves the deployment, and what an erasure does to each store. The [GDPR](https://docs.labs.ai/security-and-compliance/gdpr-compliance.md), [HIPAA](https://docs.labs.ai/security-and-compliance/hipaa-compliance.md) and [incident response](https://docs.labs.ai/security-and-compliance/incident-response.md) guides link here instead of repeating it.

***

## System Data Flow

```
┌──────────────────────────────────────────────────────────────────────────────┐
│                                EDDI Platform                                │
│                                                                             │
│  ┌──────────┐    ┌────────────────┐    ┌──────────────────────────────────┐ │
│  │ Keycloak │───▶│  REST API /    │───▶│     Conversation Pipeline        │ │
│  │  (OIDC)  │    │  SSE / MCP     │    │                                  │ │
│  │          │    │                │    │  Input → Parser → Behavior Rules │ │
│  │ JWT auth │    │  TLS required  │    │  → LLM Task → Output Generation │ │
│  └──────────┘    └────────────────┘    └──────────┬───────────────────────┘ │
│                                                   │                         │
│                    ┌──────────────────────────────┼──────────────────┐      │
│                    │              │               │                  │      │
│              ┌─────▼─────┐ ┌─────▼────┐  ┌──────▼──────┐  ┌───────▼────┐ │
│              │ Conversa- │ │  User    │  │   Audit     │  │  Secrets   │ │
│              │ tion      │ │ Memory   │  │   Ledger    │  │  Vault     │ │
│              │ Memory    │ │ Store    │  │             │  │            │ │
│              │           │ │          │  │  HMAC-signed│  │ AES-256-GCM│ │
│              │ PII: Yes  │ │ PII: Yes │  │  (when a key│  │ Envelope   │ │
│              │ Encrypted:│ │ Encrypted│  │  is set)    │  │ encryption │ │
│              │ TDE*      │ │ TDE*     │  │  PII: Yes** │  │            │ │
│              │           │ │          │  │  TDE*       │  │ PII: No    │ │
│              └─────┬─────┘ └─────┬───┘  └──────┬──────┘  └────────────┘ │
│                    │             │              │                         │
│                    └─────────────┼──────────────┘                         │
│                                 │                                         │
│                          ┌──────▼──────┐                                  │
│                          │  MongoDB /  │                                  │
│                          │ PostgreSQL  │                                  │
│                          │             │                                  │
│                          │ TDE* = DB-  │                                  │
│                          │ level       │                                  │
│                          │ encryption  │                                  │
│                          └─────────────┘                                  │
│                                                                           │
│   ** Audit rows are kept on GDPR erasure; their content is redacted and   │
│      the userId pseudonymized (see "What an erasure does" below)          │
└──────────────────────────────┬────────────────────────────────────────────┘
                               │
                               │ Configured transport (https:// or http://
                               │ base URL) — conversation content LEAVES
                               │ the deployment whenever an LLM call runs
                               ▼
                    ┌──────────────────────┐
                    │    LLM Provider      │
                    │  (per agent config)  │
                    │                      │
                    │  See "What reaches   │
                    │  the LLM provider"   │
                    └──────────────────────┘
```

***

## What reaches the LLM provider

**Conversation content leaves your deployment.** Every LLM call sends data to the provider configured on the agent — unless that provider is self-hosted (Ollama, jlama, or an OpenAI-compatible server you run), it is a third party and a processor (GDPR) or sub-Business Associate (HIPAA) that needs a contract. EDDI does not store what it sends beyond its own conversation memory and audit ledger, but the provider's retention is governed by the provider's terms.

**Transport security is the configured endpoint's.** The hosted providers' SDK defaults are `https://`, but a `baseUrl` an agent configures (Ollama, an OpenAI-compatible server, a proxy) is used as given — EDDI blocks cloud-metadata addresses there, not plain HTTP. A remote `http://` base URL sends the prompt, the history and tool results unencrypted. Use `https://` for any endpoint reached over a network you do not control.

**Always sent, on every LLM call:**

* the agent's system prompt, **after templating**
* the current user message, and any attachment passed to a multimodal model
* the conversation history window of **this** conversation (rolling summaries included, when the agent uses them)
* the names, descriptions and argument schemas of the tools the agent may call

**Sent when the agent's configuration makes it so:**

| Source                             | When                                              | What it can contain                                                                                                                                                                                                                                                                                                     |
| ---------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Prompt templates                   | the system prompt or a tool body references it    | `{properties.*}` — the user's persistent memories are loaded into the properties at conversation start, including `global` memories **other agents** wrote about the same user and `group` memories of the conversation's group; `{userInfo.userId}`; `{conversationInfo.*}`, `{context.*}`, `{snippets.*}`, `{vars.*}` |
| Memory tools (`enableMemoryTools`) | the model calls `recallMemories` / `searchMemory` | this agent's own memories, plus every `global` memory any agent wrote about the user and the group's `group` memories                                                                                                                                                                                                   |
| RAG                                | a knowledge base is attached                      | chunks retrieved from the ingested documents                                                                                                                                                                                                                                                                            |
| HTTP calls, MCP and A2A tools      | the model calls the tool                          | the tool's response, whatever the external service returns                                                                                                                                                                                                                                                              |
| Group discussions                  | the agent is a group member                       | other members' contributions and the group context                                                                                                                                                                                                                                                                      |
| Background LLM jobs                | configured on the agent or group                  | conversation summarization sends conversation history; [Dream](https://docs.labs.ai/architecture-and-concepts/user-memory.md#dream-consolidation) consolidation sends the user's memories; cascade judges and group facilitators/summarizers send the turn they assess                                                  |

**Not sent by EDDI on its own:** a user identifier or account metadata (EDDI sets no `user` field on provider requests — an identifier reaches the provider only if a template or a tool result puts it into the text), other users' conversations, vault secrets (unless a template resolves one into prompt text) and API keys other than the provider's own authentication.

"No data from other agents reaches the LLM" is therefore **not** a property of EDDI: `global` user memories are shared across agents by design, and a `longTerm` property of an agent **without a `userMemoryConfig`** is stored as `global` — so by default it is loaded into every other agent's conversations with that user. To keep an agent's memories private, give it a `userMemoryConfig` (its `defaultVisibility` is `self`) and leave `global` out of `guardrails.allowedVisibilities` (the default) — and check what the agents you deploy alongside it write as `global`.

Provider data residency: [PRIVACY.md](https://docs.labs.ai/security-and-compliance/privacy.md); provider BAAs: [HIPAA guide](https://docs.labs.ai/security-and-compliance/hipaa-compliance.md#llm-provider-baa-requirements).

***

## Other outbound recipients

Besides the LLM provider, these leave the deployment as a side effect of a feature, not of a conversation's content:

| Recipient                                                                                                                                         | When                                                                                                                                                                      | What it receives                                                                                                                                                                                                                                                                                                                             | Notes                                                                                                                                                                                                                                                                                               |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Mailbox host** of an [email integration](https://docs.labs.ai/protocols-and-integration/email-integration.md) (Exchange Online, an IMAP hoster) | Every poll of an integration                                                                                                                                              | EDDI logs in with the integration's connection and reads new mail, sets the `$EddiProcessed` keyword and may move mail to a folder. In trigger mode EDDI sends the host nothing but the login and these IMAP commands — no conversation content, no reply                                                                                    | The host is a **processor** of the mail itself (it stores what senders sent). The integration's OAuth app or app password should reach that one mailbox only ([email-integration.md](https://docs.labs.ai/protocols-and-integration/email-integration.md#exchange-online-oauth-client-credentials)) |
| **SMTP relay** of a conversation-mode email integration (`smtpHost`, e.g. Exchange Online)                                                        | Every reply, pause notice and continuation of a [conversation-mode](https://docs.labs.ai/protocols-and-integration/email-integration.md#conversation-mode) integration    | The reply itself: the agent's answer (through `replyTemplate`), the AI disclosure, the answered mail's subject and Message-IDs, and the one recipient — the authenticated sender of the mail it answers                                                                                                                                      | A **processor**; usually the same provider as the mailbox host. It delivers the reply to the recipient's mail server, which receives it as any mail. The relay must sign with DKIM for the mailbox's domain                                                                                         |
| **DNS resolver** (`eddi.email.dns.servers`, else the system's resolvers)                                                                          | Mail sender authentication checks a received message ([security.md](https://docs.labs.ai/security-and-compliance/security.md#mail-sender-authentication--dkim-and-dmarc)) | TXT queries naming the signing domain and selector of each DKIM signature (`<selector>._domainkey.<d>`) and the `From` domain or its organisational domain (`_dmarc.<domain>`) — so the resolver, and the authoritative servers it asks, learn which domains mail arrived from and when. Never an address, a local part, a subject or a body | Answers are cached (TTL-honouring, capped at `eddi.email.dns.cache-max-ttl`), which also limits how often a sender's domain is queried. Queries are plain DNS, unencrypted; point `eddi.email.dns.servers` at a resolver inside the deployment's network to keep them there                         |

***

## Data Store Inventory

| Data Store                                                                                        | Contains PII                                                                                                                                                                                               | Encryption                                | Retention                                                                                              | On GDPR erasure                                                                                                                                | Regulatory Notes                                                                                                                                                                                                                                          |
| ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Conversation Memory**                                                                           | ✅ userId, chat content                                                                                                                                                                                     | TDE (deployer)                            | Ended conversations: 365 days default (`eddi.conversations.deleteEndedConversationsOnceOlderThanDays`) | ✅ Deleted                                                                                                                                      | Primary PII store                                                                                                                                                                                                                                         |
| **User Memory**                                                                                   | ✅ userId, structured facts                                                                                                                                                                                 | TDE (deployer)                            | Kept until deleted; `eddi.usermemories.deleteOlderThanDays` (default `-1`, off)                        | ✅ Deleted                                                                                                                                      | Cross-conversation state                                                                                                                                                                                                                                  |
| **Managed Conversations**                                                                         | ✅ userId, intent mappings                                                                                                                                                                                  | TDE (deployer)                            | Until deleted                                                                                          | ✅ Deleted                                                                                                                                      | Routing metadata                                                                                                                                                                                                                                          |
| **Attachments**                                                                                   | ✅ user-uploaded images, PDFs, audio                                                                                                                                                                        | TDE (deployer)                            | With owning conversation                                                                               | ✅ Deleted                                                                                                                                      | GridFS or PostgreSQL blobs; potential PHI                                                                                                                                                                                                                 |
| **Conversation Checkpoints**                                                                      | ✅ copy of conversation properties                                                                                                                                                                          | TDE (deployer)                            | With owning conversation                                                                               | ✅ Deleted                                                                                                                                      | Same PII as the conversation                                                                                                                                                                                                                              |
| **Group Conversations**                                                                           | ✅ multi-agent transcripts                                                                                                                                                                                  | TDE (deployer)                            | Until deleted                                                                                          | ✅ Deleted                                                                                                                                      | Group discussion content                                                                                                                                                                                                                                  |
| **Shared Artifacts**                                                                              | ✅ user/agent-authored content                                                                                                                                                                              | TDE (deployer)                            | Until deleted                                                                                          | ✅ Deleted                                                                                                                                      | Owned by the creating user                                                                                                                                                                                                                                |
| **HITL Tool Journal**                                                                             | ✅ tool name, capped tool result, approver identity                                                                                                                                                         | TDE (deployer)                            | With owning conversation                                                                               | ✅ Deleted                                                                                                                                      | Human-approval audit trail                                                                                                                                                                                                                                |
| **Email message ledger**                                                                          | ✅ recipient address of each HITL notification and conversation-mode reply, the sender address of each inbound mail of an email integration, conversation id, Message-ID — never a body or subject          | TDE (deployer)                            | `eddi.email.ledger-retention` (default `30d`)                                                          | ✅ Deleted (matched by the user id's address, an `email:<integrationId>:<address>` identity's address, and the user directory's recorded email) | The SMTP relay of `eddi.mail.smtp.*` is a processor: it receives each notification's recipient, the agent name, pause reason and redacted tool arguments; see [hitl.md](https://docs.labs.ai/conversations-and-orchestration/hitl.md#email-notifications) |
| **Email suppression list**                                                                        | ✅ per integration: SHA-256 of an address a reply bounced from permanently, its domain, the bounce status, when                                                                                             | TDE (deployer)                            | Until an administrator clears it, the integration is deleted, or the address is erased                 | ✅ Deleted (matched like the ledger)                                                                                                            | Never mailed while listed; see [email-integration.md](https://docs.labs.ai/protocols-and-integration/email-integration.md#bounces-and-the-suppression-list)                                                                                               |
| **Email mailbox state**                                                                           | ❌ per mailbox: IMAP cursor, poll back-off, the day's spend, notice keys                                                                                                                                    | TDE (deployer)                            | Until the integration is deleted                                                                       | —                                                                                                                                              | No address, subject or body                                                                                                                                                                                                                               |
| **Conversation ratings, tool quotas, alerts, A2A tasks, user directory, workspace notifications** | ✅ userId-keyed rows (ratings and the step they rate; per-user tool counters; alert rule authors, acknowledgers and stored `OUTPUT_MATCH` values; a peer's A2A tasks; directory entries; notification rows) | TDE (deployer)                            | Until deleted (alert events: `eddi.alerts.event-retention`, default `90d`)                             | ✅ Deleted by the matching GDPR participant (alerts: values blanked, the user forgotten as author and acknowledger)                             | Erased in step 15 of the cascade, see [gdpr-compliance.md](https://docs.labs.ai/security-and-compliance/gdpr-compliance.md)                                                                                                                               |
| **Team outcome events**                                                                           | ✅ the principal that posted an outcome                                                                                                                                                                     | TDE (deployer)                            | Until deleted                                                                                          | ⚠️ Kept; the poster is **pseudonymized** (`groupOutcomes` participant)                                                                         | An outcome describes a team's work, not a person                                                                                                                                                                                                          |
| **Personal-space secrets and variables; OAuth connection grants**                                 | ✅ per-user values; a refresh token for the user's account at the provider                                                                                                                                  | AES-256-GCM (secrets); TDE (deployer)     | Until deleted                                                                                          | ✅ Deleted (provider-side consent is not revoked)                                                                                               | Includes `scope: secret` property slots                                                                                                                                                                                                                   |
| **Cluster erased-markers**                                                                        | ❌ the erased conversation's id only                                                                                                                                                                        | NATS KV (deployer)                        | `eddi.cluster.erasure-marker-ttl` (default 7 days)                                                     | —                                                                                                                                              | Written (`ERASED` bucket) so a node cut off from NATS does not republish a dead letter of an erased conversation                                                                                                                                          |
| **Schedules**                                                                                     | ✅ userId, trigger payloads                                                                                                                                                                                 | TDE (deployer)                            | Until deleted                                                                                          | ✅ Deleted                                                                                                                                      | Owned by the creating user                                                                                                                                                                                                                                |
| **Audit Ledger**                                                                                  | ✅ userId, prompts, responses, LLM detail, tool calls; admin actions name their actor                                                                                                                       | TDE (deployer) + HMAC (when a key is set) | Indefinite — no application-level expiry                                                               | ⚠️ Rows kept; content **redacted**, userId **pseudonymized** (`eddi.audit.erasure-mode`)                                                       | EU AI Act Arts. 12/19 record-keeping                                                                                                                                                                                                                      |
| **Audit dead-letter sink**                                                                        | ✅ whole audit entries                                                                                                                                                                                      | Deployer                                  | Never expired by EDDI                                                                                  | ❌ Not reached                                                                                                                                  | Only written when the ledger cannot persist; see [GDPR guide](https://docs.labs.ai/security-and-compliance/gdpr-compliance.md#the-audit-dead-letter-sink-holds-personal-data-and-erasure-does-not-reach-it)                                               |
| **Database Logs**                                                                                 | ✅ userId; WARN/ERROR message text                                                                                                                                                                          | TDE (deployer)                            | `eddi.logs.db-retention-days` (default `-1`: kept until deleted)                                       | ⚠️ userId pseudonymized; message text kept                                                                                                     | Operational data                                                                                                                                                                                                                                          |
| **Secrets Vault**                                                                                 | ❌ API keys only (plus `scope: "secret"` property values, deleted on erasure)                                                                                                                               | AES-256-GCM (application-level)           | Until rotated/deleted                                                                                  | ✅ Auto-vaulted user secrets deleted                                                                                                            | Credentials only                                                                                                                                                                                                                                          |

***

## PII Lifecycle

```
User Input (may contain PII)
    │
    ├──▶ Stored in Conversation Memory (MongoDB/PostgreSQL)
    │        └─ Ended conversations deleted after the retention period (default 365 days)
    │        └─ Or: GDPR erasure (immediate)
    │
    ├──▶ Extracted to User Memory (property setter longTerm, memory tools)
    │        └─ Retention: until deleted (optional age-based sweep)
    │        └─ Or: GDPR erasure (immediate)
    │
    ├──▶ Sent to the LLM provider (whenever an LLM call runs — see above)
    │        └─ Not stored by EDDI beyond its own memory and audit ledger
    │        └─ Provider retention: per the provider's terms
    │
    ├──▶ Recorded in the Audit Ledger (userId + task input/output/LLM detail/tool calls)
    │        └─ Retention: indefinite (EU AI Act record-keeping)
    │        └─ GDPR erasure: rows kept, content redacted, userId pseudonymized
    │        └─ HMAC integrity hash (when a key is configured) detects tampering
    │
    ├──▶ Mentioned in Database Logs (WARN/ERROR entries for the turn)
    │        └─ Retention: eddi.logs.db-retention-days (off by default)
    │        └─ GDPR erasure: userId pseudonymized
    │
    └──▶ Secret-scoped values → Secrets Vault
             └─ Vault reference replaces plaintext in memory
             └─ Raw input scrubbed from conversation step
```

***

## Encryption Summary

| Layer                     | Mechanism                         | Managed By                                                                                                                                                        | Covers                               |
| ------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| **In Transit**            | TLS 1.2+                          | Deployer (reverse proxy or direct)                                                                                                                                | All HTTP/SSE/MCP traffic             |
| **At Rest (credentials)** | AES-256-GCM envelope encryption   | EDDI Secrets Vault                                                                                                                                                | API keys, tokens, passwords          |
| **At Rest (data)**        | Transparent Data Encryption (TDE) | Deployer (database config)                                                                                                                                        | Conversations, memories, audit, logs |
| **Audit Integrity**       | HMAC-SHA256                       | EDDI — **only when** `EDDI_VAULT_MASTER_KEY` or `EDDI_AUDIT_HMAC_KEY` is set; `eddi.compliance.audit-signing-required=true` makes a missing key a startup failure | Tamper detection on audit entries    |

***

## What an erasure does

`DELETE /admin/gdpr/{userId}` deletes the user's data from every store above marked "Deleted" — the step-by-step list, the response fields and how to read `complete` are in the [GDPR guide](https://docs.labs.ai/security-and-compliance/gdpr-compliance.md#1-right-to-erasure-gdpr-art-17--ccpa-1798105). Two stores are kept rather than deleted:

* **Audit ledger** — every row stays, so the record of what ran and when, and the per-conversation chain, survive. The row's prompt, response, LLM detail and tool calls are replaced by a redaction marker and its userId by a keyed pseudonym; a row that verified before is re-signed so it still verifies. See [audit-ledger.md](https://docs.labs.ai/security-and-compliance/audit-ledger.md#gdpr-erasure-redaction-not-deletion).
* **Database logs** — the userId is replaced by a keyed pseudonym (`gdpr-erased:k1:…`, the audit ledger's key) or, when the ledger does not sign, a per-deployment-salted one (`gdpr-erased:s1:…`). Only rows from older releases, or an erasure that could not read the salt, carry the unsalted `gdpr-erased:<sha256(userId)>`. The log message text is not rewritten; bound it with `eddi.logs.db-retention-days`.

> **This is pseudonymisation, not anonymisation, and the distinction is legal as well as technical.** The database-log pseudonym is keyed or salted (`gdpr-erased:k1:…` / `gdpr-erased:s1:…`), so a list of candidate user IDs cannot be hashed and matched against it without the key or salt. Rows from older releases, and the fallback used when the salt cannot be read, hold an unsalted digest that can be matched that way. The audit ledger's v5 pseudonym is keyed (`gdpr-erased:k1:<HMAC(key, userId)>`) and cannot be recomputed without the ledger's key — but whoever holds the key can.
>
> Under GDPR Art. 4(5) pseudonymised data remains **personal data** and stays in scope. Do not treat these records as anonymised, and do not disclose them on that basis.

***

## See Also

* [PRIVACY.md](https://docs.labs.ai/security-and-compliance/privacy.md) — Data processing overview
* [hipaa-compliance.md](https://docs.labs.ai/security-and-compliance/hipaa-compliance.md) — HIPAA deployment guide
* [eu-ai-act-compliance.md](https://docs.labs.ai/security-and-compliance/eu-ai-act-compliance.md) — EU AI Act compliance
* [gdpr-compliance.md](https://docs.labs.ai/security-and-compliance/gdpr-compliance.md) — GDPR/CCPA operations
* [secrets-vault.md](https://docs.labs.ai/security-and-compliance/secrets-vault.md) — Encryption architecture
* [audit-ledger.md](https://docs.labs.ai/security-and-compliance/audit-ledger.md) — Audit trail details
