> For the complete documentation index, see [llms.txt](https://docs.labs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.labs.ai/agent-configuration/attachments-guide.md).

# Multimodal Attachments

> EDDI's attachment pipeline enables multimodal conversations — users can send images, files, and documents alongside text input. Attachments flow through the lifecycle pipeline and are automatically forwarded to vision-capable LLMs.

## Quick Start

### Send an Image via URL

```bash
POST /agents/{conversationId}/say?message=What%20is%20in%20this%20image?
Content-Type: application/json

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "url": "https://example.com/photo.png",
      "fileName": "photo.png"
    }
  }
}
```

### Send an Image via Base64

```bash
POST /agents/{conversationId}/say?message=Describe%20this%20icon
Content-Type: application/json

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAAAAE...",
      "fileName": "icon.png"
    }
  }
}
```

The image is automatically forwarded to the LLM as multimodal content. The LLM "sees" the image alongside the text message.

***

## How It Works

```
Client sends context with attachment_* keys
           │
           ▼
┌──────────────────────────────────┐
│  Conversation.prepareLifecycleData()  │
│                                       │
│  AttachmentContextExtractor parses    │
│  attachment_0, attachment_1, ...      │
│  into List<Attachment> objects        │
│                                       │
│  Stored in memory: "attachments"      │
└──────────────┬───────────────────┘
               │
    ┌──────────┼──────────┐
    ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────────┐
│BehaviorRules│ │LlmTask│ │Other Tasks │
│             │ │       │ │            │
│ContentType- │ │Multi- │ │Read from   │
│Matcher      │ │modal  │ │memory key  │
│condition    │ │Message│ │"attachments"│
│             │ │Enhancer││            │
└────────┘ └────────┘ └────────────┘
```

### Pipeline Stages

1. **Context Extraction** — `AttachmentContextExtractor` parses `attachment_*` context keys into `Attachment` objects
2. **Memory Storage** — Attachments are stored as `List<Attachment>` in the `attachments` memory key
3. **Rule Matching** — `ContentTypeMatcher` condition matches on MIME types for routing
4. **LLM Forwarding** — `MultimodalMessageEnhancer` converts attachments to langchain4j `ImageContent` and enhances the user message

***

## Input Paths

### Path A: URL Reference (Recommended)

Best for images already hosted somewhere. The LLM provider fetches the image directly from the URL.

```json
{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/jpeg",
      "url": "https://cdn.example.com/photos/sunset.jpg",
      "fileName": "sunset.jpg"
    }
  }
}
```

### Path B: Base64 Inline

Best for small images (< 5MB). Data is sent inline as a base64-encoded string.

```json
{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk...",
      "fileName": "icon.png"
    }
  }
}
```

> **Note:** `base64Data` is transient — it's never persisted to MongoDB. For large files, use the upload endpoint (Path C below) or URL references.

### Path C: File Upload

For large files, upload them to the storage backend and receive a storage reference:

```bash
POST /conversations/{conversationId}/attachments
Content-Type: multipart/form-data

# Form field: file (the binary file)
```

**Response (201):**

```json
{
  "storageRef": "gridfs://68abc123def456",
  "fileName": "report.pdf",
  "mimeType": "application/pdf",
  "sizeBytes": 524288
}
```

The returned `storageRef` can then be used in subsequent conversation turns by setting it as the `url` in an attachment context key. The storage backend (GridFS or PostgreSQL) is selected automatically based on the configured datastore.

| Response Code | Meaning                          |
| ------------- | -------------------------------- |
| `201`         | File stored successfully         |
| `400`         | No file provided                 |
| `503`         | No attachment storage configured |

***

## Context Key Format

Attachment context keys must match the pattern `attachment_*`:

| Key                     | Valid?           |
| ----------------------- | ---------------- |
| `attachment_0`          | ✅                |
| `attachment_screenshot` | ✅                |
| `attachment_`           | ✅                |
| `image_0`               | ❌ (wrong prefix) |
| `attachment`            | ❌ (no suffix)    |

### Required Fields

| Field      | Required        | Description                                      |
| ---------- | --------------- | ------------------------------------------------ |
| `mimeType` | Yes             | MIME type (e.g., `image/png`, `application/pdf`) |
| `url`      | One of url/data | External URL reference                           |
| `data`     | One of url/data | Base64-encoded content                           |
| `fileName` | No              | Original filename (for metadata/logging)         |

If both `url` and `data` are present, `url` takes precedence.

***

## Multiple Attachments

Send multiple attachments by incrementing the key index:

```json
{
  "attachment_0": {
    "type": "object",
    "value": { "mimeType": "image/png", "url": "https://example.com/page1.png" }
  },
  "attachment_1": {
    "type": "object",
    "value": { "mimeType": "image/png", "url": "https://example.com/page2.png" }
  }
}
```

All attachments are forwarded to the LLM in a single multimodal user message.

***

## LLM Multimodal Support

The `MultimodalMessageEnhancer` automatically converts attachments to the appropriate langchain4j content type:

| MIME Type         | langchain4j Content                      | Provider Support                                |
| ----------------- | ---------------------------------------- | ----------------------------------------------- |
| `image/*`         | `ImageContent`                           | OpenAI GPT-4o, Gemini, Claude 3, Ollama (LLaVA) |
| `application/pdf` | Metadata text (future: `PdfFileContent`) | Gemini                                          |
| `audio/*`         | Metadata text (future: `AudioContent`)   | Gemini                                          |
| Other             | Metadata text description                | All (text-only)                                 |

For unsupported MIME types, a text description is injected so the LLM knows an attachment was present:

```
[Attachment: report.csv (text/csv, 15240 bytes)]
```

***

## Routing with Behavior Rules

Use `contentTypeMatcher` to create different workflows based on attachment type:

### Route Images to Vision Agent

```json
{
  "name": "Image received",
  "actions": ["analyze_image"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "image/*",
        "minCount": "1"
      }
    }
  ]
}
```

### Route PDFs to Document Processor

```json
{
  "name": "Document received",
  "actions": ["process_document"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "application/pdf",
        "minCount": "1"
      }
    }
  ]
}
```

### Require Specific Attachment Count

```json
{
  "name": "Comparison ready",
  "actions": ["compare_images"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "image/*",
        "minCount": "2"
      }
    }
  ]
}
```

***

## Template Access

Attachments are available in templates via the memory namespace:

```
Current step attachments: {{memory.current.attachments}}
```

This can be useful for logging, debugging, or constructing custom prompts that reference attachment metadata.

***

## Architecture Notes

* **No inline storage**: Attachment payloads are never stored inline in conversation memory documents. Only metadata references are persisted.
* **Transient base64**: The `base64Data` field is `transient` — it exists only during the pipeline turn. For persistence, use the upload endpoint with `IAttachmentStorage`.
* **DB-agnostic**: The `IAttachmentStorage` SPI supports MongoDB (GridFS) and PostgreSQL (bytea) implementations.
* **GDPR cleanup**: `IAttachmentStorage.deleteByConversation()` removes all attachments when a conversation is deleted.