Multimodal Attachments

EDDI's attachment pipeline enables multimodal conversations — users can send images, files, and documents alongside text input. Attachments flow through the lifecycle pipeline and are automatically forwarded to vision-capable LLMs.

Quick Start

Send an Image via URL

POST /agents/{conversationId}/say?message=What%20is%20in%20this%20image?
Content-Type: application/json

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "url": "https://example.com/photo.png",
      "fileName": "photo.png"
    }
  }
}

Send an Image via Base64

POST /agents/{conversationId}/say?message=Describe%20this%20icon
Content-Type: application/json

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAAAAE...",
      "fileName": "icon.png"
    }
  }
}

The image is automatically forwarded to the LLM as multimodal content. The LLM "sees" the image alongside the text message.

---

How It Works

Client sends context with attachment_* keys
           │
           ▼
┌──────────────────────────────────┐
│  Conversation.prepareLifecycleData()  │
│                                       │
│  AttachmentContextExtractor parses    │
│  attachment_0, attachment_1, ...      │
│  into List<Attachment> objects        │
│                                       │
│  Stored in memory: "attachments"      │
└──────────────┬───────────────────┘
               │
    ┌──────────┼──────────┐
    ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────────┐
│BehaviorRules│ │LlmTask│ │Other Tasks │
│             │ │       │ │            │
│ContentType- │ │Multi- │ │Read from   │
│Matcher      │ │modal  │ │memory key  │
│condition    │ │Message│ │"attachments"│
│             │ │Enhancer││            │
└────────┘ └────────┘ └────────────┘

Pipeline Stages

1. Context Extraction — AttachmentContextExtractor parses attachment_* context keys into Attachment objects

2. Memory Storage — Attachments are stored as List<Attachment> in the attachments memory key

3. Rule Matching — ContentTypeMatcher condition matches on MIME types for routing

4. LLM Forwarding — MultimodalMessageEnhancer converts attachments to langchain4j ImageContent and enhances the user message

---

Input Paths

Path A: URL Reference (Recommended)

Best for images already hosted somewhere. The LLM provider fetches the image directly from the URL.

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/jpeg",
      "url": "https://cdn.example.com/photos/sunset.jpg",
      "fileName": "sunset.jpg"
    }
  }
}

Path B: Base64 Inline

Best for small images (< 5MB). Data is sent inline as a base64-encoded string.

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk...",
      "fileName": "icon.png"
    }
  }
}

Note: base64Data is transient — it's never persisted to MongoDB. For large files, use the upload endpoint (Path C below) or URL references.

Path C: File Upload

For large files, upload them to the storage backend and receive a storage reference:

POST /conversations/{conversationId}/attachments
Content-Type: multipart/form-data

# Form field: file (the binary file)

Response (201):

{
  "storageRef": "gridfs://68abc123def456",
  "fileName": "report.pdf",
  "mimeType": "application/pdf",
  "sizeBytes": 524288
}

The returned storageRef can then be used in subsequent conversation turns by setting it as the url in an attachment context key. The storage backend (GridFS or PostgreSQL) is selected automatically based on the configured datastore.

Response Code	Meaning
201	File stored successfully
400	No file provided
503	No attachment storage configured

---

Context Key Format

Attachment context keys must match the pattern attachment_*:

Key	Valid?
attachment_0	✅
attachment_screenshot	✅
attachment_	✅
image_0	❌ (wrong prefix)
attachment	❌ (no suffix)

Required Fields

Field	Required	Description
mimeType	Yes	MIME type (e.g., image/png, application/pdf)
url	One of url/data	External URL reference
data	One of url/data	Base64-encoded content
fileName	No	Original filename (for metadata/logging)

If both url and data are present, url takes precedence.

---

Multiple Attachments

Send multiple attachments by incrementing the key index:

{
  "attachment_0": {
    "type": "object",
    "value": { "mimeType": "image/png", "url": "https://example.com/page1.png" }
  },
  "attachment_1": {
    "type": "object",
    "value": { "mimeType": "image/png", "url": "https://example.com/page2.png" }
  }
}

All attachments are forwarded to the LLM in a single multimodal user message.

---

LLM Multimodal Support

The MultimodalMessageEnhancer automatically converts attachments to the appropriate langchain4j content type:

MIME Type	langchain4j Content	Provider Support
image/*	ImageContent	OpenAI GPT-4o, Gemini, Claude 3, Ollama (LLaVA)
application/pdf	Metadata text (future: PdfFileContent)	Gemini
audio/*	Metadata text (future: AudioContent)	Gemini
Other	Metadata text description	All (text-only)

For unsupported MIME types, a text description is injected so the LLM knows an attachment was present:

[Attachment: report.csv (text/csv, 15240 bytes)]

---

Routing with Behavior Rules

Use contentTypeMatcher to create different workflows based on attachment type:

Route Images to Vision Agent

{
  "name": "Image received",
  "actions": ["analyze_image"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "image/*",
        "minCount": "1"
      }
    }
  ]
}

Route PDFs to Document Processor

{
  "name": "Document received",
  "actions": ["process_document"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "application/pdf",
        "minCount": "1"
      }
    }
  ]
}

Require Specific Attachment Count

{
  "name": "Comparison ready",
  "actions": ["compare_images"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "image/*",
        "minCount": "2"
      }
    }
  ]
}

---

Template Access

Attachments are available in templates via the memory namespace:

Current step attachments: {{memory.current.attachments}}

This can be useful for logging, debugging, or constructing custom prompts that reference attachment metadata.

---

Architecture Notes

• No inline storage: Attachment payloads are never stored inline in conversation memory documents. Only metadata references are persisted.

• Transient base64: The base64Data field is transient — it exists only during the pipeline turn. For persistence, use the upload endpoint with IAttachmentStorage.

• DB-agnostic: The IAttachmentStorage SPI supports MongoDB (GridFS) and PostgreSQL (bytea) implementations.

• GDPR cleanup: IAttachmentStorage.deleteByConversation() removes all attachments when a conversation is deleted.