Multimodal Attachments
EDDI's attachment pipeline enables multimodal conversations — users can send images, files, and documents alongside text input. Attachments flow through the lifecycle pipeline and are automatically forwarded to vision-capable LLMs.
Quick Start
Send an Image via URL
POST /agents/{conversationId}
Content-Type: application/json
{
"input": "What is in this image?",
"context": {
"attachment_0": {
"type": "object",
"value": {
"mimeType": "image/png",
"url": "https://example.com/photo.png",
"fileName": "photo.png"
}
}
}
}Send an Image via Base64
The image is automatically forwarded to the LLM as multimodal content. The LLM "sees" the image alongside the text message.
How It Works
Pipeline Stages
Context Extraction —
AttachmentContextExtractorparsesattachment_*context keys intoAttachmentobjectsMemory Storage — Attachments are stored as
List<Attachment>in theattachmentsmemory keyRule Matching —
ContentTypeMatchercondition matches on MIME types for routingLLM Forwarding —
AttachmentForwarderresolves each attachment's bytes, gates it onModelCapabilityService, and converts it to the right langchain4jContenton the outgoing user message
Input Paths
Path A: URL Reference (Recommended)
Best for images already hosted somewhere. The LLM provider fetches the image directly from the URL.
Path B: Base64 Inline
Best for small images (< 5MB). Data is sent inline as a base64-encoded string.
Note:
base64Datais transient — it's never persisted to MongoDB. For large files, use the upload endpoint (Path C below) or URL references.
Path C: File Upload
For large files, upload them to the storage backend and receive a storage reference:
Response (201):
The returned storageRef can then be used in subsequent conversation turns by setting it as the storageRef field in an attachment context key. It takes precedence over url and data, and the authoritative mimeType and sizeBytes are resolved server-side from the store, so they need not be supplied:
The storage backend (GridFS or PostgreSQL) is selected automatically based on the configured datastore.
201
File stored successfully
400
No file provided, file exceeds eddi.attachments.max-size-bytes, or the store rejected the file
500
Storage or I/O error
Context Key Format
Attachment context keys must match the pattern attachment_*:
attachment_0
✅
attachment_screenshot
✅
attachment_
✅
image_0
❌ (wrong prefix)
attachment
❌ (no suffix)
Required Fields
storageRef
One of storageRef/url/data
Reference to a previously uploaded blob (see Path C)
mimeType
Yes, unless storageRef is used
MIME type (e.g., image/png, application/pdf)
url
One of storageRef/url/data
External URL reference
data
One of storageRef/url/data
Base64-encoded content
fileName
No
Original filename (for metadata/logging)
storageRef has the highest precedence; if both url and data are present, url takes precedence.
Multiple Attachments
Send multiple attachments by incrementing the key index:
All attachments are forwarded to the LLM in a single multimodal user message.
LLM Multimodal Support
AttachmentForwarder is the single place an attachment becomes langchain4j Content on the outgoing user message. For each attachment it resolves the bytes from whichever source supplied them (stored blob, URL, inline base64) under uniform per-file and aggregate byte caps, asks ModelCapabilityService what the configured provider/model can actually accept, and emits accordingly:
image/*
ImageContent — the URL is passed through when the provider fetches URLs itself, otherwise the bytes are downloaded and inlined as base64
Metadata note
application/pdf
Native PdfFileContent
PDFBox text extraction, inlined as TextContent
audio/*
AudioContent
Metadata note
text-like (text/*, JSON, XML, CSV, YAML)
Decoded and inlined as TextContent — no capability required, so this works on every model
—
Anything else
Metadata note pointing the model at the readAttachment tool
Metadata note
Note the difference between the last two rows: a CSV is read into the prompt, whereas a .zip is only announced. The metadata note looks like this:
Nothing is dropped silently. Text extracted from a PDF is persisted to the attachments:extracts memory key so later turns can stitch it back into the history, and every drop, cap-skip and capability gate is appended to attachments:errors — which is where to look first when a model claims it cannot see a file you attached.
Observability
eddi.attachment.forwarded
Counter
Attachments converted to Content
eddi.attachment.reinlined
Counter
Extracted text stitched back into conversation history
eddi.attachment.errors
Counter
Drops, cap-skips and capability gates
Routing with Behavior Rules
Use contentTypeMatcher to create different workflows based on attachment type:
Route Images to Vision Agent
Route PDFs to Document Processor
Require Specific Attachment Count
Template Access
{memory.current.attachments} renders as an empty string. Attachments are written to the current step's data store under the attachments memory key (storeData), while {memory.current.*} resolves against the step's ConversationOutput — a different map.
The attachment_* context keys are a different matter: MemoryItemConverter publishes the request context into the template model, so {context.attachment_0.mimeType}, {context.attachment_0.url} and {context.attachment_0.fileName} do resolve for attachments supplied that way.
To act on attachments, read the attachments memory key from a task, or match on them declaratively with contentTypeMatcher in a behavior rule.
Group Conversations
Group discussions accept the same three input shapes on POST /groups/{groupId}/conversations. Inline base64Data is stored in the blob store owned by the group conversation; hosted url references and pre-uploaded storageRefs pass through.
Each member's private conversation is granted access on its first turn and receives the files as attachment_* context; from there everything on this page applies unchanged. Later turns rely on the member's own history plus the auto-enabled readAttachment tool, so a member does not lose the file after phase one.
Two group-specific bounds: the per-turn cap (eddi.attachments.max-per-turn) applies per member turn, and anything dropped is reported in that member's attachments:errors, not in the group transcript. See group-conversations.md → Attachments.
Architecture Notes
No inline storage: Attachment payloads are never stored inline in conversation memory documents. Only metadata references are persisted.
Transient base64: The
base64Datafield istransient— it exists only during the pipeline turn. For persistence, use the upload endpoint withIAttachmentStore.DB-agnostic: The
IAttachmentStoreSPI supports MongoDB (GridFS) and PostgreSQL (bytea) implementations.GDPR cleanup:
IAttachmentStore.deleteByConversation()removes all attachments when a conversation is deleted.
Last updated
Was this helpful?