Multimodal

Overview

Many models now support multimodal inputs, including images, audio, video, and PDFs. This article describes how to how to create evaluations that include these data types.

The following providers currently have support for multimodal inputs:

Provider	Images	Audio	Video	PDF
OpenAI	•	•		•
Anthropic	•			•
Google	•	•	•	•
Mistral	•	•		•
Grok	•
Bedrock	•
AzureAI	•
Groq	•

Note that model providers only support multimodal inputs for a subset of their models. In the sections below on images, audio, and video we’ll enumerate which models can handle these input types. It’s also always a good idea to check the provider documentation for the most up to date compatibility matrix.

Some OpenAI and Google models additionally support Multimodal Output.

Media References and Authority

Paths and URLs in the fixed task dataset—including a SampleSource’s initial_samples() seed—are materialized before they enter the sample’s runtime message state. The dataset seed is therefore trusted configuration when it contains non-inline media references.

This is an authority decision, not a safety check. Fixed-dataset HTTP references follow redirects and are not screened for private destination addresses, ports, or response size. A third-party dataset can therefore make the evaluator access destinations named by the dataset or its redirects. Review untrusted datasets or run them with suitable host network and filesystem isolation. Hardening this trusted fetch path is tracked separately from the runtime media policy.

WarningRuntime Media Must Be Inline

Media introduced later by a tool, solver, dynamically-created task, or SampleSource (next_samples(), sample_complete(), or enqueue_sample()) must already be a data URL. The same requirement applies to generate(), count_tokens(), and compact(). Sandboxed agent bridges enforce the same rule.

Trusted runtime code can explicitly convert an authorized path or URL with await materialize_media(reference). Do not call materialize_media() on model- or tool-controlled references unless filesystem or network access is intentionally authorized.

In-process bridges explicitly materialize non-inline media before model submission; they already share the evaluator’s filesystem and network. Trusted runtime code can materialize media in the same way:

from inspect_ai.model import ContentImage
from inspect_ai.util import materialize_media

image = ContentImage(
    image=await materialize_media("picture.png")
)

materialize_media() can read files, fetch URLs, and invoke configured media resolvers. Model media data URLs may declare a compatible MIME type (for example, data:image/png;base64,...). When it is omitted, Inspect uses an image signature or PNG compatibility default, the format of audio and video content, or a document’s mime_type. HTTP materialization follows redirects and rejects error responses before their bodies are used. For extensionless media, Inspect uses a specific response MIME type, content metadata, or a bounded signature check for common image formats.

Extension Migration

The public agent_bridge() context manager continues to materialize non-inline media because an in-process scaffold already shares host access. Code that constructs AgentBridge directly must now pass allow_remote_media=True to make the same grant explicit.

Task-resolution helpers default to input_media_policy="inline_only". An embedder that calls resolve_tasks() or eval_resolve_tasks() at a genuine pre-run boundary can preserve fixed-dataset path and URL handling by passing input_media_policy="trusted_pre_run". Do not use that policy for tasks created during an eval.

See Media Reference Authority for the full threat model, source table, and residual risks.

Images

Please see provider specific documentation on which models support image input:

• OpenAI Images and Vision
• Anthropic Vision
• Gemni Image Understanding
• Mistral Vision
• Grok Image Understanding

To include an image in a dataset you should use JSON input format (either standard JSON or JSON Lines). For example, here we include an image alongside some text content:

"input": [
  {
    "role": "user",
    "content": [
        { "type": "image", "image": "picture.png"},
        { "type": "text", "text": "What is this a picture of?"}
    ]
  }
]

The "picture.png" path is resolved relative to the directory containing the dataset file. The image can be specified either as a file path or a base64 encoded Data URL.

If you are constructing chat messages programmatically, then the equivalent to the above would be:

input = [
    ChatMessageUser(content = [
        ContentImage(image="picture.png"),
        ContentText(text="What is this a picture of?")
    ])
]

Detail

Some providers support a detail option that control over how the model processes the image and generates its textual understanding. Valid options are auto (the default), low, and high. See the Open AI documentation for more information on using this option. The Mistral, AzureAI, and Groq APIs also support the detail parameter. For example, here we explicitly specify image detail:

ContentImage(image="picture.png", detail="low")

Audio

The following models currently support audio inputs:

• Open AI: gpt-4o-audio-preview
• Google: All Gemini models
• Mistral: All Voxtral models

To include audio in a dataset you should use JSON input format (either standard JSON or JSON Lines). For example, here we include audio alongside some text content:

"input": [
  {
    "role": "user",
    "content": [
        { "type": "audio", "audio": "sample.mp3", "format": "mp3" },
        { "type": "text", "text": "What words are spoken in this audio sample?"}
    ]
  }
]

The “sample.mp3” path is resolved relative to the directory containing the dataset file. The audio file can be specified either as a file path or a base64 encoded Data URL.

If you are constructing chat messages programmatically, then the equivalent to the above would be:

input = [
    ChatMessageUser(content = [
        ContentAudio(audio="sample.mp3", format="mp3"),
        ContentText(text="What words are spoken in this audio sample?")
    ])
]

Formats

You can provide audio files in one of two formats:

• MP3
• WAV

As demonstrated above, you should specify the format explicitly when including audio input.

Video

The following models currently support video inputs:

• Google: All Gemini models.

To include video in a dataset you should use JSON input format (either standard JSON or JSON Lines). For example, here we include video alongside some text content:

"input": [
  {
    "role": "user",
    "content": [
        { "type": "video", "video": "video.mp4", "format": "mp4" },
        { "type": "text", "text": "Can you please describe the attached video?"}
    ]
  }
]

The “video.mp4” path is resolved relative to the directory containing the dataset file. The video file can be specified either as a file path or a base64 encoded Data URL.

If you are constructing chat messages programmatically, then the equivalent to the above would be:

input = [
    ChatMessageUser(content = [
        ContentVideo(video="video.mp4", format="mp4"),
        ContentText(text="Can you please describe the attached video?")
    ])
]

Formats

You can provide video files in one of three formats:

• MP4
• MPEG
• MOV

As demonstrated above, you should specify the format explicitly when including video input.

PDF

The following model providers support PDF inputs:

• OpenAI
• Anthropic
• Google
• Mistral

To include PDF in a dataset you should use JSON input format (either standard JSON or JSON Lines). For example, here we include a PDF alongside some text content:

"input": [
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "Please describe the contents of the attached PDF."
      },
      {
        "type": "document",
        "document": "attention.pdf"
      }
    ]
  }
]

The “attention.pdf” path is resolved relative to the directory containing the dataset file. The video file can be specified either as a file path or a base64 encoded Data URL.

If you are constructing chat messages programmatically, then the equivalent to the above would be:

input = [
    ChatMessageUser(content=[
         ContentText(text="Please describe the contents of the attached PDF."),
        ContentDocument(document="attention.pdf")
    ])
]

Output

Some models can generate multimodal output along with text:

• OpenAI gpt-4o and gpt-5 models support image generation

• Google models gemini-2.5-flash-image, gemini-3-pro-image-preview, and gemini-3.1-flash-image-preview support image generation.

Enable image output by setting modalities=["image"] in your GenerateConfig:

config = GenerateConfig(modalities=["image"])

Text output is always implicitly included—you only need to specify additional modalities beyond text.

OpenAI

Image generation uses gpt-image-1 / gpt-image-1.5 under the hood (you can custmize this using ImageOutput options).

model = get_model("openai/gpt-5.4")
output = await model.generate(
    input=[ChatMessageUser(content="Generate an image of a sunset")],
    config=GenerateConfig(modalities=["image"]),
)

For more control over image generation, use ImageOutput with provider-specific options:

from inspect_ai.model import ImageOutput

config = GenerateConfig(modalities=[
    ImageOutput(options={
        "openai": {
            "quality": "high", 
            "size": "1024x1024", 
            "output_format": "png",
            "model": "gpt-image-1.5"
        }
    })
])

Google

model = get_model("google/gemini-3.1-flash-image-preview")
output = await model.generate(
    input=[ChatMessageUser(content="Generate an image of a sunset")],
    config=GenerateConfig(modalities=["image"]),
)

Response Format

Image output appears as ContentImage in the assistant message’s content list, with a data:image/png;base64,... data URI:

for content in output.choices[0].message.content:
    if isinstance(content, ContentImage):
        # content.image contains a data URI like "data:image/png;base64,..."
        pass

Uploads

When using audio and video with the Google Gemini API, media is first uploaded using the File API and then the URL to the uploaded file is referenced in the chat message. This results in much faster performance for subsequent uses of the media file.

The File API lets you store up to 20GB of files per project, with a per-file maximum size of 2GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. The File API is available at no cost in all regions where the Gemini API is available.

Logging

By default, inline media bytes are retained in the log file. Media logging does not fetch unresolved runtime paths or URLs. Media file logging will not create performance problems when using .eval logs, however if you are using .json logs then large numbers of media files could become unwieldy (i.e. if your .json log file grows to 100MB or larger as a result).

You can disable all media logging using the --no-log-images flag. For example, here we enable the .json log format and disable media logging:

inspect eval images.py --log-format=json --no-log-images

This controls retention only. Trusted media in the fixed task input is still materialized for submission to the model, then removed from the completed log.

You can also use the INSPECT_EVAL_LOG_IMAGES environment variable to set a global default in your .env configuration file.