Skip to content

Multimodal Requests Not Supported #48

Closed
@JuanmaBM

Description

@JuanmaBM

Currently, the simulator does not support multimodal request payloads, as used by frameworks like Llama Stack or any OpenAI-compatible client that sends structured content blocks (e.g., text, image_url, video_url).

These clients send a content array of objects instead of a plain string. When such a request is received, the simulator fails to parse the body, expecting content to be of type string, not []object.

🔁 Example Request (Valid OpenAI format)

curl -k -X POST http://localhost:8000/v1/chat/completions \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "inference-simulator-maas",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Can you describe this image?" },
          { "type": "image_url", "url": "https://example.com/image.png" }
        ]
      }
    ]
  }'

❌ Current Result

E0604 09:16:14.284161       1 simulator.go:264] "failed to read and parse request body" err="json: cannot unmarshal array into Go struct field message.messages.content of type string"

✅ Expected Behavior

The simulator should:

  • Accept content as an array of typed blocks (type: "text", type: "image_url", etc.)
  • Gracefully handle or skip unsupported types (e.g., images or videos) if not yet implemented
  • Provide fallback or mock behavior where needed for inference simulation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions