Using Vision Language Models

Aphrodite provides experimental support for Vision Language Models (VLMs). See the list of supported VLMs here. This document shows you how to run and serve these models using Aphrodite.

We are actively working on improving the VLM support in Aphrodite. Expect breaking changes in the future without any deprecation warnings.

Currently, the support for VLMs has the following limitation:

• Only single image input is supported per text prompt.

We are continuously improving user & developer experience. If you have any feedback or feature requests, please open an issue.

Offline Batched Inference

To initialize a VLM, the aforementioned arguments must be passed to the LLM class for instantiating the engine.

llm = LLM(model="llava-hf/llava-1.5-7b-hf")

To pass an image to the model, note the following in aphrodite.inputs.PromptInputs:

• prompt: The prompt should follow the format that is documented on Hugging Face.
• multi_modal_data: This is a dictionary that follows the schema defined in aphrodite.multimodal.MultiModalDataDict

# Refer to the HuggingFace repo for the correct format to use
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

# Load the image using PIL.Image
image = PIL.Image.open(...)

# Single prompt inference
outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {"image": image},
})

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

# Batch inference
image_1 = PIL.Image.open(...)
image_2 = PIL.Image.open(...)
outputs = llm.generate(
    [
        {
            "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
            "multi_modal_data": {"image": image_1},
        },
        {
            "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
            "multi_modal_data": {"image": image_2},
        }
    ]
)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

Online OpenAI Vision API Inference

You can serve vision language models with Aphrodite’s OpenAI server.

Tip

Currently, Aphrodite supports only single image_url input per messages. Support for multi-image inputs will be added in the future.

Below is an example on how to launch the same llava-hf/llava-1.5-7b-hf with Aphrodite API server.

Danger

Since OpenAI Vision API is based on Chat API, a chat template is required to launch the API server if the model’s tokenizer does not come with one. In this example, we use the HuggingFace Llava chat template that you can find in the example folder here.

aphrodite run llava-hf/llava-1.5-7b-hf --chat-template llava.jinja

To send a request to the server, you can use the following code:

from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:2242/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
chat_response = client.chat.completions.create(
    model="llava-hf/llava-1.5-7b-hf",
    messages=[{
        "role": "user",
        "content": [
            # NOTE: The prompt formatting with the image token `<image>` is not needed
            # since the prompt will be processed automatically by the API server.
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
                },
            },
        ],
    }],
)
print("Chat response:", chat_response)

Tip

By default, the timeout for fetching images through http url is 10 seconds. You can override this by setting this env variable:

export APHRODITE_IMAGE_FETCH_TIMEOUT=<timeout>

Video Decoding Backend

Aphrodite decodes video bytes into frames using a selectable decoding backend. The following decoding backends are supported:

• opencv (default): OpenCV-based decoder.
• pyav: PyAV decoder.
• torchcodec: TorchCodec (PyTorch-native) decoder.
• pynvvideocodec: NVIDIA NVDEC-based decoder.
• deepstream: NVIDIA DeepStream (NVDEC) GPU decoder.

The CPU backends are backed by FFmpeg. torchcodec lets you choose which FFmpeg version is used, while opencv and pyav rely on whichever FFmpeg build they were linked against.

Select the codec backend by passing backend via --media-io-kwargs:

aphrodite run Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"backend": "torchcodec"}}'

TorchCodec-specific parameters:

• num_ffmpeg_threads: Number of FFmpeg decoding threads. 0 (default) uses the FFmpeg default.
• seek_mode: Seek mode for the decoder. "exact" (default) guarantees frame-accurate sampling. "approximate" skips the initial scan and relies on the file’s metadata.

aphrodite run Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"backend": "torchcodec", "seek_mode": "approximate", "num_ffmpeg_threads": 4}}'

PyNvVideoCodec-specific parameters:

• hw_decoders: Maximum number of concurrent hardware decoder slots retained by each API server process. It must be a positive integer and defaults to 2, which is the recommended starting point for concurrent video workloads. Because Aphrodite reserves GPU memory for these slots at startup, this value cannot be overridden per request. Benchmark before increasing it because each additional slot increases the GPU memory reservation.

# Example: explicitly use the recommended 2 hardware decoders
aphrodite run Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"backend": "pynvvideocodec", "hw_decoders": 2}}'

GPU Video Decoding with DeepStream (NVDEC)

By default Aphrodite decodes video on the CPU. On NVIDIA GPUs you can instead decode directly on the hardware video engine (NVDEC) with the DeepStream backend, which keeps decoding off the CPU and can significantly increase video throughput.

Install the backend (Linux x86-64 only):

pip install aphrodite-engine[deepstream]

The pip wheel bundles the DeepStream libraries but still relies on a few system packages that pip cannot install. On Ubuntu:

apt-get install -y \
  gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad gstreamer1.0-libav \
  python3-gi python3-gst-1.0 libv4l-0 cuda-libraries-13-0

Select the backend either with an environment variable:

export APHRODITE_VIDEO_LOADER_BACKEND=deepstream
aphrodite run Qwen/Qwen3-VL-30B-A3B-Instruct

or per request via --media-io-kwargs:

aphrodite run Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"backend": "deepstream"}}'

DeepStream-specific parameters:

• pool_size: Number of GPU decode workers in the process-wide decode pool (clamped to [1, 16]). When unset it defaults to APHRODITE_MEDIA_LOADING_THREAD_COUNT (default 8). The pool is a singleton, so the first request’s value wins.

aphrodite run Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"backend": "deepstream", "pool_size": 12}}'

Tip

There is no need to format the prompt in the API request since it’ll be handled by the server.

Here’s a curl example:

curl -X POST "http://localhost:2242/v1/chat/completions" -H "Content-Type: application/json" \
    -H "Authorization : Bearer $OPENAI_API_KEY" \
    -d '{
        "model": "llava-hf/llava-1.5-7b-hf",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "What's in this image?"},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
                        },
                    },
                ],
            }
        ]
    }'