Unverified Commit 120ae649 authored by Indrajit Bhosale's avatar Indrajit Bhosale Committed by GitHub
Browse files

fix: llama4 vllm agg multimodal script (#6103)

parent df8fd92b
...@@ -224,7 +224,7 @@ flowchart LR ...@@ -224,7 +224,7 @@ flowchart LR
```bash ```bash
cd $DYNAMO_HOME/examples/backends/vllm cd $DYNAMO_HOME/examples/backends/vllm
bash launch/agg_multimodal_llama.sh bash launch/agg_multimodal.sh --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
``` ```
**Client:** **Client:**
......
...@@ -58,6 +58,8 @@ if [[ "$MODEL_NAME" == "Qwen/Qwen2.5-VL-7B-Instruct" ]]; then ...@@ -58,6 +58,8 @@ if [[ "$MODEL_NAME" == "Qwen/Qwen2.5-VL-7B-Instruct" ]]; then
MODEL_SPECIFIC_ARGS="--gpu-memory-utilization 0.85 --max-model-len 4096" MODEL_SPECIFIC_ARGS="--gpu-memory-utilization 0.85 --max-model-len 4096"
elif [[ "$MODEL_NAME" == "llava-hf/llava-1.5-7b-hf" ]]; then elif [[ "$MODEL_NAME" == "llava-hf/llava-1.5-7b-hf" ]]; then
MODEL_SPECIFIC_ARGS="--gpu-memory-utilization 0.85 --max-model-len 4096" MODEL_SPECIFIC_ARGS="--gpu-memory-utilization 0.85 --max-model-len 4096"
elif [[ "$MODEL_NAME" == "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8" ]]; then
MODEL_SPECIFIC_ARGS="--tensor-parallel-size=8 --gpu-memory-utilization 0.85 --max-model-len=108960"
fi fi
# Start vLLM worker with vision model # Start vLLM worker with vision model
......
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
set -ex
trap 'echo Cleaning up...; kill 0' EXIT
MODEL_NAME="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8"
# run ingress
# dynamo.frontend accepts either --http-port flag or DYN_HTTP_PORT env var (defaults to 8000)
python -m dynamo.frontend &
# run processor
python -m dynamo.vllm --multimodal-processor --enable-multimodal --model $MODEL_NAME &
# Llama 4 doesn't support image embedding input, so use encode+prefill worker
# that handles image encoding inline
python -m dynamo.vllm --multimodal-encode-prefill-worker --enable-multimodal --model $MODEL_NAME --tensor-parallel-size=8 --max-model-len=208960 --gpu-memory-utilization 0.80 &
# Wait for all background processes to complete
wait
...@@ -212,7 +212,7 @@ flowchart LR ...@@ -212,7 +212,7 @@ flowchart LR
```bash ```bash
cd $DYNAMO_HOME/examples/backends/vllm cd $DYNAMO_HOME/examples/backends/vllm
bash launch/agg_multimodal_llama.sh bash launch/agg_multimodal.sh --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
``` ```
**Client:** **Client:**
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment