torch compile compatibility issue
#23
by
jhmun
- opened
trying to do inference with vllm, getting this error:
[config.py:3785] torch.compile is turned on, but the model meta-llama/Llama-4-Maverick-17B-128E-Instruct does not support it.
Anyone to address this issue?
Below is the cmd that I tried.
VLLM_DISABLE_TORCH_COMPILE=1 vllm serve {Llama-4-Maverick-17B-128E-Instruct} --tensor-parallel-size 8 --max-model-len 12000 --disable_custom_all_reduce --limit_mm_per_prompt="image=10" --enforce_eager
I am getting same issue even with VLLM_DISABLE_TORCH_COMPILE=1. The Llama4 Scout is working well with similar cmd. It's weird!
Any workaround? Thanks!
The warning is erroneous, if the cmd fails for you there is likely something else that is wrong. Can you scroll up and see what else the logs are saying?