unsloth

/

Qwen2.5-VL-32B-Instruct-unsloth-bnb-4bit

like 8

Follow

Unsloth AI 4.83k

Hi, how to run this model using vLLM?

v1 engine return error that bitsandbytes is not supported in v1

v0 returns error
docker run --runtime nvidia --gpus all -d --name vllm-Qwen-VL-32b-v2 --restart unless-stopped -v ~/.cache/huggingface:/root/.cache/huggingface -e VLLM_USE_V1=0 -p 8000:8000 vllm/vllm-openai:v0.8.2 --model unsloth/Qwen2.5-VL-32B-Instruct-unsloth-bnb-4bit --dtype auto --served-model-name llm --max-model-len 27740 --max-num-batched-tokens 27740

INFO 03-27 14:06:32 [weight_utils.py:265] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
ERROR 03-27 14:06:33 [engine.py:448] 'layers.54.mlp.down_proj.weight.absmax'
ERROR 03-27 14:06:33 [engine.py:448] Traceback (most recent call last):
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
ERROR 03-27 14:06:33 [engine.py:448] engine = MQLLMEngine.from_vllm_config(
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
ERROR 03-27 14:06:33 [engine.py:448] return cls(
ERROR 03-27 14:06:33 [engine.py:448] ^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
ERROR 03-27 14:06:33 [engine.py:448] self.engine = LLMEngine(*args, **kwargs)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 280, in init
ERROR 03-27 14:06:33 [engine.py:448] self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
ERROR 03-27 14:06:33 [engine.py:448] self._init_executor()
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
ERROR 03-27 14:06:33 [engine.py:448] self.collective_rpc("load_model")
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-27 14:06:33 [engine.py:448] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2255, in run_method
ERROR 03-27 14:06:33 [engine.py:448] return func(*args, **kwargs)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 183, in load_model
ERROR 03-27 14:06:33 [engine.py:448] self.model_runner.load_model()
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1113, in load_model
ERROR 03-27 14:06:33 [engine.py:448] self.model = get_model(vllm_config=self.vllm_config)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
Process SpawnProcess-1:
ERROR 03-27 14:06:33 [engine.py:448] return loader.load_model(vllm_config=vllm_config)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 444, in load_model
ERROR 03-27 14:06:33 [engine.py:448] loaded_weights = model.load_weights(
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1113, in load_weights
ERROR 03-27 14:06:33 [engine.py:448] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
ERROR 03-27 14:06:33 [engine.py:448] autoloaded_weights = set(self._load_module("", self.module, weights))
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
ERROR 03-27 14:06:33 [engine.py:448] yield from self._load_module(prefix,
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 173, in _load_module
ERROR 03-27 14:06:33 [engine.py:448] loaded_params = module_load_weights(weights)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 490, in load_weights
ERROR 03-27 14:06:33 [engine.py:448] return loader.load_weights(weights)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
ERROR 03-27 14:06:33 [engine.py:448] autoloaded_weights = set(self._load_module("", self.module, weights))
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
ERROR 03-27 14:06:33 [engine.py:448] yield from self._load_module(prefix,
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 173, in _load_module
ERROR 03-27 14:06:33 [engine.py:448] loaded_params = module_load_weights(weights)
ERROR 03-27 14:06:33 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 400, in load_weights
ERROR 03-27 14:06:33 [engine.py:448] param = params_dict[name]
ERROR 03-27 14:06:33 [engine.py:448] ~~~~~~~~~~~^^^^^^
ERROR 03-27 14:06:33 [engine.py:448] KeyError: 'layers.54.mlp.down_proj.weight.absmax'