What is the mininmum VRAM required to deploy this model?
I wanted to test the llama-4 model with this 4bit quantization on a on free T4 GPU (15GB VRAM) however I am running out of VRAM .
What is the minimum vram required under 4bit quantization?
At least 79GB VRAM
Thank you for the answer. Could you explain why this is the case?
Using the classic formula Min VRAM required: (4×P)/(32/Q)×1.2, 10.2GB VRAM should be enough.
Is it because of the MoE? Even though the inference uses 17b parameter, does the model still download all the parameters in the RAM?
Im running out of memory on 140gb..
Yes you need the full 109B parameters in VRAM (or even if you built a system which swapped in parameters from system memory as needed, it would potentially be swapping for every single token and thus would be extremely slow).
Im running out of memory on 140gb..
Really? Oh my god...
Im running out of memory on 140gb..
Really? Oh my god...
yup an H200 lol
I am using NVIDIA 1 L40 GPU with 48vRAM with lLLM docker container. for me also OutOfMemoryError
Torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.50 GiB. GPU 0 has a total capacity of 44.31 GiB of which 728.31 MiB is free. Process 2790767 has 43.59 GiB memory in use. Of the allocated memory 43.10 GiB is allocated by PyTorch, and 5.57 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.