Hugging Face's logo

unsloth
/
Meta-Llama-3.1-405B-bnb-4bit

Text Generation
Transformers
Safetensors
English
llama
llama-3
meta
facebook
unsloth
text-generation-inference
4-bit precision
bitsandbytes
Model card Files Community
1

Why did num_key_value_heads change from 16 to 8?

#1
by codys12 - opened
Discussion
codys12

Was the previous version of the model even usable or was it bugged out?

codys12

Problem after config change: RuntimeError: shape '[25, 1024, 8, 128]' is invalid for input of size 52428800

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment