mobiuslabsgmbh
/

Mixtral-8x7B-Instruct-v0.1-hf-attn-4bit-moe-2bit-metaoffload-HQQ

Text Generation

Mixture of Experts

Model card Files Files and versions Community

not run in colab t4

#4

by rakmik - opened Feb 6

rakmik

Feb 6

https://github.com/werruww/hqq-/blob/main/%D9%92%D9%92%D9%92%D9%92%D9%92%D9%92XXXXXXXXXXXXXXXXXXXX.ipynb

rakmik

Feb 6

https://github.com/huggingface/transformers/pull/29637

how to use device_map="auto"

rakmik

Feb 6

it run in kagel
32vram
32 ram

not run in colab t4
16vram
12 ram

rakmik

Feb 6

thank you
mobiuslabsgmbh

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment