Hugging Face's logo

tokyotech-llm
/
Llama-3.1-Swallow-8B-Instruct-v0.1

Text Generation
Transformers
Safetensors
English
Japanese
llama
conversational
text-generation-inference
Model card Files Community
Taishi-N324 commited on
Commit
9138aa9
·
verified ·
1 Parent(s): 97f791e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +6 -5
README.md CHANGED
@@ -66,16 +66,17 @@ The website [https://swallow-llm.github.io/](https://swallow-llm.github.io/) pro
66
  |---|---|---|---|---|---|---|---|---|---|---|
67
  | |4-shot|4-shot|4-shot|4-shot|4-shot|5-shot|4-shot|3-shot|0-shot| |
68
  | |Acc|EM acc|Acc|EM acc|Acc|Acc|EM acc|CoT EM Acc|pass@1| |
69
- | Qwen2-7B-Instruct | 0.4000 | 0.5468 | 0.6146 | 0.3518 | 0.8852 | 0.7073 | 0.6300 | 0.3101 | **0.6354** | 0.5646 |
70
- | Qwen2.5-7B-Instruct | **0.4280** | 0.5187 | 0.6240 | 0.2626 | 0.8761 | **0.7419** | **0.7415** | 0.2150 | 0.6360 | 0.5604 |
 
71
  | Tanuki-8B-dpo-v1.0 | 0.3340 | 0.2838 | 0.4696 | 0.2395 | 0.8168 | 0.3772 | 0.4867 | 0.3350 | 0.2805 | 0.4026 |
72
- | Llama 3 8B Instruct | 0.3880 | 0.6687 | 0.5834 | 0.3743 | 0.8903 | 0.6567 | 0.7453 | **0.6478** | 0.5415 | 0.6107 |
73
- | Llama 3.1 8B Instruct | 0.3700 | 0.6994 | 0.5920 | 0.3783 | **0.9037** | 0.6809 | 0.7430 | 0.6928 | 0.6293 | **0.6321** |
74
  | Llama 3 Youko 8B Instruct | 0.4080 | 0.6129 | 0.5983 | 0.3370 | 0.8981 | 0.5964 | 0.5618 | 0.4012 | 0.2750 | 0.5209 |
75
  | Llama-3-ELYZA-JP-8B | 0.3200 | 0.5502 | 0.5224 | 0.3631 | 0.8809 | 0.5875 | 0.5701 | 0.3213 | 0.4604 | 0.5084 |
76
  | Llama 3 heron brain 8B v0.3 | 0.3580 | 0.6563 | 0.5686 | 0.3726 | 0.9002 | 0.6213 | 0.5777 | 0.6409 | 0.3720 | 0.5631 |
77
  | Llama 3 Swallow 8B Instruct | 0.3720 | 0.6557 | 0.5861 | 0.3648 | 0.9002 | 0.6315 | 0.5959 | 0.6391 | 0.4238 | 0.5743 |
78
- | Llama 3.1 Swallow 8B Instruct | 0.3900 | **0.6488** | **0.6151** | **0.3553** | 0.8912 | 0.6237 | 0.6050 | 0.6417 | 0.3787 | 0.5722 |
79
 
80
  ## MT-Bench JA
81
 
 
66
  |---|---|---|---|---|---|---|---|---|---|---|
67
  | |4-shot|4-shot|4-shot|4-shot|4-shot|5-shot|4-shot|3-shot|0-shot| |
68
  | |Acc|EM acc|Acc|EM acc|Acc|Acc|EM acc|CoT EM Acc|pass@1| |
69
+ | RakutenAI-7B-chat | 0.4160 | 0.5971 | 0.6465 | 0.3091 | 0.8886 | 0.5757 | 0.3139 | 0.4958 | 0.2671 | 0.5011 |
70
+ | Qwen2-7B-Instruct | 0.4000 | 0.5468 | 0.6146 | 0.3518 | 0.8852 | 0.7073 | 0.6300 | 0.3101 | 0.6354 | 0.5646 |
71
+ | Qwen2.5-7B-Instruct | 0.4280 | 0.5187 | 0.6240 | 0.2626 | 0.8761 | 0.7419 | 0.7415 | 0.2150 | 0.6360 | 0.5604 |
72
  | Tanuki-8B-dpo-v1.0 | 0.3340 | 0.2838 | 0.4696 | 0.2395 | 0.8168 | 0.3772 | 0.4867 | 0.3350 | 0.2805 | 0.4026 |
73
+ | Llama 3 8B Instruct | 0.3880 | 0.6687 | 0.5834 | 0.3743 | 0.8903 | 0.6567 | 0.7453 | 0.6478 | 0.5415 | 0.6107 |
74
+ | Llama 3.1 8B Instruct | 0.3700 | 0.6994 | 0.5920 | 0.3783 | 0.9037 | 0.6809 | 0.7430 | 0.6928 | 0.6293 | 0.6321 |
75
  | Llama 3 Youko 8B Instruct | 0.4080 | 0.6129 | 0.5983 | 0.3370 | 0.8981 | 0.5964 | 0.5618 | 0.4012 | 0.2750 | 0.5209 |
76
  | Llama-3-ELYZA-JP-8B | 0.3200 | 0.5502 | 0.5224 | 0.3631 | 0.8809 | 0.5875 | 0.5701 | 0.3213 | 0.4604 | 0.5084 |
77
  | Llama 3 heron brain 8B v0.3 | 0.3580 | 0.6563 | 0.5686 | 0.3726 | 0.9002 | 0.6213 | 0.5777 | 0.6409 | 0.3720 | 0.5631 |
78
  | Llama 3 Swallow 8B Instruct | 0.3720 | 0.6557 | 0.5861 | 0.3648 | 0.9002 | 0.6315 | 0.5959 | 0.6391 | 0.4238 | 0.5743 |
79
+ | Llama 3.1 Swallow 8B Instruct | 0.3900 | 0.6488 | 0.6151 | 0.3553 | 0.8912 | 0.6237 | 0.6050 | 0.6417 | 0.3787 | 0.5722 |
80
 
81
  ## MT-Bench JA
82