danielhanchen commited on
Commit
d8c4425
·
verified ·
1 Parent(s): 79b08d0

Add files using upload-large-folder tool

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +1 -0
  2. README.md +364 -0
  3. added_tokens.json +28 -0
  4. chat_template.jinja +98 -0
  5. config.json +332 -0
  6. generation_config.json +14 -0
  7. merges.txt +0 -0
  8. model-00001-of-00048.safetensors +3 -0
  9. model-00002-of-00048.safetensors +3 -0
  10. model-00003-of-00048.safetensors +3 -0
  11. model-00004-of-00048.safetensors +3 -0
  12. model-00005-of-00048.safetensors +3 -0
  13. model-00006-of-00048.safetensors +3 -0
  14. model-00007-of-00048.safetensors +3 -0
  15. model-00008-of-00048.safetensors +3 -0
  16. model-00009-of-00048.safetensors +3 -0
  17. model-00010-of-00048.safetensors +3 -0
  18. model-00011-of-00048.safetensors +3 -0
  19. model-00012-of-00048.safetensors +3 -0
  20. model-00013-of-00048.safetensors +3 -0
  21. model-00014-of-00048.safetensors +3 -0
  22. model-00015-of-00048.safetensors +3 -0
  23. model-00016-of-00048.safetensors +3 -0
  24. model-00017-of-00048.safetensors +3 -0
  25. model-00018-of-00048.safetensors +3 -0
  26. model-00019-of-00048.safetensors +3 -0
  27. model-00020-of-00048.safetensors +3 -0
  28. model-00021-of-00048.safetensors +3 -0
  29. model-00022-of-00048.safetensors +3 -0
  30. model-00023-of-00048.safetensors +3 -0
  31. model-00024-of-00048.safetensors +3 -0
  32. model-00025-of-00048.safetensors +3 -0
  33. model-00026-of-00048.safetensors +3 -0
  34. model-00027-of-00048.safetensors +3 -0
  35. model-00028-of-00048.safetensors +3 -0
  36. model-00029-of-00048.safetensors +3 -0
  37. model-00030-of-00048.safetensors +3 -0
  38. model-00031-of-00048.safetensors +3 -0
  39. model-00032-of-00048.safetensors +3 -0
  40. model-00033-of-00048.safetensors +3 -0
  41. model-00034-of-00048.safetensors +3 -0
  42. model-00035-of-00048.safetensors +3 -0
  43. model-00036-of-00048.safetensors +3 -0
  44. model-00037-of-00048.safetensors +3 -0
  45. model-00038-of-00048.safetensors +3 -0
  46. model-00039-of-00048.safetensors +3 -0
  47. model-00040-of-00048.safetensors +3 -0
  48. model-00041-of-00048.safetensors +3 -0
  49. model-00042-of-00048.safetensors +3 -0
  50. model-00043-of-00048.safetensors +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,364 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - unsloth
4
+ library_name: transformers
5
+ license: apache-2.0
6
+ license_link: https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8/blob/main/LICENSE
7
+ pipeline_tag: text-generation
8
+ base_model:
9
+ - Qwen/Qwen3-235B-A22B-FP8
10
+ ---
11
+
12
+ # Qwen3-235B-A22B-FP8
13
+ <a href="https://chat.qwen.ai/" target="_blank" style="margin: 2px;">
14
+ <img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
15
+ </a>
16
+
17
+ ## Qwen3 Highlights
18
+
19
+ Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support, with the following key features:
20
+
21
+ - **Uniquely support of seamless switching between thinking mode** (for complex logical reasoning, math, and coding) and **non-thinking mode** (for efficient, general-purpose dialogue) **within single model**, ensuring optimal performance across various scenarios.
22
+ - **Significantly enhancement in its reasoning capabilities**, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning.
23
+ - **Superior human preference alignment**, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience.
24
+ - **Expertise in agent capabilities**, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks.
25
+ - **Support of 100+ languages and dialects** with strong capabilities for **multilingual instruction following** and **translation**.
26
+
27
+ ## Model Overview
28
+
29
+ This repo contains the FP8 version of **Qwen3-235B-A22B**, which has the following features:
30
+ - Type: Causal Language Models
31
+ - Training Stage: Pretraining & Post-training
32
+ - Number of Parameters: 235B in total and 22B activated
33
+ - Number of Paramaters (Non-Embedding): 234B
34
+ - Number of Layers: 94
35
+ - Number of Attention Heads (GQA): 64 for Q and 4 for KV
36
+ - Number of Experts: 128
37
+ - Number of Activated Experts: 8
38
+ - Context Length: 32,768 natively and [131,072 tokens with YaRN](#processing-long-texts).
39
+
40
+ For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our [blog](https://qwenlm.github.io/blog/qwen3/), [GitHub](https://github.com/QwenLM/Qwen3), and [Documentation](https://qwen.readthedocs.io/en/latest/).
41
+
42
+ ## Quickstart
43
+
44
+ The code of Qwen3-MoE has been in the latest Hugging Face `transformers` and we advise you to use the latest version of `transformers`.
45
+
46
+ With `transformers<4.51.0`, you will encounter the following error:
47
+ ```
48
+ KeyError: 'qwen3_moe'
49
+ ```
50
+
51
+ The following contains a code snippet illustrating how to use the model generate content based on given inputs.
52
+ ```python
53
+ from transformers import AutoModelForCausalLM, AutoTokenizer
54
+
55
+ model_name = "Qwen/Qwen3-235B-A22B-FP8"
56
+
57
+ # load the tokenizer and the model
58
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
59
+ model = AutoModelForCausalLM.from_pretrained(
60
+ model_name,
61
+ torch_dtype="auto",
62
+ device_map="auto"
63
+ )
64
+
65
+ # prepare the model input
66
+ prompt = "Give me a short introduction to large language model."
67
+ messages = [
68
+ {"role": "user", "content": prompt}
69
+ ]
70
+ text = tokenizer.apply_chat_template(
71
+ messages,
72
+ tokenize=False,
73
+ add_generation_prompt=True,
74
+ enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
75
+ )
76
+ model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
77
+
78
+ # conduct text completion
79
+ generated_ids = model.generate(
80
+ **model_inputs,
81
+ max_new_tokens=32768
82
+ )
83
+ output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
84
+
85
+ # parsing thinking content
86
+ try:
87
+ # rindex finding 151668 (</think>)
88
+ index = len(output_ids) - output_ids[::-1].index(151668)
89
+ except ValueError:
90
+ index = 0
91
+
92
+ thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
93
+ content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
94
+
95
+ print("thinking content:", thinking_content)
96
+ print("content:", content)
97
+ ```
98
+
99
+ For deployment, you can use `sglang>=0.4.6.post1` or `vllm>=0.8.5` or to create an OpenAI-compatible API endpoint:
100
+ - SGLang:
101
+ ```shell
102
+ python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --reasoning-parser qwen3
103
+ ```
104
+ - vLLM:
105
+ ```shell
106
+ vllm serve Qwen/Qwen3-235B-A22B-FP8 --enable-reasoning --reasoning-parser deepseek_r1
107
+ ```
108
+
109
+ For local use, applications such as Ollama, LMStudio, MLX-LM, llama.cpp, and KTransformers have also supported Qwen3.
110
+
111
+ ## Note on FP8
112
+
113
+ For convenience and performance, we have provided `fp8`-quantized model checkpoint for Qwen3, whose name ends with `-FP8`. The quantization method is fine-grained `fp8` quantization with block size of 128. You can find more details in the `quantization_config` field in `config.json`.
114
+
115
+ You can use the Qwen3-235B-A22B-FP8 model with serveral inference frameworks, including `transformers`, `sglang`, and `vllm`, as the original bfloat16 model.
116
+ However, please pay attention to the following known issues:
117
+ - `transformers`:
118
+ - there are currently issues with the "fine-grained fp8" method in `transformers` for distributed inference. You may need to set the environment variable `CUDA_LAUNCH_BLOCKING=1` if multiple devices are used in inference.
119
+
120
+ ## Switching Between Thinking and Non-Thinking Mode
121
+
122
+ > [!TIP]
123
+ > The `enable_thinking` switch is also available in APIs created by SGLang and vLLM.
124
+ > Please refer to our documentation for [SGLang](https://qwen.readthedocs.io/en/latest/deployment/sglang.html#thinking-non-thinking-modes) and [vLLM](https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking-non-thinking-modes) users.
125
+
126
+ ### `enable_thinking=True`
127
+
128
+ By default, Qwen3 has thinking capabilities enabled, similar to QwQ-32B. This means the model will use its reasoning abilities to enhance the quality of generated responses. For example, when explicitly setting `enable_thinking=True` or leaving it as the default value in `tokenizer.apply_chat_template`, the model will engage its thinking mode.
129
+
130
+ ```python
131
+ text = tokenizer.apply_chat_template(
132
+ messages,
133
+ tokenize=False,
134
+ add_generation_prompt=True,
135
+ enable_thinking=True # True is the default value for enable_thinking
136
+ )
137
+ ```
138
+
139
+ In this mode, the model will generate think content wrapped in a `<think>...</think>` block, followed by the final response.
140
+
141
+ > [!NOTE]
142
+ > For thinking mode, use `Temperature=0.6`, `TopP=0.95`, `TopK=20`, and `MinP=0` (the default setting in `generation_config.json`). **DO NOT use greedy decoding**, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the [Best Practices](#best-practices) section.
143
+
144
+
145
+ ### `enable_thinking=False`
146
+
147
+ We provide a hard switch to strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5-Instruct models. This mode is particularly useful in scenarios where disabling thinking is essential for enhancing efficiency.
148
+
149
+ ```python
150
+ text = tokenizer.apply_chat_template(
151
+ messages,
152
+ tokenize=False,
153
+ add_generation_prompt=True,
154
+ enable_thinking=False # Setting enable_thinking=False disables thinking mode
155
+ )
156
+ ```
157
+
158
+ In this mode, the model will not generate any think content and will not include a `<think>...</think>` block.
159
+
160
+ > [!NOTE]
161
+ > For non-thinking mode, we suggest using `Temperature=0.7`, `TopP=0.8`, `TopK=20`, and `MinP=0`. For more detailed guidance, please refer to the [Best Practices](#best-practices) section.
162
+
163
+ ### Advanced Usage: Switching Between Thinking and Non-Thinking Modes via User Input
164
+
165
+ We provide a soft switch mechanism that allows users to dynamically control the model's behavior when `enable_thinking=True`. Specifically, you can add `/think` and `/no_think` to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations.
166
+
167
+ Here is an example of a multi-turn conversation:
168
+
169
+ ```python
170
+ from transformers import AutoModelForCausalLM, AutoTokenizer
171
+
172
+ class QwenChatbot:
173
+ def __init__(self, model_name="Qwen/Qwen3-235B-A22B-FP8"):
174
+ self.tokenizer = AutoTokenizer.from_pretrained(model_name)
175
+ self.model = AutoModelForCausalLM.from_pretrained(model_name)
176
+ self.history = []
177
+
178
+ def generate_response(self, user_input):
179
+ messages = self.history + [{"role": "user", "content": user_input}]
180
+
181
+ text = self.tokenizer.apply_chat_template(
182
+ messages,
183
+ tokenize=False,
184
+ add_generation_prompt=True
185
+ )
186
+
187
+ inputs = self.tokenizer(text, return_tensors="pt")
188
+ response_ids = self.model.generate(**inputs, max_new_tokens=32768)[0][len(inputs.input_ids[0]):].tolist()
189
+ response = self.tokenizer.decode(response_ids, skip_special_tokens=True)
190
+
191
+ # Update history
192
+ self.history.append({"role": "user", "content": user_input})
193
+ self.history.append({"role": "assistant", "content": response})
194
+
195
+ return response
196
+
197
+ # Example Usage
198
+ if __name__ == "__main__":
199
+ chatbot = QwenChatbot()
200
+
201
+ # First input (without /think or /no_think tags, thinking mode is enabled by default)
202
+ user_input_1 = "How many r's in strawberries?"
203
+ print(f"User: {user_input_1}")
204
+ response_1 = chatbot.generate_response(user_input_1)
205
+ print(f"Bot: {response_1}")
206
+ print("----------------------")
207
+
208
+ # Second input with /no_think
209
+ user_input_2 = "Then, how many r's in blueberries? /no_think"
210
+ print(f"User: {user_input_2}")
211
+ response_2 = chatbot.generate_response(user_input_2)
212
+ print(f"Bot: {response_2}")
213
+ print("----------------------")
214
+
215
+ # Third input with /think
216
+ user_input_3 = "Really? /think"
217
+ print(f"User: {user_input_3}")
218
+ response_3 = chatbot.generate_response(user_input_3)
219
+ print(f"Bot: {response_3}")
220
+ ```
221
+
222
+ > [!NOTE]
223
+ > For API compatibility, when `enable_thinking=True`, regardless of whether the user uses `/think` or `/no_think`, the model will always output a block wrapped in `<think>...</think>`. However, the content inside this block may be empty if thinking is disabled.
224
+ > When `enable_thinking=False`, the soft switches are not valid. Regardless of any `/think` or `/no_think` tags input by the user, the model will not generate think content and will not include a `<think>...</think>` block.
225
+
226
+ ## Agentic Use
227
+
228
+ Qwen3 excels in tool calling capabilities. We recommend using [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) to make the best use of agentic ability of Qwen3. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity.
229
+
230
+ To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself.
231
+ ```python
232
+ from qwen_agent.agents import Assistant
233
+
234
+ # Define LLM
235
+ llm_cfg = {
236
+ 'model': 'Qwen3-235B-A22B-FP8',
237
+
238
+ # Use the endpoint provided by Alibaba Model Studio:
239
+ # 'model_type': 'qwen_dashscope',
240
+ # 'api_key': os.getenv('DASHSCOPE_API_KEY'),
241
+
242
+ # Use a custom endpoint compatible with OpenAI API:
243
+ 'model_server': 'http://localhost:8000/v1', # api_base
244
+ 'api_key': 'EMPTY',
245
+
246
+ # Other parameters:
247
+ # 'generate_cfg': {
248
+ # # Add: When the response content is `<think>this is the thought</think>this is the answer;
249
+ # # Do not add: When the response has been separated by reasoning_content and content.
250
+ # 'thought_in_content': True,
251
+ # },
252
+ }
253
+
254
+ # Define Tools
255
+ tools = [
256
+ {'mcpServers': { # You can specify the MCP configuration file
257
+ 'time': {
258
+ 'command': 'uvx',
259
+ 'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']
260
+ },
261
+ "fetch": {
262
+ "command": "uvx",
263
+ "args": ["mcp-server-fetch"]
264
+ }
265
+ }
266
+ },
267
+ 'code_interpreter', # Built-in tools
268
+ ]
269
+
270
+ # Define Agent
271
+ bot = Assistant(llm=llm_cfg, function_list=tools)
272
+
273
+ # Streaming generation
274
+ messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]
275
+ for responses in bot.run(messages=messages):
276
+ pass
277
+ print(responses)
278
+ ```
279
+
280
+ ## Processing Long Texts
281
+
282
+ Qwen3 natively supports context lengths of up to 32,768 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 131,072 tokens using the [YaRN](https://arxiv.org/abs/2309.00071) method.
283
+
284
+ YaRN is currently supported by several inference frameworks, e.g., `transformers` and `llama.cpp` for local use, `vllm` and `sglang` for deployment. In general, there are two approaches to enabling YaRN for supported frameworks:
285
+
286
+ - Modifying the model files:
287
+ In the `config.json` file, add the `rope_scaling` fields:
288
+ ```json
289
+ {
290
+ ...,
291
+ "rope_scaling": {
292
+ "rope_type": "yarn",
293
+ "factor": 4.0,
294
+ "original_max_position_embeddings": 32768
295
+ }
296
+ }
297
+ ```
298
+ For `llama.cpp`, you need to regenerate the GGUF file after the modification.
299
+
300
+ - Passing command line arguments:
301
+
302
+ For `vllm`, you can use
303
+ ```shell
304
+ vllm serve ... --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' --max-model-len 131072
305
+ ```
306
+
307
+ For `sglang`, you can use
308
+ ```shell
309
+ python -m sglang.launch_server ... --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
310
+ ```
311
+
312
+ For `llama-server` from `llama.cpp`, you can use
313
+ ```shell
314
+ llama-server ... --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768
315
+ ```
316
+
317
+ > [!IMPORTANT]
318
+ > If you encounter the following warning
319
+ > ```
320
+ > Unrecognized keys in `rope_scaling` for 'rope_type'='yarn': {'original_max_position_embeddings'}
321
+ > ```
322
+ > please upgrade `transformers>=4.51.0`.
323
+
324
+ > [!NOTE]
325
+ > All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts.**
326
+ > We advise adding the `rope_scaling` configuration only when processing long contexts is required.
327
+ > It is also recommended to modify the `factor` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set `factor` as 2.0.
328
+
329
+ > [!NOTE]
330
+ > The default `max_position_embeddings` in `config.json` is set to 40,960. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance.
331
+
332
+ > [!TIP]
333
+ > The endpoint provided by Alibaba Model Studio supports dynamic YaRN by default and no extra configuration is needed.
334
+
335
+ ## Best Practices
336
+
337
+ To achieve optimal performance, we recommend the following settings:
338
+
339
+ 1. **Sampling Parameters**:
340
+ - For thinking mode (`enable_thinking=True`), use `Temperature=0.6`, `TopP=0.95`, `TopK=20`, and `MinP=0`. **DO NOT use greedy decoding**, as it can lead to performance degradation and endless repetitions.
341
+ - For non-thinking mode (`enable_thinking=False`), we suggest using `Temperature=0.7`, `TopP=0.8`, `TopK=20`, and `MinP=0`.
342
+ - For supported frameworks, you can adjust the `presence_penalty` parameter between 0 and 2 to reduce endless repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
343
+
344
+ 2. **Adequate Output Length**: We recommend using an output length of 32,768 tokens for most queries. For benchmarking on highly complex problems, such as those found in math and programming competitions, we suggest setting the max output length to 38,912 tokens. This provides the model with sufficient space to generate detailed and comprehensive responses, thereby enhancing its overall performance.
345
+
346
+ 3. **Standardize Output Format**: We recommend using prompts to standardize model outputs when benchmarking.
347
+ - **Math Problems**: Include "Please reason step by step, and put your final answer within \boxed{}." in the prompt.
348
+ - **Multiple-Choice Questions**: Add the following JSON structure to the prompt to standardize responses: "Please show your choice in the `answer` field with only the choice letter, e.g., `"answer": "C"`."
349
+
350
+ 4. **No Thinking Content in History**: In multi-turn conversations, the historical model output should only include the final output part and does not need to include the thinking content. It is implemented in the provided chat template in Jinja2. However, for frameworks that do not directly use the Jinja2 chat template, it is up to the developers to ensure that the best practice is followed.
351
+
352
+ ### Citation
353
+
354
+ If you find our work helpful, feel free to give us a cite.
355
+
356
+ ```
357
+ @misc{qwen3,
358
+ title = {Qwen3},
359
+ url = {https://qwenlm.github.io/blog/qwen3/},
360
+ author = {Qwen Team},
361
+ month = {April},
362
+ year = {2025}
363
+ }
364
+ ```
added_tokens.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</think>": 151668,
3
+ "</tool_call>": 151658,
4
+ "</tool_response>": 151666,
5
+ "<think>": 151667,
6
+ "<tool_call>": 151657,
7
+ "<tool_response>": 151665,
8
+ "<|box_end|>": 151649,
9
+ "<|box_start|>": 151648,
10
+ "<|endoftext|>": 151643,
11
+ "<|file_sep|>": 151664,
12
+ "<|fim_middle|>": 151660,
13
+ "<|fim_pad|>": 151662,
14
+ "<|fim_prefix|>": 151659,
15
+ "<|fim_suffix|>": 151661,
16
+ "<|im_end|>": 151645,
17
+ "<|im_start|>": 151644,
18
+ "<|image_pad|>": 151655,
19
+ "<|object_ref_end|>": 151647,
20
+ "<|object_ref_start|>": 151646,
21
+ "<|quad_end|>": 151651,
22
+ "<|quad_start|>": 151650,
23
+ "<|repo_name|>": 151663,
24
+ "<|video_pad|>": 151656,
25
+ "<|vision_end|>": 151653,
26
+ "<|vision_pad|>": 151654,
27
+ "<|vision_start|>": 151652
28
+ }
chat_template.jinja ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0].role == 'system' %}
4
+ {{- messages[0].content + '\n\n' }}
5
+ {%- endif %}
6
+ {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
7
+ {%- for tool in tools %}
8
+ {{- "\n" }}
9
+ {{- tool | tojson }}
10
+ {%- endfor %}
11
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
12
+ {%- else %}
13
+ {%- if messages[0].role == 'system' %}
14
+ {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
15
+ {%- endif %}
16
+ {%- endif %}
17
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
18
+ {%- for forward_message in messages %}
19
+ {%- set index = (messages|length - 1) - loop.index0 %}
20
+ {%- set message = messages[index] %}
21
+ {%- set current_content = message.content if message.content is not none else '' %}
22
+ {%- set tool_start = '<tool_response>' %}
23
+ {%- set tool_start_length = tool_start|length %}
24
+ {%- set start_of_message = current_content[:tool_start_length] %}
25
+ {%- set tool_end = '</tool_response>' %}
26
+ {%- set tool_end_length = tool_end|length %}
27
+ {%- set start_pos = (current_content|length) - tool_end_length %}
28
+ {%- if start_pos < 0 %}
29
+ {%- set start_pos = 0 %}
30
+ {%- endif %}
31
+ {%- set end_of_message = current_content[start_pos:] %}
32
+ {%- if ns.multi_step_tool and message.role == "user" and not(start_of_message == tool_start and end_of_message == tool_end) %}
33
+ {%- set ns.multi_step_tool = false %}
34
+ {%- set ns.last_query_index = index %}
35
+ {%- endif %}
36
+ {%- endfor %}
37
+ {%- for message in messages %}
38
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
39
+ {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
40
+ {%- elif message.role == "assistant" %}
41
+ {%- set content = message.content %}
42
+ {%- set reasoning_content = '' %}
43
+ {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
44
+ {%- set reasoning_content = message.reasoning_content %}
45
+ {%- else %}
46
+ {%- if '</think>' in message.content %}
47
+ {%- set content = (message.content.split('</think>')|last).lstrip('\n') %}
48
+ {%- set reasoning_content = (message.content.split('</think>')|first).rstrip('\n') %}
49
+ {%- set reasoning_content = (reasoning_content.split('<think>')|last).lstrip('\n') %}
50
+ {%- endif %}
51
+ {%- endif %}
52
+ {%- if loop.index0 > ns.last_query_index %}
53
+ {%- if loop.last or (not loop.last and reasoning_content) %}
54
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
55
+ {%- else %}
56
+ {{- '<|im_start|>' + message.role + '\n' + content }}
57
+ {%- endif %}
58
+ {%- else %}
59
+ {{- '<|im_start|>' + message.role + '\n' + content }}
60
+ {%- endif %}
61
+ {%- if message.tool_calls %}
62
+ {%- for tool_call in message.tool_calls %}
63
+ {%- if (loop.first and content) or (not loop.first) %}
64
+ {{- '\n' }}
65
+ {%- endif %}
66
+ {%- if tool_call.function %}
67
+ {%- set tool_call = tool_call.function %}
68
+ {%- endif %}
69
+ {{- '<tool_call>\n{"name": "' }}
70
+ {{- tool_call.name }}
71
+ {{- '", "arguments": ' }}
72
+ {%- if tool_call.arguments is string %}
73
+ {{- tool_call.arguments }}
74
+ {%- else %}
75
+ {{- tool_call.arguments | tojson }}
76
+ {%- endif %}
77
+ {{- '}\n</tool_call>' }}
78
+ {%- endfor %}
79
+ {%- endif %}
80
+ {{- '<|im_end|>\n' }}
81
+ {%- elif message.role == "tool" %}
82
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
83
+ {{- '<|im_start|>user' }}
84
+ {%- endif %}
85
+ {{- '\n<tool_response>\n' }}
86
+ {{- message.content }}
87
+ {{- '\n</tool_response>' }}
88
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
89
+ {{- '<|im_end|>\n' }}
90
+ {%- endif %}
91
+ {%- endif %}
92
+ {%- endfor %}
93
+ {%- if add_generation_prompt %}
94
+ {{- '<|im_start|>assistant\n' }}
95
+ {%- if enable_thinking is defined and enable_thinking is false %}
96
+ {{- '<think>\n\n</think>\n\n' }}
97
+ {%- endif %}
98
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,332 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3MoeForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "decoder_sparse_step": 1,
8
+ "eos_token_id": 151645,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 4096,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 12288,
14
+ "max_position_embeddings": 40960,
15
+ "max_window_layers": 94,
16
+ "mlp_only_layers": [],
17
+ "model_type": "qwen3_moe",
18
+ "moe_intermediate_size": 1536,
19
+ "norm_topk_prob": true,
20
+ "num_attention_heads": 64,
21
+ "num_experts": 128,
22
+ "num_experts_per_tok": 8,
23
+ "num_hidden_layers": 94,
24
+ "num_key_value_heads": 4,
25
+ "output_router_logits": false,
26
+ "pad_token_id": 151654,
27
+ "quantization_config": {
28
+ "activation_scheme": "dynamic",
29
+ "modules_to_not_convert": [
30
+ "lm_head",
31
+ "model.layers.0.input_layernorm",
32
+ "model.layers.0.mlp.gate",
33
+ "model.layers.0.post_attention_layernorm",
34
+ "model.layers.1.input_layernorm",
35
+ "model.layers.1.mlp.gate",
36
+ "model.layers.1.post_attention_layernorm",
37
+ "model.layers.2.input_layernorm",
38
+ "model.layers.2.mlp.gate",
39
+ "model.layers.2.post_attention_layernorm",
40
+ "model.layers.3.input_layernorm",
41
+ "model.layers.3.mlp.gate",
42
+ "model.layers.3.post_attention_layernorm",
43
+ "model.layers.4.input_layernorm",
44
+ "model.layers.4.mlp.gate",
45
+ "model.layers.4.post_attention_layernorm",
46
+ "model.layers.5.input_layernorm",
47
+ "model.layers.5.mlp.gate",
48
+ "model.layers.5.post_attention_layernorm",
49
+ "model.layers.6.input_layernorm",
50
+ "model.layers.6.mlp.gate",
51
+ "model.layers.6.post_attention_layernorm",
52
+ "model.layers.7.input_layernorm",
53
+ "model.layers.7.mlp.gate",
54
+ "model.layers.7.post_attention_layernorm",
55
+ "model.layers.8.input_layernorm",
56
+ "model.layers.8.mlp.gate",
57
+ "model.layers.8.post_attention_layernorm",
58
+ "model.layers.9.input_layernorm",
59
+ "model.layers.9.mlp.gate",
60
+ "model.layers.9.post_attention_layernorm",
61
+ "model.layers.10.input_layernorm",
62
+ "model.layers.10.mlp.gate",
63
+ "model.layers.10.post_attention_layernorm",
64
+ "model.layers.11.input_layernorm",
65
+ "model.layers.11.mlp.gate",
66
+ "model.layers.11.post_attention_layernorm",
67
+ "model.layers.12.input_layernorm",
68
+ "model.layers.12.mlp.gate",
69
+ "model.layers.12.post_attention_layernorm",
70
+ "model.layers.13.input_layernorm",
71
+ "model.layers.13.mlp.gate",
72
+ "model.layers.13.post_attention_layernorm",
73
+ "model.layers.14.input_layernorm",
74
+ "model.layers.14.mlp.gate",
75
+ "model.layers.14.post_attention_layernorm",
76
+ "model.layers.15.input_layernorm",
77
+ "model.layers.15.mlp.gate",
78
+ "model.layers.15.post_attention_layernorm",
79
+ "model.layers.16.input_layernorm",
80
+ "model.layers.16.mlp.gate",
81
+ "model.layers.16.post_attention_layernorm",
82
+ "model.layers.17.input_layernorm",
83
+ "model.layers.17.mlp.gate",
84
+ "model.layers.17.post_attention_layernorm",
85
+ "model.layers.18.input_layernorm",
86
+ "model.layers.18.mlp.gate",
87
+ "model.layers.18.post_attention_layernorm",
88
+ "model.layers.19.input_layernorm",
89
+ "model.layers.19.mlp.gate",
90
+ "model.layers.19.post_attention_layernorm",
91
+ "model.layers.20.input_layernorm",
92
+ "model.layers.20.mlp.gate",
93
+ "model.layers.20.post_attention_layernorm",
94
+ "model.layers.21.input_layernorm",
95
+ "model.layers.21.mlp.gate",
96
+ "model.layers.21.post_attention_layernorm",
97
+ "model.layers.22.input_layernorm",
98
+ "model.layers.22.mlp.gate",
99
+ "model.layers.22.post_attention_layernorm",
100
+ "model.layers.23.input_layernorm",
101
+ "model.layers.23.mlp.gate",
102
+ "model.layers.23.post_attention_layernorm",
103
+ "model.layers.24.input_layernorm",
104
+ "model.layers.24.mlp.gate",
105
+ "model.layers.24.post_attention_layernorm",
106
+ "model.layers.25.input_layernorm",
107
+ "model.layers.25.mlp.gate",
108
+ "model.layers.25.post_attention_layernorm",
109
+ "model.layers.26.input_layernorm",
110
+ "model.layers.26.mlp.gate",
111
+ "model.layers.26.post_attention_layernorm",
112
+ "model.layers.27.input_layernorm",
113
+ "model.layers.27.mlp.gate",
114
+ "model.layers.27.post_attention_layernorm",
115
+ "model.layers.28.input_layernorm",
116
+ "model.layers.28.mlp.gate",
117
+ "model.layers.28.post_attention_layernorm",
118
+ "model.layers.29.input_layernorm",
119
+ "model.layers.29.mlp.gate",
120
+ "model.layers.29.post_attention_layernorm",
121
+ "model.layers.30.input_layernorm",
122
+ "model.layers.30.mlp.gate",
123
+ "model.layers.30.post_attention_layernorm",
124
+ "model.layers.31.input_layernorm",
125
+ "model.layers.31.mlp.gate",
126
+ "model.layers.31.post_attention_layernorm",
127
+ "model.layers.32.input_layernorm",
128
+ "model.layers.32.mlp.gate",
129
+ "model.layers.32.post_attention_layernorm",
130
+ "model.layers.33.input_layernorm",
131
+ "model.layers.33.mlp.gate",
132
+ "model.layers.33.post_attention_layernorm",
133
+ "model.layers.34.input_layernorm",
134
+ "model.layers.34.mlp.gate",
135
+ "model.layers.34.post_attention_layernorm",
136
+ "model.layers.35.input_layernorm",
137
+ "model.layers.35.mlp.gate",
138
+ "model.layers.35.post_attention_layernorm",
139
+ "model.layers.36.input_layernorm",
140
+ "model.layers.36.mlp.gate",
141
+ "model.layers.36.post_attention_layernorm",
142
+ "model.layers.37.input_layernorm",
143
+ "model.layers.37.mlp.gate",
144
+ "model.layers.37.post_attention_layernorm",
145
+ "model.layers.38.input_layernorm",
146
+ "model.layers.38.mlp.gate",
147
+ "model.layers.38.post_attention_layernorm",
148
+ "model.layers.39.input_layernorm",
149
+ "model.layers.39.mlp.gate",
150
+ "model.layers.39.post_attention_layernorm",
151
+ "model.layers.40.input_layernorm",
152
+ "model.layers.40.mlp.gate",
153
+ "model.layers.40.post_attention_layernorm",
154
+ "model.layers.41.input_layernorm",
155
+ "model.layers.41.mlp.gate",
156
+ "model.layers.41.post_attention_layernorm",
157
+ "model.layers.42.input_layernorm",
158
+ "model.layers.42.mlp.gate",
159
+ "model.layers.42.post_attention_layernorm",
160
+ "model.layers.43.input_layernorm",
161
+ "model.layers.43.mlp.gate",
162
+ "model.layers.43.post_attention_layernorm",
163
+ "model.layers.44.input_layernorm",
164
+ "model.layers.44.mlp.gate",
165
+ "model.layers.44.post_attention_layernorm",
166
+ "model.layers.45.input_layernorm",
167
+ "model.layers.45.mlp.gate",
168
+ "model.layers.45.post_attention_layernorm",
169
+ "model.layers.46.input_layernorm",
170
+ "model.layers.46.mlp.gate",
171
+ "model.layers.46.post_attention_layernorm",
172
+ "model.layers.47.input_layernorm",
173
+ "model.layers.47.mlp.gate",
174
+ "model.layers.47.post_attention_layernorm",
175
+ "model.layers.48.input_layernorm",
176
+ "model.layers.48.mlp.gate",
177
+ "model.layers.48.post_attention_layernorm",
178
+ "model.layers.49.input_layernorm",
179
+ "model.layers.49.mlp.gate",
180
+ "model.layers.49.post_attention_layernorm",
181
+ "model.layers.50.input_layernorm",
182
+ "model.layers.50.mlp.gate",
183
+ "model.layers.50.post_attention_layernorm",
184
+ "model.layers.51.input_layernorm",
185
+ "model.layers.51.mlp.gate",
186
+ "model.layers.51.post_attention_layernorm",
187
+ "model.layers.52.input_layernorm",
188
+ "model.layers.52.mlp.gate",
189
+ "model.layers.52.post_attention_layernorm",
190
+ "model.layers.53.input_layernorm",
191
+ "model.layers.53.mlp.gate",
192
+ "model.layers.53.post_attention_layernorm",
193
+ "model.layers.54.input_layernorm",
194
+ "model.layers.54.mlp.gate",
195
+ "model.layers.54.post_attention_layernorm",
196
+ "model.layers.55.input_layernorm",
197
+ "model.layers.55.mlp.gate",
198
+ "model.layers.55.post_attention_layernorm",
199
+ "model.layers.56.input_layernorm",
200
+ "model.layers.56.mlp.gate",
201
+ "model.layers.56.post_attention_layernorm",
202
+ "model.layers.57.input_layernorm",
203
+ "model.layers.57.mlp.gate",
204
+ "model.layers.57.post_attention_layernorm",
205
+ "model.layers.58.input_layernorm",
206
+ "model.layers.58.mlp.gate",
207
+ "model.layers.58.post_attention_layernorm",
208
+ "model.layers.59.input_layernorm",
209
+ "model.layers.59.mlp.gate",
210
+ "model.layers.59.post_attention_layernorm",
211
+ "model.layers.60.input_layernorm",
212
+ "model.layers.60.mlp.gate",
213
+ "model.layers.60.post_attention_layernorm",
214
+ "model.layers.61.input_layernorm",
215
+ "model.layers.61.mlp.gate",
216
+ "model.layers.61.post_attention_layernorm",
217
+ "model.layers.62.input_layernorm",
218
+ "model.layers.62.mlp.gate",
219
+ "model.layers.62.post_attention_layernorm",
220
+ "model.layers.63.input_layernorm",
221
+ "model.layers.63.mlp.gate",
222
+ "model.layers.63.post_attention_layernorm",
223
+ "model.layers.64.input_layernorm",
224
+ "model.layers.64.mlp.gate",
225
+ "model.layers.64.post_attention_layernorm",
226
+ "model.layers.65.input_layernorm",
227
+ "model.layers.65.mlp.gate",
228
+ "model.layers.65.post_attention_layernorm",
229
+ "model.layers.66.input_layernorm",
230
+ "model.layers.66.mlp.gate",
231
+ "model.layers.66.post_attention_layernorm",
232
+ "model.layers.67.input_layernorm",
233
+ "model.layers.67.mlp.gate",
234
+ "model.layers.67.post_attention_layernorm",
235
+ "model.layers.68.input_layernorm",
236
+ "model.layers.68.mlp.gate",
237
+ "model.layers.68.post_attention_layernorm",
238
+ "model.layers.69.input_layernorm",
239
+ "model.layers.69.mlp.gate",
240
+ "model.layers.69.post_attention_layernorm",
241
+ "model.layers.70.input_layernorm",
242
+ "model.layers.70.mlp.gate",
243
+ "model.layers.70.post_attention_layernorm",
244
+ "model.layers.71.input_layernorm",
245
+ "model.layers.71.mlp.gate",
246
+ "model.layers.71.post_attention_layernorm",
247
+ "model.layers.72.input_layernorm",
248
+ "model.layers.72.mlp.gate",
249
+ "model.layers.72.post_attention_layernorm",
250
+ "model.layers.73.input_layernorm",
251
+ "model.layers.73.mlp.gate",
252
+ "model.layers.73.post_attention_layernorm",
253
+ "model.layers.74.input_layernorm",
254
+ "model.layers.74.mlp.gate",
255
+ "model.layers.74.post_attention_layernorm",
256
+ "model.layers.75.input_layernorm",
257
+ "model.layers.75.mlp.gate",
258
+ "model.layers.75.post_attention_layernorm",
259
+ "model.layers.76.input_layernorm",
260
+ "model.layers.76.mlp.gate",
261
+ "model.layers.76.post_attention_layernorm",
262
+ "model.layers.77.input_layernorm",
263
+ "model.layers.77.mlp.gate",
264
+ "model.layers.77.post_attention_layernorm",
265
+ "model.layers.78.input_layernorm",
266
+ "model.layers.78.mlp.gate",
267
+ "model.layers.78.post_attention_layernorm",
268
+ "model.layers.79.input_layernorm",
269
+ "model.layers.79.mlp.gate",
270
+ "model.layers.79.post_attention_layernorm",
271
+ "model.layers.80.input_layernorm",
272
+ "model.layers.80.mlp.gate",
273
+ "model.layers.80.post_attention_layernorm",
274
+ "model.layers.81.input_layernorm",
275
+ "model.layers.81.mlp.gate",
276
+ "model.layers.81.post_attention_layernorm",
277
+ "model.layers.82.input_layernorm",
278
+ "model.layers.82.mlp.gate",
279
+ "model.layers.82.post_attention_layernorm",
280
+ "model.layers.83.input_layernorm",
281
+ "model.layers.83.mlp.gate",
282
+ "model.layers.83.post_attention_layernorm",
283
+ "model.layers.84.input_layernorm",
284
+ "model.layers.84.mlp.gate",
285
+ "model.layers.84.post_attention_layernorm",
286
+ "model.layers.85.input_layernorm",
287
+ "model.layers.85.mlp.gate",
288
+ "model.layers.85.post_attention_layernorm",
289
+ "model.layers.86.input_layernorm",
290
+ "model.layers.86.mlp.gate",
291
+ "model.layers.86.post_attention_layernorm",
292
+ "model.layers.87.input_layernorm",
293
+ "model.layers.87.mlp.gate",
294
+ "model.layers.87.post_attention_layernorm",
295
+ "model.layers.88.input_layernorm",
296
+ "model.layers.88.mlp.gate",
297
+ "model.layers.88.post_attention_layernorm",
298
+ "model.layers.89.input_layernorm",
299
+ "model.layers.89.mlp.gate",
300
+ "model.layers.89.post_attention_layernorm",
301
+ "model.layers.90.input_layernorm",
302
+ "model.layers.90.mlp.gate",
303
+ "model.layers.90.post_attention_layernorm",
304
+ "model.layers.91.input_layernorm",
305
+ "model.layers.91.mlp.gate",
306
+ "model.layers.91.post_attention_layernorm",
307
+ "model.layers.92.input_layernorm",
308
+ "model.layers.92.mlp.gate",
309
+ "model.layers.92.post_attention_layernorm",
310
+ "model.layers.93.input_layernorm",
311
+ "model.layers.93.mlp.gate",
312
+ "model.layers.93.post_attention_layernorm"
313
+ ],
314
+ "quant_method": "fp8",
315
+ "weight_block_size": [
316
+ 128,
317
+ 128
318
+ ]
319
+ },
320
+ "rms_norm_eps": 1e-06,
321
+ "rope_scaling": null,
322
+ "rope_theta": 1000000.0,
323
+ "router_aux_loss_coef": 0.001,
324
+ "sliding_window": null,
325
+ "tie_word_embeddings": false,
326
+ "torch_dtype": "bfloat16",
327
+ "transformers_version": "4.52.0.dev0",
328
+ "unsloth_fixed": true,
329
+ "use_cache": true,
330
+ "use_sliding_window": false,
331
+ "vocab_size": 151936
332
+ }
generation_config.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 151645,
6
+ 151643
7
+ ],
8
+ "max_length": 40960,
9
+ "pad_token_id": 151654,
10
+ "temperature": 0.6,
11
+ "top_k": 20,
12
+ "top_p": 0.95,
13
+ "transformers_version": "4.52.0.dev0"
14
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model-00001-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8114e04ef7cff97c74002e992e0492ca20dd81482a8351055f5913ce9921c08c
3
+ size 4994993024
model-00002-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6dc1597a5ec29d6e0b2e334cc52f748989d1857dd0264e1ceadac5f51679f64f
3
+ size 4996254872
model-00003-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4c71a3df4c07bf368bd51553b307e21bd42f2cd97961c5c207dbc30d93b8347f
3
+ size 4996254872
model-00004-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2652d22416c04dd8bf8ac6b6256546a3be5c12d5b63da6a574887a422526e1b7
3
+ size 4996254880
model-00005-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d72e12a688a22ec57c550ba36f138b29f4386e6aadc1715caf98e5523321789e
3
+ size 4996254888
model-00006-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:551da3c7c5fa019b62282e58ac10e885d9dfae2e33ca89b1efb10b1b6dc11416
3
+ size 4996256464
model-00007-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e215ffa321fe8b0609ee621b4c170664526ca59207420b821a45c0cf0fc8567c
3
+ size 4996256464
model-00008-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67aa7aa8cfe88cafbf1cf203ce92908f29614d30eaa2c6aefc5ed8ff85606d1b
3
+ size 4996256472
model-00009-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b221dda93609fbe58a7d27def21d8f438424559b24eb042a35e65c676490dfe5
3
+ size 4996256480
model-00010-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:365c8c102d14d91462bf72b576616a2bdace7023c33fdfbc7e2781ce6328d768
3
+ size 4996256488
model-00011-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b6cf23242a5ef8eb7292299a779d0f1b7a68fb93f72eac4f6c6816567c61b5b
3
+ size 4996256488
model-00012-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dbaaf62876ba027f71571f6a6a765e4fdb3deb828fa0b9feea40a24da4c82b7f
3
+ size 4996256496
model-00013-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e2c542b9a5668fdc200a7e47fa20ba96193c92a3e8181c33239cff2ba5ee1333
3
+ size 4996256504
model-00014-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:269df049f7bc06e6ba95365164aed703b20be30fffbcd7637e65e8db124e61eb
3
+ size 4996256512
model-00015-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:093ddc15436112e0102e7315acb3d976b44c104f54277fed1cac58732bfb8969
3
+ size 4996256512
model-00016-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19b971223e907f7e8e5c098b2732902b443913c09b18b3456982429af9738746
3
+ size 4996256520
model-00017-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:438b13925b30b6902dc3145355c7e9192d3f584a5da6d8beb51e6dc50750cb90
3
+ size 4996256528
model-00018-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca7a07cce4f237a17084e2deddda880d8eaf510ccb84582a4a3698255932014a
3
+ size 4996256536
model-00019-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b13b20c5cc63a7cd10caed3b25baca71a0258d671e3ff3eb82258f8f58401864
3
+ size 4996256536
model-00020-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38666307cf34e8c997d509440225973c21c3d17cd5b1bbaa7e92db3eaf7a7599
3
+ size 4996256544
model-00021-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd2dac4cd2d149cf321239e16d6df6b3768992bd83d746eb2604ee34f931f315
3
+ size 4996256552
model-00022-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2558c6950a114abb2928699d3c712861e1e91ae9df07404aa0ce1da6b9912ec
3
+ size 4996256560
model-00023-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:332b0a1101aac9dd8f08c03f7e5d764e54b9fff2101b167e88d5539883fe35ac
3
+ size 4996256560
model-00024-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0fb9cf659ae3ef39e445d2411d57445d5405c06c37967c2063c4c8ae8efd081e
3
+ size 4996256568
model-00025-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fab6ea2520f4179c7224f42b8f0ed23b59229f86924bcde461bf7798a20a0207
3
+ size 4996256576
model-00026-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93037fc59a0b08c1ee875af6d53a350e2af7ef22d607ea6fde3e773e63cbd5f9
3
+ size 4996256584
model-00027-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8cc84983f7d92742423630ecde1fece3aa85cea1543611e37a9008b6626793d7
3
+ size 4996256584
model-00028-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:70ad1ed9be50b7b307ce2d989f9856386eb2ee7108561e28333fe735193b6f1c
3
+ size 4996256600
model-00029-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bf5a6f5d2415326e7908c0871c8047c224fa816a2d6f4125538c5853f53074bd
3
+ size 4996256608
model-00030-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ec077c99b430299196876fcf2f5804e98419ced258b9c1a7d2a0c0d82f3668f3
3
+ size 4996256624
model-00031-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:833098cb5e6e0968769e3a960df6a6e8c7a9dacfe90f796f9bcb32777e1dbd6f
3
+ size 4996256632
model-00032-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c2db15a3dab9f232026223755aee1021cccfffa5bd1fccdc0458b9afe435ee7
3
+ size 4996256640
model-00033-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8922e40d8386a6ee531a9ef96b96f8724a4de6d7052838abd2e8164bca7a1ff
3
+ size 4996256656
model-00034-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88e85308b8545ff96a2b507f74dab747853f021d1574f8d30ce6c4cf655e615e
3
+ size 4996256664
model-00035-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ce5237fdfa095442d7d5abc7ecc6c4d72f81ba87ddaadf8d423a2b98ef01ad48
3
+ size 4996256672
model-00036-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0a1b4de9e122c22aa6ade29ed72138ba16024e8ce16c4f9c1d0a8da6036a2f58
3
+ size 4996256680
model-00037-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2829a96cf87bde87a8f54008da337a2f2ada666224d7df273972723fd55a337f
3
+ size 4996256688
model-00038-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5a644d091848e22692432ac6d3ed2bc396bee0fdffb02862e7a8704a9cdd6f02
3
+ size 4996256688
model-00039-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc4a26fc1377898a3629ab8a1e1dad79c14e4b9555dafa039cf2112578f26518
3
+ size 4996256704
model-00040-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:23b7bc3e85f77b90b1067172764b083096219420be3e901395da498ecd82b9d8
3
+ size 4996256712
model-00041-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6befa99bc6e7a6d0481de21c7fb13fa77f24bd02c8494aa04aef5693c16a1a92
3
+ size 4996256712
model-00042-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:190f5faf960488cda1ad44b2bf72c2c9809c9f3fb8d58e8841b4d6ec41589727
3
+ size 4996256720
model-00043-of-00048.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c1ce61868d533c0eb741eb1839d78d0966505f2cd4fc54999e7feffe7661ef4
3
+ size 4996256728