See axolotl config

axolotl version: 0.8.0.dev0

base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-ChatML-Mod
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

trust_remote_code:

# wandb configuration
wandb_project: 12b-mn-dans-reasoning-test
wandb_watch:

wandb_run_id: V0.0.4-1-2 # V{Version}-{Run Number}-{Attempt Number}
wandb_log_model:

# push checkpoints to hub
hub_model_id: Dans-DiscountModels/12b-mn-dans-reasoning-test-5
# how to push checkpoints to hub
# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy
hub_strategy: "every_save"
# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
# Required to be true when used in combination with `push_dataset_to_hub`
hf_use_auth_token: true

# where to save the finished model to
output_dir: ./12b-mn-dans-reasoning-test

save_safetensors: true

# dataset settings (local or huggingface repo)
datasets:
  - path: PocketDoc/Dans-Reasoningmaxx-NaturalReasoning
    type: dan-chat-advanced
  - path: PocketDoc/Dans-Reasoningmaxx-WebInstruct
    type: dan-chat-advanced
  - path: PocketDoc/Dans-Reasoningmaxx-GeneralReasoning
    type: dan-chat-advanced
  - path: PocketDoc/Dans-Benchmaxx-COT
    type: dan-chat-advanced
  - path: PocketDoc/Dans-Logicmaxx-SAT-AP
    type: dan-chat-advanced
  - path: PocketDoc/Dans-Assistantmaxx-Opus-Merge
    type: dan-chat-advanced
  # - path: PocketDoc/Dans-Assistantmaxx-sonnetorca-subset
  #   type: dan-chat-advanced

plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: false
cut_cross_entropy: true

load_in_8bit: false
load_in_4bit: false
strict: false

adapter:
lora_model_dir:

lora_r: 128
lora_alpha: 128
lora_dropout: 0.1
lora_target_linear: True
lora_target_modules:
lora_modules_to_save:
  - embed_tokens
  - lm_head
lora_fan_in_fan_out:
peft_use_rslora: true

dataset_prepared_path: ./12b-mn-dans-reasoning-test-data
val_set_size: 0.005

sequence_len: 8192

sample_packing: true
eval_sample_packing: true

pad_to_sequence_len: true

gradient_checkpointing: true
# gradient_checkpointing_kwargs:
# use_reentrant: false

gradient_accumulation_steps: 1
micro_batch_size: 4

num_epochs: 2

optimizer: came_pytorch

lr_scheduler: rex
learning_rate: 0.0000015
cosine_min_lr_ratio: 0.1

weight_decay: 0.1

max_grad_norm: 0.1

train_on_inputs: false
group_by_length: true

bf16: true
fp16: false
tf32: false

early_stopping_patience:

resume_from_checkpoint:
auto_resume_from_checkpoints: true

local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_ratio: 0.05

evals_per_epoch: 16
eval_table_size:
eval_max_new_tokens:

saves_per_epoch: 4
save_total_limit: 1

debug: false

deepspeed: deepspeed_configs/zero3_bf16.json

fsdp:
fsdp_config:

special_tokens:

12b-mn-dans-reasoning-test-5

This model is a fine-tuned version of Dans-DiscountModels/Mistral-Nemo-Base-2407-ChatML-Mod on the PocketDoc/Dans-Reasoningmaxx-NaturalReasoning, the PocketDoc/Dans-Reasoningmaxx-WebInstruct, the PocketDoc/Dans-Reasoningmaxx-GeneralReasoning, the PocketDoc/Dans-Benchmaxx-COT, the PocketDoc/Dans-Logicmaxx-SAT-AP and the PocketDoc/Dans-Assistantmaxx-Opus-Merge datasets. It achieves the following results on the evaluation set:

• Loss: 0.5607

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1.5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 16
• total_eval_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_HF with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 72
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.7429	0.0014	1	0.7557
0.6704	0.0634	46	0.6654
0.632	0.1269	92	0.6255
0.6256	0.1903	138	0.6108
0.5898	0.2538	184	0.6022
0.6078	0.3172	230	0.5967
0.5585	0.3807	276	0.5925
0.609	0.4441	322	0.5887
0.6203	0.5076	368	0.5854
0.5651	0.5710	414	0.5825
0.5886	0.6345	460	0.5802
0.5479	0.6979	506	0.5783
0.5924	0.7614	552	0.5764
0.5515	0.8248	598	0.5745
0.5891	0.8883	644	0.5727
0.5757	0.9517	690	0.5712
0.5241	1.0152	736	0.5718
0.5303	1.0786	782	0.5723
0.5586	1.1421	828	0.5713
0.5087	1.2055	874	0.5702
0.5251	1.2690	920	0.5694
0.5336	1.3324	966	0.5689
0.4921	1.3959	1012	0.5689
0.5164	1.4593	1058	0.5670
0.5546	1.5228	1104	0.5665
0.49	1.5862	1150	0.5649
0.4836	1.6497	1196	0.5647
0.4768	1.7131	1242	0.5642
0.5207	1.7766	1288	0.5630
0.5127	1.8400	1334	0.5624
0.5111	1.9034	1380	0.5615
0.4927	1.9669	1426	0.5607

Framework versions

• Transformers 4.49.0
• Pytorch 2.4.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.1