End of training

README.md

@@ -50,19 +50,19 @@ flash_attention: true
 fp16: false
 fsdp: null
 fsdp_config: null
-gradient_accumulation_steps: 1
+gradient_accumulation_steps: 4
 gradient_checkpointing: true
 group_by_length: false
 hub_model_id: error577/ed864f34-7a77-4fe8-98e8-0903340e71dd
 hub_repo: null
 hub_strategy: checkpoint
 hub_token: null
-learning_rate: 0.0002
+learning_rate: 0.00000002
 load_in_4bit: false
 load_in_8bit: true
 local_rank: null
 logging_steps: 1
-lora_alpha: 64
+lora_alpha: 16
 lora_dropout: 0.0
 lora_fan_in_fan_out: null
 lora_model_dir: null
@@ -107,7 +107,7 @@ xformers_attention: null
 
 This model is a fine-tuned version of [unsloth/tinyllama](https://huggingface.co/unsloth/tinyllama) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.9738
+- Loss: 1.9212
 
 ## Model description
 
@@ -126,10 +126,12 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.0002
+- learning_rate: 2e-08
 - train_batch_size: 8
 - eval_batch_size: 8
 - seed: 42
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 32
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 30
@@ -139,11 +141,11 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| 1.9255        | 0.0002 | 1    | 1.9194          |
-| 1.9687        | 0.0306 | 200  | 1.9738          |
-| 2.0093        | 0.0611 | 400  | 1.9738          |
-| 2.0179        | 0.0917 | 600  | 1.9738          |
-| 1.8811        | 0.1223 | 800  | 1.9738          |
+| 1.9788        | 0.0006 | 1    | 1.9194          |
+| 1.9983        | 0.1223 | 200  | 1.9197          |
+| 1.8519        | 0.2446 | 400  | 1.9200          |
+| 1.9556        | 0.3669 | 600  | 1.9202          |
+| 1.8386        | 0.4891 | 800  | 1.9212          |
 
 
 ### Framework versions

adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3711ce1dcd486f12e01f0d432e856f7c41657e16284034c53638290e571515de
+oid sha256:6a04d1c08472b6d1b8e07ffe378a084feb20442c77f4900286ee4db9b4bd94f1
 size 101036698

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fe90e90343251a89f8ed6d19a7f0b201f6a1c075b7d1cae726a534b0480e2c74
+oid sha256:ef8b36a6665e6e85d4b7cefcfca344ad529964178a10485e2ff9e300343a84ef
 size 100966336