KETI_b1_s4_e1_training_log.log

03/31/2025 01:23:04 - INFO  - Train data file: finetuning_data.json
03/31/2025 01:23:04 - INFO  - Output Directory: output/gemma-3-27b-pt/20250331_012249_gemma-3-27b-pt_LoRA
03/31/2025 01:23:04 - INFO  - Experiment name: KETI_b1_s4_e1
03/31/2025 01:23:04 - INFO  - Using 6 GPU(s): NVIDIA A100-SXM4-80GB
03/31/2025 01:23:05 - INFO  - 🔢 Training samples: 101601
03/31/2025 01:23:05 - INFO  - 🔍 Evaluation samples: 11289
03/31/2025 01:23:05 - INFO  - 📊 Steps per epoch: 4233
03/31/2025 01:23:05 - INFO  - 🪜 Total training steps: 4233
03/31/2025 01:23:05 - INFO  - torch_dtype: torch.bfloat16
03/31/2025 01:23:05 - INFO  - ✅ FFT or LoRA 모드로 학습합니다.
03/31/2025 01:23:25 - INFO  - Initializing LORA model...
03/31/2025 01:23:32 - INFO  - gcc -pthread -B /root/pai/envs/llm-finetuning/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -c /tmp/tmpb98cgay3/test.c -o /tmp/tmpb98cgay3/test.o
03/31/2025 01:23:32 - INFO  - gcc -pthread -B /root/pai/envs/llm-finetuning/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -O2 -isystem /root/pai/envs/llm-finetuning/include -fPIC -c /tmp/tmp8kmegipo/test.c -o /tmp/tmp8kmegipo/test.o
03/31/2025 01:23:33 - INFO  - Start Training !
03/31/2025 01:24:01 - INFO  - [Epoch 0.07] [Step 10] loss: 3.8061
03/31/2025 01:24:24 - INFO  - [Epoch 0.13] [Step 20] loss: 3.0707
03/31/2025 01:24:47 - INFO  - [Epoch 0.20] [Step 30] loss: 2.9570
03/31/2025 01:25:10 - INFO  - [Epoch 0.26] [Step 40] loss: 2.9072
03/31/2025 01:25:33 - INFO  - [Epoch 0.33] [Step 50] loss: 2.8972
03/31/2025 01:25:55 - INFO  - [Epoch 0.40] [Step 60] loss: 2.8640
03/31/2025 01:26:18 - INFO  - [Epoch 0.46] [Step 70] loss: 2.8533
03/31/2025 01:26:41 - INFO  - [Epoch 0.53] [Step 80] loss: 2.8426
03/31/2025 01:27:03 - INFO  - [Epoch 0.59] [Step 90] loss: 2.8257
03/31/2025 01:27:26 - INFO  - [Epoch 0.66] [Step 100] loss: 2.8181
03/31/2025 01:27:49 - INFO  - [Epoch 0.72] [Step 110] loss: 2.8075
03/31/2025 01:28:11 - INFO  - [Epoch 0.79] [Step 120] loss: 2.8010
03/31/2025 01:28:33 - INFO  - [Epoch 0.86] [Step 130] loss: 2.8073
03/31/2025 01:28:56 - INFO  - [Epoch 0.92] [Step 140] loss: 2.7980
03/31/2025 01:29:18 - INFO  - [Epoch 0.99] [Step 150] loss: 2.8007
03/31/2025 01:29:39 - INFO  - ✅ Training complete. Logging system usage...
03/31/2025 01:29:39 - INFO  - >> System Usage - CPU: 3.2%, RAM: 2.6%, SSD: 75.74GB / 1888.43GB
03/31/2025 01:29:39 - INFO  - >> GPU 0: 71.28 GB used
03/31/2025 01:29:39 - INFO  - >> GPU 1: 79.90 GB used
03/31/2025 01:29:39 - INFO  - >> GPU 2: 70.95 GB used
03/31/2025 01:29:39 - INFO  - >> GPU 3: 78.04 GB used
03/31/2025 01:29:39 - INFO  - >> GPU 4: 79.88 GB used
03/31/2025 01:29:39 - INFO  - >> GPU 5: 77.00 GB used
03/31/2025 01:29:39 - INFO  - >> Total GPU Memory Used: 457.05 GB
03/31/2025 01:29:39 - INFO  - >> Total GPU Power Consumption: 540.07 W