[2024-10-20 11:18:17,804] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
10/20/2024 11:19:54 - INFO - llamafactory.cli - Initializing distributed tasks at: 127.0.0.1:22328
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] 
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] *****************************************
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] *****************************************
[2024-10-20 11:22:00,125] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-10-20 11:22:00,125] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-10-20 11:23:12,639] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-10-20 11:23:12,639] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-10-20 11:23:12,639] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
10/20/2024 11:23:12 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
10/20/2024 11:23:12 - INFO - llamafactory.hparams.parser - Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
[INFO|configuration_utils.py:673] 2024-10-20 11:23:12,820 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/config.json
[INFO|configuration_utils.py:742] 2024-10-20 11:23:12,849 >> Model config Qwen2Config {
  "_name_or_path": "/mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274",
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.45.2",
  "use_cache": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2470] 2024-10-20 11:23:13,325 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|configuration_utils.py:673] 2024-10-20 11:23:13,327 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/config.json
[INFO|configuration_utils.py:742] 2024-10-20 11:23:13,328 >> Model config Qwen2Config {
  "_name_or_path": "/mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274",
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.45.2",
  "use_cache": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,340 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file tokenizer_config.json
10/20/2024 11:23:13 - WARNING - llamafactory.model.loader - Processor was not found: 'Qwen2Config' object has no attribute 'vision_config'.
10/20/2024 11:23:13 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
[rank1]:[W1020 11:23:13.167659749 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[INFO|tokenization_utils_base.py:2470] 2024-10-20 11:23:13,765 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
10/20/2024 11:23:13 - WARNING - llamafactory.model.loader - Processor was not found: 'Qwen2Config' object has no attribute 'vision_config'.
10/20/2024 11:23:13 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
10/20/2024 11:23:13 - INFO - llamafactory.data.loader - Loading dataset re_arc_v3.json...

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 15661 examples [00:01, 9407.36 examples/s]
Generating train split: 15661 examples [00:01, 9346.62 examples/s]

Converting format of dataset (num_proc=16):   0%|          | 0/15661 [00:00<?, ? examples/s]
Converting format of dataset (num_proc=16):   5%|▍         | 709/15661 [00:00<00:02, 6827.04 examples/s]
Converting format of dataset (num_proc=16):  87%|████████▋ | 13688/15661 [00:00<00:00, 77938.57 examples/s]
Converting format of dataset (num_proc=16): 100%|██████████| 15661/15661 [00:00<00:00, 54667.82 examples/s]
[rank0]:[W1020 11:23:17.104395895 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
10/20/2024 11:23:18 - INFO - llamafactory.data.loader - Loading dataset re_arc_v3.json...

Running tokenizer on dataset (num_proc=16):   0%|          | 0/15661 [00:00<?, ? examples/s]
Running tokenizer on dataset (num_proc=16):   6%|▌         | 978/15661 [00:11<02:48, 87.40 examples/s]
Running tokenizer on dataset (num_proc=16):  12%|█▏        | 1957/15661 [00:13<01:26, 158.93 examples/s]
Running tokenizer on dataset (num_proc=16):  25%|██▍       | 3915/15661 [00:14<00:28, 414.72 examples/s]
Running tokenizer on dataset (num_proc=16):  31%|███       | 4894/15661 [00:14<00:19, 566.10 examples/s]
Running tokenizer on dataset (num_proc=16):  44%|████▍     | 6852/15661 [00:14<00:08, 1038.87 examples/s]
Running tokenizer on dataset (num_proc=16):  56%|█████▋    | 8810/15661 [00:14<00:04, 1674.62 examples/s]
Running tokenizer on dataset (num_proc=16):  69%|██████▉   | 10768/15661 [00:14<00:02, 2163.92 examples/s]
Running tokenizer on dataset (num_proc=16):  81%|████████▏ | 12726/15661 [00:15<00:01, 2598.01 examples/s]
Running tokenizer on dataset (num_proc=16):  88%|████████▊ | 13705/15661 [00:15<00:00, 2878.08 examples/s]
Running tokenizer on dataset (num_proc=16):  94%|█████████▍| 14683/15661 [00:16<00:00, 2606.21 examples/s]
Running tokenizer on dataset (num_proc=16): 100%|██████████| 15661/15661 [00:16<00:00, 2390.36 examples/s]
Running tokenizer on dataset (num_proc=16): 100%|██████████| 15661/15661 [00:16<00:00, 935.41 examples/s] 
training example:
input_ids:
[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 429, 646, 11625, 32711, 9079, 553, 1667, 264, 7199, 738, 315, 45558, 5746, 429, 525, 11537, 304, 13027, 13, 715, 12210, 5430, 4008, 75241, 12, 8886, 3383, 17167, 315, 2163, 264, 22955, 315, 4862, 10295, 11, 1380, 458, 4862, 3110, 17167, 315, 458, 1946, 5827, 323, 458, 2550, 5827, 13, 715, 12, 1752, 1817, 4862, 3110, 11, 279, 2550, 5827, 374, 279, 1102, 315, 18950, 279, 1852, 3383, 18906, 17991, 311, 279, 1946, 5827, 13, 715, 12, 576, 5795, 374, 311, 23583, 279, 17991, 504, 279, 2421, 4862, 10295, 624, 12, 576, 17991, 374, 264, 3383, 18906, 5827, 17991, 11, 892, 646, 387, 28502, 3865, 1119, 264, 8500, 315, 279, 45558, 5746, 624, 12210, 45558, 4008, 75241, 12, 20768, 323, 16605, 198, 220, 481, 3070, 4173, 95518, 18614, 5257, 821, 4494, 1075, 1565, 3543, 7808, 1565, 1190, 7808, 1565, 31941, 7808, 323, 803, 311, 27596, 5827, 7525, 624, 220, 481, 3070, 9386, 95518, 29734, 1894, 18021, 320, 68, 1302, 2572, 1565, 73956, 7808, 1565, 5225, 63, 701, 2710, 18021, 28654, 51, 7808, 1565, 37, 63, 701, 323, 72845, 22879, 320, 68, 1302, 2572, 1565, 3124, 7808, 1565, 22554, 63, 4292, 12, 2340, 35209, 198, 220, 481, 3070, 8815, 24883, 95518, 23550, 1075, 1565, 718, 7808, 1565, 59442, 7808, 1565, 64648, 7808, 323, 1565, 59394, 63, 2736, 6770, 34784, 389, 25780, 476, 45225, 624, 220, 481, 3070, 64312, 24883, 95518, 23550, 1741, 438, 1565, 16788, 7808, 1565, 39017, 7808, 323, 1565, 21028, 63, 3705, 19819, 55081, 624, 220, 481, 3070, 1043, 24883, 95518, 23550, 1075, 1565, 16912, 7808, 1565, 1358, 7808, 1565, 19052, 7808, 1565, 59251, 7808, 323, 1565, 9789, 85477, 63, 10091, 821, 23853, 624, 12, 10587, 323, 3002, 60911, 2914, 198, 220, 481, 3070, 3543, 34286, 95518, 1565, 19943, 63, 11450, 56349, 448, 5189, 15336, 323, 2750, 624, 220, 481, 3070, 3543, 53652, 95518, 23550, 1075, 1565, 4640, 24, 15, 7808, 1565, 71, 73225, 7808, 1565, 8602, 2246, 7808, 323, 1565, 2923, 12445, 63, 5165, 56349, 304, 5257, 5510, 624, 220, 481, 3070, 3136, 4203, 24883, 95518, 1565, 34147, 7808, 1565, 4997, 2292, 7808, 1565, 85, 6960, 7808, 323, 1565, 10666, 63, 8649, 476, 5602, 5479, 315, 56349, 624, 220, 481, 3070, 1190, 323, 30412, 55713, 95518, 23550, 1075, 1565, 19210, 7808, 1565, 30590, 7808, 1565, 13418, 7808, 1565, 983, 14987, 7808, 323, 1565, 2758, 789, 63, 3705, 5827, 28660, 323, 6171, 624, 12, 18320, 323, 81531, 198, 220, 481, 3070, 1636, 18320, 95518, 23550, 1741, 438, 1565, 3562, 3423, 7808, 1565, 55271, 3423, 7808, 1565, 3423, 1830, 7808, 323, 1565, 59674, 63, 23643, 1894, 42685, 624, 220, 481, 3070, 1190, 81531, 95518, 1565, 3423, 5315, 63, 323, 1565, 2141, 5315, 63, 4051, 6171, 553, 1894, 476, 1379, 624, 220, 481, 3070, 90618, 18320, 95518, 23550, 1075, 1565, 3057, 7808, 1565, 3487, 7808, 1565, 1515, 25337, 7808, 323, 1565, 24739, 18181, 63, 23643, 27979, 11871, 624, 12, 96054, 323, 425, 13586, 198, 220, 481, 3070, 14611, 1927, 95518, 1565, 6459, 7808, 1565, 79488, 7808, 1565, 67, 79488, 7808, 323, 1565, 482, 24101, 63, 8253, 13234, 1948, 5827, 14937, 624, 220, 481, 3070, 37909, 95518, 23550, 1075, 1565, 2011, 7808, 1565, 84699, 7808, 1565, 411, 2011, 7808, 323, 1565, 38630, 388, 63, 10091, 30618, 5671, 315, 28660, 624, 12, 17954, 198, 220, 481, 3070, 13999, 4440, 23470, 95518, 1565, 359, 333, 396, 63, 26885, 4194, 25780, 2878, 5189, 14262, 323, 16829, 5866, 624, 220, 481, 3070, 3543, 18954, 95518, 1565, 285, 15604, 63, 12341, 421, 458, 1946, 374, 264, 2697, 5827, 624, 220, 481, 3070, 3543, 89588, 95518, 1565, 2243, 15604, 63, 56033, 11469, 311, 279, 5827, 943, 624, 12210, 15042, 315, 279, 7907, 2038, 75241, 12, 576, 1172, 5420, 7525, 525, 27572, 279, 1102, 315, 264, 729, 1618, 304, 264, 3890, 11, 1380, 678, 5977, 1969, 2987, 387, 279, 1946, 5827, 11, 1045, 18021, 1741, 438, 25780, 476, 4185, 22879, 18860, 17961, 11, 476, 264, 3890, 8597, 24182, 2878, 279, 1852, 28961, 11, 323, 1817, 729, 429, 374, 1660, 2598, 1969, 2987, 387, 264, 45558, 729, 476, 264, 3890, 8597, 20346, 2878, 279, 1852, 28961, 13, 715, 12, 1096, 1083, 3363, 429, 1817, 1555, 315, 2038, 374, 44321, 311, 387, 264, 3175, 729, 1618, 624, 4416, 11, 498, 525, 2661, 264, 3383, 323, 264, 738, 315, 10295, 11, 498, 1184, 311, 6923, 264, 2038, 429, 646, 11625, 279, 3383, 624, 151645, 198, 151644, 872, 198, 334, 13383, 220, 16, 3070, 715, 1946, 25, 320, 24, 553, 220, 23, 8, 11631, 715, 19, 91, 17, 91, 17, 91, 19, 91, 15, 91, 17, 91, 19, 91, 17, 198, 17, 91, 19, 91, 19, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 19, 91, 17, 91, 19, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 198, 19, 91, 19, 91, 19, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 15, 91, 17, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 198, 2550, 25, 320, 23, 553, 220, 24, 8, 11631, 715, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 19, 91, 19, 91, 19, 91, 17, 91, 19, 91, 19, 91, 17, 91, 17, 91, 19, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 19, 91, 17, 91, 17, 91, 17, 91, 19, 91, 19, 91, 17, 91, 15, 198, 17, 91, 17, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 19, 198, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 17, 91, 19, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 17, 91, 17, 91, 17, 91, 19, 271, 334, 13383, 220, 17, 3070, 715, 1946, 25, 320, 16, 17, 553, 220, 16, 22, 8, 11631, 715, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 15, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 198, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 198, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 2550, 25, 320, 16, 22, 553, 220, 16, 17, 8, 11631, 715, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 23, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 198, 23, 91, 23, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 23, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 15, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 271, 334, 13383, 220, 18, 3070, 715, 1946, 25, 320, 23, 553, 220, 21, 8, 11631, 715, 22, 91, 16, 91, 22, 91, 16, 91, 22, 91, 22, 198, 22, 91, 22, 91, 22, 91, 16, 91, 16, 91, 22, 198, 22, 91, 22, 91, 22, 91, 16, 91, 22, 91, 22, 198, 22, 91, 22, 91, 22, 91, 24, 91, 22, 91, 22, 198, 16, 91, 16, 91, 22, 91, 16, 91, 22, 91, 16, 198, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 198, 16, 91, 16, 91, 22, 91, 24, 91, 22, 91, 24, 198, 22, 91, 16, 91, 16, 91, 22, 91, 22, 91, 22, 198, 2550, 25, 320, 21, 553, 220, 23, 8, 11631, 715, 22, 91, 24, 91, 22, 91, 16, 91, 22, 91, 22, 91, 22, 91, 22, 198, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 16, 91, 22, 198, 22, 91, 24, 91, 22, 91, 16, 91, 24, 91, 16, 91, 16, 91, 16, 198, 16, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 198, 16, 91, 16, 91, 22, 91, 16, 91, 22, 91, 22, 91, 22, 91, 16, 198, 22, 91, 16, 91, 22, 91, 16, 91, 22, 91, 22, 91, 22, 91, 22, 271, 151645, 198, 151644, 77091, 198, 750, 28961, 8972, 25, 10587, 8, 1464, 10587, 510, 262, 856, 15, 284, 5749, 24, 15, 8972, 340, 262, 856, 16, 284, 305, 73225, 2075, 15, 340, 262, 470, 856, 16, 151645]
inputs:
<|im_start|>system
You are a helpful assistant that can solve reasoning tasks by using a limited set of DSL functions that are implemented in Python. 
*** Task description ***
- Each task consists of around a handful of training examples, where an training example consists of an input grid and an output grid. 
- For each training example, the output grid is the result of applying the same task-specific transformation to the input grid. 
- The goal is to infer the transformation from the few training examples.
- The transformation is a task-specific grid transformation, which can be decomposed into a sequence of the DSL functions.
*** DSL description ***
- Types and Constants
  - **Types**: Define various data types like `Grid`, `Object`, `Indices`, and more to facilitate grid operations.
  - **Constants**: Include color constants (e.g., `ZERO`, `ONE`), boolean constants (`T`, `F`), and directional vectors (e.g., `UP`, `DOWN`).
- Primitives
  - **Math Operations**: Functions like `add`, `subtract`, `multiply`, and `divide` perform basic arithmetic on integers or tuples.
  - **Logical Operations**: Functions such as `even`, `flip`, and `both` handle logical evaluations.
  - **Data Operations**: Functions like `identity`, `order`, `merge`, `difference`, and `dedupe` manage data containers.
- Grid and Object Manipulation
  - **Grid Creation**: `canvas` creates grids with specified dimensions and values.
  - **Grid Transformation**: Functions like `rot90`, `hmirror`, `upscale`, and `downscale` transform grids in various ways.
  - **Subgrid Operations**: `crop`, `hsplit`, `vsplit`, and `trim` extract or modify parts of grids.
  - **Object and Patch Handling**: Functions like `objects`, `normalize`, `shift`, `toindices`, and `recolor` handle grid patches and objects.
- Analysis and Filtering
  - **Color Analysis**: Functions such as `mostcolor`, `leastcolor`, `colorcount`, and `palette` analyze color distributions.
  - **Object Filtering**: `colorfilter` and `sizefilter` filter objects by color or size.
  - **Spatial Analysis**: Functions like `center`, `position`, `manhattan`, and `adjacent` analyze spatial relationships.
- Connectivity and Bounding
  - **Connectivity**: `connect`, `neighbors`, `dneighbors`, and `ineighbors` determine connections between grid indices.
  - **Bounding**: Functions like `box`, `inbox`, `outbox`, and `corners` manage bounding areas of patches.
- Utils
  - **Random Integer Generation**: `unifint` generates random integers within specified bounds and difficulty levels.
  - **Grid Validation**: `is_grid` checks if an input is a valid grid.
  - **Grid Formatting**: `format_grid` casts lists to the grid type.
*** Format of the generated code ***
- The only allowed operations are storing the result of a function call in a variable, where all arguments must either be the input grid, some constants such as integers or common vectors indicating directions, or a variable previously computed within the same solver, and each function that is being called must either be a DSL function or a variable previously constructed within the same solver. 
- This also means that each line of code is enforced to be a single function call.
So, you are given a task and a set of examples, you need to generate a code that can solve the task.
<|im_end|>
<|im_start|>user
** Example 1 ** 
 input: (9 by 8) Matrix 
4|2|2|4|0|2|4|2
2|4|4|0|2|2|2|2
2|2|4|2|4|2|2|2
2|2|2|2|4|2|4|2
2|2|2|2|2|2|4|2
4|4|4|2|2|2|2|2
2|2|2|0|2|2|4|2
2|2|2|2|4|2|4|2
2|2|2|2|2|2|4|2
 output: (8 by 9) Matrix 
2|2|2|2|2|2|2|2|2
4|4|4|2|4|4|2|2|4
2|2|2|2|2|2|2|2|2
2|4|2|2|2|4|4|2|0
2|2|0|2|2|2|2|0|4
2|2|2|4|2|2|4|4|2
2|2|2|4|2|2|2|4|2
2|2|2|4|2|2|2|2|4

** Example 2 ** 
 input: (12 by 17) Matrix 
2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|8
2|2|2|2|2|0|2|2|2|2|8|2|8|2|2|0|2
2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2
8|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2
2|8|2|8|2|2|2|2|2|2|2|2|8|2|2|2|2
0|2|2|2|2|8|2|2|2|2|2|2|2|2|0|2|2
2|2|2|2|2|2|2|2|2|2|2|2|2|2|8|2|8
8|2|2|2|2|8|2|2|2|2|2|2|2|8|2|2|2
2|2|8|2|2|2|2|2|8|2|2|2|2|2|2|2|2
2|2|8|2|2|2|2|2|2|8|2|2|2|2|2|2|2
2|2|2|2|2|2|2|2|2|8|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|8|2|2|2|2|2|2|2
 output: (17 by 12) Matrix 
2|2|2|2|2|8|2|2|2|2|2|8
2|2|2|2|2|2|2|2|2|2|0|2
2|2|2|2|2|8|0|2|2|2|2|2
2|2|2|2|8|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|2|8|2
2|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|2|2|2|8|2
8|8|8|2|2|2|2|2|2|2|2|2
2|2|2|8|2|2|2|2|2|2|2|2
8|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|8|2|8|2|2|2|0|2
2|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|2|2|2
2|2|8|8|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|2|2|2
2|2|2|2|8|2|0|2|8|2|2|2

** Example 3 ** 
 input: (8 by 6) Matrix 
7|1|7|1|7|7
7|7|7|1|1|7
7|7|7|1|7|7
7|7|7|9|7|7
1|1|7|1|7|1
7|7|7|7|7|7
1|1|7|9|7|9
7|1|1|7|7|7
 output: (6 by 8) Matrix 
7|9|7|1|7|7|7|7
7|7|7|7|7|7|1|7
7|9|7|1|9|1|1|1
1|7|7|7|7|7|7|7
1|1|7|1|7|7|7|1
7|1|7|1|7|7|7|7

<|im_end|>
<|im_start|>assistant
def solver(I: Grid) -> Grid:
    x0 = rot90(I)
    x1 = hmirror(x0)
    return x1<|im_end|>
label_ids:
[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 750, 28961, 8972, 25, 10587, 8, 1464, 10587, 510, 262, 856, 15, 284, 5749, 24, 15, 8972, 340, 262, 856, 16, 284, 305, 73225, 2075, 15, 340, 262, 470, 856, 16, 151645]
labels:
def solver(I: Grid) -> Grid:
    x0 = rot90(I)
    x1 = hmirror(x0)
    return x1<|im_end|>
[INFO|configuration_utils.py:673] 2024-10-20 11:23:35,214 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/config.json
[INFO|configuration_utils.py:742] 2024-10-20 11:23:35,215 >> Model config Qwen2Config {
  "_name_or_path": "/mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274",
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.45.2",
  "use_cache": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

[INFO|modeling_utils.py:3729] 2024-10-20 11:23:41,683 >> loading weights file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/model.safetensors.index.json
[INFO|modeling_utils.py:3874] 2024-10-20 11:23:41,685 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
[2024-10-20 11:23:41,685] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-10-20 11:23:41,685] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[INFO|configuration_utils.py:1099] 2024-10-20 11:23:41,692 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151643,
  "use_cache": false
}

[2024-10-20 11:23:41,939] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 339, num_elems = 7.62B

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:03<00:10,  3.40s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:03<00:11,  3.76s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:06<00:06,  3.42s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:07<00:07,  3.70s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:09<00:03,  3.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00,  2.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00,  2.53s/it]
10/20/2024 11:23:52 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
10/20/2024 11:23:52 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
10/20/2024 11:23:52 - INFO - llamafactory.model.adapter - ZeRO3 / FSDP detected, remaining trainable params in float32.
10/20/2024 11:23:52 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
10/20/2024 11:23:52 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 7,615,616,512 || trainable%: 100.0000

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:10<00:03,  3.37s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00,  2.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00,  2.75s/it]
[INFO|modeling_utils.py:4574] 2024-10-20 11:23:52,977 >> All model checkpoint weights were used when initializing Qwen2ForCausalLM.

[INFO|modeling_utils.py:4582] 2024-10-20 11:23:52,977 >> All the weights of Qwen2ForCausalLM were initialized from the model checkpoint at /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen2ForCausalLM for predictions without further training.
[INFO|configuration_utils.py:1052] 2024-10-20 11:23:53,013 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/generation_config.json
[INFO|configuration_utils.py:1099] 2024-10-20 11:23:53,013 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "do_sample": true,
  "eos_token_id": [
    151645,
    151643
  ],
  "pad_token_id": 151643,
  "repetition_penalty": 1.1,
  "temperature": 0.7,
  "top_k": 20,
  "top_p": 0.8
}

10/20/2024 11:23:53 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
10/20/2024 11:23:53 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
10/20/2024 11:23:53 - INFO - llamafactory.model.adapter - ZeRO3 / FSDP detected, remaining trainable params in float32.
10/20/2024 11:23:53 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
10/20/2024 11:23:53 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 7,615,616,512 || trainable%: 100.0000
Detected kernel version 4.19.91, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:667] 2024-10-20 11:23:53,139 >> Using auto half precision backend
[2024-10-20 11:23:53,339] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.15.2, git-hash=unknown, git-branch=unknown
[2024-10-20 11:23:53,340] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-10-20 11:23:53,349] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-10-20 11:23:53,350] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-10-20 11:23:53,350] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-10-20 11:23:53,363] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW
[2024-10-20 11:23:53,363] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=AdamW type=<class 'torch.optim.adamw.AdamW'>
[2024-10-20 11:23:53,363] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-10-20 11:23:53,363] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2024-10-20 11:23:53,537] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2024-10-20 11:23:53,538] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB         Max_MA 9.65 GB         CA 8.89 GB         Max_CA 10 GB 
[2024-10-20 11:23:53,538] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:53,540] [INFO] [stage3.py:165:__init__] Reduce bucket size 12845056
[2024-10-20 11:23:53,540] [INFO] [stage3.py:166:__init__] Prefetch bucket size 11560550
[2024-10-20 11:23:53,716] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-10-20 11:23:53,717] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB         Max_MA 7.11 GB         CA 8.89 GB         Max_CA 9 GB 
[2024-10-20 11:23:53,717] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
Parameter Offload: Total persistent parameters: 333312 in 141 params
[2024-10-20 11:23:53,913] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-10-20 11:23:53,914] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB         Max_MA 7.11 GB         CA 8.89 GB         Max_CA 9 GB 
[2024-10-20 11:23:53,914] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:54,093] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions
[2024-10-20 11:23:54,094] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB         Max_MA 7.11 GB         CA 8.89 GB         Max_CA 9 GB 
[2024-10-20 11:23:54,094] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:57,802] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 5
[2024-10-20 11:23:57,803] [INFO] [utils.py:782:see_memory_usage] MA 7.09 GB         Max_MA 7.11 GB         CA 7.1 GB         Max_CA 9 GB 
[2024-10-20 11:23:57,803] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:57,979] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions
[2024-10-20 11:23:57,980] [INFO] [utils.py:782:see_memory_usage] MA 7.09 GB         Max_MA 7.09 GB         CA 7.1 GB         Max_CA 7 GB 
[2024-10-20 11:23:57,980] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,175] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions
[2024-10-20 11:23:58,176] [INFO] [utils.py:782:see_memory_usage] MA 21.28 GB         Max_MA 22.72 GB         CA 23.19 GB         Max_CA 23 GB 
[2024-10-20 11:23:58,176] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,350] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2024-10-20 11:23:58,351] [INFO] [utils.py:782:see_memory_usage] MA 21.28 GB         Max_MA 21.28 GB         CA 23.19 GB         Max_CA 23 GB 
[2024-10-20 11:23:58,351] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,527] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2024-10-20 11:23:58,527] [INFO] [utils.py:782:see_memory_usage] MA 21.28 GB         Max_MA 25.08 GB         CA 26.99 GB         Max_CA 27 GB 
[2024-10-20 11:23:58,528] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,528] [INFO] [stage3.py:520:_setup_for_real_optimizer] optimizer state initialized
[2024-10-20 11:23:58,815] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2024-10-20 11:23:58,816] [INFO] [utils.py:782:see_memory_usage] MA 28.4 GB         Max_MA 30.43 GB         CA 34.08 GB         Max_CA 34 GB 
[2024-10-20 11:23:58,816] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,816] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3
[2024-10-20 11:23:58,816] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = None
[2024-10-20 11:23:58,816] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-10-20 11:23:58,817] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999)]
[2024-10-20 11:23:58,818] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2024-10-20 11:23:58,818] [INFO] [config.py:1003:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2024-10-20 11:23:58,818] [INFO] [config.py:1003:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2024-10-20 11:23:58,818] [INFO] [config.py:1003:print]   amp_enabled .................. False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   amp_params ................... False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   bfloat16_enabled ............. True
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   bfloat16_immediate_grad_update  False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   checkpoint_parallel_write_pipeline  False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   checkpoint_tag_validation_enabled  True
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   checkpoint_tag_validation_fail  False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f9ec0617850>
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   communication_data_type ...... None
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   curriculum_enabled_legacy .... False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   curriculum_params_legacy ..... False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print]   data_efficiency_enabled ...... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   dataloader_drop_last ......... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   disable_allgather ............ False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   dump_state ................... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   dynamic_loss_scale_args ...... None
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_enabled ........... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_gas_boundary_resolution  1
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_layer_num ......... 0
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_max_iter .......... 100
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_stability ......... 1e-06
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_tol ............... 0.01
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   eigenvalue_verbose ........... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   elasticity_enabled ........... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   fp16_auto_cast ............... None
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   fp16_enabled ................. False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   fp16_master_weights_and_gradients  False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   global_rank .................. 0
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print]   grad_accum_dtype ............. None
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   gradient_accumulation_steps .. 4
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   gradient_clipping ............ 1.0
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   gradient_predivide_factor .... 1.0
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   graph_harvesting ............. False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   initial_dynamic_scale ........ 1
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   load_universal_checkpoint .... False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   loss_scale ................... 1.0
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   memory_breakdown ............. False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   mics_hierarchial_params_gather  False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   mics_shard_size .............. -1
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   optimizer_legacy_fusion ...... False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   optimizer_name ............... None
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   optimizer_params ............. None
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   pld_enabled .................. False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   pld_params ................... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   prescale_gradients ........... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   scheduler_name ............... None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   scheduler_params ............. None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   seq_parallel_communication_data_type  torch.float32
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   sparse_attention ............. None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   sparse_gradients_enabled ..... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   steps_per_print .............. inf
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   timers_config ................ enabled=True synchronized=True
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   train_batch_size ............. 8
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   train_micro_batch_size_per_gpu  1
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   use_data_before_expert_parallel_  False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   use_node_local_storage ....... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   wall_clock_breakdown ......... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   weight_quantization_config ... None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   world_size ................... 2
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   zero_allow_untested_optimizer  True
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=12845056 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=11560550 param_persistence_threshold=35840 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-10-20 11:23:58,823] [INFO] [config.py:1003:print]   zero_enabled ................. True
[2024-10-20 11:23:58,823] [INFO] [config.py:1003:print]   zero_force_ds_cpu_optimizer .. True
[2024-10-20 11:23:58,823] [INFO] [config.py:1003:print]   zero_optimization_stage ...... 3
[2024-10-20 11:23:58,823] [INFO] [config.py:989:print_user_config]   json = {
    "train_batch_size": 8, 
    "train_micro_batch_size_per_gpu": 1, 
    "gradient_accumulation_steps": 4, 
    "gradient_clipping": 1.0, 
    "zero_allow_untested_optimizer": true, 
    "fp16": {
        "enabled": false, 
        "loss_scale": 0, 
        "loss_scale_window": 1000, 
        "initial_scale_power": 16, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "bf16": {
        "enabled": true
    }, 
    "zero_optimization": {
        "stage": 3, 
        "overlap_comm": true, 
        "contiguous_gradients": true, 
        "sub_group_size": 1.000000e+09, 
        "reduce_bucket_size": 1.284506e+07, 
        "stage3_prefetch_bucket_size": 1.156055e+07, 
        "stage3_param_persistence_threshold": 3.584000e+04, 
        "stage3_max_live_parameters": 1.000000e+09, 
        "stage3_max_reuse_distance": 1.000000e+09, 
        "stage3_gather_16bit_weights_on_model_save": true
    }, 
    "steps_per_print": inf
}
[INFO|trainer.py:2243] 2024-10-20 11:23:58,823 >> ***** Running training *****
[INFO|trainer.py:2244] 2024-10-20 11:23:58,823 >>   Num examples = 14,094
[INFO|trainer.py:2245] 2024-10-20 11:23:58,823 >>   Num Epochs = 4
[INFO|trainer.py:2246] 2024-10-20 11:23:58,823 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2249] 2024-10-20 11:23:58,823 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:2250] 2024-10-20 11:23:58,823 >>   Gradient Accumulation steps = 4
[INFO|trainer.py:2251] 2024-10-20 11:23:58,823 >>   Total optimization steps = 7,044
[INFO|trainer.py:2252] 2024-10-20 11:23:58,824 >>   Number of trainable parameters = 7,615,616,512

  0%|          | 0/7044 [00:00<?, ?it/s][2024-10-20 11:24:15,553] [WARNING] [stage3.py:2104:step] 1 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time

  0%|          | 1/7044 [00:16<32:21:12, 16.54s/it][rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>
[rank0]:     launch()
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]:     run_exp()
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp
[rank0]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 96, in run_sft
[rank0]:     train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2052, in train
[rank0]:     return inner_training_loop(
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2388, in _inner_training_loop
[rank0]:     tr_loss_step = self.training_step(model, inputs)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3518, in training_step
[rank0]:     self.accelerator.backward(loss, **kwargs)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 2188, in backward
[rank0]:     self.deepspeed_engine_wrapped.backward(loss, **kwargs)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 166, in backward
[rank0]:     self.engine.backward(loss, **kwargs)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]:     ret_val = func(*args, **kwargs)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2020, in backward
[rank0]:     self.optimizer.backward(loss, retain_graph=retain_graph)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]:     ret_val = func(*args, **kwargs)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 2249, in backward
[rank0]:     self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
[rank0]:     scaled_loss.backward(retain_graph=retain_graph)
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
[rank0]:     torch.autograd.backward(
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.34 GiB. GPU 0 has a total capacity of 79.35 GiB of which 2.95 GiB is free. Process 22118 has 76.39 GiB memory in use. Of the allocated memory 67.73 GiB is allocated by PyTorch, and 7.82 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

  0%|          | 1/7044 [00:20<40:23:26, 20.65s/it]
W1020 11:24:20.981000 5294 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 5344 closing signal SIGTERM
E1020 11:24:21.272000 5294 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 5343) of binary: /mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/bin/python
Traceback (most recent call last):
  File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-10-20_11:24:20
  host      : dsw-116518-6475cdf9d-vpqks
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 5343)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================