nohup: ignoring input grep: warning: GREP_OPTIONS is deprecated; please use an alias or script [2024/07/08 01:49:14] ppocr INFO: Architecture : [2024/07/08 01:49:14] ppocr INFO: Backbone : [2024/07/08 01:49:14] ppocr INFO: depth : 4 [2024/07/08 01:49:14] ppocr INFO: embed_dim : 256 [2024/07/08 01:49:14] ppocr INFO: img_size : [192, 672] [2024/07/08 01:49:14] ppocr INFO: input_channel : 1 [2024/07/08 01:49:14] ppocr INFO: name : HybridTransformer [2024/07/08 01:49:14] ppocr INFO: num_classes : 0 [2024/07/08 01:49:14] ppocr INFO: num_heads : 8 [2024/07/08 01:49:14] ppocr INFO: patch_size : 16 [2024/07/08 01:49:14] ppocr INFO: Head : [2024/07/08 01:49:14] ppocr INFO: decoder_args : [2024/07/08 01:49:14] ppocr INFO: attn_on_attn : True [2024/07/08 01:49:14] ppocr INFO: cross_attend : True [2024/07/08 01:49:14] ppocr INFO: ff_glu : True [2024/07/08 01:49:14] ppocr INFO: rel_pos_bias : False [2024/07/08 01:49:14] ppocr INFO: use_scalenorm : False [2024/07/08 01:49:14] ppocr INFO: ignore_index : -100 [2024/07/08 01:49:14] ppocr INFO: name : LaTeXOCRHead [2024/07/08 01:49:14] ppocr INFO: pad_value : 0 [2024/07/08 01:49:14] ppocr INFO: Transform : None [2024/07/08 01:49:14] ppocr INFO: algorithm : LaTeXOCR [2024/07/08 01:49:14] ppocr INFO: in_channels : 1 [2024/07/08 01:49:14] ppocr INFO: model_type : rec [2024/07/08 01:49:14] ppocr INFO: Eval : [2024/07/08 01:49:14] ppocr INFO: dataset : [2024/07/08 01:49:14] ppocr INFO: batch_size_per_pair : 10 [2024/07/08 01:49:14] ppocr INFO: data : /workspace/code/paddle_ocr/mycode/pickle_file_paddle/latexocr_val.pkl [2024/07/08 01:49:14] ppocr INFO: keep_smaller_batches : True [2024/07/08 01:49:14] ppocr INFO: max_dimensions : [672, 192] [2024/07/08 01:49:14] ppocr INFO: min_dimensions : [32, 32] [2024/07/08 01:49:14] ppocr INFO: name : LaTeXOCRDataSet [2024/07/08 01:49:14] ppocr INFO: transforms : [2024/07/08 01:49:14] ppocr INFO: DecodeImage : [2024/07/08 01:49:14] ppocr INFO: channel_first : False [2024/07/08 01:49:14] ppocr INFO: LatexTestTransform : None [2024/07/08 01:49:14] ppocr INFO: NormalizeImage : [2024/07/08 01:49:14] ppocr INFO: mean : [0.7931, 0.7931, 0.7931] [2024/07/08 01:49:14] ppocr INFO: order : hwc [2024/07/08 01:49:14] ppocr INFO: std : [0.1738, 0.1738, 0.1738] [2024/07/08 01:49:14] ppocr INFO: LatexImageFormat : None [2024/07/08 01:49:14] ppocr INFO: KeepKeys : [2024/07/08 01:49:14] ppocr INFO: keep_keys : ['image'] [2024/07/08 01:49:14] ppocr INFO: loader : [2024/07/08 01:49:14] ppocr INFO: batch_size_per_card : 1 [2024/07/08 01:49:14] ppocr INFO: collate_fn : LaTeXOCRCollator [2024/07/08 01:49:14] ppocr INFO: drop_last : False [2024/07/08 01:49:14] ppocr INFO: num_workers : 0 [2024/07/08 01:49:14] ppocr INFO: shuffle : False [2024/07/08 01:49:14] ppocr INFO: Global : [2024/07/08 01:49:14] ppocr INFO: cal_metric_during_train : True [2024/07/08 01:49:14] ppocr INFO: checkpoints : None [2024/07/08 01:49:14] ppocr INFO: distributed : False [2024/07/08 01:49:14] ppocr INFO: epoch_num : 300 [2024/07/08 01:49:14] ppocr INFO: eval_batch_step : [0, 30000] [2024/07/08 01:49:14] ppocr INFO: fast_tokenizer_file : ppocr/utils/dict/latex_ocr_tokenizer.json [2024/07/08 01:49:14] ppocr INFO: infer_img : doc/datasets/crohme_demo/hme_00.jpg [2024/07/08 01:49:14] ppocr INFO: infer_mode : False [2024/07/08 01:49:14] ppocr INFO: log_smooth_window : 20 [2024/07/08 01:49:14] ppocr INFO: max_seq_len : 512 [2024/07/08 01:49:14] ppocr INFO: pretrained_model : None [2024/07/08 01:49:14] ppocr INFO: print_batch_step : 100 [2024/07/08 01:49:14] ppocr INFO: save_epoch_step : 5 [2024/07/08 01:49:14] ppocr INFO: save_inference_dir : None [2024/07/08 01:49:14] ppocr INFO: save_model_dir : ./output/rec/latex_ocr/ [2024/07/08 01:49:14] ppocr INFO: save_res_path : ./output/rec/predicts_latexocr.txt [2024/07/08 01:49:14] ppocr INFO: use_gpu : True [2024/07/08 01:49:14] ppocr INFO: use_space_char : False [2024/07/08 01:49:14] ppocr INFO: use_visualdl : False [2024/07/08 01:49:14] ppocr INFO: Loss : [2024/07/08 01:49:14] ppocr INFO: name : LaTeXOCRLoss [2024/07/08 01:49:14] ppocr INFO: Metric : [2024/07/08 01:49:14] ppocr INFO: cal_blue_score : False [2024/07/08 01:49:14] ppocr INFO: main_indicator : exp_rate [2024/07/08 01:49:14] ppocr INFO: name : LaTeXOCRMetric [2024/07/08 01:49:14] ppocr INFO: Optimizer : [2024/07/08 01:49:14] ppocr INFO: beta1 : 0.9 [2024/07/08 01:49:14] ppocr INFO: beta2 : 0.999 [2024/07/08 01:49:14] ppocr INFO: lr : [2024/07/08 01:49:14] ppocr INFO: learning_rate : 0.0001 [2024/07/08 01:49:14] ppocr INFO: name : Const [2024/07/08 01:49:14] ppocr INFO: name : AdamW [2024/07/08 01:49:14] ppocr INFO: PostProcess : [2024/07/08 01:49:14] ppocr INFO: fast_tokenizer_file : ppocr/utils/dict/latex_ocr_tokenizer.json [2024/07/08 01:49:14] ppocr INFO: name : LaTeXOCRDecode [2024/07/08 01:49:14] ppocr INFO: Train : [2024/07/08 01:49:14] ppocr INFO: dataset : [2024/07/08 01:49:14] ppocr INFO: batch_size_per_pair : 56 [2024/07/08 01:49:14] ppocr INFO: data : /workspace/code/paddle_ocr/mycode/pickle_file_paddle/latexocr_train.pkl [2024/07/08 01:49:14] ppocr INFO: keep_smaller_batches : False [2024/07/08 01:49:14] ppocr INFO: max_dimensions : [672, 192] [2024/07/08 01:49:14] ppocr INFO: min_dimensions : [32, 32] [2024/07/08 01:49:14] ppocr INFO: name : LaTeXOCRDataSet [2024/07/08 01:49:14] ppocr INFO: transforms : [2024/07/08 01:49:14] ppocr INFO: DecodeImage : [2024/07/08 01:49:14] ppocr INFO: channel_first : False [2024/07/08 01:49:14] ppocr INFO: LatexTrainTransform : [2024/07/08 01:49:14] ppocr INFO: bitmap_prob : 0.04 [2024/07/08 01:49:14] ppocr INFO: NormalizeImage : [2024/07/08 01:49:14] ppocr INFO: mean : [0.7931, 0.7931, 0.7931] [2024/07/08 01:49:14] ppocr INFO: order : hwc [2024/07/08 01:49:14] ppocr INFO: std : [0.1738, 0.1738, 0.1738] [2024/07/08 01:49:14] ppocr INFO: LatexImageFormat : None [2024/07/08 01:49:14] ppocr INFO: KeepKeys : [2024/07/08 01:49:14] ppocr INFO: keep_keys : ['image'] [2024/07/08 01:49:14] ppocr INFO: loader : [2024/07/08 01:49:14] ppocr INFO: batch_size_per_card : 1 [2024/07/08 01:49:14] ppocr INFO: collate_fn : LaTeXOCRCollator [2024/07/08 01:49:14] ppocr INFO: drop_last : False [2024/07/08 01:49:14] ppocr INFO: num_workers : 0 [2024/07/08 01:49:14] ppocr INFO: shuffle : True [2024/07/08 01:49:14] ppocr INFO: profiler_options : None [2024/07/08 01:49:14] ppocr INFO: train with paddle 2.6.1 and device Place(gpu:0) W0708 01:49:14.920226 81417 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 12.0, Runtime API Version: 11.7 W0708 01:49:14.926788 81417 gpu_resources.cc:164] device: 0, cuDNN Version: 8.5. load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range load_config {'shuffle': False, 'drop_last': False, 'batch_size_per_card': 1, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} out_channel 256 [2024/07/08 01:49:18] ppocr INFO: train dataloader has 2740 iters [2024/07/08 01:49:18] ppocr INFO: valid dataloader has 716 iters [2024/07/08 01:49:18] ppocr INFO: train from scratch [2024/07/08 01:49:18] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 30000 iterations load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 01:49:38] ppocr INFO: epoch: [1/300], global_step: 100, lr: 0.000100, edit distance: 0.964775, exp_rate: 0.000000, exp_rate<=1 : 0.395994, exp_rate<=2 : 0.607130, exp_rate<=3 : 0.744106, loss: 3.385197, avg_reader_cost: 0.01022 s, avg_batch_cost: 0.20571 s, avg_samples: 56.0, ips: 272.22858 samples/s, eta: 1 day, 22:57:52, max_mem_reserved: 24892 MB, max_mem_allocated: 18323 MB [2024/07/08 01:49:57] ppocr INFO: epoch: [1/300], global_step: 200, lr: 0.000100, edit distance: 0.880169, exp_rate: 0.000000, exp_rate<=1 : 0.227504, exp_rate<=2 : 0.364832, exp_rate<=3 : 0.454446, loss: 1.956195, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18867 s, avg_samples: 56.0, ips: 296.81683 samples/s, eta: 1 day, 21:00:49, max_mem_reserved: 28952 MB, max_mem_allocated: 18323 MB [2024/07/08 01:50:17] ppocr INFO: epoch: [1/300], global_step: 300, lr: 0.000100, edit distance: 0.838722, exp_rate: 0.000000, exp_rate<=1 : 0.149251, exp_rate<=2 : 0.239920, exp_rate<=3 : 0.301206, loss: 1.523986, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19556 s, avg_samples: 56.0, ips: 286.35626 samples/s, eta: 1 day, 20:53:04, max_mem_reserved: 28952 MB, max_mem_allocated: 18323 MB [2024/07/08 01:50:35] ppocr INFO: epoch: [1/300], global_step: 400, lr: 0.000100, edit distance: 0.813923, exp_rate: 0.000000, exp_rate<=1 : 0.111304, exp_rate<=2 : 0.181315, exp_rate<=3 : 0.234727, loss: 1.406363, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.17940 s, avg_samples: 56.0, ips: 312.15498 samples/s, eta: 1 day, 19:53:41, max_mem_reserved: 28952 MB, max_mem_allocated: 18323 MB [2024/07/08 01:50:54] ppocr INFO: epoch: [1/300], global_step: 500, lr: 0.000100, edit distance: 0.794035, exp_rate: 0.000000, exp_rate<=1 : 0.088758, exp_rate<=2 : 0.146862, exp_rate<=3 : 0.193971, loss: 1.304527, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19214 s, avg_samples: 56.0, ips: 291.45352 samples/s, eta: 1 day, 19:52:50, max_mem_reserved: 28952 MB, max_mem_allocated: 18323 MB [2024/07/08 01:51:14] ppocr INFO: epoch: [1/300], global_step: 600, lr: 0.000100, edit distance: 0.779183, exp_rate: 0.000000, exp_rate<=1 : 0.074075, exp_rate<=2 : 0.125348, exp_rate<=3 : 0.171646, loss: 1.159215, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20270 s, avg_samples: 56.0, ips: 276.26918 samples/s, eta: 1 day, 20:16:15, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:51:33] ppocr INFO: epoch: [1/300], global_step: 700, lr: 0.000100, edit distance: 0.766732, exp_rate: 0.000000, exp_rate<=1 : 0.063968, exp_rate<=2 : 0.110183, exp_rate<=3 : 0.155550, loss: 1.196347, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18202 s, avg_samples: 56.0, ips: 307.65058 samples/s, eta: 1 day, 19:52:28, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:51:52] ppocr INFO: epoch: [1/300], global_step: 800, lr: 0.000100, edit distance: 0.754668, exp_rate: 0.000000, exp_rate<=1 : 0.057073, exp_rate<=2 : 0.101785, exp_rate<=3 : 0.150306, loss: 1.089586, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18945 s, avg_samples: 56.0, ips: 295.59009 samples/s, eta: 1 day, 19:47:14, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:52:11] ppocr INFO: epoch: [1/300], global_step: 900, lr: 0.000100, edit distance: 0.743967, exp_rate: 0.000000, exp_rate<=1 : 0.051456, exp_rate<=2 : 0.093507, exp_rate<=3 : 0.142396, loss: 1.066910, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19262 s, avg_samples: 56.0, ips: 290.72859 samples/s, eta: 1 day, 19:47:56, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:52:29] ppocr INFO: epoch: [1/300], global_step: 1000, lr: 0.000100, edit distance: 0.734705, exp_rate: 0.000000, exp_rate<=1 : 0.046982, exp_rate<=2 : 0.087834, exp_rate<=3 : 0.138242, loss: 1.179451, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18212 s, avg_samples: 56.0, ips: 307.48310 samples/s, eta: 1 day, 19:34:03, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:52:48] ppocr INFO: epoch: [1/300], global_step: 1100, lr: 0.000100, edit distance: 0.726776, exp_rate: 0.000000, exp_rate<=1 : 0.043329, exp_rate<=2 : 0.082308, exp_rate<=3 : 0.132507, loss: 1.076200, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19104 s, avg_samples: 56.0, ips: 293.13330 samples/s, eta: 1 day, 19:33:44, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:53:05] ppocr INFO: epoch: [1/300], global_step: 1200, lr: 0.000100, edit distance: 0.719688, exp_rate: 0.000000, exp_rate<=1 : 0.040619, exp_rate<=2 : 0.079401, exp_rate<=3 : 0.131527, loss: 1.019825, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16806 s, avg_samples: 56.0, ips: 333.21874 samples/s, eta: 1 day, 19:07:13, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:53:24] ppocr INFO: epoch: [1/300], global_step: 1300, lr: 0.000100, edit distance: 0.712087, exp_rate: 0.000000, exp_rate<=1 : 0.037970, exp_rate<=2 : 0.076009, exp_rate<=3 : 0.128290, loss: 0.977357, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19424 s, avg_samples: 56.0, ips: 288.29622 samples/s, eta: 1 day, 19:12:17, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:53:42] ppocr INFO: epoch: [1/300], global_step: 1400, lr: 0.000100, edit distance: 0.704719, exp_rate: 0.000013, exp_rate<=1 : 0.036151, exp_rate<=2 : 0.074703, exp_rate<=3 : 0.129540, loss: 1.067336, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17479 s, avg_samples: 56.0, ips: 320.37804 samples/s, eta: 1 day, 18:57:35, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:54:00] ppocr INFO: epoch: [1/300], global_step: 1500, lr: 0.000100, edit distance: 0.697601, exp_rate: 0.000012, exp_rate<=1 : 0.034540, exp_rate<=2 : 0.072952, exp_rate<=3 : 0.128524, loss: 0.899725, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18296 s, avg_samples: 56.0, ips: 306.07619 samples/s, eta: 1 day, 18:52:15, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:54:20] ppocr INFO: epoch: [1/300], global_step: 1600, lr: 0.000100, edit distance: 0.690881, exp_rate: 0.000011, exp_rate<=1 : 0.033062, exp_rate<=2 : 0.071811, exp_rate<=3 : 0.128140, loss: 0.875717, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19778 s, avg_samples: 56.0, ips: 283.13619 samples/s, eta: 1 day, 19:00:13, max_mem_reserved: 30648 MB, max_mem_allocated: 23148 MB [2024/07/08 01:54:39] ppocr INFO: epoch: [1/300], global_step: 1700, lr: 0.000100, edit distance: 0.684694, exp_rate: 0.000011, exp_rate<=1 : 0.031774, exp_rate<=2 : 0.071037, exp_rate<=3 : 0.128537, loss: 0.915901, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18874 s, avg_samples: 56.0, ips: 296.69793 samples/s, eta: 1 day, 18:59:56, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:54:56] ppocr INFO: epoch: [1/300], global_step: 1800, lr: 0.000100, edit distance: 0.678998, exp_rate: 0.000010, exp_rate<=1 : 0.030518, exp_rate<=2 : 0.070055, exp_rate<=3 : 0.128765, loss: 0.944362, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17448 s, avg_samples: 56.0, ips: 320.95677 samples/s, eta: 1 day, 18:48:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:55:15] ppocr INFO: epoch: [1/300], global_step: 1900, lr: 0.000100, edit distance: 0.673731, exp_rate: 0.000009, exp_rate<=1 : 0.029316, exp_rate<=2 : 0.069018, exp_rate<=3 : 0.128695, loss: 0.955739, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18235 s, avg_samples: 56.0, ips: 307.09417 samples/s, eta: 1 day, 18:44:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:55:32] ppocr INFO: epoch: [1/300], global_step: 2000, lr: 0.000100, edit distance: 0.668607, exp_rate: 0.000009, exp_rate<=1 : 0.028726, exp_rate<=2 : 0.069258, exp_rate<=3 : 0.130306, loss: 0.851141, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17501 s, avg_samples: 56.0, ips: 319.97483 samples/s, eta: 1 day, 18:35:34, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:55:50] ppocr INFO: epoch: [1/300], global_step: 2100, lr: 0.000100, edit distance: 0.663282, exp_rate: 0.000009, exp_rate<=1 : 0.028291, exp_rate<=2 : 0.069798, exp_rate<=3 : 0.131873, loss: 0.831465, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17878 s, avg_samples: 56.0, ips: 313.22559 samples/s, eta: 1 day, 18:29:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:56:11] ppocr INFO: epoch: [1/300], global_step: 2200, lr: 0.000100, edit distance: 0.657978, exp_rate: 0.000008, exp_rate<=1 : 0.027204, exp_rate<=2 : 0.068243, exp_rate<=3 : 0.130006, loss: 0.901045, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.21018 s, avg_samples: 56.0, ips: 266.43428 samples/s, eta: 1 day, 18:44:15, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:56:30] ppocr INFO: epoch: [1/300], global_step: 2300, lr: 0.000100, edit distance: 0.653906, exp_rate: 0.000008, exp_rate<=1 : 0.026679, exp_rate<=2 : 0.068115, exp_rate<=3 : 0.131141, loss: 0.871360, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18520 s, avg_samples: 56.0, ips: 302.37196 samples/s, eta: 1 day, 18:42:28, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:56:49] ppocr INFO: epoch: [1/300], global_step: 2400, lr: 0.000100, edit distance: 0.649486, exp_rate: 0.000007, exp_rate<=1 : 0.026251, exp_rate<=2 : 0.068207, exp_rate<=3 : 0.132095, loss: 0.903299, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19425 s, avg_samples: 56.0, ips: 288.29252 samples/s, eta: 1 day, 18:45:57, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:57:09] ppocr INFO: epoch: [1/300], global_step: 2500, lr: 0.000100, edit distance: 0.645161, exp_rate: 0.000007, exp_rate<=1 : 0.025628, exp_rate<=2 : 0.067714, exp_rate<=3 : 0.131950, loss: 0.814229, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19974 s, avg_samples: 56.0, ips: 280.35918 samples/s, eta: 1 day, 18:52:09, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:57:28] ppocr INFO: epoch: [1/300], global_step: 2600, lr: 0.000100, edit distance: 0.640922, exp_rate: 0.000007, exp_rate<=1 : 0.025307, exp_rate<=2 : 0.067890, exp_rate<=3 : 0.132788, loss: 0.795772, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18448 s, avg_samples: 56.0, ips: 303.55635 samples/s, eta: 1 day, 18:49:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:57:46] ppocr INFO: epoch: [1/300], global_step: 2700, lr: 0.000100, edit distance: 0.636994, exp_rate: 0.000007, exp_rate<=1 : 0.024782, exp_rate<=2 : 0.067566, exp_rate<=3 : 0.132880, loss: 0.842802, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18575 s, avg_samples: 56.0, ips: 301.47885 samples/s, eta: 1 day, 18:48:17, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:57:54] ppocr INFO: epoch: [1/300], global_step: 2740, lr: 0.000100, edit distance: 0.635499, exp_rate: 0.000007, exp_rate<=1 : 0.024810, exp_rate<=2 : 0.068025, exp_rate<=3 : 0.134028, loss: 0.899766, avg_reader_cost: 0.00004 s, avg_batch_cost: 0.07937 s, avg_samples: 22.4, ips: 282.20603 samples/s, eta: 1 day, 18:50:13, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:57:55] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 01:58:08] ppocr INFO: epoch: [2/300], global_step: 2800, lr: 0.000100, edit distance: 0.523637, exp_rate: 0.000000, exp_rate<=1 : 0.012906, exp_rate<=2 : 0.055563, exp_rate<=3 : 0.131324, loss: 0.849735, avg_reader_cost: 0.01186 s, avg_batch_cost: 0.13815 s, avg_samples: 33.6, ips: 243.20742 samples/s, eta: 1 day, 19:02:19, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:58:28] ppocr INFO: epoch: [2/300], global_step: 2900, lr: 0.000100, edit distance: 0.519591, exp_rate: 0.000119, exp_rate<=1 : 0.017314, exp_rate<=2 : 0.074247, exp_rate<=3 : 0.160836, loss: 0.796270, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.65210 samples/s, eta: 1 day, 19:05:35, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:58:47] ppocr INFO: epoch: [2/300], global_step: 3000, lr: 0.000100, edit distance: 0.517214, exp_rate: 0.000071, exp_rate<=1 : 0.016414, exp_rate<=2 : 0.074710, exp_rate<=3 : 0.166487, loss: 0.838358, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19004 s, avg_samples: 56.0, ips: 294.68150 samples/s, eta: 1 day, 19:05:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:59:06] ppocr INFO: epoch: [2/300], global_step: 3100, lr: 0.000100, edit distance: 0.515433, exp_rate: 0.000051, exp_rate<=1 : 0.015118, exp_rate<=2 : 0.071632, exp_rate<=3 : 0.159819, loss: 0.812214, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19385 s, avg_samples: 56.0, ips: 288.88787 samples/s, eta: 1 day, 19:07:11, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:59:24] ppocr INFO: epoch: [2/300], global_step: 3200, lr: 0.000100, edit distance: 0.511004, exp_rate: 0.000040, exp_rate<=1 : 0.015823, exp_rate<=2 : 0.074870, exp_rate<=3 : 0.166217, loss: 0.822068, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18179 s, avg_samples: 56.0, ips: 308.05156 samples/s, eta: 1 day, 19:03:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 01:59:44] ppocr INFO: epoch: [2/300], global_step: 3300, lr: 0.000100, edit distance: 0.507428, exp_rate: 0.000032, exp_rate<=1 : 0.016122, exp_rate<=2 : 0.075452, exp_rate<=3 : 0.167340, loss: 0.822428, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19310 s, avg_samples: 56.0, ips: 290.00600 samples/s, eta: 1 day, 19:04:48, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:00:05] ppocr INFO: epoch: [2/300], global_step: 3400, lr: 0.000100, edit distance: 0.504362, exp_rate: 0.000027, exp_rate<=1 : 0.015645, exp_rate<=2 : 0.072540, exp_rate<=3 : 0.161149, loss: 0.765157, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21044 s, avg_samples: 56.0, ips: 266.11281 samples/s, eta: 1 day, 19:12:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:00:24] ppocr INFO: epoch: [2/300], global_step: 3500, lr: 0.000100, edit distance: 0.502535, exp_rate: 0.000024, exp_rate<=1 : 0.015551, exp_rate<=2 : 0.071796, exp_rate<=3 : 0.159737, loss: 0.765810, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19452 s, avg_samples: 56.0, ips: 287.89410 samples/s, eta: 1 day, 19:14:21, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:00:43] ppocr INFO: epoch: [2/300], global_step: 3600, lr: 0.000100, edit distance: 0.500242, exp_rate: 0.000021, exp_rate<=1 : 0.016192, exp_rate<=2 : 0.073218, exp_rate<=3 : 0.161985, loss: 0.754396, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18386 s, avg_samples: 56.0, ips: 304.58081 samples/s, eta: 1 day, 19:11:38, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:01:02] ppocr INFO: epoch: [2/300], global_step: 3700, lr: 0.000100, edit distance: 0.497626, exp_rate: 0.000019, exp_rate<=1 : 0.015503, exp_rate<=2 : 0.071137, exp_rate<=3 : 0.157820, loss: 0.717921, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19717 s, avg_samples: 56.0, ips: 284.01357 samples/s, eta: 1 day, 19:13:57, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:01:22] ppocr INFO: epoch: [2/300], global_step: 3800, lr: 0.000100, edit distance: 0.494900, exp_rate: 0.000034, exp_rate<=1 : 0.015722, exp_rate<=2 : 0.072690, exp_rate<=3 : 0.160308, loss: 0.686552, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19253 s, avg_samples: 56.0, ips: 290.86783 samples/s, eta: 1 day, 19:14:29, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:01:41] ppocr INFO: epoch: [2/300], global_step: 3900, lr: 0.000100, edit distance: 0.491491, exp_rate: 0.000078, exp_rate<=1 : 0.016887, exp_rate<=2 : 0.076326, exp_rate<=3 : 0.165420, loss: 0.702812, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19809 s, avg_samples: 56.0, ips: 282.69956 samples/s, eta: 1 day, 19:16:54, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:02:00] ppocr INFO: epoch: [2/300], global_step: 4000, lr: 0.000100, edit distance: 0.488334, exp_rate: 0.000071, exp_rate<=1 : 0.016931, exp_rate<=2 : 0.077082, exp_rate<=3 : 0.166875, loss: 0.733292, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18321 s, avg_samples: 56.0, ips: 305.66618 samples/s, eta: 1 day, 19:14:07, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:02:17] ppocr INFO: epoch: [2/300], global_step: 4100, lr: 0.000100, edit distance: 0.484918, exp_rate: 0.000079, exp_rate<=1 : 0.017378, exp_rate<=2 : 0.078440, exp_rate<=3 : 0.169290, loss: 0.663393, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.17613 s, avg_samples: 56.0, ips: 317.93945 samples/s, eta: 1 day, 19:09:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:02:36] ppocr INFO: epoch: [2/300], global_step: 4200, lr: 0.000100, edit distance: 0.482204, exp_rate: 0.000086, exp_rate<=1 : 0.018271, exp_rate<=2 : 0.080301, exp_rate<=3 : 0.171849, loss: 0.642726, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18743 s, avg_samples: 56.0, ips: 298.77941 samples/s, eta: 1 day, 19:07:58, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:02:55] ppocr INFO: epoch: [2/300], global_step: 4300, lr: 0.000100, edit distance: 0.479873, exp_rate: 0.000081, exp_rate<=1 : 0.018225, exp_rate<=2 : 0.079253, exp_rate<=3 : 0.170811, loss: 0.619364, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19063 s, avg_samples: 56.0, ips: 293.76078 samples/s, eta: 1 day, 19:07:53, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:03:14] ppocr INFO: epoch: [2/300], global_step: 4400, lr: 0.000100, edit distance: 0.477175, exp_rate: 0.000097, exp_rate<=1 : 0.018735, exp_rate<=2 : 0.081085, exp_rate<=3 : 0.173643, loss: 0.584330, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18938 s, avg_samples: 56.0, ips: 295.69718 samples/s, eta: 1 day, 19:07:25, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:03:32] ppocr INFO: epoch: [2/300], global_step: 4500, lr: 0.000100, edit distance: 0.473993, exp_rate: 0.000092, exp_rate<=1 : 0.019753, exp_rate<=2 : 0.084583, exp_rate<=3 : 0.178806, loss: 0.586982, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18153 s, avg_samples: 56.0, ips: 308.48787 samples/s, eta: 1 day, 19:04:34, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:03:51] ppocr INFO: epoch: [2/300], global_step: 4600, lr: 0.000100, edit distance: 0.471481, exp_rate: 0.000096, exp_rate<=1 : 0.020450, exp_rate<=2 : 0.086629, exp_rate<=3 : 0.181651, loss: 0.698251, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18880 s, avg_samples: 56.0, ips: 296.60729 samples/s, eta: 1 day, 19:03:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:04:11] ppocr INFO: epoch: [2/300], global_step: 4700, lr: 0.000100, edit distance: 0.468552, exp_rate: 0.000110, exp_rate<=1 : 0.020776, exp_rate<=2 : 0.087252, exp_rate<=3 : 0.182690, loss: 0.622576, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19360 s, avg_samples: 56.0, ips: 289.25725 samples/s, eta: 1 day, 19:04:48, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:04:28] ppocr INFO: epoch: [2/300], global_step: 4800, lr: 0.000100, edit distance: 0.465704, exp_rate: 0.000165, exp_rate<=1 : 0.022324, exp_rate<=2 : 0.090835, exp_rate<=3 : 0.188249, loss: 0.598428, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17154 s, avg_samples: 56.0, ips: 326.45029 samples/s, eta: 1 day, 18:59:19, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:04:46] ppocr INFO: epoch: [2/300], global_step: 4900, lr: 0.000100, edit distance: 0.462966, exp_rate: 0.000183, exp_rate<=1 : 0.023640, exp_rate<=2 : 0.093950, exp_rate<=3 : 0.192067, loss: 0.600865, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18415 s, avg_samples: 56.0, ips: 304.09478 samples/s, eta: 1 day, 18:57:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:05:04] ppocr INFO: epoch: [2/300], global_step: 5000, lr: 0.000100, edit distance: 0.460386, exp_rate: 0.000198, exp_rate<=1 : 0.024614, exp_rate<=2 : 0.096613, exp_rate<=3 : 0.195782, loss: 0.537568, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17444 s, avg_samples: 56.0, ips: 321.03475 samples/s, eta: 1 day, 18:53:12, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:05:23] ppocr INFO: epoch: [2/300], global_step: 5100, lr: 0.000100, edit distance: 0.456800, exp_rate: 0.000220, exp_rate<=1 : 0.024891, exp_rate<=2 : 0.097097, exp_rate<=3 : 0.196539, loss: 0.563829, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19723 s, avg_samples: 56.0, ips: 283.92661 samples/s, eta: 1 day, 18:55:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:05:42] ppocr INFO: epoch: [2/300], global_step: 5200, lr: 0.000100, edit distance: 0.453860, exp_rate: 0.000255, exp_rate<=1 : 0.026210, exp_rate<=2 : 0.100329, exp_rate<=3 : 0.201256, loss: 0.538376, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18453 s, avg_samples: 56.0, ips: 303.47936 samples/s, eta: 1 day, 18:53:34, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:06:00] ppocr INFO: epoch: [2/300], global_step: 5300, lr: 0.000100, edit distance: 0.450930, exp_rate: 0.000321, exp_rate<=1 : 0.027330, exp_rate<=2 : 0.103441, exp_rate<=3 : 0.205693, loss: 0.564374, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18064 s, avg_samples: 56.0, ips: 310.00099 samples/s, eta: 1 day, 18:51:06, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:06:19] ppocr INFO: epoch: [2/300], global_step: 5400, lr: 0.000100, edit distance: 0.447706, exp_rate: 0.000357, exp_rate<=1 : 0.028032, exp_rate<=2 : 0.105545, exp_rate<=3 : 0.208749, loss: 0.534193, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18844 s, avg_samples: 56.0, ips: 297.17656 samples/s, eta: 1 day, 18:50:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:06:34] ppocr INFO: epoch: [2/300], global_step: 5480, lr: 0.000100, edit distance: 0.445272, exp_rate: 0.000386, exp_rate<=1 : 0.028895, exp_rate<=2 : 0.107624, exp_rate<=3 : 0.211404, loss: 0.470474, avg_reader_cost: 0.00008 s, avg_batch_cost: 0.15040 s, avg_samples: 44.8, ips: 297.86433 samples/s, eta: 1 day, 18:50:14, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:06:36] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:06:40] ppocr INFO: epoch: [3/300], global_step: 5500, lr: 0.000100, edit distance: 0.347317, exp_rate: 0.003409, exp_rate<=1 : 0.075812, exp_rate<=2 : 0.253896, exp_rate<=3 : 0.402841, loss: 0.507990, avg_reader_cost: 0.02391 s, avg_batch_cost: 0.05886 s, avg_samples: 11.2, ips: 190.26701 samples/s, eta: 1 day, 18:55:24, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:06:57] ppocr INFO: epoch: [3/300], global_step: 5600, lr: 0.000100, edit distance: 0.347322, exp_rate: 0.002082, exp_rate<=1 : 0.061364, exp_rate<=2 : 0.197445, exp_rate<=3 : 0.334377, loss: 0.473079, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17553 s, avg_samples: 56.0, ips: 319.03285 samples/s, eta: 1 day, 18:51:45, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:07:16] ppocr INFO: epoch: [3/300], global_step: 5700, lr: 0.000100, edit distance: 0.346405, exp_rate: 0.002799, exp_rate<=1 : 0.066035, exp_rate<=2 : 0.199325, exp_rate<=3 : 0.336133, loss: 0.494086, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18740 s, avg_samples: 56.0, ips: 298.82059 samples/s, eta: 1 day, 18:51:03, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:07:35] ppocr INFO: epoch: [3/300], global_step: 5800, lr: 0.000100, edit distance: 0.345647, exp_rate: 0.002571, exp_rate<=1 : 0.068564, exp_rate<=2 : 0.198062, exp_rate<=3 : 0.334638, loss: 0.548589, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18407 s, avg_samples: 56.0, ips: 304.23102 samples/s, eta: 1 day, 18:49:35, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:07:53] ppocr INFO: epoch: [3/300], global_step: 5900, lr: 0.000100, edit distance: 0.343578, exp_rate: 0.002789, exp_rate<=1 : 0.069593, exp_rate<=2 : 0.197706, exp_rate<=3 : 0.333057, loss: 0.447227, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18656 s, avg_samples: 56.0, ips: 300.17871 samples/s, eta: 1 day, 18:48:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:08:13] ppocr INFO: epoch: [3/300], global_step: 6000, lr: 0.000100, edit distance: 0.340808, exp_rate: 0.002738, exp_rate<=1 : 0.068233, exp_rate<=2 : 0.196986, exp_rate<=3 : 0.331626, loss: 0.411847, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19699 s, avg_samples: 56.0, ips: 284.27489 samples/s, eta: 1 day, 18:50:15, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:08:31] ppocr INFO: epoch: [3/300], global_step: 6100, lr: 0.000100, edit distance: 0.337516, exp_rate: 0.003276, exp_rate<=1 : 0.071121, exp_rate<=2 : 0.202947, exp_rate<=3 : 0.338614, loss: 0.455211, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17876 s, avg_samples: 56.0, ips: 313.26158 samples/s, eta: 1 day, 18:47:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:08:50] ppocr INFO: epoch: [3/300], global_step: 6200, lr: 0.000100, edit distance: 0.334317, exp_rate: 0.003501, exp_rate<=1 : 0.072947, exp_rate<=2 : 0.206432, exp_rate<=3 : 0.343243, loss: 0.471775, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18564 s, avg_samples: 56.0, ips: 301.66096 samples/s, eta: 1 day, 18:46:39, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:09:08] ppocr INFO: epoch: [3/300], global_step: 6300, lr: 0.000100, edit distance: 0.331937, exp_rate: 0.003874, exp_rate<=1 : 0.073413, exp_rate<=2 : 0.206737, exp_rate<=3 : 0.344178, loss: 0.401549, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18963 s, avg_samples: 56.0, ips: 295.31128 samples/s, eta: 1 day, 18:46:31, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:09:29] ppocr INFO: epoch: [3/300], global_step: 6400, lr: 0.000100, edit distance: 0.328997, exp_rate: 0.004386, exp_rate<=1 : 0.074503, exp_rate<=2 : 0.208943, exp_rate<=3 : 0.346839, loss: 0.442144, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20082 s, avg_samples: 56.0, ips: 278.85227 samples/s, eta: 1 day, 18:48:46, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:09:47] ppocr INFO: epoch: [3/300], global_step: 6500, lr: 0.000100, edit distance: 0.325627, exp_rate: 0.004640, exp_rate<=1 : 0.075141, exp_rate<=2 : 0.210174, exp_rate<=3 : 0.347360, loss: 0.424626, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18553 s, avg_samples: 56.0, ips: 301.83367 samples/s, eta: 1 day, 18:47:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:10:06] ppocr INFO: epoch: [3/300], global_step: 6600, lr: 0.000100, edit distance: 0.323426, exp_rate: 0.004708, exp_rate<=1 : 0.076182, exp_rate<=2 : 0.213019, exp_rate<=3 : 0.350182, loss: 0.454042, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19146 s, avg_samples: 56.0, ips: 292.48488 samples/s, eta: 1 day, 18:47:56, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:10:26] ppocr INFO: epoch: [3/300], global_step: 6700, lr: 0.000100, edit distance: 0.320854, exp_rate: 0.004904, exp_rate<=1 : 0.076415, exp_rate<=2 : 0.213122, exp_rate<=3 : 0.349745, loss: 0.411485, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.19764 s, avg_samples: 56.0, ips: 283.34999 samples/s, eta: 1 day, 18:49:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:10:44] ppocr INFO: epoch: [3/300], global_step: 6800, lr: 0.000100, edit distance: 0.318022, exp_rate: 0.005192, exp_rate<=1 : 0.077869, exp_rate<=2 : 0.215987, exp_rate<=3 : 0.354186, loss: 0.378094, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18335 s, avg_samples: 56.0, ips: 305.42546 samples/s, eta: 1 day, 18:47:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:11:03] ppocr INFO: epoch: [3/300], global_step: 6900, lr: 0.000100, edit distance: 0.315076, exp_rate: 0.005389, exp_rate<=1 : 0.078269, exp_rate<=2 : 0.216070, exp_rate<=3 : 0.355416, loss: 0.388167, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18659 s, avg_samples: 56.0, ips: 300.12941 samples/s, eta: 1 day, 18:47:08, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:11:22] ppocr INFO: epoch: [3/300], global_step: 7000, lr: 0.000100, edit distance: 0.312606, exp_rate: 0.005507, exp_rate<=1 : 0.079789, exp_rate<=2 : 0.218196, exp_rate<=3 : 0.358520, loss: 0.407685, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18733 s, avg_samples: 56.0, ips: 298.94061 samples/s, eta: 1 day, 18:46:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:11:42] ppocr INFO: epoch: [3/300], global_step: 7100, lr: 0.000100, edit distance: 0.309994, exp_rate: 0.005790, exp_rate<=1 : 0.080780, exp_rate<=2 : 0.219302, exp_rate<=3 : 0.359572, loss: 0.359091, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19984 s, avg_samples: 56.0, ips: 280.22771 samples/s, eta: 1 day, 18:48:16, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:12:00] ppocr INFO: epoch: [3/300], global_step: 7200, lr: 0.000100, edit distance: 0.307517, exp_rate: 0.006141, exp_rate<=1 : 0.082455, exp_rate<=2 : 0.221976, exp_rate<=3 : 0.362437, loss: 0.390039, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18164 s, avg_samples: 56.0, ips: 308.29857 samples/s, eta: 1 day, 18:46:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:12:19] ppocr INFO: epoch: [3/300], global_step: 7300, lr: 0.000100, edit distance: 0.305238, exp_rate: 0.006585, exp_rate<=1 : 0.084433, exp_rate<=2 : 0.224968, exp_rate<=3 : 0.365987, loss: 0.400022, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18709 s, avg_samples: 56.0, ips: 299.32398 samples/s, eta: 1 day, 18:45:52, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:12:38] ppocr INFO: epoch: [3/300], global_step: 7400, lr: 0.000100, edit distance: 0.302589, exp_rate: 0.006957, exp_rate<=1 : 0.086202, exp_rate<=2 : 0.227082, exp_rate<=3 : 0.368326, loss: 0.340600, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19623 s, avg_samples: 56.0, ips: 285.37667 samples/s, eta: 1 day, 18:46:54, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:12:58] ppocr INFO: epoch: [3/300], global_step: 7500, lr: 0.000100, edit distance: 0.300126, exp_rate: 0.007503, exp_rate<=1 : 0.087693, exp_rate<=2 : 0.229341, exp_rate<=3 : 0.370514, loss: 0.318161, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19618 s, avg_samples: 56.0, ips: 285.45787 samples/s, eta: 1 day, 18:47:52, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:13:16] ppocr INFO: epoch: [3/300], global_step: 7600, lr: 0.000100, edit distance: 0.297595, exp_rate: 0.007873, exp_rate<=1 : 0.089704, exp_rate<=2 : 0.232647, exp_rate<=3 : 0.374061, loss: 0.331724, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18326 s, avg_samples: 56.0, ips: 305.57942 samples/s, eta: 1 day, 18:46:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:13:35] ppocr INFO: epoch: [3/300], global_step: 7700, lr: 0.000100, edit distance: 0.295203, exp_rate: 0.008304, exp_rate<=1 : 0.091609, exp_rate<=2 : 0.235751, exp_rate<=3 : 0.377527, loss: 0.321450, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18895 s, avg_samples: 56.0, ips: 296.37346 samples/s, eta: 1 day, 18:46:09, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:13:54] ppocr INFO: epoch: [3/300], global_step: 7800, lr: 0.000100, edit distance: 0.293122, exp_rate: 0.008755, exp_rate<=1 : 0.093409, exp_rate<=2 : 0.238535, exp_rate<=3 : 0.380492, loss: 0.342589, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19021 s, avg_samples: 56.0, ips: 294.41553 samples/s, eta: 1 day, 18:46:02, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:14:14] ppocr INFO: epoch: [3/300], global_step: 7900, lr: 0.000100, edit distance: 0.291002, exp_rate: 0.009210, exp_rate<=1 : 0.095174, exp_rate<=2 : 0.241536, exp_rate<=3 : 0.383762, loss: 0.334330, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19269 s, avg_samples: 56.0, ips: 290.62762 samples/s, eta: 1 day, 18:46:20, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:14:33] ppocr INFO: epoch: [3/300], global_step: 8000, lr: 0.000100, edit distance: 0.288468, exp_rate: 0.009759, exp_rate<=1 : 0.096954, exp_rate<=2 : 0.244332, exp_rate<=3 : 0.387147, loss: 0.316873, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18850 s, avg_samples: 56.0, ips: 297.07839 samples/s, eta: 1 day, 18:45:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:14:52] ppocr INFO: epoch: [3/300], global_step: 8100, lr: 0.000100, edit distance: 0.286331, exp_rate: 0.010312, exp_rate<=1 : 0.098896, exp_rate<=2 : 0.247214, exp_rate<=3 : 0.390281, loss: 0.357692, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19585 s, avg_samples: 56.0, ips: 285.93005 samples/s, eta: 1 day, 18:46:43, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:15:11] ppocr INFO: epoch: [3/300], global_step: 8200, lr: 0.000100, edit distance: 0.284282, exp_rate: 0.010714, exp_rate<=1 : 0.100488, exp_rate<=2 : 0.249516, exp_rate<=3 : 0.393045, loss: 0.307594, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18584 s, avg_samples: 56.0, ips: 301.34072 samples/s, eta: 1 day, 18:45:51, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:15:14] ppocr INFO: epoch: [3/300], global_step: 8220, lr: 0.000100, edit distance: 0.283847, exp_rate: 0.010830, exp_rate<=1 : 0.101070, exp_rate<=2 : 0.250418, exp_rate<=3 : 0.394031, loss: 0.281341, avg_reader_cost: 0.00002 s, avg_batch_cost: 0.03290 s, avg_samples: 11.2, ips: 340.42188 samples/s, eta: 1 day, 18:44:58, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:15:16] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:15:33] ppocr INFO: epoch: [4/300], global_step: 8300, lr: 0.000100, edit distance: 0.209776, exp_rate: 0.029973, exp_rate<=1 : 0.161616, exp_rate<=2 : 0.340929, exp_rate<=3 : 0.486555, loss: 0.266500, avg_reader_cost: 0.02385 s, avg_batch_cost: 0.18766 s, avg_samples: 44.8, ips: 238.72545 samples/s, eta: 1 day, 18:50:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:15:51] ppocr INFO: epoch: [4/300], global_step: 8400, lr: 0.000100, edit distance: 0.207951, exp_rate: 0.031810, exp_rate<=1 : 0.166245, exp_rate<=2 : 0.342972, exp_rate<=3 : 0.492391, loss: 0.257990, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18124 s, avg_samples: 56.0, ips: 308.97794 samples/s, eta: 1 day, 18:49:00, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:16:09] ppocr INFO: epoch: [4/300], global_step: 8500, lr: 0.000100, edit distance: 0.204883, exp_rate: 0.034876, exp_rate<=1 : 0.176723, exp_rate<=2 : 0.363424, exp_rate<=3 : 0.510858, loss: 0.291786, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18031 s, avg_samples: 56.0, ips: 310.58395 samples/s, eta: 1 day, 18:47:14, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:16:28] ppocr INFO: epoch: [4/300], global_step: 8600, lr: 0.000100, edit distance: 0.203329, exp_rate: 0.035666, exp_rate<=1 : 0.179342, exp_rate<=2 : 0.367877, exp_rate<=3 : 0.516919, loss: 0.281453, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18887 s, avg_samples: 56.0, ips: 296.50072 samples/s, eta: 1 day, 18:46:50, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:16:46] ppocr INFO: epoch: [4/300], global_step: 8700, lr: 0.000100, edit distance: 0.203382, exp_rate: 0.034194, exp_rate<=1 : 0.175682, exp_rate<=2 : 0.362858, exp_rate<=3 : 0.512692, loss: 0.289966, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18348 s, avg_samples: 56.0, ips: 305.21518 samples/s, eta: 1 day, 18:45:37, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:17:05] ppocr INFO: epoch: [4/300], global_step: 8800, lr: 0.000100, edit distance: 0.203482, exp_rate: 0.035368, exp_rate<=1 : 0.175806, exp_rate<=2 : 0.362092, exp_rate<=3 : 0.510821, loss: 0.282130, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18596 s, avg_samples: 56.0, ips: 301.13851 samples/s, eta: 1 day, 18:44:47, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:17:24] ppocr INFO: epoch: [4/300], global_step: 8900, lr: 0.000100, edit distance: 0.201924, exp_rate: 0.035406, exp_rate<=1 : 0.175202, exp_rate<=2 : 0.360289, exp_rate<=3 : 0.510178, loss: 0.236218, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19316 s, avg_samples: 56.0, ips: 289.90820 samples/s, eta: 1 day, 18:45:04, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:17:43] ppocr INFO: epoch: [4/300], global_step: 9000, lr: 0.000100, edit distance: 0.200606, exp_rate: 0.036256, exp_rate<=1 : 0.178698, exp_rate<=2 : 0.365458, exp_rate<=3 : 0.516440, loss: 0.232962, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18426 s, avg_samples: 56.0, ips: 303.91667 samples/s, eta: 1 day, 18:44:00, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:18:01] ppocr INFO: epoch: [4/300], global_step: 9100, lr: 0.000100, edit distance: 0.199784, exp_rate: 0.037654, exp_rate<=1 : 0.181980, exp_rate<=2 : 0.369424, exp_rate<=3 : 0.520684, loss: 0.263835, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18557 s, avg_samples: 56.0, ips: 301.77777 samples/s, eta: 1 day, 18:43:08, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:18:20] ppocr INFO: epoch: [4/300], global_step: 9200, lr: 0.000100, edit distance: 0.198430, exp_rate: 0.038623, exp_rate<=1 : 0.184613, exp_rate<=2 : 0.373707, exp_rate<=3 : 0.525729, loss: 0.263151, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18353 s, avg_samples: 56.0, ips: 305.12693 samples/s, eta: 1 day, 18:41:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:18:39] ppocr INFO: epoch: [4/300], global_step: 9300, lr: 0.000100, edit distance: 0.197420, exp_rate: 0.038982, exp_rate<=1 : 0.185262, exp_rate<=2 : 0.374048, exp_rate<=3 : 0.525691, loss: 0.251930, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19449 s, avg_samples: 56.0, ips: 287.93882 samples/s, eta: 1 day, 18:42:27, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:18:58] ppocr INFO: epoch: [4/300], global_step: 9400, lr: 0.000100, edit distance: 0.196586, exp_rate: 0.039621, exp_rate<=1 : 0.187019, exp_rate<=2 : 0.376599, exp_rate<=3 : 0.528281, loss: 0.283673, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19028 s, avg_samples: 56.0, ips: 294.29560 samples/s, eta: 1 day, 18:42:18, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:19:18] ppocr INFO: epoch: [4/300], global_step: 9500, lr: 0.000100, edit distance: 0.195479, exp_rate: 0.040864, exp_rate<=1 : 0.189308, exp_rate<=2 : 0.380372, exp_rate<=3 : 0.531343, loss: 0.235014, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19396 s, avg_samples: 56.0, ips: 288.72454 samples/s, eta: 1 day, 18:42:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:19:38] ppocr INFO: epoch: [4/300], global_step: 9600, lr: 0.000100, edit distance: 0.194169, exp_rate: 0.040305, exp_rate<=1 : 0.186951, exp_rate<=2 : 0.377221, exp_rate<=3 : 0.528940, loss: 0.219719, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20490 s, avg_samples: 56.0, ips: 273.29789 samples/s, eta: 1 day, 18:44:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:19:57] ppocr INFO: epoch: [4/300], global_step: 9700, lr: 0.000100, edit distance: 0.193295, exp_rate: 0.040377, exp_rate<=1 : 0.187749, exp_rate<=2 : 0.377838, exp_rate<=3 : 0.530371, loss: 0.244691, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19360 s, avg_samples: 56.0, ips: 289.24918 samples/s, eta: 1 day, 18:44:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:20:17] ppocr INFO: epoch: [4/300], global_step: 9800, lr: 0.000100, edit distance: 0.192052, exp_rate: 0.040960, exp_rate<=1 : 0.189058, exp_rate<=2 : 0.379429, exp_rate<=3 : 0.531927, loss: 0.233239, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19315 s, avg_samples: 56.0, ips: 289.92961 samples/s, eta: 1 day, 18:45:01, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:20:37] ppocr INFO: epoch: [4/300], global_step: 9900, lr: 0.000100, edit distance: 0.191007, exp_rate: 0.040863, exp_rate<=1 : 0.187919, exp_rate<=2 : 0.377975, exp_rate<=3 : 0.530632, loss: 0.214335, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20368 s, avg_samples: 56.0, ips: 274.93540 samples/s, eta: 1 day, 18:46:39, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:20:55] ppocr INFO: epoch: [4/300], global_step: 10000, lr: 0.000100, edit distance: 0.189952, exp_rate: 0.041614, exp_rate<=1 : 0.190486, exp_rate<=2 : 0.381431, exp_rate<=3 : 0.533837, loss: 0.221936, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17921 s, avg_samples: 56.0, ips: 312.48276 samples/s, eta: 1 day, 18:44:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:21:14] ppocr INFO: epoch: [4/300], global_step: 10100, lr: 0.000100, edit distance: 0.189254, exp_rate: 0.041994, exp_rate<=1 : 0.192359, exp_rate<=2 : 0.383909, exp_rate<=3 : 0.536264, loss: 0.214971, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18763 s, avg_samples: 56.0, ips: 298.45745 samples/s, eta: 1 day, 18:44:21, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:21:37] ppocr INFO: epoch: [4/300], global_step: 10200, lr: 0.000100, edit distance: 0.188320, exp_rate: 0.042693, exp_rate<=1 : 0.193662, exp_rate<=2 : 0.385194, exp_rate<=3 : 0.537737, loss: 0.227539, avg_reader_cost: 0.00017 s, avg_batch_cost: 0.22986 s, avg_samples: 56.0, ips: 243.62954 samples/s, eta: 1 day, 18:49:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:21:56] ppocr INFO: epoch: [4/300], global_step: 10300, lr: 0.000100, edit distance: 0.186961, exp_rate: 0.043517, exp_rate<=1 : 0.194994, exp_rate<=2 : 0.387256, exp_rate<=3 : 0.539764, loss: 0.228508, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18680 s, avg_samples: 56.0, ips: 299.78148 samples/s, eta: 1 day, 18:48:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:22:13] ppocr INFO: epoch: [4/300], global_step: 10400, lr: 0.000100, edit distance: 0.185759, exp_rate: 0.044265, exp_rate<=1 : 0.196923, exp_rate<=2 : 0.390618, exp_rate<=3 : 0.543497, loss: 0.217764, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.16867 s, avg_samples: 56.0, ips: 332.00114 samples/s, eta: 1 day, 18:45:35, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:22:32] ppocr INFO: epoch: [4/300], global_step: 10500, lr: 0.000100, edit distance: 0.184921, exp_rate: 0.044712, exp_rate<=1 : 0.197951, exp_rate<=2 : 0.391464, exp_rate<=3 : 0.544311, loss: 0.227694, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19176 s, avg_samples: 56.0, ips: 292.02983 samples/s, eta: 1 day, 18:45:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:22:50] ppocr INFO: epoch: [4/300], global_step: 10600, lr: 0.000100, edit distance: 0.183786, exp_rate: 0.045672, exp_rate<=1 : 0.199824, exp_rate<=2 : 0.393546, exp_rate<=3 : 0.546307, loss: 0.223798, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18433 s, avg_samples: 56.0, ips: 303.80757 samples/s, eta: 1 day, 18:44:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:23:09] ppocr INFO: epoch: [4/300], global_step: 10700, lr: 0.000100, edit distance: 0.182703, exp_rate: 0.046632, exp_rate<=1 : 0.201722, exp_rate<=2 : 0.396087, exp_rate<=3 : 0.549067, loss: 0.242836, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19067 s, avg_samples: 56.0, ips: 293.70519 samples/s, eta: 1 day, 18:44:21, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:23:29] ppocr INFO: epoch: [4/300], global_step: 10800, lr: 0.000100, edit distance: 0.181602, exp_rate: 0.047233, exp_rate<=1 : 0.202537, exp_rate<=2 : 0.396950, exp_rate<=3 : 0.549976, loss: 0.208638, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19749 s, avg_samples: 56.0, ips: 283.55475 samples/s, eta: 1 day, 18:45:01, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:23:47] ppocr INFO: epoch: [4/300], global_step: 10900, lr: 0.000100, edit distance: 0.180730, exp_rate: 0.047735, exp_rate<=1 : 0.203747, exp_rate<=2 : 0.398337, exp_rate<=3 : 0.551585, loss: 0.203497, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17924 s, avg_samples: 56.0, ips: 312.43400 samples/s, eta: 1 day, 18:43:24, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:23:57] ppocr INFO: epoch: [4/300], global_step: 10960, lr: 0.000100, edit distance: 0.180086, exp_rate: 0.048406, exp_rate<=1 : 0.205137, exp_rate<=2 : 0.400041, exp_rate<=3 : 0.553313, loss: 0.229597, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.10459 s, avg_samples: 33.6, ips: 321.25913 samples/s, eta: 1 day, 18:42:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:23:59] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:24:07] ppocr INFO: epoch: [5/300], global_step: 11000, lr: 0.000100, edit distance: 0.141388, exp_rate: 0.076096, exp_rate<=1 : 0.244728, exp_rate<=2 : 0.463376, exp_rate<=3 : 0.633542, loss: 0.197742, avg_reader_cost: 0.02493 s, avg_batch_cost: 0.09746 s, avg_samples: 22.4, ips: 229.84260 samples/s, eta: 1 day, 18:44:37, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:24:26] ppocr INFO: epoch: [5/300], global_step: 11100, lr: 0.000100, edit distance: 0.142528, exp_rate: 0.085672, exp_rate<=1 : 0.258827, exp_rate<=2 : 0.460077, exp_rate<=3 : 0.616444, loss: 0.178756, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18754 s, avg_samples: 56.0, ips: 298.59810 samples/s, eta: 1 day, 18:44:02, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:24:46] ppocr INFO: epoch: [5/300], global_step: 11200, lr: 0.000100, edit distance: 0.140011, exp_rate: 0.084897, exp_rate<=1 : 0.258031, exp_rate<=2 : 0.459398, exp_rate<=3 : 0.612924, loss: 0.166799, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19969 s, avg_samples: 56.0, ips: 280.43984 samples/s, eta: 1 day, 18:44:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:25:05] ppocr INFO: epoch: [5/300], global_step: 11300, lr: 0.000100, edit distance: 0.138891, exp_rate: 0.086005, exp_rate<=1 : 0.263971, exp_rate<=2 : 0.468667, exp_rate<=3 : 0.621358, loss: 0.176698, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19132 s, avg_samples: 56.0, ips: 292.70194 samples/s, eta: 1 day, 18:44:47, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:25:23] ppocr INFO: epoch: [5/300], global_step: 11400, lr: 0.000100, edit distance: 0.139557, exp_rate: 0.085152, exp_rate<=1 : 0.262846, exp_rate<=2 : 0.466894, exp_rate<=3 : 0.620591, loss: 0.173578, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18202 s, avg_samples: 56.0, ips: 307.65707 samples/s, eta: 1 day, 18:43:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:25:42] ppocr INFO: epoch: [5/300], global_step: 11500, lr: 0.000100, edit distance: 0.138328, exp_rate: 0.087490, exp_rate<=1 : 0.268694, exp_rate<=2 : 0.476442, exp_rate<=3 : 0.629002, loss: 0.188714, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18640 s, avg_samples: 56.0, ips: 300.43524 samples/s, eta: 1 day, 18:42:50, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:26:03] ppocr INFO: epoch: [5/300], global_step: 11600, lr: 0.000100, edit distance: 0.137269, exp_rate: 0.085327, exp_rate<=1 : 0.264806, exp_rate<=2 : 0.471228, exp_rate<=3 : 0.622961, loss: 0.191564, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21026 s, avg_samples: 56.0, ips: 266.33547 samples/s, eta: 1 day, 18:44:54, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:26:21] ppocr INFO: epoch: [5/300], global_step: 11700, lr: 0.000100, edit distance: 0.136627, exp_rate: 0.086040, exp_rate<=1 : 0.267357, exp_rate<=2 : 0.473881, exp_rate<=3 : 0.624559, loss: 0.180316, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18224 s, avg_samples: 56.0, ips: 307.28580 samples/s, eta: 1 day, 18:43:42, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:26:39] ppocr INFO: epoch: [5/300], global_step: 11800, lr: 0.000100, edit distance: 0.136495, exp_rate: 0.087145, exp_rate<=1 : 0.270029, exp_rate<=2 : 0.477776, exp_rate<=3 : 0.628316, loss: 0.172970, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17717 s, avg_samples: 56.0, ips: 316.08431 samples/s, eta: 1 day, 18:41:56, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:26:57] ppocr INFO: epoch: [5/300], global_step: 11900, lr: 0.000100, edit distance: 0.136112, exp_rate: 0.088652, exp_rate<=1 : 0.272534, exp_rate<=2 : 0.480602, exp_rate<=3 : 0.630952, loss: 0.193291, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18411 s, avg_samples: 56.0, ips: 304.17317 samples/s, eta: 1 day, 18:40:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:27:15] ppocr INFO: epoch: [5/300], global_step: 12000, lr: 0.000100, edit distance: 0.135817, exp_rate: 0.089260, exp_rate<=1 : 0.274346, exp_rate<=2 : 0.483987, exp_rate<=3 : 0.633729, loss: 0.187962, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17085 s, avg_samples: 56.0, ips: 327.77344 samples/s, eta: 1 day, 18:38:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:27:34] ppocr INFO: epoch: [5/300], global_step: 12100, lr: 0.000100, edit distance: 0.135347, exp_rate: 0.090021, exp_rate<=1 : 0.275782, exp_rate<=2 : 0.485841, exp_rate<=3 : 0.635740, loss: 0.200911, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19270 s, avg_samples: 56.0, ips: 290.61272 samples/s, eta: 1 day, 18:38:35, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:27:53] ppocr INFO: epoch: [5/300], global_step: 12200, lr: 0.000100, edit distance: 0.134932, exp_rate: 0.089795, exp_rate<=1 : 0.275196, exp_rate<=2 : 0.484480, exp_rate<=3 : 0.634139, loss: 0.181511, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19154 s, avg_samples: 56.0, ips: 292.37326 samples/s, eta: 1 day, 18:38:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:28:12] ppocr INFO: epoch: [5/300], global_step: 12300, lr: 0.000100, edit distance: 0.134632, exp_rate: 0.089602, exp_rate<=1 : 0.274929, exp_rate<=2 : 0.484241, exp_rate<=3 : 0.634177, loss: 0.165005, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18636 s, avg_samples: 56.0, ips: 300.49671 samples/s, eta: 1 day, 18:37:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:28:29] ppocr INFO: epoch: [5/300], global_step: 12400, lr: 0.000100, edit distance: 0.134110, exp_rate: 0.090032, exp_rate<=1 : 0.275566, exp_rate<=2 : 0.485109, exp_rate<=3 : 0.635614, loss: 0.147896, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17522 s, avg_samples: 56.0, ips: 319.60210 samples/s, eta: 1 day, 18:35:57, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:28:49] ppocr INFO: epoch: [5/300], global_step: 12500, lr: 0.000100, edit distance: 0.133732, exp_rate: 0.090108, exp_rate<=1 : 0.275105, exp_rate<=2 : 0.484663, exp_rate<=3 : 0.635257, loss: 0.192571, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19510 s, avg_samples: 56.0, ips: 287.03585 samples/s, eta: 1 day, 18:36:15, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:29:08] ppocr INFO: epoch: [5/300], global_step: 12600, lr: 0.000100, edit distance: 0.133378, exp_rate: 0.090622, exp_rate<=1 : 0.276086, exp_rate<=2 : 0.485194, exp_rate<=3 : 0.635292, loss: 0.182591, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18802 s, avg_samples: 56.0, ips: 297.84435 samples/s, eta: 1 day, 18:35:47, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:29:28] ppocr INFO: epoch: [5/300], global_step: 12700, lr: 0.000100, edit distance: 0.133186, exp_rate: 0.090872, exp_rate<=1 : 0.276763, exp_rate<=2 : 0.485788, exp_rate<=3 : 0.635965, loss: 0.172385, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20268 s, avg_samples: 56.0, ips: 276.29583 samples/s, eta: 1 day, 18:36:52, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:29:46] ppocr INFO: epoch: [5/300], global_step: 12800, lr: 0.000100, edit distance: 0.132763, exp_rate: 0.091756, exp_rate<=1 : 0.277940, exp_rate<=2 : 0.487107, exp_rate<=3 : 0.636968, loss: 0.177386, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18018 s, avg_samples: 56.0, ips: 310.80491 samples/s, eta: 1 day, 18:35:34, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:30:05] ppocr INFO: epoch: [5/300], global_step: 12900, lr: 0.000100, edit distance: 0.132284, exp_rate: 0.092223, exp_rate<=1 : 0.278897, exp_rate<=2 : 0.488294, exp_rate<=3 : 0.637935, loss: 0.164454, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19289 s, avg_samples: 56.0, ips: 290.31742 samples/s, eta: 1 day, 18:35:36, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:30:24] ppocr INFO: epoch: [5/300], global_step: 13000, lr: 0.000100, edit distance: 0.132092, exp_rate: 0.093159, exp_rate<=1 : 0.280763, exp_rate<=2 : 0.490158, exp_rate<=3 : 0.639442, loss: 0.160563, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18738 s, avg_samples: 56.0, ips: 298.85023 samples/s, eta: 1 day, 18:35:04, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:30:43] ppocr INFO: epoch: [5/300], global_step: 13100, lr: 0.000100, edit distance: 0.131954, exp_rate: 0.094276, exp_rate<=1 : 0.282950, exp_rate<=2 : 0.492420, exp_rate<=3 : 0.641215, loss: 0.155602, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18598 s, avg_samples: 56.0, ips: 301.10199 samples/s, eta: 1 day, 18:34:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:31:01] ppocr INFO: epoch: [5/300], global_step: 13200, lr: 0.000100, edit distance: 0.131375, exp_rate: 0.094754, exp_rate<=1 : 0.284171, exp_rate<=2 : 0.493945, exp_rate<=3 : 0.642721, loss: 0.181767, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18493 s, avg_samples: 56.0, ips: 302.81834 samples/s, eta: 1 day, 18:33:36, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:31:20] ppocr INFO: epoch: [5/300], global_step: 13300, lr: 0.000100, edit distance: 0.131006, exp_rate: 0.095441, exp_rate<=1 : 0.285240, exp_rate<=2 : 0.495441, exp_rate<=3 : 0.644068, loss: 0.179081, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18848 s, avg_samples: 56.0, ips: 297.11483 samples/s, eta: 1 day, 18:33:12, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:31:40] ppocr INFO: epoch: [5/300], global_step: 13400, lr: 0.000100, edit distance: 0.130545, exp_rate: 0.095501, exp_rate<=1 : 0.285703, exp_rate<=2 : 0.495456, exp_rate<=3 : 0.644293, loss: 0.143707, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20088 s, avg_samples: 56.0, ips: 278.77990 samples/s, eta: 1 day, 18:34:02, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:32:00] ppocr INFO: epoch: [5/300], global_step: 13500, lr: 0.000100, edit distance: 0.130054, exp_rate: 0.095663, exp_rate<=1 : 0.285961, exp_rate<=2 : 0.495359, exp_rate<=3 : 0.644373, loss: 0.154641, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19981 s, avg_samples: 56.0, ips: 280.27110 samples/s, eta: 1 day, 18:34:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:32:19] ppocr INFO: epoch: [5/300], global_step: 13600, lr: 0.000100, edit distance: 0.129665, exp_rate: 0.096319, exp_rate<=1 : 0.286976, exp_rate<=2 : 0.496243, exp_rate<=3 : 0.645268, loss: 0.163147, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18966 s, avg_samples: 56.0, ips: 295.26013 samples/s, eta: 1 day, 18:34:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:32:37] ppocr INFO: epoch: [5/300], global_step: 13700, lr: 0.000100, edit distance: 0.129190, exp_rate: 0.097212, exp_rate<=1 : 0.288415, exp_rate<=2 : 0.498094, exp_rate<=3 : 0.646718, loss: 0.148148, avg_reader_cost: 0.00015 s, avg_batch_cost: 0.17817 s, avg_samples: 56.0, ips: 314.30168 samples/s, eta: 1 day, 18:33:00, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:32:39] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/08 02:32:40] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_5 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:32:58] ppocr INFO: epoch: [6/300], global_step: 13800, lr: 0.000100, edit distance: 0.100210, exp_rate: 0.153848, exp_rate<=1 : 0.366998, exp_rate<=2 : 0.589165, exp_rate<=3 : 0.727484, loss: 0.125918, avg_reader_cost: 0.03208 s, avg_batch_cost: 0.20997 s, avg_samples: 56.0, ips: 266.70012 samples/s, eta: 1 day, 18:34:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:33:16] ppocr INFO: epoch: [6/300], global_step: 13900, lr: 0.000100, edit distance: 0.100817, exp_rate: 0.150468, exp_rate<=1 : 0.363735, exp_rate<=2 : 0.585649, exp_rate<=3 : 0.724909, loss: 0.139491, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17625 s, avg_samples: 56.0, ips: 317.73876 samples/s, eta: 1 day, 18:33:03, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:33:36] ppocr INFO: epoch: [6/300], global_step: 14000, lr: 0.000100, edit distance: 0.102940, exp_rate: 0.149016, exp_rate<=1 : 0.357113, exp_rate<=2 : 0.574922, exp_rate<=3 : 0.712744, loss: 0.143587, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.20287 s, avg_samples: 56.0, ips: 276.04427 samples/s, eta: 1 day, 18:34:01, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:33:55] ppocr INFO: epoch: [6/300], global_step: 14100, lr: 0.000100, edit distance: 0.103019, exp_rate: 0.146952, exp_rate<=1 : 0.355094, exp_rate<=2 : 0.569728, exp_rate<=3 : 0.706905, loss: 0.128321, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18853 s, avg_samples: 56.0, ips: 297.04260 samples/s, eta: 1 day, 18:33:36, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:34:14] ppocr INFO: epoch: [6/300], global_step: 14200, lr: 0.000100, edit distance: 0.102632, exp_rate: 0.143185, exp_rate<=1 : 0.350026, exp_rate<=2 : 0.563671, exp_rate<=3 : 0.703871, loss: 0.143492, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19340 s, avg_samples: 56.0, ips: 289.55953 samples/s, eta: 1 day, 18:33:38, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:34:32] ppocr INFO: epoch: [6/300], global_step: 14300, lr: 0.000100, edit distance: 0.102548, exp_rate: 0.141877, exp_rate<=1 : 0.349299, exp_rate<=2 : 0.563459, exp_rate<=3 : 0.703459, loss: 0.119568, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17838 s, avg_samples: 56.0, ips: 313.94442 samples/s, eta: 1 day, 18:32:15, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:34:50] ppocr INFO: epoch: [6/300], global_step: 14400, lr: 0.000100, edit distance: 0.102750, exp_rate: 0.145019, exp_rate<=1 : 0.354161, exp_rate<=2 : 0.567363, exp_rate<=3 : 0.706789, loss: 0.132288, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18453 s, avg_samples: 56.0, ips: 303.48038 samples/s, eta: 1 day, 18:31:28, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:35:09] ppocr INFO: epoch: [6/300], global_step: 14500, lr: 0.000100, edit distance: 0.103488, exp_rate: 0.143965, exp_rate<=1 : 0.354027, exp_rate<=2 : 0.566777, exp_rate<=3 : 0.705498, loss: 0.142562, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18127 s, avg_samples: 56.0, ips: 308.93245 samples/s, eta: 1 day, 18:30:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:35:29] ppocr INFO: epoch: [6/300], global_step: 14600, lr: 0.000100, edit distance: 0.103432, exp_rate: 0.141466, exp_rate<=1 : 0.349116, exp_rate<=2 : 0.561296, exp_rate<=3 : 0.701179, loss: 0.132807, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.20013 s, avg_samples: 56.0, ips: 279.81193 samples/s, eta: 1 day, 18:31:03, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:35:47] ppocr INFO: epoch: [6/300], global_step: 14700, lr: 0.000100, edit distance: 0.102595, exp_rate: 0.144107, exp_rate<=1 : 0.352810, exp_rate<=2 : 0.565582, exp_rate<=3 : 0.704864, loss: 0.135598, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18246 s, avg_samples: 56.0, ips: 306.91808 samples/s, eta: 1 day, 18:30:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:36:07] ppocr INFO: epoch: [6/300], global_step: 14800, lr: 0.000100, edit distance: 0.102863, exp_rate: 0.140908, exp_rate<=1 : 0.347686, exp_rate<=2 : 0.560172, exp_rate<=3 : 0.700465, loss: 0.124016, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19983 s, avg_samples: 56.0, ips: 280.24383 samples/s, eta: 1 day, 18:30:42, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:36:25] ppocr INFO: epoch: [6/300], global_step: 14900, lr: 0.000100, edit distance: 0.102667, exp_rate: 0.141644, exp_rate<=1 : 0.348068, exp_rate<=2 : 0.561647, exp_rate<=3 : 0.702441, loss: 0.136334, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17800 s, avg_samples: 56.0, ips: 314.60166 samples/s, eta: 1 day, 18:29:20, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:36:44] ppocr INFO: epoch: [6/300], global_step: 15000, lr: 0.000100, edit distance: 0.102456, exp_rate: 0.143453, exp_rate<=1 : 0.350159, exp_rate<=2 : 0.563004, exp_rate<=3 : 0.703113, loss: 0.133139, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19703 s, avg_samples: 56.0, ips: 284.21392 samples/s, eta: 1 day, 18:29:42, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:37:05] ppocr INFO: epoch: [6/300], global_step: 15100, lr: 0.000100, edit distance: 0.102256, exp_rate: 0.142315, exp_rate<=1 : 0.348461, exp_rate<=2 : 0.560523, exp_rate<=3 : 0.700840, loss: 0.136386, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.20276 s, avg_samples: 56.0, ips: 276.18809 samples/s, eta: 1 day, 18:30:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:37:24] ppocr INFO: epoch: [6/300], global_step: 15200, lr: 0.000100, edit distance: 0.102021, exp_rate: 0.142052, exp_rate<=1 : 0.347838, exp_rate<=2 : 0.559706, exp_rate<=3 : 0.699668, loss: 0.123735, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19131 s, avg_samples: 56.0, ips: 292.71499 samples/s, eta: 1 day, 18:30:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:37:42] ppocr INFO: epoch: [6/300], global_step: 15300, lr: 0.000100, edit distance: 0.101885, exp_rate: 0.143721, exp_rate<=1 : 0.350100, exp_rate<=2 : 0.562441, exp_rate<=3 : 0.702086, loss: 0.133089, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18237 s, avg_samples: 56.0, ips: 307.07495 samples/s, eta: 1 day, 18:29:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:38:03] ppocr INFO: epoch: [6/300], global_step: 15400, lr: 0.000100, edit distance: 0.102027, exp_rate: 0.142572, exp_rate<=1 : 0.348209, exp_rate<=2 : 0.560382, exp_rate<=3 : 0.700037, loss: 0.123240, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20752 s, avg_samples: 56.0, ips: 269.85085 samples/s, eta: 1 day, 18:30:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:38:23] ppocr INFO: epoch: [6/300], global_step: 15500, lr: 0.000100, edit distance: 0.101801, exp_rate: 0.142812, exp_rate<=1 : 0.348050, exp_rate<=2 : 0.559987, exp_rate<=3 : 0.700100, loss: 0.126303, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19778 s, avg_samples: 56.0, ips: 283.14332 samples/s, eta: 1 day, 18:31:03, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:38:42] ppocr INFO: epoch: [6/300], global_step: 15600, lr: 0.000100, edit distance: 0.101507, exp_rate: 0.142824, exp_rate<=1 : 0.348678, exp_rate<=2 : 0.560620, exp_rate<=3 : 0.700794, loss: 0.123166, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18974 s, avg_samples: 56.0, ips: 295.13417 samples/s, eta: 1 day, 18:30:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:39:00] ppocr INFO: epoch: [6/300], global_step: 15700, lr: 0.000100, edit distance: 0.101367, exp_rate: 0.142463, exp_rate<=1 : 0.348450, exp_rate<=2 : 0.561207, exp_rate<=3 : 0.701659, loss: 0.118021, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18475 s, avg_samples: 56.0, ips: 303.11153 samples/s, eta: 1 day, 18:29:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:39:19] ppocr INFO: epoch: [6/300], global_step: 15800, lr: 0.000100, edit distance: 0.101204, exp_rate: 0.143373, exp_rate<=1 : 0.349641, exp_rate<=2 : 0.562385, exp_rate<=3 : 0.702842, loss: 0.126802, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18619 s, avg_samples: 56.0, ips: 300.76097 samples/s, eta: 1 day, 18:29:22, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:39:39] ppocr INFO: epoch: [6/300], global_step: 15900, lr: 0.000100, edit distance: 0.100996, exp_rate: 0.143321, exp_rate<=1 : 0.348933, exp_rate<=2 : 0.561474, exp_rate<=3 : 0.702159, loss: 0.118266, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20075 s, avg_samples: 56.0, ips: 278.95585 samples/s, eta: 1 day, 18:29:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:39:58] ppocr INFO: epoch: [6/300], global_step: 16000, lr: 0.000100, edit distance: 0.100875, exp_rate: 0.144518, exp_rate<=1 : 0.350599, exp_rate<=2 : 0.562896, exp_rate<=3 : 0.703377, loss: 0.124146, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18695 s, avg_samples: 56.0, ips: 299.55212 samples/s, eta: 1 day, 18:29:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:40:16] ppocr INFO: epoch: [6/300], global_step: 16100, lr: 0.000100, edit distance: 0.100632, exp_rate: 0.144645, exp_rate<=1 : 0.350801, exp_rate<=2 : 0.563553, exp_rate<=3 : 0.704253, loss: 0.117458, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18537 s, avg_samples: 56.0, ips: 302.09766 samples/s, eta: 1 day, 18:28:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:40:34] ppocr INFO: epoch: [6/300], global_step: 16200, lr: 0.000100, edit distance: 0.100402, exp_rate: 0.145138, exp_rate<=1 : 0.351461, exp_rate<=2 : 0.564672, exp_rate<=3 : 0.705452, loss: 0.128275, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17893 s, avg_samples: 56.0, ips: 312.96387 samples/s, eta: 1 day, 18:27:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:40:51] ppocr INFO: epoch: [6/300], global_step: 16300, lr: 0.000100, edit distance: 0.100223, exp_rate: 0.146201, exp_rate<=1 : 0.353284, exp_rate<=2 : 0.566868, exp_rate<=3 : 0.707385, loss: 0.123448, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17159 s, avg_samples: 56.0, ips: 326.36787 samples/s, eta: 1 day, 18:25:43, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:41:10] ppocr INFO: epoch: [6/300], global_step: 16400, lr: 0.000100, edit distance: 0.100071, exp_rate: 0.145894, exp_rate<=1 : 0.353327, exp_rate<=2 : 0.567042, exp_rate<=3 : 0.707670, loss: 0.127149, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18558 s, avg_samples: 56.0, ips: 301.74890 samples/s, eta: 1 day, 18:25:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:41:17] ppocr INFO: epoch: [6/300], global_step: 16440, lr: 0.000100, edit distance: 0.099965, exp_rate: 0.146233, exp_rate<=1 : 0.353914, exp_rate<=2 : 0.567572, exp_rate<=3 : 0.708155, loss: 0.121662, avg_reader_cost: 0.00006 s, avg_batch_cost: 0.07498 s, avg_samples: 22.4, ips: 298.75774 samples/s, eta: 1 day, 18:24:53, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:41:20] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:41:33] ppocr INFO: epoch: [7/300], global_step: 16500, lr: 0.000100, edit distance: 0.085787, exp_rate: 0.176224, exp_rate<=1 : 0.383329, exp_rate<=2 : 0.588158, exp_rate<=3 : 0.716549, loss: 0.109762, avg_reader_cost: 0.03081 s, avg_batch_cost: 0.15490 s, avg_samples: 33.6, ips: 216.91954 samples/s, eta: 1 day, 18:28:03, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:41:51] ppocr INFO: epoch: [7/300], global_step: 16600, lr: 0.000100, edit distance: 0.082347, exp_rate: 0.193493, exp_rate<=1 : 0.402467, exp_rate<=2 : 0.614214, exp_rate<=3 : 0.743331, loss: 0.099891, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17914 s, avg_samples: 56.0, ips: 312.60972 samples/s, eta: 1 day, 18:26:52, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:42:09] ppocr INFO: epoch: [7/300], global_step: 16700, lr: 0.000100, edit distance: 0.081662, exp_rate: 0.189159, exp_rate<=1 : 0.400615, exp_rate<=2 : 0.615312, exp_rate<=3 : 0.746885, loss: 0.103172, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18591 s, avg_samples: 56.0, ips: 301.22404 samples/s, eta: 1 day, 18:26:14, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:42:28] ppocr INFO: epoch: [7/300], global_step: 16800, lr: 0.000100, edit distance: 0.081375, exp_rate: 0.187247, exp_rate<=1 : 0.401237, exp_rate<=2 : 0.615139, exp_rate<=3 : 0.747464, loss: 0.102611, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18882 s, avg_samples: 56.0, ips: 296.58540 samples/s, eta: 1 day, 18:25:51, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:42:48] ppocr INFO: epoch: [7/300], global_step: 16900, lr: 0.000100, edit distance: 0.082654, exp_rate: 0.187372, exp_rate<=1 : 0.401500, exp_rate<=2 : 0.613466, exp_rate<=3 : 0.745600, loss: 0.122380, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19910 s, avg_samples: 56.0, ips: 281.26545 samples/s, eta: 1 day, 18:26:17, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:43:08] ppocr INFO: epoch: [7/300], global_step: 17000, lr: 0.000100, edit distance: 0.083067, exp_rate: 0.185990, exp_rate<=1 : 0.398652, exp_rate<=2 : 0.608632, exp_rate<=3 : 0.740967, loss: 0.094393, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19407 s, avg_samples: 56.0, ips: 288.56111 samples/s, eta: 1 day, 18:26:18, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:43:26] ppocr INFO: epoch: [7/300], global_step: 17100, lr: 0.000100, edit distance: 0.082747, exp_rate: 0.185530, exp_rate<=1 : 0.399264, exp_rate<=2 : 0.612641, exp_rate<=3 : 0.745180, loss: 0.106796, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18042 s, avg_samples: 56.0, ips: 310.39512 samples/s, eta: 1 day, 18:25:15, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:43:44] ppocr INFO: epoch: [7/300], global_step: 17200, lr: 0.000100, edit distance: 0.082852, exp_rate: 0.183935, exp_rate<=1 : 0.396879, exp_rate<=2 : 0.609950, exp_rate<=3 : 0.743019, loss: 0.107001, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18624 s, avg_samples: 56.0, ips: 300.69456 samples/s, eta: 1 day, 18:24:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:44:03] ppocr INFO: epoch: [7/300], global_step: 17300, lr: 0.000100, edit distance: 0.082888, exp_rate: 0.186005, exp_rate<=1 : 0.399568, exp_rate<=2 : 0.612140, exp_rate<=3 : 0.745278, loss: 0.108214, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18495 s, avg_samples: 56.0, ips: 302.79097 samples/s, eta: 1 day, 18:23:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:44:21] ppocr INFO: epoch: [7/300], global_step: 17400, lr: 0.000100, edit distance: 0.082828, exp_rate: 0.186763, exp_rate<=1 : 0.399966, exp_rate<=2 : 0.613618, exp_rate<=3 : 0.746810, loss: 0.113845, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18469 s, avg_samples: 56.0, ips: 303.21305 samples/s, eta: 1 day, 18:23:17, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:44:39] ppocr INFO: epoch: [7/300], global_step: 17500, lr: 0.000100, edit distance: 0.082917, exp_rate: 0.187521, exp_rate<=1 : 0.401443, exp_rate<=2 : 0.615803, exp_rate<=3 : 0.748353, loss: 0.100183, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17468 s, avg_samples: 56.0, ips: 320.59099 samples/s, eta: 1 day, 18:21:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:44:58] ppocr INFO: epoch: [7/300], global_step: 17600, lr: 0.000100, edit distance: 0.083058, exp_rate: 0.186526, exp_rate<=1 : 0.400916, exp_rate<=2 : 0.615156, exp_rate<=3 : 0.747929, loss: 0.112929, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19233 s, avg_samples: 56.0, ips: 291.16681 samples/s, eta: 1 day, 18:21:43, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:45:16] ppocr INFO: epoch: [7/300], global_step: 17700, lr: 0.000100, edit distance: 0.082953, exp_rate: 0.187440, exp_rate<=1 : 0.401619, exp_rate<=2 : 0.616484, exp_rate<=3 : 0.748875, loss: 0.111430, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17838 s, avg_samples: 56.0, ips: 313.92801 samples/s, eta: 1 day, 18:20:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:45:36] ppocr INFO: epoch: [7/300], global_step: 17800, lr: 0.000100, edit distance: 0.083123, exp_rate: 0.185689, exp_rate<=1 : 0.398768, exp_rate<=2 : 0.613618, exp_rate<=3 : 0.745911, loss: 0.107201, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20336 s, avg_samples: 56.0, ips: 275.37529 samples/s, eta: 1 day, 18:21:16, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:45:56] ppocr INFO: epoch: [7/300], global_step: 17900, lr: 0.000100, edit distance: 0.082970, exp_rate: 0.184112, exp_rate<=1 : 0.396170, exp_rate<=2 : 0.611271, exp_rate<=3 : 0.744382, loss: 0.107847, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19671 s, avg_samples: 56.0, ips: 284.68029 samples/s, eta: 1 day, 18:21:29, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:46:16] ppocr INFO: epoch: [7/300], global_step: 18000, lr: 0.000100, edit distance: 0.082888, exp_rate: 0.183263, exp_rate<=1 : 0.395025, exp_rate<=2 : 0.609430, exp_rate<=3 : 0.742558, loss: 0.102960, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20201 s, avg_samples: 56.0, ips: 277.20749 samples/s, eta: 1 day, 18:22:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:46:36] ppocr INFO: epoch: [7/300], global_step: 18100, lr: 0.000100, edit distance: 0.082852, exp_rate: 0.184012, exp_rate<=1 : 0.396028, exp_rate<=2 : 0.610628, exp_rate<=3 : 0.743475, loss: 0.111190, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19681 s, avg_samples: 56.0, ips: 284.54314 samples/s, eta: 1 day, 18:22:18, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:46:55] ppocr INFO: epoch: [7/300], global_step: 18200, lr: 0.000100, edit distance: 0.082931, exp_rate: 0.183997, exp_rate<=1 : 0.396077, exp_rate<=2 : 0.610274, exp_rate<=3 : 0.743815, loss: 0.110247, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18628 s, avg_samples: 56.0, ips: 300.62189 samples/s, eta: 1 day, 18:21:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:47:14] ppocr INFO: epoch: [7/300], global_step: 18300, lr: 0.000100, edit distance: 0.082780, exp_rate: 0.184040, exp_rate<=1 : 0.396296, exp_rate<=2 : 0.610474, exp_rate<=3 : 0.744223, loss: 0.105305, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19222 s, avg_samples: 56.0, ips: 291.32538 samples/s, eta: 1 day, 18:21:36, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:47:34] ppocr INFO: epoch: [7/300], global_step: 18400, lr: 0.000100, edit distance: 0.082750, exp_rate: 0.183440, exp_rate<=1 : 0.395054, exp_rate<=2 : 0.609514, exp_rate<=3 : 0.743242, loss: 0.100600, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.20163 s, avg_samples: 56.0, ips: 277.73699 samples/s, eta: 1 day, 18:22:09, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:47:53] ppocr INFO: epoch: [7/300], global_step: 18500, lr: 0.000100, edit distance: 0.082791, exp_rate: 0.183190, exp_rate<=1 : 0.395159, exp_rate<=2 : 0.609673, exp_rate<=3 : 0.743366, loss: 0.109128, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18722 s, avg_samples: 56.0, ips: 299.11143 samples/s, eta: 1 day, 18:21:38, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:48:11] ppocr INFO: epoch: [7/300], global_step: 18600, lr: 0.000100, edit distance: 0.082920, exp_rate: 0.183220, exp_rate<=1 : 0.395413, exp_rate<=2 : 0.610119, exp_rate<=3 : 0.743371, loss: 0.125066, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18618 s, avg_samples: 56.0, ips: 300.78587 samples/s, eta: 1 day, 18:21:04, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:48:29] ppocr INFO: epoch: [7/300], global_step: 18700, lr: 0.000100, edit distance: 0.082888, exp_rate: 0.184587, exp_rate<=1 : 0.397546, exp_rate<=2 : 0.611963, exp_rate<=3 : 0.744895, loss: 0.098394, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17523 s, avg_samples: 56.0, ips: 319.57350 samples/s, eta: 1 day, 18:19:42, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:48:47] ppocr INFO: epoch: [7/300], global_step: 18800, lr: 0.000100, edit distance: 0.082616, exp_rate: 0.185921, exp_rate<=1 : 0.398960, exp_rate<=2 : 0.613463, exp_rate<=3 : 0.746312, loss: 0.104623, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.18198 s, avg_samples: 56.0, ips: 307.71902 samples/s, eta: 1 day, 18:18:51, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:49:05] ppocr INFO: epoch: [7/300], global_step: 18900, lr: 0.000100, edit distance: 0.082466, exp_rate: 0.187256, exp_rate<=1 : 0.400481, exp_rate<=2 : 0.615386, exp_rate<=3 : 0.748064, loss: 0.106183, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17382 s, avg_samples: 56.0, ips: 322.17619 samples/s, eta: 1 day, 18:17:24, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:49:23] ppocr INFO: epoch: [7/300], global_step: 19000, lr: 0.000100, edit distance: 0.082274, exp_rate: 0.187970, exp_rate<=1 : 0.401220, exp_rate<=2 : 0.616353, exp_rate<=3 : 0.748753, loss: 0.099529, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18119 s, avg_samples: 56.0, ips: 309.06664 samples/s, eta: 1 day, 18:16:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:49:41] ppocr INFO: epoch: [7/300], global_step: 19100, lr: 0.000100, edit distance: 0.082073, exp_rate: 0.188592, exp_rate<=1 : 0.401989, exp_rate<=2 : 0.617350, exp_rate<=3 : 0.749936, loss: 0.082872, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18087 s, avg_samples: 56.0, ips: 309.61496 samples/s, eta: 1 day, 18:15:35, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:49:56] ppocr INFO: epoch: [7/300], global_step: 19180, lr: 0.000100, edit distance: 0.081956, exp_rate: 0.188991, exp_rate<=1 : 0.402458, exp_rate<=2 : 0.617483, exp_rate<=3 : 0.750160, loss: 0.104262, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.15395 s, avg_samples: 44.8, ips: 291.00424 samples/s, eta: 1 day, 18:15:29, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:49:58] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:50:03] ppocr INFO: epoch: [8/300], global_step: 19200, lr: 0.000100, edit distance: 0.073517, exp_rate: 0.207540, exp_rate<=1 : 0.417634, exp_rate<=2 : 0.642894, exp_rate<=3 : 0.759425, loss: 0.098842, avg_reader_cost: 0.02526 s, avg_batch_cost: 0.06805 s, avg_samples: 11.2, ips: 164.58336 samples/s, eta: 1 day, 18:17:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:50:21] ppocr INFO: epoch: [8/300], global_step: 19300, lr: 0.000100, edit distance: 0.070007, exp_rate: 0.230612, exp_rate<=1 : 0.449218, exp_rate<=2 : 0.655908, exp_rate<=3 : 0.777433, loss: 0.079190, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18020 s, avg_samples: 56.0, ips: 310.77214 samples/s, eta: 1 day, 18:16:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:50:40] ppocr INFO: epoch: [8/300], global_step: 19400, lr: 0.000100, edit distance: 0.069393, exp_rate: 0.232690, exp_rate<=1 : 0.451238, exp_rate<=2 : 0.658430, exp_rate<=3 : 0.786830, loss: 0.095506, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19068 s, avg_samples: 56.0, ips: 293.68847 samples/s, eta: 1 day, 18:16:19, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:50:58] ppocr INFO: epoch: [8/300], global_step: 19500, lr: 0.000100, edit distance: 0.067707, exp_rate: 0.234873, exp_rate<=1 : 0.454242, exp_rate<=2 : 0.663103, exp_rate<=3 : 0.788408, loss: 0.074719, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17662 s, avg_samples: 56.0, ips: 317.06655 samples/s, eta: 1 day, 18:15:06, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:51:17] ppocr INFO: epoch: [8/300], global_step: 19600, lr: 0.000100, edit distance: 0.067863, exp_rate: 0.238046, exp_rate<=1 : 0.456808, exp_rate<=2 : 0.664692, exp_rate<=3 : 0.789405, loss: 0.091809, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18888 s, avg_samples: 56.0, ips: 296.47682 samples/s, eta: 1 day, 18:14:45, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:51:36] ppocr INFO: epoch: [8/300], global_step: 19700, lr: 0.000100, edit distance: 0.068395, exp_rate: 0.237668, exp_rate<=1 : 0.456692, exp_rate<=2 : 0.665574, exp_rate<=3 : 0.787893, loss: 0.081053, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19202 s, avg_samples: 56.0, ips: 291.63616 samples/s, eta: 1 day, 18:14:36, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:51:55] ppocr INFO: epoch: [8/300], global_step: 19800, lr: 0.000100, edit distance: 0.068360, exp_rate: 0.233898, exp_rate<=1 : 0.450111, exp_rate<=2 : 0.660480, exp_rate<=3 : 0.784690, loss: 0.089247, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19406 s, avg_samples: 56.0, ips: 288.56634 samples/s, eta: 1 day, 18:14:35, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:52:15] ppocr INFO: epoch: [8/300], global_step: 19900, lr: 0.000100, edit distance: 0.068579, exp_rate: 0.230472, exp_rate<=1 : 0.446112, exp_rate<=2 : 0.656321, exp_rate<=3 : 0.781153, loss: 0.079826, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19635 s, avg_samples: 56.0, ips: 285.21173 samples/s, eta: 1 day, 18:14:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:52:35] ppocr INFO: epoch: [8/300], global_step: 20000, lr: 0.000100, edit distance: 0.068700, exp_rate: 0.230524, exp_rate<=1 : 0.446905, exp_rate<=2 : 0.658555, exp_rate<=3 : 0.782792, loss: 0.094256, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19481 s, avg_samples: 56.0, ips: 287.45948 samples/s, eta: 1 day, 18:14:45, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:52:53] ppocr INFO: epoch: [8/300], global_step: 20100, lr: 0.000100, edit distance: 0.068755, exp_rate: 0.230265, exp_rate<=1 : 0.445789, exp_rate<=2 : 0.656550, exp_rate<=3 : 0.781131, loss: 0.070172, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18189 s, avg_samples: 56.0, ips: 307.87088 samples/s, eta: 1 day, 18:13:56, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:53:12] ppocr INFO: epoch: [8/300], global_step: 20200, lr: 0.000100, edit distance: 0.068779, exp_rate: 0.230588, exp_rate<=1 : 0.447926, exp_rate<=2 : 0.659367, exp_rate<=3 : 0.782740, loss: 0.082770, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18989 s, avg_samples: 56.0, ips: 294.91008 samples/s, eta: 1 day, 18:13:38, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:53:31] ppocr INFO: epoch: [8/300], global_step: 20300, lr: 0.000100, edit distance: 0.069093, exp_rate: 0.229608, exp_rate<=1 : 0.446942, exp_rate<=2 : 0.658204, exp_rate<=3 : 0.781999, loss: 0.089337, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18804 s, avg_samples: 56.0, ips: 297.80694 samples/s, eta: 1 day, 18:13:13, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:53:49] ppocr INFO: epoch: [8/300], global_step: 20400, lr: 0.000100, edit distance: 0.069165, exp_rate: 0.229204, exp_rate<=1 : 0.447697, exp_rate<=2 : 0.659004, exp_rate<=3 : 0.782706, loss: 0.090350, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18261 s, avg_samples: 56.0, ips: 306.66505 samples/s, eta: 1 day, 18:12:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:54:08] ppocr INFO: epoch: [8/300], global_step: 20500, lr: 0.000100, edit distance: 0.069198, exp_rate: 0.227131, exp_rate<=1 : 0.445295, exp_rate<=2 : 0.656765, exp_rate<=3 : 0.781122, loss: 0.090924, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19468 s, avg_samples: 56.0, ips: 287.64834 samples/s, eta: 1 day, 18:12:27, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:54:28] ppocr INFO: epoch: [8/300], global_step: 20600, lr: 0.000100, edit distance: 0.069458, exp_rate: 0.225989, exp_rate<=1 : 0.443655, exp_rate<=2 : 0.655090, exp_rate<=3 : 0.779680, loss: 0.093127, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19873 s, avg_samples: 56.0, ips: 281.78905 samples/s, eta: 1 day, 18:12:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:54:48] ppocr INFO: epoch: [8/300], global_step: 20700, lr: 0.000100, edit distance: 0.069802, exp_rate: 0.224300, exp_rate<=1 : 0.442633, exp_rate<=2 : 0.654374, exp_rate<=3 : 0.778854, loss: 0.095620, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19802 s, avg_samples: 56.0, ips: 282.79881 samples/s, eta: 1 day, 18:12:58, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:55:06] ppocr INFO: epoch: [8/300], global_step: 20800, lr: 0.000100, edit distance: 0.070066, exp_rate: 0.222091, exp_rate<=1 : 0.440657, exp_rate<=2 : 0.652910, exp_rate<=3 : 0.777922, loss: 0.088080, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18343 s, avg_samples: 56.0, ips: 305.29257 samples/s, eta: 1 day, 18:12:15, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:55:25] ppocr INFO: epoch: [8/300], global_step: 20900, lr: 0.000100, edit distance: 0.069993, exp_rate: 0.224084, exp_rate<=1 : 0.442628, exp_rate<=2 : 0.655175, exp_rate<=3 : 0.779571, loss: 0.102963, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18196 s, avg_samples: 56.0, ips: 307.76453 samples/s, eta: 1 day, 18:11:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:55:45] ppocr INFO: epoch: [8/300], global_step: 21000, lr: 0.000100, edit distance: 0.069902, exp_rate: 0.223246, exp_rate<=1 : 0.441114, exp_rate<=2 : 0.653426, exp_rate<=3 : 0.778056, loss: 0.087366, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19910 s, avg_samples: 56.0, ips: 281.26525 samples/s, eta: 1 day, 18:11:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:56:03] ppocr INFO: epoch: [8/300], global_step: 21100, lr: 0.000100, edit distance: 0.069804, exp_rate: 0.224331, exp_rate<=1 : 0.442284, exp_rate<=2 : 0.654617, exp_rate<=3 : 0.779041, loss: 0.098291, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18691 s, avg_samples: 56.0, ips: 299.61662 samples/s, eta: 1 day, 18:11:14, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:56:25] ppocr INFO: epoch: [8/300], global_step: 21200, lr: 0.000100, edit distance: 0.069776, exp_rate: 0.222408, exp_rate<=1 : 0.439450, exp_rate<=2 : 0.651755, exp_rate<=3 : 0.777070, loss: 0.098949, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.21302 s, avg_samples: 56.0, ips: 262.88756 samples/s, eta: 1 day, 18:12:24, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:56:43] ppocr INFO: epoch: [8/300], global_step: 21300, lr: 0.000100, edit distance: 0.069792, exp_rate: 0.223290, exp_rate<=1 : 0.440075, exp_rate<=2 : 0.652220, exp_rate<=3 : 0.777191, loss: 0.088416, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18583 s, avg_samples: 56.0, ips: 301.34676 samples/s, eta: 1 day, 18:11:50, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:57:01] ppocr INFO: epoch: [8/300], global_step: 21400, lr: 0.000100, edit distance: 0.069730, exp_rate: 0.223093, exp_rate<=1 : 0.440647, exp_rate<=2 : 0.653462, exp_rate<=3 : 0.778378, loss: 0.081450, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17882 s, avg_samples: 56.0, ips: 313.16312 samples/s, eta: 1 day, 18:10:50, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:57:21] ppocr INFO: epoch: [8/300], global_step: 21500, lr: 0.000100, edit distance: 0.069827, exp_rate: 0.223178, exp_rate<=1 : 0.440773, exp_rate<=2 : 0.653206, exp_rate<=3 : 0.778004, loss: 0.089776, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19490 s, avg_samples: 56.0, ips: 287.32742 samples/s, eta: 1 day, 18:10:51, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:57:37] ppocr INFO: epoch: [8/300], global_step: 21600, lr: 0.000100, edit distance: 0.069777, exp_rate: 0.224660, exp_rate<=1 : 0.442936, exp_rate<=2 : 0.655247, exp_rate<=3 : 0.779814, loss: 0.083431, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16534 s, avg_samples: 56.0, ips: 338.69659 samples/s, eta: 1 day, 18:09:01, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:57:54] ppocr INFO: epoch: [8/300], global_step: 21700, lr: 0.000100, edit distance: 0.069729, exp_rate: 0.225509, exp_rate<=1 : 0.443729, exp_rate<=2 : 0.656143, exp_rate<=3 : 0.780656, loss: 0.087226, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17214 s, avg_samples: 56.0, ips: 325.31644 samples/s, eta: 1 day, 18:07:38, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:58:14] ppocr INFO: epoch: [8/300], global_step: 21800, lr: 0.000100, edit distance: 0.069724, exp_rate: 0.225623, exp_rate<=1 : 0.443697, exp_rate<=2 : 0.655919, exp_rate<=3 : 0.780567, loss: 0.086465, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19116 s, avg_samples: 56.0, ips: 292.94459 samples/s, eta: 1 day, 18:07:25, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:58:31] ppocr INFO: epoch: [8/300], global_step: 21900, lr: 0.000100, edit distance: 0.069669, exp_rate: 0.226237, exp_rate<=1 : 0.444469, exp_rate<=2 : 0.656962, exp_rate<=3 : 0.781389, loss: 0.075628, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17918 s, avg_samples: 56.0, ips: 312.52829 samples/s, eta: 1 day, 18:06:29, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:58:36] ppocr INFO: epoch: [8/300], global_step: 21920, lr: 0.000100, edit distance: 0.069647, exp_rate: 0.226676, exp_rate<=1 : 0.444866, exp_rate<=2 : 0.657478, exp_rate<=3 : 0.781762, loss: 0.089446, avg_reader_cost: 0.00003 s, avg_batch_cost: 0.04374 s, avg_samples: 11.2, ips: 256.07391 samples/s, eta: 1 day, 18:06:46, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:58:38] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 02:58:54] ppocr INFO: epoch: [9/300], global_step: 22000, lr: 0.000100, edit distance: 0.058446, exp_rate: 0.243707, exp_rate<=1 : 0.454819, exp_rate<=2 : 0.669574, exp_rate<=3 : 0.801328, loss: 0.068862, avg_reader_cost: 0.02577 s, avg_batch_cost: 0.18240 s, avg_samples: 44.8, ips: 245.62003 samples/s, eta: 1 day, 18:08:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:59:14] ppocr INFO: epoch: [9/300], global_step: 22100, lr: 0.000100, edit distance: 0.056825, exp_rate: 0.256793, exp_rate<=1 : 0.465761, exp_rate<=2 : 0.677664, exp_rate<=3 : 0.805215, loss: 0.072673, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19430 s, avg_samples: 56.0, ips: 288.22111 samples/s, eta: 1 day, 18:08:21, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:59:33] ppocr INFO: epoch: [9/300], global_step: 22200, lr: 0.000100, edit distance: 0.057554, exp_rate: 0.258830, exp_rate<=1 : 0.470206, exp_rate<=2 : 0.681168, exp_rate<=3 : 0.805300, loss: 0.069074, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18933 s, avg_samples: 56.0, ips: 295.78458 samples/s, eta: 1 day, 18:08:01, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 02:59:52] ppocr INFO: epoch: [9/300], global_step: 22300, lr: 0.000100, edit distance: 0.057973, exp_rate: 0.259185, exp_rate<=1 : 0.471403, exp_rate<=2 : 0.683079, exp_rate<=3 : 0.806833, loss: 0.082535, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19014 s, avg_samples: 56.0, ips: 294.52432 samples/s, eta: 1 day, 18:07:44, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:00:09] ppocr INFO: epoch: [9/300], global_step: 22400, lr: 0.000100, edit distance: 0.058645, exp_rate: 0.264730, exp_rate<=1 : 0.480269, exp_rate<=2 : 0.691983, exp_rate<=3 : 0.812709, loss: 0.080071, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.16959 s, avg_samples: 56.0, ips: 330.20982 samples/s, eta: 1 day, 18:06:13, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:00:27] ppocr INFO: epoch: [9/300], global_step: 22500, lr: 0.000100, edit distance: 0.058747, exp_rate: 0.267176, exp_rate<=1 : 0.484123, exp_rate<=2 : 0.695268, exp_rate<=3 : 0.814545, loss: 0.077411, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17956 s, avg_samples: 56.0, ips: 311.86974 samples/s, eta: 1 day, 18:05:18, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:00:46] ppocr INFO: epoch: [9/300], global_step: 22600, lr: 0.000100, edit distance: 0.059211, exp_rate: 0.266697, exp_rate<=1 : 0.484292, exp_rate<=2 : 0.694797, exp_rate<=3 : 0.813083, loss: 0.084239, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19077 s, avg_samples: 56.0, ips: 293.54456 samples/s, eta: 1 day, 18:05:04, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:01:04] ppocr INFO: epoch: [9/300], global_step: 22700, lr: 0.000100, edit distance: 0.059214, exp_rate: 0.264852, exp_rate<=1 : 0.483323, exp_rate<=2 : 0.695020, exp_rate<=3 : 0.814363, loss: 0.072776, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17883 s, avg_samples: 56.0, ips: 313.15163 samples/s, eta: 1 day, 18:04:07, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:01:23] ppocr INFO: epoch: [9/300], global_step: 22800, lr: 0.000100, edit distance: 0.059669, exp_rate: 0.261769, exp_rate<=1 : 0.480301, exp_rate<=2 : 0.690411, exp_rate<=3 : 0.810006, loss: 0.072725, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19409 s, avg_samples: 56.0, ips: 288.53125 samples/s, eta: 1 day, 18:04:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:01:42] ppocr INFO: epoch: [9/300], global_step: 22900, lr: 0.000100, edit distance: 0.059765, exp_rate: 0.262015, exp_rate<=1 : 0.478903, exp_rate<=2 : 0.689676, exp_rate<=3 : 0.808926, loss: 0.070088, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18671 s, avg_samples: 56.0, ips: 299.93628 samples/s, eta: 1 day, 18:03:36, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:02:01] ppocr INFO: epoch: [9/300], global_step: 23000, lr: 0.000100, edit distance: 0.059801, exp_rate: 0.259961, exp_rate<=1 : 0.476962, exp_rate<=2 : 0.688887, exp_rate<=3 : 0.808534, loss: 0.071733, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19200 s, avg_samples: 56.0, ips: 291.66476 samples/s, eta: 1 day, 18:03:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:02:19] ppocr INFO: epoch: [9/300], global_step: 23100, lr: 0.000100, edit distance: 0.059863, exp_rate: 0.260828, exp_rate<=1 : 0.477608, exp_rate<=2 : 0.689809, exp_rate<=3 : 0.809067, loss: 0.069704, avg_reader_cost: 0.00009 s, avg_batch_cost: 0.18337 s, avg_samples: 56.0, ips: 305.39561 samples/s, eta: 1 day, 18:02:46, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:02:38] ppocr INFO: epoch: [9/300], global_step: 23200, lr: 0.000100, edit distance: 0.060009, exp_rate: 0.259835, exp_rate<=1 : 0.476534, exp_rate<=2 : 0.688122, exp_rate<=3 : 0.807167, loss: 0.076912, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19073 s, avg_samples: 56.0, ips: 293.60650 samples/s, eta: 1 day, 18:02:31, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:02:57] ppocr INFO: epoch: [9/300], global_step: 23300, lr: 0.000100, edit distance: 0.059994, exp_rate: 0.259617, exp_rate<=1 : 0.476468, exp_rate<=2 : 0.688847, exp_rate<=3 : 0.807389, loss: 0.078508, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18959 s, avg_samples: 56.0, ips: 295.36899 samples/s, eta: 1 day, 18:02:12, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:03:15] ppocr INFO: epoch: [9/300], global_step: 23400, lr: 0.000100, edit distance: 0.059978, exp_rate: 0.260584, exp_rate<=1 : 0.477355, exp_rate<=2 : 0.690030, exp_rate<=3 : 0.808566, loss: 0.080604, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17859 s, avg_samples: 56.0, ips: 313.56404 samples/s, eta: 1 day, 18:01:16, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:03:33] ppocr INFO: epoch: [9/300], global_step: 23500, lr: 0.000100, edit distance: 0.059928, exp_rate: 0.263021, exp_rate<=1 : 0.479592, exp_rate<=2 : 0.692055, exp_rate<=3 : 0.810007, loss: 0.071059, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18011 s, avg_samples: 56.0, ips: 310.92436 samples/s, eta: 1 day, 18:00:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:03:52] ppocr INFO: epoch: [9/300], global_step: 23600, lr: 0.000100, edit distance: 0.059975, exp_rate: 0.262871, exp_rate<=1 : 0.479421, exp_rate<=2 : 0.691598, exp_rate<=3 : 0.809123, loss: 0.082692, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19078 s, avg_samples: 56.0, ips: 293.53258 samples/s, eta: 1 day, 18:00:12, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:04:13] ppocr INFO: epoch: [9/300], global_step: 23700, lr: 0.000100, edit distance: 0.060181, exp_rate: 0.260742, exp_rate<=1 : 0.476913, exp_rate<=2 : 0.688698, exp_rate<=3 : 0.806896, loss: 0.082737, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.21012 s, avg_samples: 56.0, ips: 266.50824 samples/s, eta: 1 day, 18:01:02, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:04:32] ppocr INFO: epoch: [9/300], global_step: 23800, lr: 0.000100, edit distance: 0.060255, exp_rate: 0.259512, exp_rate<=1 : 0.476297, exp_rate<=2 : 0.687703, exp_rate<=3 : 0.806289, loss: 0.085318, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18983 s, avg_samples: 56.0, ips: 294.99364 samples/s, eta: 1 day, 18:00:45, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:04:52] ppocr INFO: epoch: [9/300], global_step: 23900, lr: 0.000100, edit distance: 0.060297, exp_rate: 0.258517, exp_rate<=1 : 0.475256, exp_rate<=2 : 0.686726, exp_rate<=3 : 0.805565, loss: 0.077942, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19418 s, avg_samples: 56.0, ips: 288.38894 samples/s, eta: 1 day, 18:00:41, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:05:11] ppocr INFO: epoch: [9/300], global_step: 24000, lr: 0.000100, edit distance: 0.060459, exp_rate: 0.258896, exp_rate<=1 : 0.475420, exp_rate<=2 : 0.687084, exp_rate<=3 : 0.805836, loss: 0.080880, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19170 s, avg_samples: 56.0, ips: 292.11945 samples/s, eta: 1 day, 18:00:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:05:31] ppocr INFO: epoch: [9/300], global_step: 24100, lr: 0.000100, edit distance: 0.060533, exp_rate: 0.258492, exp_rate<=1 : 0.474855, exp_rate<=2 : 0.686002, exp_rate<=3 : 0.804912, loss: 0.076224, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.19832 s, avg_samples: 56.0, ips: 282.37625 samples/s, eta: 1 day, 18:00:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:05:50] ppocr INFO: epoch: [9/300], global_step: 24200, lr: 0.000100, edit distance: 0.060625, exp_rate: 0.258279, exp_rate<=1 : 0.474457, exp_rate<=2 : 0.686006, exp_rate<=3 : 0.805010, loss: 0.073391, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19308 s, avg_samples: 56.0, ips: 290.03996 samples/s, eta: 1 day, 18:00:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:06:08] ppocr INFO: epoch: [9/300], global_step: 24300, lr: 0.000100, edit distance: 0.060731, exp_rate: 0.258304, exp_rate<=1 : 0.475042, exp_rate<=2 : 0.686278, exp_rate<=3 : 0.805126, loss: 0.084180, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17882 s, avg_samples: 56.0, ips: 313.15590 samples/s, eta: 1 day, 17:59:38, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:06:27] ppocr INFO: epoch: [9/300], global_step: 24400, lr: 0.000100, edit distance: 0.060881, exp_rate: 0.258320, exp_rate<=1 : 0.475305, exp_rate<=2 : 0.686634, exp_rate<=3 : 0.805144, loss: 0.077939, avg_reader_cost: 0.00014 s, avg_batch_cost: 0.19013 s, avg_samples: 56.0, ips: 294.54003 samples/s, eta: 1 day, 17:59:21, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:06:47] ppocr INFO: epoch: [9/300], global_step: 24500, lr: 0.000100, edit distance: 0.060856, exp_rate: 0.257476, exp_rate<=1 : 0.474549, exp_rate<=2 : 0.685902, exp_rate<=3 : 0.804411, loss: 0.078483, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19590 s, avg_samples: 56.0, ips: 285.85608 samples/s, eta: 1 day, 17:59:23, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:07:05] ppocr INFO: epoch: [9/300], global_step: 24600, lr: 0.000100, edit distance: 0.060777, exp_rate: 0.257929, exp_rate<=1 : 0.475140, exp_rate<=2 : 0.686515, exp_rate<=3 : 0.804998, loss: 0.068390, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18429 s, avg_samples: 56.0, ips: 303.87193 samples/s, eta: 1 day, 17:58:47, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:07:16] ppocr INFO: epoch: [9/300], global_step: 24660, lr: 0.000100, edit distance: 0.060805, exp_rate: 0.257949, exp_rate<=1 : 0.475250, exp_rate<=2 : 0.686439, exp_rate<=3 : 0.804785, loss: 0.074608, avg_reader_cost: 0.00007 s, avg_batch_cost: 0.11317 s, avg_samples: 33.6, ips: 296.89437 samples/s, eta: 1 day, 17:58:34, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:07:19] ppocr INFO: save model in ./output/rec/latex_ocr/latest load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 03:07:26] ppocr INFO: epoch: [10/300], global_step: 24700, lr: 0.000100, edit distance: 0.048871, exp_rate: 0.340689, exp_rate<=1 : 0.571429, exp_rate<=2 : 0.768761, exp_rate<=3 : 0.866654, loss: 0.063517, avg_reader_cost: 0.02794 s, avg_batch_cost: 0.09887 s, avg_samples: 22.4, ips: 226.55271 samples/s, eta: 1 day, 17:59:41, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:07:44] ppocr INFO: epoch: [10/300], global_step: 24800, lr: 0.000100, edit distance: 0.050742, exp_rate: 0.320950, exp_rate<=1 : 0.545205, exp_rate<=2 : 0.752565, exp_rate<=3 : 0.855682, loss: 0.071576, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17435 s, avg_samples: 56.0, ips: 321.18711 samples/s, eta: 1 day, 17:58:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:08:04] ppocr INFO: epoch: [10/300], global_step: 24900, lr: 0.000100, edit distance: 0.052285, exp_rate: 0.309204, exp_rate<=1 : 0.531095, exp_rate<=2 : 0.736813, exp_rate<=3 : 0.840728, loss: 0.077688, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19845 s, avg_samples: 56.0, ips: 282.18650 samples/s, eta: 1 day, 17:58:42, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:08:23] ppocr INFO: epoch: [10/300], global_step: 25000, lr: 0.000100, edit distance: 0.052595, exp_rate: 0.297435, exp_rate<=1 : 0.515295, exp_rate<=2 : 0.723857, exp_rate<=3 : 0.831648, loss: 0.058958, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19786 s, avg_samples: 56.0, ips: 283.02693 samples/s, eta: 1 day, 17:58:50, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:08:43] ppocr INFO: epoch: [10/300], global_step: 25100, lr: 0.000100, edit distance: 0.053139, exp_rate: 0.296174, exp_rate<=1 : 0.510922, exp_rate<=2 : 0.719090, exp_rate<=3 : 0.827032, loss: 0.085667, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19344 s, avg_samples: 56.0, ips: 289.49542 samples/s, eta: 1 day, 17:58:43, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:09:02] ppocr INFO: epoch: [10/300], global_step: 25200, lr: 0.000100, edit distance: 0.053095, exp_rate: 0.295733, exp_rate<=1 : 0.508718, exp_rate<=2 : 0.715817, exp_rate<=3 : 0.825737, loss: 0.056513, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19074 s, avg_samples: 56.0, ips: 293.58606 samples/s, eta: 1 day, 17:58:28, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:09:22] ppocr INFO: epoch: [10/300], global_step: 25300, lr: 0.000100, edit distance: 0.053521, exp_rate: 0.293948, exp_rate<=1 : 0.508137, exp_rate<=2 : 0.714145, exp_rate<=3 : 0.824248, loss: 0.073784, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19673 s, avg_samples: 56.0, ips: 284.64945 samples/s, eta: 1 day, 17:58:31, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:09:40] ppocr INFO: epoch: [10/300], global_step: 25400, lr: 0.000100, edit distance: 0.053522, exp_rate: 0.293512, exp_rate<=1 : 0.508775, exp_rate<=2 : 0.715094, exp_rate<=3 : 0.825475, loss: 0.064651, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17953 s, avg_samples: 56.0, ips: 311.91745 samples/s, eta: 1 day, 17:57:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:09:58] ppocr INFO: epoch: [10/300], global_step: 25500, lr: 0.000100, edit distance: 0.053240, exp_rate: 0.291602, exp_rate<=1 : 0.507143, exp_rate<=2 : 0.714524, exp_rate<=3 : 0.825869, loss: 0.060084, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18528 s, avg_samples: 56.0, ips: 302.24236 samples/s, eta: 1 day, 17:57:08, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:10:17] ppocr INFO: epoch: [10/300], global_step: 25600, lr: 0.000100, edit distance: 0.053225, exp_rate: 0.291174, exp_rate<=1 : 0.506096, exp_rate<=2 : 0.713483, exp_rate<=3 : 0.825237, loss: 0.067358, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.19155 s, avg_samples: 56.0, ips: 292.35473 samples/s, eta: 1 day, 17:56:55, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:10:36] ppocr INFO: epoch: [10/300], global_step: 25700, lr: 0.000100, edit distance: 0.053265, exp_rate: 0.290815, exp_rate<=1 : 0.505982, exp_rate<=2 : 0.713708, exp_rate<=3 : 0.825642, loss: 0.068283, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18417 s, avg_samples: 56.0, ips: 304.06023 samples/s, eta: 1 day, 17:56:19, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:10:55] ppocr INFO: epoch: [10/300], global_step: 25800, lr: 0.000100, edit distance: 0.053390, exp_rate: 0.290806, exp_rate<=1 : 0.506423, exp_rate<=2 : 0.714515, exp_rate<=3 : 0.826132, loss: 0.066647, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19232 s, avg_samples: 56.0, ips: 291.17554 samples/s, eta: 1 day, 17:56:08, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:11:13] ppocr INFO: epoch: [10/300], global_step: 25900, lr: 0.000100, edit distance: 0.053360, exp_rate: 0.289659, exp_rate<=1 : 0.505577, exp_rate<=2 : 0.714082, exp_rate<=3 : 0.826343, loss: 0.066825, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17790 s, avg_samples: 56.0, ips: 314.78437 samples/s, eta: 1 day, 17:55:13, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:11:33] ppocr INFO: epoch: [10/300], global_step: 26000, lr: 0.000100, edit distance: 0.053301, exp_rate: 0.287557, exp_rate<=1 : 0.503880, exp_rate<=2 : 0.712794, exp_rate<=3 : 0.824861, loss: 0.066941, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19912 s, avg_samples: 56.0, ips: 281.23966 samples/s, eta: 1 day, 17:55:24, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:11:52] ppocr INFO: epoch: [10/300], global_step: 26100, lr: 0.000100, edit distance: 0.053396, exp_rate: 0.287750, exp_rate<=1 : 0.503341, exp_rate<=2 : 0.711814, exp_rate<=3 : 0.823607, loss: 0.069762, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19703 s, avg_samples: 56.0, ips: 284.22638 samples/s, eta: 1 day, 17:55:27, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:12:09] ppocr INFO: epoch: [10/300], global_step: 26200, lr: 0.000100, edit distance: 0.053338, exp_rate: 0.289648, exp_rate<=1 : 0.506005, exp_rate<=2 : 0.714484, exp_rate<=3 : 0.825530, loss: 0.066944, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.16749 s, avg_samples: 56.0, ips: 334.35353 samples/s, eta: 1 day, 17:54:01, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:12:27] ppocr INFO: epoch: [10/300], global_step: 26300, lr: 0.000100, edit distance: 0.053271, exp_rate: 0.289174, exp_rate<=1 : 0.505424, exp_rate<=2 : 0.714569, exp_rate<=3 : 0.825807, loss: 0.067351, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18113 s, avg_samples: 56.0, ips: 309.16897 samples/s, eta: 1 day, 17:53:17, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:12:47] ppocr INFO: epoch: [10/300], global_step: 26400, lr: 0.000100, edit distance: 0.053290, exp_rate: 0.288058, exp_rate<=1 : 0.504166, exp_rate<=2 : 0.713596, exp_rate<=3 : 0.825172, loss: 0.066468, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.19256 s, avg_samples: 56.0, ips: 290.81234 samples/s, eta: 1 day, 17:53:07, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:13:05] ppocr INFO: epoch: [10/300], global_step: 26500, lr: 0.000100, edit distance: 0.053402, exp_rate: 0.288936, exp_rate<=1 : 0.505083, exp_rate<=2 : 0.714535, exp_rate<=3 : 0.826060, loss: 0.064029, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18177 s, avg_samples: 56.0, ips: 308.08818 samples/s, eta: 1 day, 17:52:25, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:13:24] ppocr INFO: epoch: [10/300], global_step: 26600, lr: 0.000100, edit distance: 0.053515, exp_rate: 0.288660, exp_rate<=1 : 0.504692, exp_rate<=2 : 0.714484, exp_rate<=3 : 0.825802, loss: 0.075949, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18655 s, avg_samples: 56.0, ips: 300.19150 samples/s, eta: 1 day, 17:51:57, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:13:43] ppocr INFO: epoch: [10/300], global_step: 26700, lr: 0.000100, edit distance: 0.053621, exp_rate: 0.288397, exp_rate<=1 : 0.504121, exp_rate<=2 : 0.713722, exp_rate<=3 : 0.825067, loss: 0.076641, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19564 s, avg_samples: 56.0, ips: 286.23875 samples/s, eta: 1 day, 17:51:56, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:14:02] ppocr INFO: epoch: [10/300], global_step: 26800, lr: 0.000100, edit distance: 0.053743, exp_rate: 0.287833, exp_rate<=1 : 0.503892, exp_rate<=2 : 0.713434, exp_rate<=3 : 0.824730, loss: 0.062639, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18580 s, avg_samples: 56.0, ips: 301.39348 samples/s, eta: 1 day, 17:51:26, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:14:21] ppocr INFO: epoch: [10/300], global_step: 26900, lr: 0.000100, edit distance: 0.053802, exp_rate: 0.287358, exp_rate<=1 : 0.503232, exp_rate<=2 : 0.712969, exp_rate<=3 : 0.823938, loss: 0.068588, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19615 s, avg_samples: 56.0, ips: 285.50172 samples/s, eta: 1 day, 17:51:27, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:14:40] ppocr INFO: epoch: [10/300], global_step: 27000, lr: 0.000100, edit distance: 0.053856, exp_rate: 0.286664, exp_rate<=1 : 0.503029, exp_rate<=2 : 0.712691, exp_rate<=3 : 0.823593, loss: 0.072353, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19100 s, avg_samples: 56.0, ips: 293.19955 samples/s, eta: 1 day, 17:51:12, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:14:59] ppocr INFO: epoch: [10/300], global_step: 27100, lr: 0.000100, edit distance: 0.053820, exp_rate: 0.286607, exp_rate<=1 : 0.503476, exp_rate<=2 : 0.713136, exp_rate<=3 : 0.824003, loss: 0.057222, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18249 s, avg_samples: 56.0, ips: 306.86006 samples/s, eta: 1 day, 17:50:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:15:17] ppocr INFO: epoch: [10/300], global_step: 27200, lr: 0.000100, edit distance: 0.053974, exp_rate: 0.286885, exp_rate<=1 : 0.503969, exp_rate<=2 : 0.713799, exp_rate<=3 : 0.824273, loss: 0.063757, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18429 s, avg_samples: 56.0, ips: 303.87445 samples/s, eta: 1 day, 17:49:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:15:36] ppocr INFO: epoch: [10/300], global_step: 27300, lr: 0.000100, edit distance: 0.053991, exp_rate: 0.286409, exp_rate<=1 : 0.503175, exp_rate<=2 : 0.713123, exp_rate<=3 : 0.823750, loss: 0.072652, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18729 s, avg_samples: 56.0, ips: 298.99996 samples/s, eta: 1 day, 17:49:33, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:15:54] ppocr INFO: epoch: [10/300], global_step: 27400, lr: 0.000100, edit distance: 0.054072, exp_rate: 0.285998, exp_rate<=1 : 0.502752, exp_rate<=2 : 0.712651, exp_rate<=3 : 0.823314, loss: 0.063492, avg_reader_cost: 0.00013 s, avg_batch_cost: 0.18450 s, avg_samples: 56.0, ips: 303.51686 samples/s, eta: 1 day, 17:49:00, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:15:57] ppocr INFO: save model in ./output/rec/latex_ocr/latest [2024/07/08 03:15:57] ppocr INFO: save model in ./output/rec/latex_ocr/iter_epoch_10 load_config {'shuffle': True, 'batch_size_per_card': 1, 'drop_last': False, 'num_workers': 0, 'collate_fn': 'LaTeXOCRCollator'} list index out of range [2024/07/08 03:16:18] ppocr INFO: epoch: [11/300], global_step: 27500, lr: 0.000100, edit distance: 0.046034, exp_rate: 0.298204, exp_rate<=1 : 0.503608, exp_rate<=2 : 0.712412, exp_rate<=3 : 0.825276, loss: 0.049939, avg_reader_cost: 0.03264 s, avg_batch_cost: 0.23311 s, avg_samples: 56.0, ips: 240.23213 samples/s, eta: 1 day, 17:50:47, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:16:36] ppocr INFO: epoch: [11/300], global_step: 27600, lr: 0.000100, edit distance: 0.046101, exp_rate: 0.317336, exp_rate<=1 : 0.532377, exp_rate<=2 : 0.737437, exp_rate<=3 : 0.845002, loss: 0.049507, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18052 s, avg_samples: 56.0, ips: 310.21680 samples/s, eta: 1 day, 17:50:02, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:16:55] ppocr INFO: epoch: [11/300], global_step: 27700, lr: 0.000100, edit distance: 0.046049, exp_rate: 0.313846, exp_rate<=1 : 0.530709, exp_rate<=2 : 0.737880, exp_rate<=3 : 0.843573, loss: 0.064323, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19261 s, avg_samples: 56.0, ips: 290.73610 samples/s, eta: 1 day, 17:49:52, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:17:15] ppocr INFO: epoch: [11/300], global_step: 27800, lr: 0.000100, edit distance: 0.046342, exp_rate: 0.309209, exp_rate<=1 : 0.523361, exp_rate<=2 : 0.731034, exp_rate<=3 : 0.837639, loss: 0.052998, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19862 s, avg_samples: 56.0, ips: 281.95230 samples/s, eta: 1 day, 17:49:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:17:34] ppocr INFO: epoch: [11/300], global_step: 27900, lr: 0.000100, edit distance: 0.046365, exp_rate: 0.314592, exp_rate<=1 : 0.527839, exp_rate<=2 : 0.733172, exp_rate<=3 : 0.838119, loss: 0.056651, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18867 s, avg_samples: 56.0, ips: 296.82102 samples/s, eta: 1 day, 17:49:37, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:17:53] ppocr INFO: epoch: [11/300], global_step: 28000, lr: 0.000100, edit distance: 0.046566, exp_rate: 0.314950, exp_rate<=1 : 0.527103, exp_rate<=2 : 0.732700, exp_rate<=3 : 0.838022, loss: 0.054738, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18884 s, avg_samples: 56.0, ips: 296.54736 samples/s, eta: 1 day, 17:49:16, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:18:12] ppocr INFO: epoch: [11/300], global_step: 28100, lr: 0.000100, edit distance: 0.046598, exp_rate: 0.315933, exp_rate<=1 : 0.529006, exp_rate<=2 : 0.732982, exp_rate<=3 : 0.838677, loss: 0.055837, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19239 s, avg_samples: 56.0, ips: 291.08006 samples/s, eta: 1 day, 17:49:05, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:18:30] ppocr INFO: epoch: [11/300], global_step: 28200, lr: 0.000100, edit distance: 0.046662, exp_rate: 0.319811, exp_rate<=1 : 0.534229, exp_rate<=2 : 0.737649, exp_rate<=3 : 0.842061, loss: 0.052795, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.17585 s, avg_samples: 56.0, ips: 318.45211 samples/s, eta: 1 day, 17:48:07, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:18:49] ppocr INFO: epoch: [11/300], global_step: 28300, lr: 0.000100, edit distance: 0.046851, exp_rate: 0.318576, exp_rate<=1 : 0.534292, exp_rate<=2 : 0.737109, exp_rate<=3 : 0.841328, loss: 0.055617, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19370 s, avg_samples: 56.0, ips: 289.10528 samples/s, eta: 1 day, 17:47:59, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:19:07] ppocr INFO: epoch: [11/300], global_step: 28400, lr: 0.000100, edit distance: 0.047167, exp_rate: 0.319824, exp_rate<=1 : 0.534974, exp_rate<=2 : 0.738623, exp_rate<=3 : 0.842640, loss: 0.059258, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18143 s, avg_samples: 56.0, ips: 308.66590 samples/s, eta: 1 day, 17:47:18, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:19:25] ppocr INFO: epoch: [11/300], global_step: 28500, lr: 0.000100, edit distance: 0.047324, exp_rate: 0.320260, exp_rate<=1 : 0.536172, exp_rate<=2 : 0.739271, exp_rate<=3 : 0.843803, loss: 0.056362, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17928 s, avg_samples: 56.0, ips: 312.36585 samples/s, eta: 1 day, 17:46:30, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:19:43] ppocr INFO: epoch: [11/300], global_step: 28600, lr: 0.000100, edit distance: 0.047448, exp_rate: 0.321826, exp_rate<=1 : 0.537460, exp_rate<=2 : 0.740046, exp_rate<=3 : 0.844616, loss: 0.060168, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.18189 s, avg_samples: 56.0, ips: 307.88668 samples/s, eta: 1 day, 17:45:50, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:20:02] ppocr INFO: epoch: [11/300], global_step: 28700, lr: 0.000100, edit distance: 0.047625, exp_rate: 0.320066, exp_rate<=1 : 0.535846, exp_rate<=2 : 0.739145, exp_rate<=3 : 0.844122, loss: 0.064980, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18974 s, avg_samples: 56.0, ips: 295.13625 samples/s, eta: 1 day, 17:45:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:20:22] ppocr INFO: epoch: [11/300], global_step: 28800, lr: 0.000100, edit distance: 0.047734, exp_rate: 0.317389, exp_rate<=1 : 0.531951, exp_rate<=2 : 0.735985, exp_rate<=3 : 0.841810, loss: 0.054911, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19909 s, avg_samples: 56.0, ips: 281.28603 samples/s, eta: 1 day, 17:45:39, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:20:40] ppocr INFO: epoch: [11/300], global_step: 28900, lr: 0.000100, edit distance: 0.047803, exp_rate: 0.317211, exp_rate<=1 : 0.532464, exp_rate<=2 : 0.737668, exp_rate<=3 : 0.843312, loss: 0.063949, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17834 s, avg_samples: 56.0, ips: 314.00798 samples/s, eta: 1 day, 17:44:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:21:00] ppocr INFO: epoch: [11/300], global_step: 29000, lr: 0.000100, edit distance: 0.047784, exp_rate: 0.316720, exp_rate<=1 : 0.531948, exp_rate<=2 : 0.737548, exp_rate<=3 : 0.843310, loss: 0.054153, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19587 s, avg_samples: 56.0, ips: 285.90956 samples/s, eta: 1 day, 17:44:48, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:21:17] ppocr INFO: epoch: [11/300], global_step: 29100, lr: 0.000100, edit distance: 0.047858, exp_rate: 0.315236, exp_rate<=1 : 0.530642, exp_rate<=2 : 0.737356, exp_rate<=3 : 0.843495, loss: 0.059771, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.17562 s, avg_samples: 56.0, ips: 318.87443 samples/s, eta: 1 day, 17:43:51, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:21:36] ppocr INFO: epoch: [11/300], global_step: 29200, lr: 0.000100, edit distance: 0.047973, exp_rate: 0.316029, exp_rate<=1 : 0.532243, exp_rate<=2 : 0.738652, exp_rate<=3 : 0.844423, loss: 0.057066, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.18278 s, avg_samples: 56.0, ips: 306.37714 samples/s, eta: 1 day, 17:43:14, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:21:54] ppocr INFO: epoch: [11/300], global_step: 29300, lr: 0.000100, edit distance: 0.048090, exp_rate: 0.317404, exp_rate<=1 : 0.533673, exp_rate<=2 : 0.738860, exp_rate<=3 : 0.844396, loss: 0.057913, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18375 s, avg_samples: 56.0, ips: 304.75387 samples/s, eta: 1 day, 17:42:40, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:22:15] ppocr INFO: epoch: [11/300], global_step: 29400, lr: 0.000100, edit distance: 0.048285, exp_rate: 0.314292, exp_rate<=1 : 0.530319, exp_rate<=2 : 0.736039, exp_rate<=3 : 0.842275, loss: 0.065078, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.21399 s, avg_samples: 56.0, ips: 261.69283 samples/s, eta: 1 day, 17:43:27, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:22:33] ppocr INFO: epoch: [11/300], global_step: 29500, lr: 0.000100, edit distance: 0.048351, exp_rate: 0.315455, exp_rate<=1 : 0.531909, exp_rate<=2 : 0.737008, exp_rate<=3 : 0.843061, loss: 0.061088, avg_reader_cost: 0.00012 s, avg_batch_cost: 0.17630 s, avg_samples: 56.0, ips: 317.63608 samples/s, eta: 1 day, 17:42:32, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:22:52] ppocr INFO: epoch: [11/300], global_step: 29600, lr: 0.000100, edit distance: 0.048396, exp_rate: 0.315031, exp_rate<=1 : 0.530651, exp_rate<=2 : 0.735954, exp_rate<=3 : 0.842294, loss: 0.061522, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18790 s, avg_samples: 56.0, ips: 298.03215 samples/s, eta: 1 day, 17:42:09, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:23:11] ppocr INFO: epoch: [11/300], global_step: 29700, lr: 0.000100, edit distance: 0.048414, exp_rate: 0.314241, exp_rate<=1 : 0.529963, exp_rate<=2 : 0.735516, exp_rate<=3 : 0.842117, loss: 0.060014, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.18901 s, avg_samples: 56.0, ips: 296.27305 samples/s, eta: 1 day, 17:41:49, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:23:30] ppocr INFO: epoch: [11/300], global_step: 29800, lr: 0.000100, edit distance: 0.048341, exp_rate: 0.314244, exp_rate<=1 : 0.529550, exp_rate<=2 : 0.735021, exp_rate<=3 : 0.841905, loss: 0.061676, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19198 s, avg_samples: 56.0, ips: 291.69884 samples/s, eta: 1 day, 17:41:37, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:23:49] ppocr INFO: epoch: [11/300], global_step: 29900, lr: 0.000100, edit distance: 0.048318, exp_rate: 0.313608, exp_rate<=1 : 0.528518, exp_rate<=2 : 0.734398, exp_rate<=3 : 0.841599, loss: 0.062101, avg_reader_cost: 0.00010 s, avg_batch_cost: 0.19334 s, avg_samples: 56.0, ips: 289.64490 samples/s, eta: 1 day, 17:41:28, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB [2024/07/08 03:24:08] ppocr INFO: epoch: [11/300], global_step: 30000, lr: 0.000100, edit distance: 0.048302, exp_rate: 0.313360, exp_rate<=1 : 0.528132, exp_rate<=2 : 0.734145, exp_rate<=3 : 0.841295, loss: 0.059429, avg_reader_cost: 0.00011 s, avg_batch_cost: 0.19042 s, avg_samples: 56.0, ips: 294.08252 samples/s, eta: 1 day, 17:41:12, max_mem_reserved: 30648 MB, max_mem_allocated: 24983 MB eval model:: 0%| | 0/716 [00:00