Recognition (en_PP_OCRv4_rec_train) finetuning questions #14387

l03126 · 2024-12-16T01:51:54Z

l03126
Dec 16, 2024

I have a few pretty beginner questions on finetuning a model. For my use case most characters are recognized well except for few characters in my font that do not look like traditional characters (OCR-A Font). Hence, I plan on finetuning the model.

To test out the training, I experimented with a very small dataset of just 10 images. YML file below:

Architecture:
  Backbone:
    name: PPLCNetV3
    scale: 0.95
  Head:
    head_list:
    - CTCHead:
        Head:
          fc_decay: 1.0e-05
        Neck:
          depth: 2
          dims: 120
          hidden_dims: 120
          kernel_size:
          - 1
          - 3
          name: svtr
          use_guide: true
    - NRTRHead:
        max_text_length: 25
        nrtr_dim: 384
    name: MultiHead
  Transform: null
  algorithm: SVTR_LCNet
  model_type: rec
Eval:
  dataset:
    data_dir: /kaggle/input/recognition-data7/Recognition7/Test/images
    label_file_list:
    - /kaggle/input/recognition-data7/Recognition7/Test/RecTestLabel.txt
    name: SimpleDataSet
    transforms:
    - DecodeImage:
        channel_first: false
        img_mode: BGR
    - MultiLabelEncode:
        gtc_encode: NRTRLabelEncode
    - RecResizeImg:
        image_shape:
        - 3
        - 48
        - 320
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_gtc
        - length
        - valid_ratio
  loader:
    batch_size_per_card: 1
    drop_last: false
    num_workers: 1
    shuffle: false
Global:
  cal_metric_during_train: true
  character_dict_path: /kaggle/input/char-dict/char_dict.txt
  checkpoints: null
  debug: false
  distributed: true
  epoch_num: 50
  eval_batch_step:
  - 0
  - 10
  infer_img: doc/imgs_words/ch/word_1.jpg
  infer_mode: false
  log_smooth_window: 20
  max_text_length: 25
  pretrained_model: null
  print_batch_step: 1
  save_epoch_step: 1
  save_inference_dir: null
  save_model_dir: /kaggle/working/finetuned_model
  save_res_path: ./output/rec/predicts_ppocrv3.txt
  use_gpu: true
  use_space_char: true
  use_visualdl: false
Loss:
  loss_config_list:
  - CTCLoss: null
  - NRTRLoss: null
  name: MultiLoss
Metric:
  ignore_space: false
  main_indicator: acc
  name: RecMetric
Optimizer:
  beta1: 0.9
  beta2: 0.999
  lr:
    learning_rate: 0.0005
    name: Cosine
    warmup_epoch: 1
  name: Adam
  regularizer:
    factor: 3.0e-05
    name: L2
PostProcess:
  name: CTCLabelDecode
Train:
  dataset:
    data_dir: /kaggle/input/recognition-data7/Recognition7/Train/images
    ds_width: false
    ext_op_transform_idx: 1
    label_file_list:
    - /kaggle/input/recognition-data7/Recognition7/Train/RecTrainLabel.txt
    name: MultiScaleDataSet
    transforms:
    - DecodeImage:
        channel_first: false
        img_mode: BGR
    - RecConAug:
        ext_data_num: 2
        image_shape:
        - 48
        - 320
        - 3
        max_text_length: 25
        prob: 0.5
    - RecAug: null
    - MultiLabelEncode:
        gtc_encode: NRTRLabelEncode
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_gtc
        - length
        - valid_ratio
  loader:
    batch_size_per_card: 1
    drop_last: true
    num_workers: 1
    shuffle: true
  sampler:
    divided_factor:
    - 8
    - 16
    first_bs: 1
    fix_bs: false
    is_training: true
    name: MultiScaleSampler
    scales:
    - - 320
      - 32
    - - 320
      - 48
    - - 320
      - 64
profiler_options: null

The results of the finetuning are as follows (first 2 epoch and last 2 epochs):

W1216 01:31:53.997754   287 gpu_resources.cc:164] device: 0, cuDNN Version: 9.0.
[2024/12/16 01:31:54] ppocr INFO: train dataloader has 9 iters
[2024/12/16 01:31:54] ppocr INFO: valid dataloader has 3 iters
[2024/12/16 01:31:54] ppocr INFO: train from scratch
[2024/12/16 01:31:54] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 10 iterations
[2024/12/16 01:31:57] ppocr INFO: epoch: [1/50], global_step: 1, lr: 0.000000, acc: 0.000000, norm_edit_dis: 0.142866, CTCLoss: 116.376183, NRTRLoss: 4.028224, loss: 120.404404, avg_reader_cost: 0.17032 s, avg_batch_cost: 2.95783 s, avg_samples: 1.0, ips: 0.33809 samples/s, eta: 0:22:08, max_mem_reserved: 427 MB, max_mem_allocated: 338 MB
[2024/12/16 01:31:57] ppocr INFO: epoch: [1/50], global_step: 2, lr: 0.000028, acc: 0.000000, norm_edit_dis: 0.139005, CTCLoss: 114.874996, NRTRLoss: 4.145246, loss: 119.020241, avg_reader_cost: 0.00316 s, avg_batch_cost: 0.15258 s, avg_samples: 1.0, ips: 6.55376 samples/s, eta: 0:11:36, max_mem_reserved: 528 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:57] ppocr INFO: epoch: [1/50], global_step: 3, lr: 0.000056, acc: 0.000000, norm_edit_dis: 0.142866, CTCLoss: 114.533569, NRTRLoss: 4.028224, loss: 118.409439, avg_reader_cost: 0.00288 s, avg_batch_cost: 0.14287 s, avg_samples: 1.0, ips: 6.99961 samples/s, eta: 0:08:04, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:57] ppocr INFO: epoch: [1/50], global_step: 4, lr: 0.000083, acc: 0.000000, norm_edit_dis: 0.139005, CTCLoss: 114.412453, NRTRLoss: 3.952049, loss: 118.022758, avg_reader_cost: 0.00306 s, avg_batch_cost: 0.13872 s, avg_samples: 1.0, ips: 7.20859 samples/s, eta: 0:06:18, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:58] ppocr INFO: epoch: [1/50], global_step: 5, lr: 0.000111, acc: 0.000000, norm_edit_dis: 0.135144, CTCLoss: 114.291336, NRTRLoss: 3.875873, loss: 117.636078, avg_reader_cost: 0.00415 s, avg_batch_cost: 0.18691 s, avg_samples: 1.0, ips: 5.35026 samples/s, eta: 0:05:18, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:58] ppocr INFO: epoch: [1/50], global_step: 6, lr: 0.000139, acc: 0.000000, norm_edit_dis: 0.128182, CTCLoss: 113.869442, NRTRLoss: 3.757446, loss: 117.631321, avg_reader_cost: 0.00335 s, avg_batch_cost: 0.14416 s, avg_samples: 1.0, ips: 6.93687 samples/s, eta: 0:04:35, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:58] ppocr INFO: epoch: [1/50], global_step: 7, lr: 0.000167, acc: 0.000000, norm_edit_dis: 0.121221, CTCLoss: 114.291336, NRTRLoss: 3.639019, loss: 117.636078, avg_reader_cost: 0.00333 s, avg_batch_cost: 0.16965 s, avg_samples: 1.0, ips: 5.89446 samples/s, eta: 0:04:06, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:58] ppocr INFO: epoch: [1/50], global_step: 8, lr: 0.000194, acc: 0.000000, norm_edit_dis: 0.123115, CTCLoss: 113.869442, NRTRLoss: 3.585448, loss: 117.631321, avg_reader_cost: 0.00354 s, avg_batch_cost: 0.15066 s, avg_samples: 1.0, ips: 6.63742 samples/s, eta: 0:03:43, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:58] ppocr INFO: epoch: [1/50], global_step: 9, lr: 0.000222, acc: 0.000000, norm_edit_dis: 0.121221, CTCLoss: 113.447548, NRTRLoss: 3.531876, loss: 117.626564, avg_reader_cost: 0.00311 s, avg_batch_cost: 0.14437 s, avg_samples: 1.0, ips: 6.92647 samples/s, eta: 0:03:25, max_mem_reserved: 531 MB, max_mem_allocated: 479 MB
[2024/12/16 01:31:59] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/16 01:32:00] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_1
[2024/12/16 01:32:00] ppocr INFO: epoch: [2/50], global_step: 10, lr: 0.000250, acc: 0.000000, norm_edit_dis: 0.123115, CTCLoss: 113.410679, NRTRLoss: 3.516903, loss: 117.356567, avg_reader_cost: 1.53860 s, avg_batch_cost: 1.76911 s, avg_samples: 1.0, ips: 0.56525 samples/s, eta: 0:04:22, max_mem_reserved: 538 MB, max_mem_allocated: 479 MB
eval model:: 100%|████████████████████████████████| 3/3 [00:00<00:00, 11.28it/s]
[2024/12/16 01:32:00] ppocr INFO: cur metric, acc: 0.0, norm_edit_dis: 0.047622222211640364, fps: 15.985060349328668
[2024/12/16 01:32:02] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/16 01:32:02] ppocr INFO: best metric, acc: 0.0, is_float16: False, norm_edit_dis: 0.047622222211640364, fps: 15.985060349328668, best_epoch: 2
[2024/12/16 01:32:02] ppocr INFO: epoch: [2/50], global_step: 11, lr: 0.000278, acc: 0.000000, norm_edit_dis: 0.121221, CTCLoss: 113.373810, NRTRLoss: 3.531876, loss: 117.086571, avg_reader_cost: 0.00338 s, avg_batch_cost: 0.14452 s, avg_samples: 1.0, ips: 6.91963 samples/s, eta: 0:04:03, max_mem_reserved: 538 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:02] ppocr INFO: epoch: [2/50], global_step: 12, lr: 0.000306, acc: 0.000000, norm_edit_dis: 0.123115, CTCLoss: 112.358006, NRTRLoss: 3.516903, loss: 115.980324, avg_reader_cost: 0.00300 s, avg_batch_cost: 0.13301 s, avg_samples: 1.0, ips: 7.51840 samples/s, eta: 0:03:47, max_mem_reserved: 540 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:02] ppocr INFO: epoch: [2/50], global_step: 13, lr: 0.000333, acc: 0.000000, norm_edit_dis: 0.125009, CTCLoss: 111.342201, NRTRLoss: 3.501930, loss: 114.874077, avg_reader_cost: 0.00281 s, avg_batch_cost: 0.13125 s, avg_samples: 1.0, ips: 7.61897 samples/s, eta: 0:03:33, max_mem_reserved: 540 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:02] ppocr INFO: epoch: [2/50], global_step: 14, lr: 0.000361, acc: 0.000000, norm_edit_dis: 0.129175, CTCLoss: 111.189186, NRTRLoss: 3.465416, loss: 114.669575, avg_reader_cost: 0.00299 s, avg_batch_cost: 0.13243 s, avg_samples: 1.0, ips: 7.55088 samples/s, eta: 0:03:22, max_mem_reserved: 540 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:02] ppocr INFO: epoch: [2/50], global_step: 15, lr: 0.000389, acc: 0.000000, norm_edit_dis: 0.125009, CTCLoss: 111.036171, NRTRLoss: 3.428901, loss: 114.465073, avg_reader_cost: 0.00300 s, avg_batch_cost: 0.12989 s, avg_samples: 1.0, ips: 7.69889 samples/s, eta: 0:03:12, max_mem_reserved: 540 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:02] ppocr INFO: epoch: [2/50], global_step: 16, lr: 0.000417, acc: 0.000000, norm_edit_dis: 0.129175, CTCLoss: 110.871017, NRTRLoss: 3.389934, loss: 114.299404, avg_reader_cost: 0.00299 s, avg_batch_cost: 0.12997 s, avg_samples: 1.0, ips: 7.69435 samples/s, eta: 0:03:03, max_mem_reserved: 540 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:03] ppocr INFO: epoch: [2/50], global_step: 17, lr: 0.000444, acc: 0.000000, norm_edit_dis: 0.133342, CTCLoss: 110.705864, NRTRLoss: 3.350966, loss: 114.133736, avg_reader_cost: 0.00288 s, avg_batch_cost: 0.13391 s, avg_samples: 1.0, ips: 7.46787 samples/s, eta: 0:02:55, max_mem_reserved: 542 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:03] ppocr INFO: epoch: [2/50], global_step: 18, lr: 0.000472, acc: 0.000000, norm_edit_dis: 0.129175, CTCLoss: 110.617493, NRTRLoss: 3.343098, loss: 114.067780, avg_reader_cost: 0.00306 s, avg_batch_cost: 0.13615 s, avg_samples: 1.0, ips: 7.34498 samples/s, eta: 0:02:48, max_mem_reserved: 542 MB, max_mem_allocated: 479 MB
[2024/12/16 01:32:04] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/16 01:32:04] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_2
.
.
.
.
.
[2024/12/16 01:35:58] ppocr INFO: epoch: [49/50], global_step: 433, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.357149, CTCLoss: 44.512390, NRTRLoss: 0.765518, loss: 45.305004, avg_reader_cost: 1.65331 s, avg_batch_cost: 1.84601 s, avg_samples: 1.0, ips: 0.54171 samples/s, eta: 0:00:06, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:58] ppocr INFO: epoch: [49/50], global_step: 434, lr: 0.000008, acc: 0.000000, norm_edit_dis: 0.357149, CTCLoss: 44.512390, NRTRLoss: 0.757762, loss: 45.305004, avg_reader_cost: 0.00333 s, avg_batch_cost: 0.14422 s, avg_samples: 1.0, ips: 6.93362 samples/s, eta: 0:00:05, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:59] ppocr INFO: epoch: [49/50], global_step: 435, lr: 0.000007, acc: 0.000000, norm_edit_dis: 0.333340, CTCLoss: 45.524193, NRTRLoss: 0.766032, loss: 46.318789, avg_reader_cost: 0.00288 s, avg_batch_cost: 0.13616 s, avg_samples: 1.0, ips: 7.34415 samples/s, eta: 0:00:05, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:59] ppocr INFO: epoch: [49/50], global_step: 436, lr: 0.000007, acc: 0.000000, norm_edit_dis: 0.357149, CTCLoss: 44.512390, NRTRLoss: 0.757762, loss: 45.305004, avg_reader_cost: 0.00311 s, avg_batch_cost: 0.13752 s, avg_samples: 1.0, ips: 7.27168 samples/s, eta: 0:00:05, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:59] ppocr INFO: epoch: [49/50], global_step: 437, lr: 0.000006, acc: 0.000000, norm_edit_dis: 0.357149, CTCLoss: 44.512390, NRTRLoss: 0.766032, loss: 45.305004, avg_reader_cost: 0.00300 s, avg_batch_cost: 0.13335 s, avg_samples: 1.0, ips: 7.49918 samples/s, eta: 0:00:04, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:59] ppocr INFO: epoch: [49/50], global_step: 438, lr: 0.000006, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 44.512390, NRTRLoss: 0.766032, loss: 45.305004, avg_reader_cost: 0.00308 s, avg_batch_cost: 0.14866 s, avg_samples: 1.0, ips: 6.72690 samples/s, eta: 0:00:04, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:59] ppocr INFO: epoch: [49/50], global_step: 439, lr: 0.000006, acc: 0.000000, norm_edit_dis: 0.404768, CTCLoss: 43.394655, NRTRLoss: 0.757762, loss: 44.171045, avg_reader_cost: 0.00322 s, avg_batch_cost: 0.14744 s, avg_samples: 1.0, ips: 6.78260 samples/s, eta: 0:00:03, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:35:59] ppocr INFO: epoch: [49/50], global_step: 440, lr: 0.000005, acc: 0.000000, norm_edit_dis: 0.404768, CTCLoss: 43.394655, NRTRLoss: 0.757762, loss: 44.171045, avg_reader_cost: 0.00325 s, avg_batch_cost: 0.15096 s, avg_samples: 1.0, ips: 6.62447 samples/s, eta: 0:00:03, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
eval model:: 100%|████████████████████████████████| 3/3 [00:00<00:00, 14.40it/s]
[2024/12/16 01:35:59] ppocr INFO: cur metric, acc: 0.0, norm_edit_dis: 3.3333222222431402e-06, fps: 22.60296429091079
[2024/12/16 01:36:01] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/16 01:36:01] ppocr INFO: best metric, acc: 0.0, is_float16: False, norm_edit_dis: 3.3333222222431402e-06, fps: 22.60296429091079, best_epoch: 49
[2024/12/16 01:36:01] ppocr INFO: epoch: [49/50], global_step: 441, lr: 0.000005, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 43.394655, NRTRLoss: 0.748861, loss: 44.171045, avg_reader_cost: 0.00355 s, avg_batch_cost: 0.14545 s, avg_samples: 1.0, ips: 6.87523 samples/s, eta: 0:00:03, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:02] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/16 01:36:03] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_49
[2024/12/16 01:36:03] ppocr INFO: epoch: [50/50], global_step: 442, lr: 0.000005, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 44.512390, NRTRLoss: 0.757762, loss: 45.305004, avg_reader_cost: 1.64218 s, avg_batch_cost: 1.83549 s, avg_samples: 1.0, ips: 0.54481 samples/s, eta: 0:00:02, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:03] ppocr INFO: epoch: [50/50], global_step: 443, lr: 0.000004, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 44.512390, NRTRLoss: 0.757762, loss: 45.305004, avg_reader_cost: 0.00297 s, avg_batch_cost: 0.13316 s, avg_samples: 1.0, ips: 7.51003 samples/s, eta: 0:00:02, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 444, lr: 0.000004, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 44.512390, NRTRLoss: 0.757762, loss: 45.305004, avg_reader_cost: 0.00858 s, avg_batch_cost: 0.14246 s, avg_samples: 1.0, ips: 7.01927 samples/s, eta: 0:00:02, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 445, lr: 0.000004, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 43.394655, NRTRLoss: 0.748861, loss: 44.171045, avg_reader_cost: 0.00297 s, avg_batch_cost: 0.14092 s, avg_samples: 1.0, ips: 7.09617 samples/s, eta: 0:00:01, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 446, lr: 0.000003, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 43.394655, NRTRLoss: 0.748861, loss: 44.171045, avg_reader_cost: 0.00317 s, avg_batch_cost: 0.14240 s, avg_samples: 1.0, ips: 7.02260 samples/s, eta: 0:00:01, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 447, lr: 0.000003, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 40.224833, NRTRLoss: 0.743349, loss: 40.959333, avg_reader_cost: 0.00352 s, avg_batch_cost: 0.14520 s, avg_samples: 1.0, ips: 6.88688 samples/s, eta: 0:00:01, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 448, lr: 0.000003, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 43.394655, NRTRLoss: 0.748861, loss: 44.171045, avg_reader_cost: 0.00357 s, avg_batch_cost: 0.14411 s, avg_samples: 1.0, ips: 6.93936 samples/s, eta: 0:00:00, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 449, lr: 0.000003, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 40.224833, NRTRLoss: 0.744121, loss: 40.959333, avg_reader_cost: 0.00306 s, avg_batch_cost: 0.13996 s, avg_samples: 1.0, ips: 7.14510 samples/s, eta: 0:00:00, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
[2024/12/16 01:36:04] ppocr INFO: epoch: [50/50], global_step: 450, lr: 0.000002, acc: 0.000000, norm_edit_dis: 0.380959, CTCLoss: 40.224833, NRTRLoss: 0.744121, loss: 40.959333, avg_reader_cost: 0.00300 s, avg_batch_cost: 0.14647 s, avg_samples: 1.0, ips: 6.82726 samples/s, eta: 0:00:00, max_mem_reserved: 554 MB, max_mem_allocated: 479 MB
eval model:: 100%|████████████████████████████████| 3/3 [00:00<00:00, 13.19it/s]
[2024/12/16 01:36:05] ppocr INFO: cur metric, acc: 0.0, norm_edit_dis: 3.3333222222431402e-06, fps: 20.616436246383127
[2024/12/16 01:36:06] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/16 01:36:06] ppocr INFO: best metric, acc: 0.0, is_float16: False, norm_edit_dis: 3.3333222222431402e-06, fps: 20.616436246383127, best_epoch: 50
[2024/12/16 01:36:07] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/16 01:36:08] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_50
[2024/12/16 01:36:08] ppocr INFO: best metric, acc: 0.0, is_float16: False, norm_edit_dis: 3.3333222222431402e-06, fps: 20.616436246383127, best_epoch: 50

Questions:

The loss and learning rates are decreasing, however the accuracy doesn't seem to increase (constant 0). I assume it's because of the extremely small dataset, or could it be something else?
At the end of the finetuning, is the result models /kaggle/working/finetuned_model/best_accuracy.states, /kaggle/working/finetuned_model/best_accuracy.pdopt and /kaggle/working/finetuned_model/best_accuracy.pdparams the same model as /kaggle/working/finetuned_model/best_model/model.pdopt and /kaggle/working/finetuned_model/best_model/model.pdparams?
Is there any other suggested way to finetune the model just for a specific font? I have already restricted the char_dict to only include the characters for my use case.
Is there an existing way of testing the finetuned model? Or must it be done manually? I plan to split my larger dataset later for Train, Test and Eval. (I have labelled the Eval set as Test in this scenario-please ignore)

Hey @VishyAnand28, hope you could provide your input for these questions. Really appreciate it

GreatV · 2024-12-16T02:01:06Z

GreatV
Dec 16, 2024
Maintainer

Answers to Your Questions:

1. The loss and learning rates are decreasing, but the accuracy is constant at 0. Could this be due to the extremely small dataset or something else?

Yes, the extremely small dataset (just 10 images) is a significant factor here. Models, especially deep learning models, require a sufficient amount of diverse data to generalize and improve accuracy. Here are a few possible reasons for the constant accuracy:

Dataset Size: A dataset with only 10 images is too small to sufficiently train the model. It may not provide enough variety for the model to learn meaningful patterns.
Overfitting: The small dataset may cause overfitting, where the model memorizes the training data but fails to generalize to unseen data.
Labeling Issues: Ensure that the labels in your dataset are accurate and correctly formatted according to your character dictionary.
Learning Rate: While your learning rate is decreasing as per schedule, the model might not have enough data to make meaningful updates to its weights.
Pretrained Model Usage: If you're starting from scratch (as noted in the logs), you may not be using the pretrained weights appropriately. Ensure the pretrained_model field in the config file is correctly pointing to the pretrained model (e.g., en_PP-OCRv4_rec_train.pdparams).

Suggestions:

Use a larger and more diverse dataset to fine-tune the model.
Ensure that the pretrained model is correctly loaded to leverage its existing knowledge.
Verify that your character dictionary matches the target font and characters you want to recognize.
Start with a smaller learning rate and verify that the model is learning effectively by monitoring the norm edit distance (norm_edit_dis) alongside the accuracy.

2. Are the models saved in `/kaggle/working/finetuned_model/best_accuracy.` the same as those in `/kaggle/working/finetuned_model/best_model/`?

No, these are not necessarily the same. Based on PaddleOCR's conventions:

/kaggle/working/finetuned_model/best_accuracy.*: These files correspond to the model checkpoint that achieved the best accuracy during training.
/kaggle/working/finetuned_model/best_model/*: This could be a separate naming convention for storing the best-performing model for another metric or another checkpoint saved manually.

Ensure to check the training logs for save best model is to ... to confirm which checkpoint corresponds to the best accuracy.

3. Is there another suggested way to fine-tune the model for a specific font?

Yes, here are some targeted suggestions for fine-tuning on a specific font:

Generate Synthetic Data: If you have a limited amount of real data, generate synthetic images using the target font. Tools like TextRecognitionDataGenerator, SynthText, or custom scripts can help create labeled data for fine-tuning.
Restrict the Dictionary: You've already restricted the character dictionary, which is an important step. Ensure it contains only the characters present in the target font.
Start from Pretrained Model: Use a pretrained model (e.g., en_PP-OCRv4_rec_train.pdparams) as your starting point. This avoids training from scratch and helps leverage existing knowledge.
Modify Augmentation Techniques: Use augmentation strategies that mimic variations in the target font, such as slight skew, noise, or scaling.
Custom Loss Weights: If certain characters are more challenging for the model, consider re-weighting the loss for specific characters or adding more examples of those characters in the dataset.

4. Is there an existing way to test the fine-tuned model, or must it be done manually?

There are existing ways to test the fine-tuned model in PaddleOCR:

Evaluation During Training: Your config already evaluates the model during training (as seen from the eval_batch_step parameter). This provides metrics like accuracy and norm edit distance.
Post-Training Inference: Use PaddleOCR's tools/infer_rec.py script to test the fine-tuned model on custom images. This script allows you to pass in the path to the trained model and a list of test images to evaluate its performance.
Example command:
```
python tools/infer_rec.py --rec_model_dir ./kaggle/working/finetuned_model/best_accuracy --image_dir ./test_images
```
Manual Testing: If you want to manually evaluate the output, split your dataset into train, test, and eval sets. Use the Test set to verify the model's predictions against the ground truth.
Visualization: Use tools like VisualDL to monitor training and evaluation metrics visually.

Additional Notes:

From the provided discussion, ensure that the pretrained parameters are being loaded correctly. If accuracy starts at 0 and loss is very high, it could indicate that the pretrained weights are not being utilized properly. Double-check the pretrained_model path and confirm it matches the model architecture.
When fine-tuning for a specific font, you don’t need an extremely large dataset, but it should have sufficient examples of the target characters in different contexts (e.g., uppercase, lowercase, spacing).

VishyAnand28
Dec 17, 2024

@l03126 thanks for the tag. @GreatV as expert many points have been covered. Let me give my piece regarding it.
Initially, I did a small batch sample of 12 images for training and 8 for eval. I got an extreme case of overfitting and in other configurations, I had acc=0.000. The model is not designed to be performed well for such small datasets. Also, try to have variance in your dataset either manually with Imgaug or using data preprocessing techniques from https://github.com/PaddlePaddle/PaddleOCR/tree/main/ppocr/data/imaug and see if you can implement them.
Also, remember that you are not employing a pretrained model so you can expect a very slow increase in accuracy over epochs. Training them minimum for 500 epochs with apt batch size and learning rate. Training the model over several config files is necessary. Hope these tips help!

2 replies

l03126 Dec 18, 2024
Author

Oh alright Vishy. Thank you so much, I understand. I tested out the training with a larger dataset. It seems that it went decently. But I am facing some issues. Hope you can help me regarding that as well.

#14414

Really sorry to tag you in my questions. The GreatV bot gives good solutions but sometimes it can't provide answers for something if it isn't specifically mentioned in the documents.

VishyAnand28 Dec 18, 2024

No problem, feel free to ask when problems arise!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recognition (en_PP_OCRv4_rec_train) finetuning questions #14387

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Recognition (en_PP_OCRv4_rec_train) finetuning questions #14387

l03126 Dec 16, 2024

Replies: 2 comments · 2 replies

GreatV Dec 16, 2024 Maintainer

Answers to Your Questions:

1. The loss and learning rates are decreasing, but the accuracy is constant at 0. Could this be due to the extremely small dataset or something else?

2. Are the models saved in /kaggle/working/finetuned_model/best_accuracy.* the same as those in /kaggle/working/finetuned_model/best_model/*?

3. Is there another suggested way to fine-tune the model for a specific font?

4. Is there an existing way to test the fine-tuned model, or must it be done manually?

Additional Notes:

Related Discussion Links:

VishyAnand28 Dec 17, 2024

l03126 Dec 18, 2024 Author

VishyAnand28 Dec 18, 2024

l03126
Dec 16, 2024

Replies: 2 comments 2 replies

GreatV
Dec 16, 2024
Maintainer

2. Are the models saved in `/kaggle/working/finetuned_model/best_accuracy.` the same as those in `/kaggle/working/finetuned_model/best_model/`?

VishyAnand28
Dec 17, 2024

l03126 Dec 18, 2024
Author