Segmentation fault when fine-tuning #13940

ep0p · 2024-10-04T10:53:02Z

ep0p
Oct 4, 2024

I am attempting to fine tune with a french dataset the model latin_PP-OCRv3_rec_train (as per recommendations from here )

However i am not sure where it goes wrong but i get the segmentation fault. I have decreased the batch_size, changed the dictionary, i get the same error.

[2024/10/04 10:40:14] ppocr INFO: Architecture : 
[2024/10/04 10:40:14] ppocr INFO:     Backbone : 
[2024/10/04 10:40:14] ppocr INFO:         last_conv_stride : [1, 2]
[2024/10/04 10:40:14] ppocr INFO:         last_pool_type : avg
[2024/10/04 10:40:14] ppocr INFO:         name : MobileNetV1Enhance
[2024/10/04 10:40:14] ppocr INFO:         scale : 0.5
[2024/10/04 10:40:14] ppocr INFO:     Head : 
[2024/10/04 10:40:14] ppocr INFO:         head_list : 
[2024/10/04 10:40:14] ppocr INFO:             CTCHead : 
[2024/10/04 10:40:14] ppocr INFO:                 Head : 
[2024/10/04 10:40:14] ppocr INFO:                     fc_decay : 1e-05
[2024/10/04 10:40:14] ppocr INFO:                 Neck : 
[2024/10/04 10:40:14] ppocr INFO:                     depth : 2
[2024/10/04 10:40:14] ppocr INFO:                     dims : 64
[2024/10/04 10:40:14] ppocr INFO:                     hidden_dims : 120
[2024/10/04 10:40:14] ppocr INFO:                     name : svtr
[2024/10/04 10:40:14] ppocr INFO:                     use_guide : True
[2024/10/04 10:40:14] ppocr INFO:             SARHead : 
[2024/10/04 10:40:14] ppocr INFO:                 enc_dim : 512
[2024/10/04 10:40:14] ppocr INFO:                 max_text_length : 25
[2024/10/04 10:40:14] ppocr INFO:         name : MultiHead
[2024/10/04 10:40:14] ppocr INFO:     Transform : None
[2024/10/04 10:40:14] ppocr INFO:     algorithm : SVTR
[2024/10/04 10:40:14] ppocr INFO:     model_type : rec
[2024/10/04 10:40:14] ppocr INFO: Eval : 
[2024/10/04 10:40:14] ppocr INFO:     dataset : 
[2024/10/04 10:40:14] ppocr INFO:         data_dir : /home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/test
[2024/10/04 10:40:14] ppocr INFO:         label_file_list : ['/home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/modified_test.txt']
[2024/10/04 10:40:14] ppocr INFO:         name : SimpleDataSet
[2024/10/04 10:40:14] ppocr INFO:         transforms : 
[2024/10/04 10:40:14] ppocr INFO:             DecodeImage : 
[2024/10/04 10:40:14] ppocr INFO:                 channel_first : False
[2024/10/04 10:40:14] ppocr INFO:                 img_mode : BGR
[2024/10/04 10:40:14] ppocr INFO:             MultiLabelEncode : None
[2024/10/04 10:40:14] ppocr INFO:             SARRecResizeImg : 
[2024/10/04 10:40:14] ppocr INFO:                 image_shape : [3, 48, 48, 320]
[2024/10/04 10:40:14] ppocr INFO:                 mode : v3
[2024/10/04 10:40:14] ppocr INFO:                 width_downsample_ratio : 0.25
[2024/10/04 10:40:14] ppocr INFO:             KeepKeys : 
[2024/10/04 10:40:14] ppocr INFO:                 keep_keys : ['image', 'label_ctc', 'label_sar', 'length', 'valid_ratio']
[2024/10/04 10:40:14] ppocr INFO:     loader : 
[2024/10/04 10:40:14] ppocr INFO:         batch_size_per_card : 64
[2024/10/04 10:40:14] ppocr INFO:         drop_last : False
[2024/10/04 10:40:14] ppocr INFO:         num_workers : 4
[2024/10/04 10:40:14] ppocr INFO:         shuffle : False
[2024/10/04 10:40:14] ppocr INFO: Global : 
[2024/10/04 10:40:14] ppocr INFO:     cal_metric_during_train : True
[2024/10/04 10:40:14] ppocr INFO:     character_dict_path : /home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/word_dict.txt
[2024/10/04 10:40:14] ppocr INFO:     checkpoints : None
[2024/10/04 10:40:14] ppocr INFO:     debug : False
[2024/10/04 10:40:14] ppocr INFO:     distributed : False
[2024/10/04 10:40:14] ppocr INFO:     epoch_num : 20
[2024/10/04 10:40:14] ppocr INFO:     eval_batch_step : [0, 2000]
[2024/10/04 10:40:14] ppocr INFO:     infer_img : None
[2024/10/04 10:40:14] ppocr INFO:     infer_mode : False
[2024/10/04 10:40:14] ppocr INFO:     log_smooth_window : 20
[2024/10/04 10:40:14] ppocr INFO:     max_text_length : 25
[2024/10/04 10:40:14] ppocr INFO:     pretrained_model : /home/epop/paddleOCR/pretrain_models/latin_PP-OCRv3_rec_train/best_accuracy.pdparams
[2024/10/04 10:40:14] ppocr INFO:     print_batch_step : 10
[2024/10/04 10:40:14] ppocr INFO:     save_epoch_step : 3
[2024/10/04 10:40:14] ppocr INFO:     save_inference_dir : None
[2024/10/04 10:40:14] ppocr INFO:     save_model_dir : ./output/rec_ppocr_v3_latin
[2024/10/04 10:40:14] ppocr INFO:     save_res_path : ./output/rec/predicts_ppocrv3.txt
[2024/10/04 10:40:14] ppocr INFO:     use_gpu : True
[2024/10/04 10:40:14] ppocr INFO:     use_space_char : True
[2024/10/04 10:40:14] ppocr INFO:     use_visualdl : False
[2024/10/04 10:40:14] ppocr INFO: Loss : 
[2024/10/04 10:40:14] ppocr INFO:     loss_config_list : 
[2024/10/04 10:40:14] ppocr INFO:         CTCLoss : None
[2024/10/04 10:40:14] ppocr INFO:         SARLoss : None
[2024/10/04 10:40:14] ppocr INFO:     name : MultiLoss
[2024/10/04 10:40:14] ppocr INFO: Metric : 
[2024/10/04 10:40:14] ppocr INFO:     keep_space : False
[2024/10/04 10:40:14] ppocr INFO:     main_indicator : acc
[2024/10/04 10:40:14] ppocr INFO:     name : RecMetric
[2024/10/04 10:40:14] ppocr INFO: Optimizer : 
[2024/10/04 10:40:14] ppocr INFO:     beta1 : 0.9
[2024/10/04 10:40:14] ppocr INFO:     beta2 : 0.999
[2024/10/04 10:40:14] ppocr INFO:     lr : 
[2024/10/04 10:40:14] ppocr INFO:         learning_rate : 0.001
[2024/10/04 10:40:14] ppocr INFO:         name : Cosine
[2024/10/04 10:40:14] ppocr INFO:         warmup_epoch : 5
[2024/10/04 10:40:14] ppocr INFO:     name : Adam
[2024/10/04 10:40:14] ppocr INFO:     regularizer : 
[2024/10/04 10:40:14] ppocr INFO:         factor : 3e-05
[2024/10/04 10:40:14] ppocr INFO:         name : L2
[2024/10/04 10:40:14] ppocr INFO: PostProcess : 
[2024/10/04 10:40:14] ppocr INFO:     name : CTCLabelDecode
[2024/10/04 10:40:14] ppocr INFO: Train : 
[2024/10/04 10:40:14] ppocr INFO:     dataset : 
[2024/10/04 10:40:14] ppocr INFO:         data_dir : /home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/train
[2024/10/04 10:40:14] ppocr INFO:         label_file_list : ['/home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/modified_train.txt']
[2024/10/04 10:40:14] ppocr INFO:         name : SimpleDataSet
[2024/10/04 10:40:14] ppocr INFO:         ratio_list : [1.0]
[2024/10/04 10:40:14] ppocr INFO:         transforms : 
[2024/10/04 10:40:14] ppocr INFO:             DecodeImage : 
[2024/10/04 10:40:14] ppocr INFO:                 channel_first : False
[2024/10/04 10:40:14] ppocr INFO:                 img_mode : BGR
[2024/10/04 10:40:14] ppocr INFO:             RecAug : None
[2024/10/04 10:40:14] ppocr INFO:             MultiLabelEncode : None
[2024/10/04 10:40:14] ppocr INFO:             SARRecResizeImg : 
[2024/10/04 10:40:14] ppocr INFO:                 image_shape : [3, 48, 48, 320]
[2024/10/04 10:40:14] ppocr INFO:                 mode : v3
[2024/10/04 10:40:14] ppocr INFO:                 width_downsample_ratio : 0.25
[2024/10/04 10:40:14] ppocr INFO:             KeepKeys : 
[2024/10/04 10:40:14] ppocr INFO:                 keep_keys : ['image', 'label_ctc', 'label_sar', 'length', 'valid_ratio']
[2024/10/04 10:40:14] ppocr INFO:     loader : 
[2024/10/04 10:40:14] ppocr INFO:         batch_size_per_card : 64
[2024/10/04 10:40:14] ppocr INFO:         drop_last : True
[2024/10/04 10:40:14] ppocr INFO:         num_workers : 4
[2024/10/04 10:40:14] ppocr INFO:         shuffle : True
[2024/10/04 10:40:14] ppocr INFO: profiler_options : None
[2024/10/04 10:40:14] ppocr INFO: train with paddle 2.6.0 and device Place(gpu:0)
[2024/10/04 10:40:14] ppocr INFO: Initialize indexs of datasets:['/home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/modified_train.txt']
[2024/10/04 10:40:16] ppocr INFO: Initialize indexs of datasets:['/home/epop/DATASET/ds_paddle_1p5mil_rec_4rep/modified_test.txt']
W1004 10:40:16.281527  5133 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 12.0
W1004 10:40:16.285171  5133 gpu_resources.cc:164] device: 0, cuDNN Version: 9.4.
INFO:root:If regularizer of a Parameter has been set by 'paddle.ParamAttr' or 'static.WeightNormParamAttr' already. The weight_decay[3e-05] in Optimizer will not take effect, and it will only be applied to other Parameters!
[2024/10/04 10:40:16] ppocr INFO: train dataloader has 18728 iters
[2024/10/04 10:40:16] ppocr INFO: valid dataloader has 2342 iters
[2024/10/04 10:40:16] ppocr WARNING: The shape of model params head.ctc_head.fc.weight [64, 264] not matched with loaded params head.ctc_head.fc.weight [64, 187] !
[2024/10/04 10:40:16] ppocr WARNING: The shape of model params head.ctc_head.fc.bias [264] not matched with loaded params head.ctc_head.fc.bias [187] !
[2024/10/04 10:40:16] ppocr WARNING: The shape of model params head.sar_head.decoder.embedding.weight [266, 512] not matched with loaded params head.sar_head.decoder.embedding.weight [189, 512] !
[2024/10/04 10:40:16] ppocr WARNING: The shape of model params head.sar_head.decoder.prediction.weight [1536, 265] not matched with loaded params head.sar_head.decoder.prediction.weight [1536, 188] !
[2024/10/04 10:40:16] ppocr WARNING: The shape of model params head.sar_head.decoder.prediction.bias [265] not matched with loaded params head.sar_head.decoder.prediction.bias [188] !
[2024/10/04 10:40:16] ppocr INFO: load pretrain successful from /home/epop/paddleOCR/pretrain_models/latin_PP-OCRv3_rec_train/best_accuracy
[2024/10/04 10:40:16] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 2000 iterations


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
No stack trace in paddle, may be caused by external reasons.

----------------------
Error Message Summary:
----------------------
FatalError: `Segmentation fault` is detected by the operating system.
  [TimeInfo: *** Aborted at 1728038423 (unix time) try "date -d @1728038423" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x0) received by PID 5133 (TID 0x7fda2687a440) from PID 0 ***]

Segmentation fault (core dumped)

Answered by ep0p

Oct 10, 2024

I have finally figured it out. This issue was caused by cuDNN. I was using cuDNN version 9.5.0, which apparently works for CUDA 12.6, while I have CUDA 12.4 installed on my system.

NVIDIA's documentation only mentions compatibility with CUDA 12, without specifying the exact sub-version.

After downgrading cuDNN to version 8, everything works perfectly now.

Hope this helps anyone facing a similar issue in the future!

View full answer

ep0p · 2024-10-10T07:03:17Z

ep0p
Oct 10, 2024
Author

I have finally figured it out. This issue was caused by cuDNN. I was using cuDNN version 9.5.0, which apparently works for CUDA 12.6, while I have CUDA 12.4 installed on my system.

NVIDIA's documentation only mentions compatibility with CUDA 12, without specifying the exact sub-version.

After downgrading cuDNN to version 8, everything works perfectly now.

Hope this helps anyone facing a similar issue in the future!

1 reply

Manchester-United-in-my-heart Dec 16, 2024

hi there,
Please let me know which exactly cuda version you used (f.e: 11.0 or 11.6). I am here waiting for your answer. Thanks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault when fine-tuning #13940

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 1 reply

{{title}}

{{title}}

Select a reply

Segmentation fault when fine-tuning #13940

ep0p Oct 4, 2024

Replies: 1 comment · 1 reply

ep0p Oct 10, 2024 Author

Manchester-United-in-my-heart Dec 16, 2024

ep0p
Oct 4, 2024

Replies: 1 comment 1 reply

ep0p
Oct 10, 2024
Author