You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I prepared a custom dataset in COCO format having 3 classes like below:
But there is no samples related to 'Resumes' class. Only 'heading' and 'text' classes are present there in my sample.
WARNING [09/09 20:18:55 d2.data.datasets.coco]:
Category ids in annotations are not in [1, #categories]! We'll apply a mapping for you.
[09/09 20:18:55 d2.data.datasets.coco]: Loaded 638 images in COCO format from ./data/train/annotations.json
[09/09 20:18:55 d2.data.build]: Removed 320 images with no usable annotations. 318 images left.
[09/09 20:18:55 d2.data.build]: Distribution of instances among all 3 categories:
| category | #instances | category | #instances | category | #instances |
|:----------:|:-------------|:----------:|:-------------|:----------:|:-------------|
| Resumes | 0 | heading | 1028 | text | 1951 |
| | | | | | |
| total | 2979 | | | | |
[09/09 20:18:55 d2.data.detection_utils]: TransformGens used in training: [ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice'), RandomFlip()]
[09/09 20:18:55 d2.data.build]: Using training sampler TrainingSampler
[09/09 20:18:56 d2.engine.train_loop]: Starting training from iteration 75500
ERROR [09/09 20:18:57 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
File "/home/ujjawal/miniconda2/envs/caffe2/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 132, in train
self.run_step()
File "/home/ujjawal/miniconda2/envs/caffe2/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 216, in run_step
self._detect_anomaly(losses, loss_dict)
File "/home/ujjawal/miniconda2/envs/caffe2/lib/python3.7/site-packages/detectron2/engine/train_loop.py", line 239, in _detect_anomaly
self.iter, loss_dict
FloatingPointError: Loss became infinite or NaN at iteration=75501!
loss_dict = {'loss_cls': tensor(nan, device='cuda:0', grad_fn=<NllLossBackward>), 'loss_box_reg': tensor(nan, device='cuda:0', grad_fn=<DivBackward0>), 'loss_mask': tensor(0.7118, device='cuda:0', grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_rpn_cls': tensor(0.6949, device='cuda:0', grad_fn=<MulBackward0>), 'loss_rpn_loc': tensor(0.4812, device='cuda:0', grad_fn=<MulBackward0>)}
I tried to change the NUM_CLASSES: 5 to 3 but no luck.
Some suggested to reduce the LR still no luck.
Can anyone please suggest a way to tackle this issue?
The text was updated successfully, but these errors were encountered:
Hi,
I prepared a custom dataset in COCO format having 3 classes like below:
But there is no samples related to 'Resumes' class. Only 'heading' and 'text' classes are present there in my sample.
I'm using this config file for finetuning
Got following error:-
I tried to change the NUM_CLASSES: 5 to 3 but no luck.
Some suggested to reduce the LR still no luck.
Can anyone please suggest a way to tackle this issue?
The text was updated successfully, but these errors were encountered: